Repository: microsoft/ptvsd

Issue:
unicode string cannot handle correct in debug console
run in Debug Console (python 2.7.16):
```
u'中'
u'\xe4\xb8\xad' # debug console
```

run in Terminal  (python 2.7.16):
```
(py27) D:\OneDrive - danpier\Python\Code>python
Python 2.7.16 |Anaconda, Inc.| (default, Mar 14 2019, 15:49:40) [MSC v.1500 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> u'中'
u'\u4e2d'
```

As you can see above, the unicode character '中' is handled as three unicode characters in Debug Console.
This could casues some problem like:
```
import re

text = re.sub(u'[中]', 's', u'中')
print(text)

# debug console: sss (unexpected result)
# terminal: s (expected result)
```

## Environment data

- VS Code version: 1.39.2
- Extension version (available under the Extensions sidebar): ms-python.python 2019.10.41019
- OS and version: Windows 7 32-bit (Windows_NT ia32 6.1.7601)
- Python version (Anaconda3-2019.07-Windows-x86): 3.7.3
- Type of virtual environment used: conda 4.7.10 (Python 2.7.16)
- Jedi or Language Server? : False

## Workspace setting:

launch.json
```
{
    "version": "0.2.0",
    "configurations": [
        ...
        {
            "name": "Python 2.7: Current File (Debug Console)",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "internalConsole",
            "pythonPath": "C:\\Users\\Zerone\\Anaconda3\\envs\\py27\\python.exe"
        },
        ...
    ]
}
```

settings.json

```
{
    "python.pythonPath": "C:\\Users\\Zerone\\Anaconda3\\python.exe",
    "python.terminal.activateEnvironment": true,
    // Lint
    "python.linting.enabled": true,
    "python.linting.flake8Enabled": true,
    "python.linting.pylintEnabled": false,
    // format
    "editor.formatOnSave": true,
    "python.formatting.provider": "yapf",
    //editor
    "workbench.editor.enablePreview": false
}
```

## Expected behaviour

debug console: '中'

## Actual behaviour

debug console: 'ä¸­'

## Steps to reproduce:
1. config env: conda create --name py27 python=2.7
2. config 'launch.json' mentioned above
3. run below code  in Debug Console:
```
print(u'中')
```


## DEBUG CONSOLE

```
Frame skipped from debugging during step-in.
Note: may have been skipped because of "justMyCode" option (default == true).
ä¸­
```



Discussion:
I'll investigate this.
Below is the result of inputs/outputs in the debug console in python 2 and python 3... 

One thing that drew my attention is that printing in Python 2 with `cp1252` gave the proper output (while the `PYTHONIOENCODING` was set to `utf-8`)... I still need to investigate a bit more to understand why we're having these results.

Python 2:
```
>>> import sys;print(sys.stdout.encoding)
UTF-8

>>> u'中'
u'\xe4\xb8\xad'

# Note: the evaluate response message is:
# - Python 2 {"result": "u'\\xe4\\xb8\\xad'" ... }
# - Python 3 {"result": "'\u4e2d'" ... }
# -- note that both are technically correct, the problem is when data is printed.

>>> u'中' == u'\xe4\xb8\xad'
True

>>> print(u'中')
ä¸­

>>> print(u'中'.encode('utf-8'))
ä¸­

>>> print(u'中'.encode('cp1252'))
中

```

Python 3 (printed output seems correct):

```
>>> import sys;print(sys.stdout.encoding)
UTF-8

>>> u'中'
'中'

>>> print(u'中')
中

>>> print(u'中'.encode('utf-8'))
b'\xe4\xb8\xad'

>>> print(u'中'.encode('cp1252'))
UnicodeEncodeError: 'charmap' codec can't encode character '\u4e2d' in position 0: character maps to <undefined>

```

Some notes:

- changing the `PYTHONIOENCODING` to `cp1252` on Python 2 did not fix things (the output is different but not correct).

- this is with the current master version.

- I believe that the out of process adapter is capturing the output right now, so, my current suspect is that the encoding used to capture the bytes does not match the output in which bytes are being printed.
Some more details on the logs (on Python 2 and Python 3):

Python 3:
```
---- print(u'中'): proper output on python 3 ---- 
{"command": "evaluate", "arguments": {"expression": "print(u'\u4e2d')"}}

{"event": "output", "body": {"category": "stdout", "output": "\u4e2d\n"}}


---- print(u'\u4e2d'): proper output on python 3 ---- 
{"command": "evaluate", "arguments": {"expression": "print(u'\\u4e2d')"}}

{"event": "output", "body": {"category": "stdout", "output": "\u4e2d\n"}}


---- print(u'\xe4\xb8\xad'): garbled output on python 3 ---- 
{"command": "evaluate", "arguments": {"expression": "print(u'\\xe4\\xb8\\xad')"}}

{"event": "output", "body": {"category": "stdout", "output": "\u00e4\u00b8\u00ad"}}
```

Python 2: 
```
---- print(u'中'): gave garbled output on python 2 ---- 
{"command": "evaluate", "arguments": {"expression": "print u'\u4e2d'"}}

{"event": "output", "body": {"category": "stdout", "output": "\u00e4\u00b8\u00ad"}}


---- print(u'\u4e2d')': gave proper output on python 2 ---- 
{"command": "evaluate", "arguments": {"expression": "print u'\\u4e2d'"}}

{"event": "output", "body": {"category": "stdout", "output": "\u4e2d\n"}}


---- print(u'\xe4\xb8\xad'): gave garbled output on python 2 ---- 
{"command": "evaluate", "arguments": {"expression": "print u'\\xe4\\xb8\\xad'"}}

{"event": "output", "body": {"category": "stdout", "output": "\u00e4\u00b8\u00ad"}}
```

Resolving patch:
diff --git a/src/ptvsd/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py b/src/ptvsd/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py
--- a/src/ptvsd/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py
+++ b/src/ptvsd/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py
@@ -954,7 +954,7 @@ def internal_evaluate_expression_json(py_db, request, thread_id):
 
     if IS_PY2 and isinstance(expression, unicode):
         try:
-            expression = expression.encode('utf-8')
+            expression.encode('utf-8')
         except:
             _evaluate_response(py_db, request, '', error_message='Expression is not valid utf-8.')
             raise
diff --git a/src/ptvsd/_vendored/pydevd/_pydevd_bundle/pydevd_vars.py b/src/ptvsd/_vendored/pydevd/_pydevd_bundle/pydevd_vars.py
--- a/src/ptvsd/_vendored/pydevd/_pydevd_bundle/pydevd_vars.py
+++ b/src/ptvsd/_vendored/pydevd/_pydevd_bundle/pydevd_vars.py
@@ -2,10 +2,12 @@
     resolution/conversion to XML.
 """
 import pickle
-from _pydevd_bundle.pydevd_constants import get_frame, get_current_thread_id, xrange
+from _pydevd_bundle.pydevd_constants import get_frame, get_current_thread_id, xrange, IS_PY2
 
 from _pydevd_bundle.pydevd_xml import ExceptionOnEvaluate, get_type, var_to_xml
 from _pydev_bundle import pydev_log
+import codecs
+import os
 
 try:
     from StringIO import StringIO
@@ -203,10 +205,21 @@ def custom_operation(dbg, thread_id, frame_id, scope, attrs, style, code_or_file
         pydev_log.exception()
 
 
+def _expression_to_evaluate(expression):
+    if IS_PY2 and isinstance(expression, unicode):
+        # In Python 2 we need to compile with bytes and not unicode (otherwise it'd use
+        # the default encoding which could be ascii).
+        # See https://github.com/microsoft/ptvsd/issues/1864 and https://bugs.python.org/issue18870
+        # for why we're using the utf-8 bom.
+        # i.e.: ... if an utf-8 bom is present, it is considered utf-8 in eval/exec.
+        expression = codecs.BOM_UTF8 + expression.encode('utf-8')
+    return expression
+
+
 def eval_in_context(expression, globals, locals):
     result = None
     try:
-        result = eval(expression, globals, locals)
+        result = eval(_expression_to_evaluate(expression), globals, locals)
     except Exception:
         s = StringIO()
         traceback.print_exc(file=s)
@@ -225,6 +238,9 @@ def eval_in_context(expression, globals, locals):
 
         # Ok, we have the initial error message, but let's see if we're dealing with a name mangling error...
         try:
+            if IS_PY2 and isinstance(expression, unicode):
+                expression = expression.encode('utf-8')
+
             if '__' in expression:
                 # Try to handle '__' name mangling...
                 split = expression.split('.')
@@ -255,19 +271,28 @@ def evaluate_expression(dbg, frame, expression, is_exec):
     updated_globals.update(frame.f_locals)  # locals later because it has precedence over the actual globals
 
     try:
-        expression = str(expression.replace('@LINE@', '\n'))
+
+        if IS_PY2 and isinstance(expression, unicode):
+            expression = expression.replace(u'@LINE@', u'\n')
+        else:
+            expression = expression.replace('@LINE@', '\n')
 
         if is_exec:
             try:
                 # try to make it an eval (if it is an eval we can print it, otherwise we'll exec it and
                 # it will have whatever the user actually did)
-                compiled = compile(expression, '<string>', 'eval')
+                compiled = compile(_expression_to_evaluate(expression), '<string>', 'eval')
             except:
-                Exec(expression, updated_globals, frame.f_locals)
+                Exec(_expression_to_evaluate(expression), updated_globals, frame.f_locals)
                 pydevd_save_locals.save_locals(frame)
             else:
                 result = eval(compiled, updated_globals, frame.f_locals)
                 if result is not None:  # Only print if it's not None (as python does)
+                    if IS_PY2 and isinstance(result, unicode):
+                        encoding = sys.stdout.encoding
+                        if not encoding:
+                            encoding = os.environ.get('PYTHONIOENCODING', 'utf-8')
+                        result = result.encode(encoding, 'replace')
                     sys.stdout.write('%s\n' % (result,))
             return
 


Test patch:
diff --git a/src/ptvsd/_vendored/pydevd/tests_python/test_debugger_json.py b/src/ptvsd/_vendored/pydevd/tests_python/test_debugger_json.py
--- a/src/ptvsd/_vendored/pydevd/tests_python/test_debugger_json.py
+++ b/src/ptvsd/_vendored/pydevd/tests_python/test_debugger_json.py
@@ -1215,6 +1215,62 @@ def test_evaluate_unicode(case_setup):
         writer.finished_ok = True
 
 
+@pytest.mark.skipif(IS_PY26, reason='Not ok on Python 2.6.')
+def test_evaluate_exec_unicode(case_setup):
+
+    def get_environ(writer):
+        env = os.environ.copy()
+
+        env["PYTHONIOENCODING"] = 'utf-8'
+        return env
+
+    with case_setup.test_file('_debugger_case_local_variables2.py', get_environ=get_environ) as writer:
+        json_facade = JsonFacade(writer)
+        writer.write_start_redirect()
+
+        writer.write_add_breakpoint(writer.get_line_index_with_content('Break here'))
+        json_facade.write_make_initial_run()
+
+        json_hit = json_facade.wait_for_thread_stopped()
+        json_hit = json_facade.get_stack_as_json_hit(json_hit.thread_id)
+
+        # Check eval
+        json_facade.evaluate(
+            "print(u'中')",
+            frameId=json_hit.frame_id,
+            context="repl",
+        )
+
+        messages = json_facade.mark_messages(
+            OutputEvent, lambda output_event: u'中' in output_event.body.output)
+        assert len(messages) == 1
+
+        # Check exec
+        json_facade.evaluate(
+            "a=10;print(u'中')",
+            frameId=json_hit.frame_id,
+            context="repl",
+        )
+
+        messages = json_facade.mark_messages(
+            OutputEvent, lambda output_event: u'中' in output_event.body.output)
+        assert len(messages) == 1
+
+        response = json_facade.evaluate(
+            "u'中'",
+            frameId=json_hit.frame_id,
+            context="repl",
+        )
+        assert response.body.result in ("u'\\u4e2d'", "'\u4e2d'")  # py2 or py3
+
+        messages = json_facade.mark_messages(
+            OutputEvent, lambda output_event: u'中' in output_event.body.output)
+        assert len(messages) == 0  # i.e.: we don't print in this case.
+
+        json_facade.write_continue()
+        writer.finished_ok = True
+
+
 def test_evaluate_variable_references(case_setup):
     from _pydevd_bundle._debug_adapter.pydevd_schema import EvaluateRequest
     from _pydevd_bundle._debug_adapter.pydevd_schema import EvaluateArguments