Repository: microsoft/ptvsd

Issue:
NetCommand: UnicodeDecodeError on French Windows 10 while reporting socket error
<!-- Please search existing issues to avoid creating duplicates. -->

## Environment data

- VS Code version: 1.36.1
- Extension version (available under the Extensions sidebar): 2019.6.24221
- OS and version: Windows 10 Enterprise 1703
- Python version (& distribution if applicable, e.g. Anaconda): Python 2.7.x
- Type of virtual environment used (N/A | venv | virtualenv | conda | ...): virtualenv 
- Relevant/affected Python packages and their versions: -
- Jedi or Language Server? (i.e. what is `"python.jediEnabled"` set to; more info microsoft/vscode-python#3977): -

## Expected behaviour

Socket server error should not make whole application crash in debug

## Actual behaviour

Application crash because of the decoding error 

## Steps to reproduce:
1. Have a django project (for example)
2. Use the django runserver command in debug mode
3. Fire a request
4. Before completion, cancel the request
5. Error is handled, and the debugger crashes

<!--
Note: If you think a GIF of what is happening would be helpful, consider tools like https://www.cockos.com/licecap/, https://github.com/phw/peek or https://www.screentogif.com/ .
-->

## Logs
Output for `Python` in the `Output` panel (`View`→`Output`, change the drop-down the upper-right of the `Output` panel to `Python`)

```
[...]
  File "D:\Code\myproject\pyvirtualenv\lib\site-packages\django\core\management\commands\runserver.py", line 61, in execute
    super(Command, self).execute(*args, **options)
  File "D:\Code\myproject\pyvirtualenv\lib\site-packages\django\core\management\base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "D:\Code\myproject\pyvirtualenv\lib\site-packages\django\core\management\commands\runserver.py", line 100, in handle
    self.run(**options)
  File "D:\Code\myproject\pyvirtualenv\lib\site-packages\django\core\management\commands\runserver.py", line 111, in run
    self.inner_run(None, **options)
  File "D:\Code\myproject\pyvirtualenv\lib\site-packages\django\core\management\commands\runserver.py", line 148, in inner_run
    ipv6=self.use_ipv6, threading=threading, server_cls=self.server_cls)
  File "D:\Code\myproject\pyvirtualenv\lib\site-packages\django\core\servers\basehttp.py", line 174, in run
    httpd.serve_forever()
  File "c:\python27\Lib\SocketServer.py", line 236, in serve_forever
    self._handle_request_noblock()
  File "c:\python27\Lib\SocketServer.py", line 295, in _handle_request_noblock
    self.handle_error(request, client_address)
  File "D:\Code\myproject\pyvirtualenv\lib\site-packages\django\core\servers\basehttp.py", line 80, in handle_error
    super(WSGIServer, self).handle_error(request, client_address)
  File "c:\python27\Lib\SocketServer.py", line 354, in handle_error
    traceback.print_exc() # XXX But this goes to stderr!
  File "c:\python27\Lib\traceback.py", line 233, in print_exc
    print_exception(etype, value, tb, limit, file)
  File "c:\python27\Lib\traceback.py", line 128, in print_exception
    _print(file, line, '')
  File "c:\python27\Lib\traceback.py", line 13, in _print
    file.write(str+terminator)
  File "c:\Users\foobar\.vscode\extensions\ms-python.python-2019.6.24221\pythonFiles\lib\python\ptvsd\_vendored\pydevd\_pydevd_bundle\pydevd_io.py", line 40, in write
    r.write(s)
  File "c:\Users\foobar\.vscode\extensions\ms-python.python-2019.6.24221\pythonFiles\lib\python\ptvsd\_vendored\pydevd\pydevd.py", line 1902, in write
    cmd = py_db.cmd_factory.make_io_message(s, self._out_ctx)
  File "c:\Users\foobar\.vscode\extensions\ms-python.python-2019.6.24221\pythonFiles\lib\python\ptvsd\_vendored\pydevd\_pydevd_bundle\pydevd_net_command_factory_json.py", line 249, in make_io_message
    return NetCommand(CMD_WRITE_TO_CONSOLE, 0, event, is_json=True)
  File "c:\Users\foobar\.vscode\extensions\ms-python.python-2019.6.24221\pythonFiles\lib\python\ptvsd\_vendored\pydevd\_pydevd_bundle\pydevd_net_command.py", line 58, in __init__
    text = json.dumps(as_dict)
  File "c:\python27\Lib\json\__init__.py", line 244, in dumps
    return _default_encoder.encode(obj)
  File "c:\python27\Lib\json\encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "c:\python27\Lib\json\encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 35: invalid continuation byte
```

I've found that there's a dict containing a str (not unicode) delivered by the OS network layer that has the value: `'error: [Errno 10053] Une connexion \xe9tablie a \xe9t\xe9 abandonn\xe9e par un logiciel de votre ordinateur h\xf4te\n'`, which is encoded either in `latin1`, `cp1252` or `cp850`.



Discussion:
Actually this is a bug with `ptvsd`.
Bug submitted to ptvsd and I made a [pull request](https://github.com/microsoft/ptvsd/pull/1634).
This should be open until the fix is in, but we can transfer it to the right repo. I don't have write permissions here tho, so I can't do a move. @DonJayamanne, can you do the transfer so that it moves it?

Resolving patch:
diff --git a/src/ptvsd/_vendored/pydevd/pydevd.py b/src/ptvsd/_vendored/pydevd/pydevd.py
--- a/src/ptvsd/_vendored/pydevd/pydevd.py
+++ b/src/ptvsd/_vendored/pydevd/pydevd.py
@@ -2280,7 +2280,10 @@ def __init__(self, out_ctx, wrap_stream, wrap_buffer, on_write=None):
             If not passed the default implementation will create an io message
             and send it through the debugger.
         '''
-        self.encoding = getattr(wrap_stream, 'encoding', os.environ.get('PYTHONIOENCODING', 'utf-8'))
+        encoding = getattr(wrap_stream, 'encoding', None)
+        if not encoding:
+            encoding = os.environ.get('PYTHONIOENCODING', 'utf-8')
+        self.encoding = encoding
         self._out_ctx = out_ctx
         if wrap_buffer:
             self.buffer = _CustomWriter(out_ctx, wrap_stream, wrap_buffer=False, on_write=on_write)
@@ -2296,10 +2299,13 @@ def write(self, s):
 
         if s:
             if IS_PY2:
-                # Need s in bytes
+                # Need s in utf-8 bytes
                 if isinstance(s, unicode):  # noqa
                     # Note: python 2.6 does not accept the "errors" keyword.
                     s = s.encode('utf-8', 'replace')
+                else:
+                    s = s.decode(self.encoding, 'replace').encode('utf-8', 'replace')
+
             else:
                 # Need s in str
                 if isinstance(s, bytes):


Test patch:
diff --git a/src/ptvsd/_vendored/pydevd/tests_python/resources/_debugger_case_redirect.py b/src/ptvsd/_vendored/pydevd/tests_python/resources/_debugger_case_redirect.py
--- a/src/ptvsd/_vendored/pydevd/tests_python/resources/_debugger_case_redirect.py
+++ b/src/ptvsd/_vendored/pydevd/tests_python/resources/_debugger_case_redirect.py
@@ -11,14 +11,19 @@
         stream.write('ação1\n')
 
         if sys.version_info[0] >= 3:
-            # sys.stdout.buffer is only available on py3.   
+            # sys.stdout.buffer is only available on py3.
             stream.buffer.write(b'binary\n')
             # Note: this will be giberish on the receiving side because when writing bytes
             # we can't be sure what's the encoding and will treat it as utf-8 (i.e.:
             # uses PYTHONIOENCODING).
             stream.buffer.write('ação2\n'.encode(encoding='latin1'))
-            
+
             # This will be ok
             stream.buffer.write('ação3\n'.encode(encoding='utf-8'))
 
-    print('TEST SUCEEDED!')
\ No newline at end of file
+        if sys.version_info[0] >= 3:
+            stream.buffer.write(b'\xe8\xF0\x80\x80\x80\n\n')
+        else:
+            stream.write(b'\xe8\xF0\x80\x80\x80\n\n')
+
+    print('TEST SUCEEDED!')
diff --git a/src/ptvsd/_vendored/pydevd/tests_python/test_debugger.py b/src/ptvsd/_vendored/pydevd/tests_python/test_debugger.py
--- a/src/ptvsd/_vendored/pydevd/tests_python/test_debugger.py
+++ b/src/ptvsd/_vendored/pydevd/tests_python/test_debugger.py
@@ -1925,10 +1925,16 @@ def get_environ(writer):
         def _ignore_stderr_line(line):
             if original_ignore_stderr_line(line):
                 return True
+
+            binary_junk = b'\xe8\xF0\x80\x80\x80'
+            if sys.version_info[0] >= 3:
+                binary_junk = binary_junk.decode('utf-8', 'replace')
+
             return line.startswith((
                 'text',
                 'binary',
-                'a'
+                'a',
+                binary_junk,
             ))
 
         writer._ignore_stderr_line = _ignore_stderr_line
@@ -1948,6 +1954,11 @@ def _ignore_stderr_line(line):
                 'ação3\n',
             ))
 
+        binary_junk = '\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\n\n'
+        if sys.version_info[0] >= 3:
+            binary_junk = "\ufffd\ufffd\ufffd\ufffd\ufffd\n\n"
+        expected.append(binary_junk)
+
         new_expected = [(x, 'stdout') for x in expected]
         new_expected.extend([(x, 'stderr') for x in expected])
 
diff --git a/src/ptvsd/_vendored/pydevd/tests_python/test_debugger_json.py b/src/ptvsd/_vendored/pydevd/tests_python/test_debugger_json.py
--- a/src/ptvsd/_vendored/pydevd/tests_python/test_debugger_json.py
+++ b/src/ptvsd/_vendored/pydevd/tests_python/test_debugger_json.py
@@ -2753,16 +2753,23 @@ def get_environ(writer):
         def _ignore_stderr_line(line):
             if original_ignore_stderr_line(line):
                 return True
+
+            binary_junk = b'\xe8\xF0\x80\x80\x80'
+            if sys.version_info[0] >= 3:
+                binary_junk = binary_junk.decode('utf-8', 'replace')
+
             return line.startswith((
                 'text',
                 'binary',
-                'a'
+                'a',
+                binary_junk,
             ))
 
         writer._ignore_stderr_line = _ignore_stderr_line
 
         # Note: writes to stdout and stderr are now synchronous (so, the order
         # must always be consistent and there's a message for each write).
+
         expected = [
             'text\n',
             'binary or text\n',
@@ -2776,6 +2783,11 @@ def _ignore_stderr_line(line):
                 'ação3\n',
             ))
 
+        binary_junk = '\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\n\n'
+        if sys.version_info[0] >= 3:
+            binary_junk = "\ufffd\ufffd\ufffd\ufffd\ufffd\n\n"
+        expected.append(binary_junk)
+
         new_expected = [(x, 'stdout') for x in expected]
         new_expected.extend([(x, 'stderr') for x in expected])
 
diff --git a/tests/ptvsd/server/test_output.py b/tests/ptvsd/server/test_output.py
--- a/tests/ptvsd/server/test_output.py
+++ b/tests/ptvsd/server/test_output.py
@@ -8,7 +8,6 @@
 
 from tests import debug
 
-
 # When debuggee exits, there's no guarantee currently that all "output" events have
 # already been sent. To ensure that they are, all tests below must set a breakpoint
 # on the last line of the debuggee, and stop on it. Since debugger sends its events
@@ -16,6 +15,7 @@
 
 
 def test_with_no_output(pyfile, target, run):
+
     @pyfile
     def code_to_debug():
         import debug_me  # noqa
@@ -36,6 +36,7 @@ def code_to_debug():
 
 
 def test_with_tab_in_output(pyfile, target, run):
+
     @pyfile
     def code_to_debug():
         import debug_me  # noqa
@@ -56,6 +57,7 @@ def code_to_debug():
 
 @pytest.mark.parametrize("redirect", ["enabled", "disabled"])
 def test_redirect_output(pyfile, target, run, redirect):
+
     @pyfile
     def code_to_debug():
         import debug_me  # noqa
@@ -78,3 +80,34 @@ def code_to_debug():
         assert session.output("stdout") == "111\n222\n333\n444\n"
     else:
         assert not session.output("stdout")
+
+
+def test_non_ascii_output(pyfile, target, run):
+
+    @pyfile
+    def code_to_debug():
+        import debug_me  # noqa
+        import sys
+        a = b'\xc3\xa9 \xc3\xa0 \xc3\xb6 \xc3\xb9\n'
+        if sys.version_info[0] >= 3:
+            sys.stdout.buffer.write(a)
+        else:
+            sys.stdout.write(a)
+        ()  # @wait_for_output
+
+    with debug.Session() as session:
+        session.config["redirectOutput"] = True
+
+        with run(session, target(code_to_debug)):
+            session.set_breakpoints(code_to_debug, all)
+
+        session.wait_for_stop()
+        session.request_continue()
+
+    output = session.output("stdout").encode('utf-8', 'replace')
+
+    assert output in (
+        b'\xc3\xa9 \xc3\xa0 \xc3\xb6 \xc3\xb9\n',
+        b'\xc3\x83\xc2\xa9 \xc3\x83\xc2\xa0 \xc3\x83\xc2\xb6 \xc3\x83\xc2\xb9\n'
+    )
+