Repository: microsoft/ptvsd

Issue:
ptvsd broken on Python 2 if python-future is installed
Issue Type: <b>Bug</b>

After I installed an interpreter of Python 2.7.16 32bit, an exception is thrown with error message below when I want to switch to the old Python 2.7.16 64bit:
```
E+00000.045: /handling microsoft/vscode-python#1 request "launch" from Adapter-1/
             Handler 'launch_request' (file u'c:\\Users\\linshimeng01\\.vscode\\extensions\\ms-python.python-2019.11.50794\\pythonFiles\\lib\\python\\new_ptvsd\\no_wheels\\ptvsd\\launcher/../../ptvsd\\launcher\\adapter.py', line 35)
             couldn't handle microsoft/vscode-python#1 request "launch" from Adapter-1:

             Traceback (most recent call last):
               File "c:\Users\linshimeng01\.vscode\extensions\ms-python.python-2019.11.50794\pythonFiles\lib\python\new_ptvsd\no_wheels\ptvsd\launcher/../../ptvsd\common\messaging.py", line 763, in _handle
                 result = handler(self)
               File "c:\Users\linshimeng01\.vscode\extensions\ms-python.python-2019.11.50794\pythonFiles\lib\python\new_ptvsd\no_wheels\ptvsd\launcher/../../ptvsd\launcher\adapter.py", line 147, in launch_request
                 debuggee.spawn(process_name, cmdline, cwd, env, redirect_output)
               File "c:\Users\linshimeng01\.vscode\extensions\ms-python.python-2019.11.50794\pythonFiles\lib\python\new_ptvsd\no_wheels\ptvsd\launcher/../../ptvsd\launcher\debuggee.py", line 64, in spawn
                 "Couldn't spawn debuggee: {0}\n\nCommand line:{1!r}", exc, cmdline
             TypeError: unbound method cant_handle() must be called with Message instance as first argument (got unicode instance instead)

             Stack where logged:
               File "C:\Python27-32\lib\threading.py", line 774, in __bootstrap
                 self.__bootstrap_inner()
               File "C:\Python27-32\lib\threading.py", line 801, in __bootstrap_inner
                 self.run()
               File "C:\Python27-32\lib\threading.py", line 754, in run
                 self.__target(*self.__args, **self.__kwargs)
               File "c:\Users\linshimeng01\.vscode\extensions\ms-python.python-2019.11.50794\pythonFiles\lib\python\new_ptvsd\no_wheels\ptvsd\launcher/../../ptvsd\common\messaging.py", line 1520, in _run_handlers
                 handler()
               File "c:\Users\linshimeng01\.vscode\extensions\ms-python.python-2019.11.50794\pythonFiles\lib\python\new_ptvsd\no_wheels\ptvsd\launcher/../../ptvsd\common\messaging.py", line 809, in _handle
                 self.describe(),
```
Now I installed an Python 3 interpreter, and I found the two Python 2 interpreter are all unavailable and throw the same error message above as well when I attempt to run a python file. How can I fix it?

Extension version: 2019.11.50794
VS Code version: Code 1.40.1 (8795a9889db74563ddd43eb0a897a2384129a619, 2019-11-13T16:49:35.976Z)
OS version: Windows_NT x64 10.0.17763

<details>
<summary>System Info</summary>

|Item|Value|
|---|---|
|CPUs|Intel(R) Core(TM) i5-9400F CPU @ 2.90GHz (6 x 2904)|
|GPU Status|2d_canvas: enabled<br>flash_3d: enabled<br>flash_stage3d: enabled<br>flash_stage3d_baseline: enabled<br>gpu_compositing: enabled<br>metal: disabled_off<br>multiple_raster_threads: enabled_on<br>oop_rasterization: disabled_off<br>protected_video_decode: unavailable_off<br>rasterization: enabled<br>skia_renderer: disabled_off<br>surface_control: disabled_off<br>surface_synchronization: enabled_on<br>video_decode: enabled<br>viz_display_compositor: enabled_on<br>viz_hit_test_surface_layer: disabled_off<br>webgl: enabled<br>webgl2: enabled|
|Load (avg)|undefined|
|Memory (System)|15.93GB (7.79GB free)|
|Process Argv||
|Screen Reader|no|
|VM|67%|
</details>
<!-- generated by issue reporter -->


Discussion:
This isn't directly caused by installation of interpreters - it just coincided with an update to the major new release of the debugger for you.

The real problem here is that it can't start the debugged program at all. Unfortunately, the code that is trying to report that error, contains a bug itself, which causes this seemingly unrelated message.

Can you please share your launch.json? I might be able to tell why it's not starting from that.
> This isn't directly caused by installation of interpreters - it just coincided with an update to the major new release of the debugger for you.
> 
> The real problem here is that it can't start the debugged program at all. Unfortunately, the code that is trying to report that error, contains a bug itself, which causes this seemingly unrelated message.
> 
> Can you please share your launch.json? I might be able to tell why it's not starting from that.


{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Current File",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "integratedTerminal"
        }
    ]
}
Since it uses "integratedTerminal", the terminal should contain the command line that was used to launch the app - can you share that, as well?
> Since it uses "integratedTerminal", the terminal should contain the command line that was used to launch the app - can you share that, as well?

${env:PTVSD_SESSION_ID}='1'; ${env:PTVSD_LAUNCHER_PORT}='65174'; & {%pythonpath%} 'c:\Users\linshimeng01\.vscode\extensions\ms-python.python-2019.11.50794\pythonFiles\lib\python\new_ptvsd\no_wheels\ptvsd\launcher' {%filename%}
Is there anything unusual about Python path? The error indicates that it fails to even spawn the process, which usually happens when it points at the wrong or non-existing Python binary, for example.
> Is there anything unusual about Python path? The error indicates that it fails to even spawn the process, which usually happens when it points at the wrong or non-existing Python binary, for example.

I am afraid it is not the path problem, because the Python paths are just 'C:\Python27\python.exe' and 'C:\Python27-32\python.exe'. which are the correct Python binaries that I am sure. 
@Danielslin can you run Python 2 directly from the terminal to see what happens? 

i.e.: in the terminal run:

`C:\Python27\python.exe <filename>`

What happens in this case? Does it work? Do you have some error message?
> @Danielslin can you run Python 2 directly from the terminal to see what happens?
> 
> i.e.: in the terminal run:
> 
> `C:\Python27\python.exe <filename>`
> 
> What happens in this case? Does it work? Do you have some error message?

Yes, In CMD and Powershell it works correctly. No error.
Can you change:

`raise messaging.Message.cant_handle(`

to

`raise AssertionError(`

in 

`c:\Users\linshimeng01\.vscode\extensions\ms-python.python-2019.11.50794\pythonFiles\lib\python\new_ptvsd\no_wheels\ptvsd\launcher/../../ptvsd\launcher\debuggee.py`

and then rerun and provide the error?
(in line 64)
It won't quite work, because that exception also does formatting. It needs to become this:
```py
            raise AssertionError(fmt(
                "Couldn't spawn debuggee: {0}\n\nCommand line:{1!r}", exc, cmdline
            ))
```
> Can you change:
> 
> `raise messaging.Message.cant_handle(`
> 
> to
> 
> `raise AssertionError(`
> 
> in
> 
> `c:\Users\linshimeng01\.vscode\extensions\ms-python.python-2019.11.50794\pythonFiles\lib\python\new_ptvsd\no_wheels\ptvsd\launcher/../../ptvsd\launcher\debuggee.py`
> 
> and then rerun and provide the error?

             Traceback (most recent call last):
               File "c:\Users\linshimeng01\.vscode\extensions\ms-python.python-2019.11.50794\pythonFiles\lib\python\new_ptvsd\no_wheels\ptvsd\launcher/../../ptvsd\common\messaging.py", line 763, in _handle
                 result = handler(self)
               File "c:\Users\linshimeng01\.vscode\extensions\ms-python.python-2019.11.50794\pythonFiles\lib\python\new_ptvsd\no_wheels\ptvsd\launcher/../../ptvsd\launcher\adapter.py", line 147, in launch_request
                 debuggee.spawn(process_name, cmdline, cwd, env, redirect_output)
               File "c:\Users\linshimeng01\.vscode\extensions\ms-python.python-2019.11.50794\pythonFiles\lib\python\new_ptvsd\no_wheels\ptvsd\launcher/../../ptvsd\launcher\debuggee.py", line 64, in spawn
                 fmt("Couldn't spawn debuggee: {0}\n\nCommand line:{1!r}", exc, cmdline)
             
AssertionError: Couldn't spawn debuggee: environment can only contain strings

Looks like a unicode problem of Python 2?
I transfer the keys and values of the environment dictionary into string type before subprocess.Popen and it works.  
.../ptvsd/launcher/debuggee.py, line 59:
```
        try:
            global process

            # add one line here
            env = {str(k):str(v) for k,v in env.iteritems()}

            process = subprocess.Popen(cmdline, cwd=cwd, env=env, bufsize=0, **kwargs)
        except Exception as exc:
            raise AssertionError(
                fmt("Couldn't spawn debuggee: {0}\n\nCommand line:{1!r}", exc, cmdline)
            )

```
Very interesting, thank you! There's already code that does (almost) exactly this thing for Python 2, it just happens to be in the function that calls this one:
https://github.com/microsoft/ptvsd/blob/5d674c49e5fc1fb150a3358379211749112aed11/src/ptvsd/launcher/adapter.py#L138-L146
So it's not clear to me as to how it ends up in this situation. Can you dump `repr(env)` at the point where you edited it, and check which variable ends up having the wrong type, and what value it has?


> Very interesting, thank you! There's already code that does (almost) exactly this thing for Python 2, it just happens to be in the function that calls this one:
> https://github.com/microsoft/ptvsd/blob/5d674c49e5fc1fb150a3358379211749112aed11/src/ptvsd/launcher/adapter.py#L138-L146
> 
> 
> So it's not clear to me as to how it ends up in this situation. Can you dump `repr(env)` at the point where you edited it, and check which variable ends up having the wrong type, and what value it has?

I check the types of the key-values in env. Their types are all  <class 'future.types.newstr.newstr'> (both key and value) except env['PYTHONUNBUFFERED'] and env['PYTHONIOENCODING'] which are <type 'str'>. 
Ah, that explains it!

`future.types.newstr.newstr` is a type from `python-future`, which is a compatibility library to write code that works the same on Python 2 and 3.

ptvsd itself does not depend on it or use it. But that library, when installed on Python 3, among other things, installs a module named `builtins` - that emulates the corresponding standard Python 3 module. And exposes various aliases from it - including `builtins.str` as an alias for `newstr`. And ptvsd has some of its own logic to adapt itself to either Python 2 or 3, that, among other things, tries to import `builtins.str`. It's expected to fail on Python 2 (in which case it falls back to `__builtin__.str`) - but with this library installed, it succeeds on Python 2, but gets `newstr`. Which, pretty much, breaks everything.

The proper fix for this is to swap the order of imports:
https://github.com/microsoft/ptvsd/blob/5d674c49e5fc1fb150a3358379211749112aed11/src/ptvsd/common/compat.py#L17-L20
i.e. it should try `__builtin__` first, and then `builtins` only if that fails:
```py
try:
    import __builtin__ as builtins
except ImportError:
    import builtins
```
Can you try doing this, and see if that fixes it for you, in lieu of manually str'ifying everything in debuggee.py?
> Ah, that explains it!
> 
> `future.types.newstr.newstr` is a type from `python-future`, which is a compatibility library to write code that works the same on Python 2 and 3.
> 
> ptvsd itself does not depend on it or use it. But that library, when installed on Python 3, among other things, installs a module named `builtins` - that emulates the corresponding standard Python 3 module. And exposes various aliases from it - including `builtins.str` as an alias for `newstr`. And ptvsd has some of its own logic to adapt itself to either Python 2 or 3, that, among other things, tries to import `builtins.str`. It's expected to fail on Python 2 (in which case it falls back to `__builtin__.str`) - but with this library installed, it succeeds on Python 2, but gets `newstr`. Which, pretty much, breaks everything.
> 
> The proper fix for this is to swap the order of imports:
> https://github.com/microsoft/ptvsd/blob/5d674c49e5fc1fb150a3358379211749112aed11/src/ptvsd/common/compat.py#L17-L20
> 
> 
> i.e. it should try `__builtin__` first, and then `builtins` only if that fails:
> ```python
> try:
>     import __builtin__ as builtins
> except ImportError:
>     import builtins
> ```
> 
> Can you try doing this, and see if that fixes it for you, in lieu of manually str'ifying everything in debuggee.py?

Yes, it works!! Thank you very much for your help.
Thank you for your patience in helping figure this out! 

Resolving patch:
diff --git a/src/ptvsd/common/compat.py b/src/ptvsd/common/compat.py
--- a/src/ptvsd/common/compat.py
+++ b/src/ptvsd/common/compat.py
@@ -15,9 +15,9 @@
 
 
 try:
-    import builtins
+    import __builtin__ as builtins
 except ImportError:
-    import __builtin__ as builtins  # noqa
+    import builtins
 
 try:
     unicode = builtins.unicode


Test patch:
diff --git a/tests/debug/session.py b/tests/debug/session.py
--- a/tests/debug/session.py
+++ b/tests/debug/session.py
@@ -512,7 +512,7 @@ def _start_channel(self, stream):
         telemetry = self.wait_for_next_event("output")
         assert telemetry == {
             "category": "telemetry",
-            "output": "ptvsd.adapter",
+            "output": "ptvsd",
             "data": {"packageVersion": some.str},
         }