Repository: microsoft/ptvsd

Issue:
breakpoint are not hit when debugger is attached to remote target
## Environment data

- PTVSD version: 4.2.0
- OS and version: Linux, Fedora 29
- Python version: Python 2.7.15
- Using VS Code or Visual Studio:
Version: 1.30.2
Commit: 61122f88f0bf01e2ac16bdb9e1bc4571755f5bd8
Date: 2019-01-07T22:49:48.319Z
Electron: 2.0.12
Chrome: 61.0.3163.100
Node.js: 8.9.3
V8: 6.1.534.41
OS: Linux x64 4.19.13-300.fc29.x86_64


Target machine:
- OS and version: Linux, Ubuntu 18.04LTS
- PTVSD version: 4.2.0
- Python 2.7.15rc1

## Actual behavior
I try to debug remotely python script with vscode. In my launch.json i have proper configuration including pathMappings.

I have try following cases:
1. Run script on target machine like:
python -m ptvsd --single-session --host 10.3.72.215 --port 3000 --wait --multiprocess script-debug
On vscode i put break point and attach to target. As result script-debug is executed without hitting break point and on target machine i see message:
pydev debugger: warning: trying to add breakpoint to file that does not exist: "settings from localRoot /script-debug"
 Some times i can manage to start debugging - i.e. debugger stops at BP, but this is less than 1% of all attempts.

2. Run script on target but modified:
`_script-debug_`
```py
import os
import sys
import ptvsd

# # Allow other computers to attach to ptvsd at this IP address and port.
ptvsd.enable_attach(address=('0.0.0.0', 3000), redirect_output=True)

# # Pause the program until a remote debugger is attached
ptvsd.wait_for_attach()
```

Same behavior like p.1 

3. Run script on target but more modified:
`_script-debug_`
```py
import os
import sys
import ptvsd

# # Allow other computers to attach to ptvsd at this IP address and port.
ptvsd.enable_attach(address=('0.0.0.0', 3000), redirect_output=True)

# # Pause the program until a remote debugger is attached
ptvsd.wait_for_attach()
ptvsd.break_into_debugger()
```

In this case 100% of attempts as successful if i remove/disable all BP from vscode and add/enable them once debugger is attached.

If i start debugger with breakpoints, that are set before attachment to remote target, again see message on remote site:
pydev debugger: warning: trying to add breakpoint to file that does not exist: "settings from localRoot /script-debug"
So each time before debug session i have to disable BP, attach to target and re-enable them in order to use debugger.

Looks like some kind of race condition when debugger is attached and pathMappings, and BP are set to remote side.

## Expected  behavior
To be possible to run target like 'python -m ptvsd --single-session --host 10.3.72.215 --port 3000 --wait --multiprocess script-debug" and not to be forced to use ptvsd.break_into_debugger() in order to set BP

Already configured BP to work without needs to be re-enabled.




Discussion:
@dim2iv Can you share the `launch.json` configuration you are using for this?
@karthiknadig following is content of lauch.json
```js
{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Attach (nfvt-up)",
            "type": "python",
            "request": "attach",
            "port": 3000,
            "host": "10.3.72.215",
            "pathMappings": [
                
                { 
                    "localRoot": "${workspaceFolder}/nfvtime-upgrade/orig/", 
                    "remoteRoot": "/opt/bin/" 
                }
            ],
            "logToFile": false,
            "redirectOutput": true,
            "subProcess": true
        },
        {
            "name": "Python: Current File (Integrated Terminal)",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "integratedTerminal"
        },
        {
            "name": "Python: Module",
            "type": "python",
            "request": "launch",
            "module": "enter-your-module-name-here",
            "console": "integratedTerminal"
        },
        {
            "name": "Python: Django",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/manage.py",
            "console": "integratedTerminal",
            "args": [
                "runserver",
                "--noreload",
                "--nothreading"
            ],
            "django": true
        },
        {
            "name": "Python: Flask",
            "type": "python",
            "request": "launch",
            "module": "flask",
            "env": {
                "FLASK_APP": "app.py"
            },
            "args": [
                "run",
                "--no-debugger",
                "--no-reload"
            ],
            "jinja": true
        },
        {
            "name": "Python: Current File (External Terminal)",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "externalTerminal"
        }
    ]
}
```
For clarification, the full path for `script-debug` on remote is `/opt/bin/script-debug`?
Yes. This path on remote (target) machine.
@dim2iv I tried this, it seems to be working. Can you share your project? This might be specific to the project.
@karthiknadig here is a file. Is it possible reason to be that some of imported modules do not exists on host (locals) system, but exists on target ( remote ) system ?
[source.zip](https://github.com/Microsoft/ptvsd/files/2748617/source.zip)


@dim2iv if the files are missing locally, you get a message box that pops up on VS Code that says 'File not found'. 

Unfortunately, I am not able to repro this at all (with the source file you provided). Can you do one more thing, set `"logToFile": true,` and attach the log file here? You may want to remove any personal info from the log file. The log file should be in  `C:\Users\<username>\.vscode\extensions\ms-python.python-2018.12.1\debug<pid>.log`.
@karthiknadig with missing files i mean, that some modules imported do not exists on host system. They are related to (like apt_pkg). They are specific for the Ubuntu (target system), while on development host they are not installed at all. Development host is Fedora.
I will will run with enabled logging and will try to collect and send files for good and bad cases. 
@karthiknadig Here You can find logs for:
Bad case 1 - script is started on target like this '_python -m ptvsd --host 10.3.72.215 --port 3000 --wait nfvt-up_'
Bad case 2 - script is modified and code is added at start of file
```py
_#! /usr/bin/env python
import os
import sys
import ptvsd

# # Allow other computers to attach to ptvsd at this IP address and port.
ptvsd.enable_attach(address=('0.0.0.0', 3000), redirect_output=True)

# Pause the program until a remote debugger is attached
ptvsd.wait_for_attach()_
```

In both cases bp is set to line 1300
In both cases debugger just catch exception on sys.exit()
 Do You need log in case where ptvsd.break_into_debugger() is called ?
[bad_case2.zip](https://github.com/Microsoft/ptvsd/files/2754350/bad_case2.zip)
[bad-case1.zip](https://github.com/Microsoft/ptvsd/files/2754351/bad-case1.zip)


Can you try the following and see if that helps?
1. Try moving the location on the remote machine to something other than `/opt/bin`. 
2. Try setting `"debugStdLib": true`.
I suspect the issue might be that the debugger is ignoring the break-point because it thinks the break-point is in a standard library and skips it (it should not).

If either of the above steps did not help then, can you give me the location of site-packages and python installs on the remote machine? Run the code below using the same python you use to launch your remote script
```py
import sys
import site
print('SysPath:' + str(sys.path))
print('Prefix: ' + sys.prefix)
if hasattr(sys, 'base_prefix'):
    print('BasePrefix: ' + sys.base_prefix)
if hasattr(sys, 'real_prefix'):
    print('RealPrefix: ' + sys.real_prefix)
if hasattr(sys, 'exec_prefix'):
    print('ExecPrefix: ' + sys.exec_prefix)

if hasattr(site, 'getusersitepackages'):
    print('User pkg: ' + str(site.getusersitepackages()))
if hasattr(site, 'getsitepackages'):
    print('Site pkg: ' + str(site.getsitepackages()))
```
Let me know how this goes.
@karthiknadig i notice something very interesting.
I have execute debug session using p.1 and p.2. I have moved my script in /root directory and start it like _python -m ptvsd --host 10.3.72.215 --port 3000 --wait nfvt-up_ 
Result - debugger hit planted BP at line 1300. All is ok and can use the debugger.
After that i decide to run another one session. Note that i do not change anything in code of debugged script or in settings. 
Result - debugger do not hit planted BP ?!?!?
After that i restart VSCode and repeat again. All was same like before - first run debug session is ok, all following - do not stops at BP. Please see attached debug log files. File with less pid is good one. And i notice by simple compare that in good case there is extra rows in log. Please take a look at line 71. The difference starts from here. 
May be key point is restart of VSCode, where some cached information is removed ?

Here is also output for site-packeges:

SysPath:['/root', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']
Prefix: /usr
ExecPrefix: /usr
User pkg: /root/.local/lib/python2.7/site-packages
Site pkg: ['/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']

Sorry for the long post but since we have big time shift i think it can help You if all information is available.
[case-first-second.zip](https://github.com/Microsoft/ptvsd/files/2758897/case-first-second.zip)



Thanks you for the logs. It looks like the breakpoint event is not getting triggered in the second case. Can you run the debugger with following env variable on the remote machine?
`PTVSD_DEBUG=True`

You should see some output like this. Please capture that output for the good and bad case.
```
sending output
output sent
<- 501 (CMD_VERSION)            [1000000000]: |1.1      WINDOWS ID|
-> 501 (CMD_VERSION)            [1000000000]: |@@BUILD_NUMBER@@\n|
<- 111 (CMD_SET_BREAK)          [1000000001]: |1        python-line     C:\GIT\pyscratch3\scratch1.py   3       None
None    None    None    None    ALL|
<- 501 (CMD_VERSION)            [1000000002]: |1.1      WINDOWS ID|
-> 501 (CMD_VERSION)            [1000000002]: |@@BUILD_NUMBER@@\n|
<- 122 (CMD_ADD_EXCEPTION_BREAK) [1000000003]: |python-BaseException    0       1       1|
<- 200 (CMD_REDIRECT_OUTPUT)    [1000000004]: |STDOUT   STDERR|
<- 156 (CMD_PYDEVD_JSON_CONFIG) [1000000006]: |{"skip_suspend_on_breakpoint_exception": ["BaseException"], "skip_print_breakpoint_exception": ["NameError"], "multi_threads_single_notification": true}|
<- 101 (CMD_RUN)                [1000000007]: ||
ready to debug
<- 102 (CMD_LIST_THREADS)       [1000000008]: ||
-> 502 (CMD_RETURN)             [1000000008]: |<xml><thread name="MainThread" id="pid_16412_id_2374191782712" /></xml>\n|
-> 103 (CMD_THREAD_CREATE)      [         2]: |<xml><thread name="MainThread" id="pid_16412_id_2374191782712" /></xml>\n|
-> 105 (CMD_THREAD_SUSPEND)     [         4]: |<xml><thread id="pid_16412_id_2374191782712" stop_reason="111" message="" suspend_type="trace"><frame id="2374208578632" name="&lt;module&gt;" file="C%3A%5CGIT%5Cpyscratch3%5Cscratch1.py" line="3"></frame></thread></xml>\n|
-> 157 (CMD_THREAD_SUSPEND_SINGLE_NOTIFICATION) [         6]: |{"thread_id": "pid_16412_id_2374191782712", "stop_reason": 111}\n|
<- 102 (CMD_LIST_THREADS)       [1000000009]: ||
-> 502 (CMD_RETURN)             [1000000009]: |<xml><thread name="MainThread" id="pid_16412_id_2374191782712" /></xml>\n|
<- 152 (CMD_GET_THREAD_STACK)   [1000000010]: |pid_16412_id_2374191782712|
.
.
.
```
@karthiknadig You can find logs attached. As i see on bad case ptvsd complains that file is unknown. Looks like mappings are not set second ( and following ) run(s).
[good.zip](https://github.com/Microsoft/ptvsd/files/2767493/good.zip)
[bad.zip](https://github.com/Microsoft/ptvsd/files/2767495/bad.zip)


@fabioz have you seen this issue. Looks like path mapping, fails often, in this scenario.

good case:
```
<- 501 (CMD_VERSION)            [1000000000]: |1.1      UNIX    ID|
-> 501 (CMD_VERSION)            [1000000000]: |@@BUILD_NUMBER@@\n|
<- 111 (CMD_SET_BREAK)          [1000000001]: |1        python-line     /home/mitko/git/nfvtime-os/packages/nfvtime-upgrade/orig/nfvt-up        1300    None    None    None    None    None       ALL|
<- 501 (CMD_VERSION)            [1000000002]: |1.1      UNIX    ID|
-> 501 (CMD_VERSION)            [1000000002]: |@@BUILD_NUMBER@@\n|
```
bad case:
```
<- 501 (CMD_VERSION)            [1000000000]: |1.1      UNIX    ID|
-> 501 (CMD_VERSION)            [1000000000]: |@@BUILD_NUMBER@@\n|
<- 111 (CMD_SET_BREAK)          [1000000001]: |1        python-line     /home/mitko/git/nfvtime-os/packages/nfvtime-upgrade/orig/nfvt-up        1300    None    None    None    None    None       ALL|
pydev debugger: warning: trying to add breakpoint to file that does not exist: /home/mitko/git/nfvtime-os/packages/nfvtime-upgrade/orig/nfvt-up (will have no effect)
<- 501 (CMD_VERSION)            [1000000002]: |1.1      UNIX    ID|
-> 501 (CMD_VERSION)            [1000000002]: |@@BUILD_NUMBER@@\n|
```  

I see a success response from DAP logs for both cases. Any idea where this might be breaking?
Unfortunately the log seems to be incomplete as there's no info on the file translation being set -- and it really seems like the translation is set on one case but not in the other...

@dim2iv, can you do the following: 

1. Set an environment variable named `DEBUG_PYDEVD_PATHS_TRANSLATION` with a value of `1`.

2. Open the file: `pydevd_file_utils.py` (to get its location you can create a program with the contents below and run it):

```
import pydevd_file_utils
print(pydevd_file_utils.__file__)
```

Then put a `sys.stderr.write('setup_client_server_paths: %s\n' % (paths,))` as the first line in the method `setup_client_server_paths` .

3. Run that and provide the output on the good/bad case.


@fabioz i  do not have installed pydevd installed on host or on target machine.
Python 2.7.15rc1 (default, Apr 15 2018, 21:51:34)
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pydevd_file_utils
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named pydevd_file_utils
>>>
In VScode is not mentioned that i have to install it in order to able to debug.
@dim2iv `ptvsd` bundles `pydevd`, so, I thought it'd be there... try to do:

```
import ptvsd
print(ptvsd.__file__)
```

then, the location of `pydevd_file_utils.py` should be under the `ptvsd` folder `/_vendored/pydevd/pydevd_file_utils.py`.
Hi @fabioz and @karthiknadig,
Requested logs are attached. Application is started on target like:
_PTVSD_DEBUG=True DEBUG_PYDEVD_PATHS_TRANSLATION=1 python -m ptvsd --host 10.3.72.215 --port 3000 --wait nfvt-up_

I think there is a race condition (look at bold)
Good
=======
<- 501 (CMD_VERSION)            [1000000000]: |1.1	UNIX	ID|
**-> 501 (CMD_VERSION)            [1000000000]: |@@BUILD_NUMBER@@\n|
setup_client_server_paths: [(u'/home/mitko/git/nfvtime-os/packages/nfvtime-upgrade/orig/', u'/root/')]
<- 111 (CMD_SET_BREAK)          [1000000001]: |1	python-line	/home/mitko/git/nfvtime-os/packages/nfvtime-upgrade/orig/nfvt-up	1300	None	None	None	None**	None	ALL|
pydev debugger: replacing to server: /home/mitko/git/nfvtime-os/packages/nfvtime-upgrade/orig/nfvt-up
pydev debugger: sent to server: /root/nfvt-up
<- 501 (CMD_VERSION)            [1000000002]: |1.1	UNIX	ID|
-> 501 (CMD_VERSION)            [1000000002]: |@@BUILD_NUMBER@@\n|
<- 122 (CMD_ADD_EXCEPTION_BREAK) [1000000003]: |python-BaseException	0	1	0|
<- 200 (CMD_REDIRECT_OUTPUT)    [1000000004]: |STDOUT	STDERR|
<- 156 (CMD_PYDEVD_JSON_CONFIG) [1000000005]: |{"skip_print_breakpoint_exception": ["NameError"], "multi_threads_single_notification": true, "skip_suspend_on_breakpoint_exception": ["BaseException"]}|
<- 101 (CMD_RUN)                [1000000006]: ||


Bad
=====
<- 501 (CMD_VERSION)            [1000000000]: |1.1	UNIX	ID|
**<- 111 (CMD_SET_BREAK)          [1000000001]: |1	python-line	/home/mitko/git/nfvtime-os/packages/nfvtime-upgrade/orig/nfvt-up	1300	None	None	None	None	None	ALL|
pydev debugger: warning: trying to add breakpoint to file that does not exist: /home/mitko/git/nfvtime-os/packages/nfvtime-upgrade/orig/nfvt-up (will have no effect)
<- 501 (CMD_VERSION)            [1000000002]: |1.1	UNIX	ID|
-> 501 (CMD_VERSION)            [1000000000]: |@@BUILD_NUMBER@@\n|
-> 501 (CMD_VERSION)            [1000000002]: |@@BUILD_NUMBER@@\n|
setup_client_server_paths: [(u'/home/mitko/git/nfvtime-os/packages/nfvtime-upgrade/orig/', u'/root/')]**
<- 122 (CMD_ADD_EXCEPTION_BREAK) [1000000003]: |python-BaseException	0	1	0|
<- 200 (CMD_REDIRECT_OUTPUT)    [1000000004]: |STDOUT	STDERR|
<- 156 (CMD_PYDEVD_JSON_CONFIG) [1000000005]: |{"skip_print_breakpoint_exception": ["NameError"], "multi_threads_single_notification": true, "skip_suspend_on_breakpoint_exception": ["BaseException"]}|
<- 101 (CMD_RUN)                [1000000006]: ||
ready to debug

[bad-ext.zip](https://github.com/Microsoft/ptvsd/files/2772070/bad-ext.zip)
[good-ext.zip](https://github.com/Microsoft/ptvsd/files/2772071/good-ext.zip)

I notice following when DEBUG_PYDEVD_PATHS_TRANSLATION is set.
pydev debugger: replacing to client: /root/nfvt-up
pydev debugger: sent to client: /home/mitko/git/nfvtime-os/packages/nfvtime-upgrade/orig/nfvt-up
pydev debugger: to client: unable to find matching prefix for: /usr/local/lib/python2.7/dist-packages/ptvsd/_local.py in ['/root']
pydev debugger: to client: unable to find matching prefix for: /usr/local/lib/python2.7/dist-packages/ptvsd/__main__.py in ['/root']
pydev debugger: to client: unable to find matching prefix for: /usr/lib/python2.7/runpy.py in ['/root']
@karthiknadig it really seems like ptvsd is sending breakpoints before setting up the translation in this case (which wasn't expected in `pydevd`)... now, there are 2 possible fixes: 
1. `ptvsd` makes sure that the paths are setup before the breakpoints.
2. `pydevd` itself can update the breakpoints if the translations are sent after breakpoints are set.

If the requests are expected to be sent in order by the `DAP`, then `1` may be reasonable and if it's out of order, `2` must be done (although `2` must be done if we expect the user to be able to change things programmatically later on).

@dim2iv I think a workaround for you for now is updating the method `setup_client_server_paths` (in `pydevd_file_utils.py`:

Put `paths = [(u'/home/mitko/git/nfvtime-os/packages/nfvtime-upgrade/orig/', u'/root/')]` as the first line of the method until this is actually fixed (this should work because the initial translation is called when that module is imported).
@fabioz proposed workaround is too hard coded, since will work just for one script in project. I able to debug for now and will wait for real fix.
@karthiknadig is this is considered as a problem ?

Resolving patch:
diff --git a/src/ptvsd/wrapper.py b/src/ptvsd/wrapper.py
--- a/src/ptvsd/wrapper.py
+++ b/src/ptvsd/wrapper.py
@@ -1213,7 +1213,7 @@ def on_attach(self, request, args):
         multiproc.root_start_request = request
         self.start_reason = 'attach'
         self._set_debug_options(args)
-        self._handle_attach(request, args)
+        self._handle_launch_or_attach(request, args)
         self.send_response(request)
 
     def on_launch(self, request, args):
@@ -1221,7 +1221,7 @@ def on_launch(self, request, args):
         self.start_reason = 'launch'
         self._set_debug_options(args)
         self._notify_launch()
-        self._handle_launch(request, args)
+        self._handle_launch_or_attach(request, args)
         self.send_response(request)
 
     def on_disconnect(self, request, args):
@@ -1302,10 +1302,7 @@ def _process_debug_options(self, opts):
     def _handle_configurationDone(self, request, args):
         pass
 
-    def _handle_attach(self, request, args):
-        pass
-
-    def _handle_launch(self, request, args):
+    def _handle_launch_or_attach(self, request, args):
         pass
 
     def _handle_detach(self):
@@ -1356,6 +1353,8 @@ def __init__(self, socket, pydevd_notify, pydevd_request,
         # adapter state
         self.path_casing = PathUnNormcase()
         self._detached = False
+        self._path_mappings_received = False
+        self._path_mappings_applied = False
 
     def _start_event_loop(self):
         self.loop = futures.EventLoop()
@@ -1609,6 +1608,8 @@ def _initialize_path_maps(self, args):
         if len(self._path_mappings) > 0:
             pydevd_file_utils.setup_client_server_paths(self._path_mappings)
 
+        self._path_mappings_applied = True
+
     def _send_cmd_version_command(self):
         cmd = pydevd_comm.CMD_VERSION
         default_os_type = 'WINDOWS' if platform.system() == 'Windows' else 'UNIX'
@@ -1618,18 +1619,9 @@ def _send_cmd_version_command(self):
         return self.pydevd_request(cmd, msg)
 
     @async_handler
-    def _handle_attach(self, request, args):
-        self.pydevd_request(pydevd_comm.CMD_SET_PROTOCOL, 'json')
-        yield self._send_cmd_version_command()
-
-        pydevd_request = copy.deepcopy(request)
-        del pydevd_request['seq']  # A new seq should be created for pydevd.
-        yield self.pydevd_request(-1, pydevd_request, is_json=True)
-
-        self._initialize_path_maps(args)
+    def _handle_launch_or_attach(self, request, args):
+        self._path_mappings_received = True
 
-    @async_handler
-    def _handle_launch(self, request, args):
         self.pydevd_request(pydevd_comm.CMD_SET_PROTOCOL, 'json')
         yield self._send_cmd_version_command()
 
@@ -2276,6 +2268,15 @@ def _get_bp_type(self, path):
 
     @async_handler
     def on_setBreakpoints(self, request, args):
+        if not self._path_mappings_received:
+            self.send_error_response(request, "'setBreakpoints' request must be issued after 'launch' or 'attach' request.")
+            return
+
+        # There might be a concurrent 'launch' or 'attach' request in flight that hasn't
+        # gotten to processing path mappings yet. If so, spin until it finishes that.
+        while not self._path_mappings_applied:
+            yield self.sleep()
+
         pydevd_request = copy.deepcopy(request)
         del pydevd_request['seq']  # A new seq should be created for pydevd.
         _, _, resp_args = yield self.pydevd_request(


Test patch: