Repository: microsoft/ptvsd

Issue:
Stop on entry hangs the debugger
## Environment data

- PTVSD version: master
- OS and version: windows
- Python version (& distribution if applicable, e.g. Anaconda): 3.6
- Using VS Code or Visual Studio: VS or VSC

## Actual behavior

Note that you have to change the default for STOP_ON_ENTRY in wrapper.py#1198 to True.
Repros almost always on VS, but infrequently on VSC.

## Expected  behavior

Should run and break on entry

## Steps to reproduce:
For VS:
1. Change wrapper.py:1198 default for STOP_ON_ENTRY  to True.
2. Use F5 to start the debugger.

For VSC:
1. Set `stopOnEntry` in launch json.




Discussion:
Note that this occurs with or without the changes below. the change below seems to cause this issues more frequently.
https://github.com/karthiknadig/ptvsd/commit/6a80f7c070c9b652a13eeac325b0056aebb8b34e
@anapo14 let's remove this from the VS Code getting started tutorial since it seems to break from time to time
@karthiknadig are you investigating this? I couldn't reproduce here.

My guess is that it's a racing condition -- it'd be nice to have the `ptvsd` as well as the `pydevd` logs to know what's happening when it fails...

To get the `pydevd` logs, it's possible to set environment variables (not sure about the last release, but it does work in the development version).

```
PYDEVD_DEBUG=True
PYDEVD_DEBUG_FILE=c:\temp\file_to_log.txt
```
[ptvsd-7560.log](https://github.com/microsoft/ptvsd/files/3163421/ptvsd-7560.log)
[pydevd_log.txt](https://github.com/microsoft/ptvsd/files/3163422/pydevd_log.txt)
[VS_adapter_host_logs.txt](https://github.com/microsoft/ptvsd/files/3163423/VS_adapter_host_logs.txt)

This is a hard repro. Today, i am not able to repro it in VSC but only in VS.


@karthiknadig I still can't reproduce it (tried running the related tests thousands of times in the ci and it didn't fail either).

If you can still reproduce it there, can you check some things?

1. It seems that `SetDebuggerPropertyRequest` was sent after the `LaunchRequest` to `pydevd` and this doesn't seem correct for me (we should say that we don't want to deal with the `ptvsd` directories before sending the launch request -- it's possible that the debugger is halting inside `ptvsd` because of that in some unrelated handled exception).

2. If you don't set exception handling, do you still get the hang?

3. Another thing to consider... I see that the `dontTraceStartPatterns` are sent as `c:\\git\\ptvsd\\src\\ptvsd\\`... when you're launching, is that the only path used? (or for instance, is it possible that you have a junction which is being used as the launch path from the IDE -- if you run locally with one path and the IDE launches from the other, the .pyc files may have outdated paths and the `dontTraceStartPatterns` would be wrong in one of those).
Since only i am able to repro, but only on VS. I will try and narrow it down to something specific. 

1. For `setDebuggerProperty` since it is not part of the main protocol yet, i don't think we can guarantee when it should occur. I completely skipped the setDebuggerProperty call and it seems to still repro.

2. Skipping `setExceptionBreakpoints` still caused the repro.

3. Deleted all *.pyc's and tried again. It still repros.
I meant the `SetDebuggerPropertyRequest` which is sent internally from `ptvsd` to `pydevd` passing the `dontTraceEndPatterns` and `dontTraceStartPatterns` (for which `ptvsd` should have full control -- see the `pydevd logs`), not the `SetDebuggerPropertyRequest` which is passed from `VS` to `ptvsd`.
Found the culprit: `module 'ptvsd' has no attribute 'main_thread'`. in wrapper.py `_process_debug_options`.
For some reason, we reach this point in VS with main_thread not set.


Since VS takes `debug` path instead of `__main__` while launching the debugger. It is not setting the `ptvsd.main_thread`. But there seems to be additional issues around how that is getting called. 
@fabioz The `debug` path also seems to trigger a stack overflow in pydevd. Not sure if this stack helps. I will share more details if available. We will be changing that path to use the same stuff in `__main__`. It should likely fix that.
```console
Fatal Python error: Cannot recover from stack overflow.

Thread 0x0000426c (most recent call first):
  File "C:\GIT\ptvsd\src\ptvsd\_vendored\pydevd\pydevd.py", line 197 in _on_run
  File "C:\GIT\ptvsd\src\ptvsd\_vendored\pydevd\_pydevd_bundle\pydevd_comm.py", line 149 in run
  File "C:\Python36\lib\threading.py", line 916 in _bootstrap_inner
  File "C:\Python36\lib\threading.py", line 884 in _bootstrap
  File "C:\GIT\ptvsd\src\ptvsd\_vendored\pydevd\_pydev_bundle\pydev_monkey.py", line 667 in __call__

Thread 0x000033e4 (most recent call first):
  File "C:\GIT\ptvsd\src\ptvsd\_vendored\pydevd\pydevd.py", line 157 in _on_run
  File "C:\GIT\ptvsd\src\ptvsd\_vendored\pydevd\_pydevd_bundle\pydevd_comm.py", line 149 in run
  File "C:\Python36\lib\threading.py", line 916 in _bootstrap_inner
  File "C:\Python36\lib\threading.py", line 884 in _bootstrap
  File "C:\GIT\ptvsd\src\ptvsd\_vendored\pydevd\_pydev_bundle\pydev_monkey.py", line 667 in __call__

Thread 0x00001ab8 (most recent call first):
  File "C:\GIT\ptvsd\src\ptvsd\wrapper.py", line 263 in recv
  File "C:\GIT\ptvsd\src\ptvsd\_vendored\pydevd\_pydevd_bundle\pydevd_comm.py", line 238 in _read_line
  File "C:\GIT\ptvsd\src\ptvsd\_vendored\pydevd\_pydevd_bundle\pydevd_comm.py", line 250 in _on_run
  File "C:\GIT\ptvsd\src\ptvsd\_vendored\pydevd\_pydevd_bundle\pydevd_comm.py", line 149 in run
  File "C:\Python36\lib\threading.py", line 916 in _bootstrap_inner
  File "C:\Python36\lib\threading.py", line 884 in _bootstrap

Thread 0x0000055c (most recent call first):
  File "C:\Python36\lib\threading.py", line 299 in wait
  File "C:\Python36\lib\queue.py", line 173 in get
  File "C:\GIT\ptvsd\src\ptvsd\_vendored\pydevd\_pydevd_bundle\pydevd_comm.py", line 348 in _on_run
  File "C:\GIT\ptvsd\src\ptvsd\_vendored\pydevd\_pydevd_bundle\pydevd_comm.py", line 149 in run
  File "C:\Python36\lib\threading.py", line 916 in _bootstrap_inner
  File "C:\Python36\lib\threading.py", line 884 in _bootstrap

Thread 0x00000870 (most recent call first):
  File "C:\GIT\ptvsd\src\ptvsd\ipcjson.py", line 113 in _buffered_read_line_as_ascii
  File "C:\GIT\ptvsd\src\ptvsd\ipcjson.py", line 154 in _wait_for_message
  File "C:\GIT\ptvsd\src\ptvsd\ipcjson.py", line 272 in process_one_message
  File "C:\GIT\ptvsd\src\ptvsd\ipcjson.py", line 258 in process_messages
  File "C:\GIT\ptvsd\src\ptvsd\wrapper.py", line 699 in process_messages
  File "C:\Python36\lib\threading.py", line 864 in run
  File "C:\Python36\lib\threading.py", line 916 in _bootstrap_inner
  File "C:\Python36\lib\threading.py", line 884 in _bootstrap

Current thread 0x00004564 (most recent call first):
  File "C:\GIT\ptvsd\src\ptvsd\_vendored\pydevd\pydevd.py", line 646 in enable_tracing
  File "C:\GIT\ptvsd\src\ptvsd\_vendored\pydevd\pydevd.py", line 1685 in run
  File "C:\GIT\ptvsd\src\ptvsd\_vendored\pydevd\pydevd.py", line 2353 in main
  File "C:\GIT\ptvsd\src\ptvsd\_local.py", line 140 in _run
  File "C:\GIT\ptvsd\src\ptvsd\_local.py", line 80 in run_file
  File "C:\GIT\ptvsd\src\ptvsd\debugger.py", line 43 in debug
  File "c:\users\karth\appdata\local\microsoft\visualstudio\16.0_8cc59729exp\extensions\microsoft corporation\python\16.0.0\ptvsd_launcher.py", line 119 in <module>

```
I wasn't even aware that there was a different entry point... 

I agree with you that this entry point should really be removed and things should work properly when that's done (the stack itself seems pretty odd, but there are many things done in the `__main__.py` module that can't really be skipped, so, I don't think it's worth investigating that crash from the `debbugger.debug` entry).

Resolving patch:
diff --git a/src/ptvsd/debugger.py b/src/ptvsd/debugger.py
--- a/src/ptvsd/debugger.py
+++ b/src/ptvsd/debugger.py
@@ -2,50 +2,38 @@
 # Licensed under the MIT License. See LICENSE in the project root
 # for license information.
 
-import sys
-
 import ptvsd.log
-from ptvsd._local import run_module, run_file, run_main
-
-
-# TODO: not needed?
-DONT_DEBUG = []
+import ptvsd.options
+from ptvsd.__main__ import run_file, run_module, run_code
 
-LOCALHOST = 'localhost'
 
 RUNNERS = {
-    'module': run_module,  # python -m spam
-    'script': run_file,  # python spam.py
-    'code': run_file,  # python -c 'print("spam")'
-    None: run_file,  # catchall
+    'module': run_module,
+    'script': run_file,
+    'code': run_code,
 }
 
+# Not actually used, but VS will try to add entries to it.
+DONT_DEBUG = []
 
-def debug(filename, port_num, debug_id, debug_options, run_as,
-          _runners=RUNNERS, _extra=None, *args, **kwargs):
 
+# A legacy entrypoint for Visual Studio, to allow older versions to work with new ptvsd.
+# All new code should use the entrypoints in __main__ directly.
+def debug(filename, port_num, debug_id, debug_options, run_as):
     ptvsd.log.to_file()
     ptvsd.log.info('debug{0!r}', (filename, port_num, debug_id, debug_options, run_as))
 
-    if _extra is None:
-        _extra = sys.argv[1:]
-    address = (LOCALHOST, port_num)
     try:
-        run = _runners[run_as]
+        run = RUNNERS[run_as]
     except KeyError:
-        # TODO: fail?
-        run = _runners[None]
-    if _extra:
-        args = _extra + list(args)
-    kwargs.setdefault('singlesession', True)
-    run(address, filename, *args, **kwargs)
-
+        raise ValueError('run_as must be one of: {0!r}'.format(tuple(RUNNERS.keys())))
 
-def run(filename, port_num, run_as,
-        *args, **kwargs):
+    ptvsd.options.target_kind = 'file' if run_as == 'script' else run_as
+    ptvsd.options.target = filename
+    ptvsd.options.port = port_num
+    ptvsd.options.client = True
 
-    ptvsd.log.to_file()
-    ptvsd.log.info('run{0!r}', (filename, port_num, run_as))
+    # debug_id is ignored because it has no meaning in DAP.
+    # debug_options are ignored, because they will be passed later via DAP "launch" request.
 
-    address = (LOCALHOST, port_num)
-    run_main(address, filename, run_as, *args, **kwargs)
+    run()


Test patch:
diff --git a/tests/func/test_run.py b/tests/func/test_run.py
--- a/tests/func/test_run.py
+++ b/tests/func/test_run.py
@@ -88,3 +88,30 @@ def code_to_debug():
             'category': 'stdout',
             'output': 'ok',
         })))
+
+
+@pytest.mark.parametrize('run_as', ['script', 'module'])
+def test_run_vs(pyfile, run_as):
+    @pyfile
+    def code_to_debug():
+        # import_and_enable_debugger
+        import backchannel
+        backchannel.write_json('ok')
+
+    @pyfile
+    def ptvsd_launcher():
+        # import_and_enable_debugger
+        import ptvsd.debugger
+        import backchannel
+        args = tuple(backchannel.read_json())
+        print('debug%r' % (args,))
+        ptvsd.debugger.debug(*args)
+
+    with DebugSession() as session:
+        filename = 'code_to_debug' if run_as == 'module' else code_to_debug
+        session.before_connect = lambda: session.write_json([filename, session.ptvsd_port, None, None, run_as])
+
+        session.initialize(target=('file', ptvsd_launcher), start_method='custom_client', use_backchannel=True)
+        session.start_debugging()
+        assert session.read_json() == 'ok'
+        session.wait_for_exit()
diff --git a/tests/helpers/session.py b/tests/helpers/session.py
--- a/tests/helpers/session.py
+++ b/tests/helpers/session.py
@@ -40,7 +40,7 @@ class DebugSession(object):
     StopInfo = namedtuple('StopInfo', 'thread_stopped, stacktrace, thread_id, frame_id')
 
     def __init__(self, start_method='launch', ptvsd_port=None, pid=None):
-        assert start_method in ('launch', 'attach_pid', 'attach_socket_cmdline', 'attach_socket_import')
+        assert start_method in ('launch', 'attach_pid', 'attach_socket_cmdline', 'attach_socket_import', 'custom_client')
         assert ptvsd_port is None or start_method.startswith('attach_socket_')
 
         print('New debug session with method %r' % str(start_method))
@@ -198,6 +198,9 @@ def _get_argv_for_attach_using_pid(self):
         # argv += ['--pid', '<pid>']  # pid value to be appended later
         return argv
 
+    def _get_argv_for_custom_client(self):
+        return [sys.executable]
+
     def _get_target(self):
         argv = []
         run_as, path_or_code = self.target
@@ -242,10 +245,18 @@ def _setup_session(self, **kwargs):
         for k, v in kwargs.items():
             setattr(self, k, v)
 
-        assert self.start_method in ('launch', 'attach_pid', 'attach_socket_cmdline', 'attach_socket_import')
+        assert self.start_method in ('launch', 'attach_pid', 'attach_socket_cmdline', 'attach_socket_import', 'custom_client')
         assert len(self.target) == 2
         assert self.target[0] in ('file', 'module', 'code')
 
+    def before_connect(self):
+        """Invoked by initialize() before connecting to the debuggee, or before waiting
+        for an incoming connection, but after all the session parameters (port number etc)
+        are determined."""
+
+        # The default implementation does nothing - this is a hook for the tests to override.
+        pass
+
     def initialize(self, **kwargs):
         """Spawns ptvsd using the configured method, telling it to execute the
         provided Python file, module, or code, and establishes a message channel
@@ -275,6 +286,9 @@ def initialize(self, **kwargs):
         elif self.start_method == 'attach_pid':
             self._listen()
             dbg_argv += self._get_argv_for_attach_using_pid()
+        elif self.start_method == 'custom_client':
+            self._listen()
+            dbg_argv += self._get_argv_for_custom_client()
         else:
             pytest.fail()
 
@@ -339,7 +353,9 @@ def initialize(self, **kwargs):
             temp_process = subprocess.Popen(dbg_argv)
             print('temp process has pid=%d' % temp_process.pid)
 
-        if self.start_method not in ('launch', 'attach_pid'):
+        self.before_connect()
+
+        if self.start_method not in ('launch', 'attach_pid', 'custom_client'):
             self.connect()
         self.connected.wait()
         assert self.ptvsd_port
diff --git a/tests/test_debugger.py b/tests/test_debugger.py
deleted file mode 100644
--- a/tests/test_debugger.py
+++ /dev/null
@@ -1,185 +0,0 @@
-# Copyright (c) Microsoft Corporation. All rights reserved.
-# Licensed under the MIT License. See LICENSE in the project root
-# for license information.
-
-import sys
-import pytest
-
-from ptvsd.debugger import debug, LOCALHOST
-from ptvsd.socket import Address
-
-
-PROG = 'eggs'
-PORT_ARGS = ['--port', '8888']
-PYDEVD_DEFAULT_ARGS = ['--qt-support=auto']
-
-
-def _get_args(*args, **kwargs):
-    ptvsd_extras = kwargs.get('ptvsd_extras', [])
-    prog = [kwargs.get('prog', PROG)]
-    port = kwargs.get('port', PORT_ARGS)
-    pydevd_args = kwargs.get('pydevd', PYDEVD_DEFAULT_ARGS)
-    return prog + port + ptvsd_extras + pydevd_args + list(args)
-
-
-class TestDebug(object):
-
-    @pytest.fixture()
-    def setUp(self):
-        def _make_run(kind):
-            def run(addr, name, *args, **kwargs):
-                self._run(kind, addr, name, *args, **kwargs)
-            return run
-        self.runners = {}
-        for kind in ('module', 'script', 'code', None):
-            self.runners[kind] = _make_run(kind)
-        self.kind = None
-        self.args = None
-        self.kwargs = None
-
-    def _run(self, kind, *args, **kwargs):
-        self.kind = kind
-        self.args = args
-        self.kwargs = kwargs
-
-    def test_module(self, setUp):
-        filename = 'spam'
-        _, port = addr = (LOCALHOST, 8888)
-        debug_id = 1
-        debug_options = {'x': 'y'}
-        debug(filename, port, debug_id, debug_options, 'module',
-              _runners=self.runners, _extra=())
-
-        assert self.kind == 'module'
-        assert self.args == (addr, filename)
-        assert self.kwargs == {'singlesession': True}
-
-    def test_script(self, setUp):
-        filename = 'spam.py'
-        _, port = addr = (LOCALHOST, 8888)
-        debug_id = 1
-        debug_options = {'x': 'y'}
-        debug(filename, port, debug_id, debug_options, 'script',
-              _runners=self.runners, _extra=())
-
-        assert self.kind == 'script'
-        assert self.args == (addr, filename)
-        assert self.kwargs == {'singlesession': True}
-
-    def test_code(self, setUp):
-        filename = "print('spam')"
-        _, port = addr = (LOCALHOST, 8888)
-        debug_id = 1
-        debug_options = {'x': 'y'}
-        debug(filename, port, debug_id, debug_options, 'code',
-              _runners=self.runners, _extra=())
-
-        assert self.kind == 'code'
-        assert self.args == (addr, filename)
-        assert self.kwargs == {'singlesession': True}
-
-    def test_unsupported(self, setUp):
-        filename = 'spam'
-        _, port = addr = (LOCALHOST, 8888)
-        debug_id = 1
-        debug_options = {'x': 'y'}
-        debug(filename, port, debug_id, debug_options, '???',
-              _runners=self.runners, _extra=())
-
-        assert self.kind is None
-        assert self.args == (addr, filename)
-        assert self.kwargs == {'singlesession': True}
-
-    def test_extra_sys_argv(self, setUp):
-        filename = 'spam.py'
-        _, port = addr = (LOCALHOST, 8888)
-        debug_id = 1
-        debug_options = {'x': 'y'}
-        extra = ['--eggs', 'abc']
-        debug(filename, port, debug_id, debug_options, 'script',
-              _runners=self.runners, _extra=extra)
-
-        assert self.args == (addr, filename, '--eggs', 'abc')
-        assert self.kwargs == {'singlesession': True}
-
-
-class TestIntegration(object):
-
-    @pytest.fixture(scope='function')
-    def setUp(self):
-        self.argv = None
-        self.addr = None
-        self.kwargs = None
-        self._sys_argv = list(sys.argv)
-        yield
-        sys.argv[:] = self._sys_argv
-
-    def _run(self, argv, addr, **kwargs):
-        self.argv = argv
-        self.addr = addr
-        self.kwargs = kwargs
-
-    def test_module(self, setUp):
-        filename = 'spam'
-        port = 8888
-        debug_id = 1
-        debug_options = {'x': 'y'}
-        sys.argv = [filename]
-        debug(filename, port, debug_id, debug_options, 'module',
-              _run=self._run, _prog='eggs')
-
-        assert self.argv == _get_args('--module', '--file', 'spam:', ptvsd_extras=['--client', LOCALHOST])
-        assert self.addr == Address.as_client(None, port)
-        assert self.kwargs == {'singlesession': True}
-
-    def test_script(self, setUp):
-        filename = 'spam.py'
-        port = 8888
-        debug_id = 1
-        debug_options = {'x': 'y'}
-        sys.argv = [filename]
-        debug(filename, port, debug_id, debug_options, 'script',
-              _run=self._run, _prog='eggs')
-
-        assert self.argv == _get_args('--file', 'spam.py', ptvsd_extras=['--client', LOCALHOST])
-        assert self.addr == Address.as_client(None, port)
-        assert self.kwargs == {'singlesession': True}
-
-    def test_code(self, setUp):
-        filename = "print('spam')"
-        port = 8888
-        debug_id = 1
-        debug_options = {'x': 'y'}
-        sys.argv = [filename]
-        debug(filename, port, debug_id, debug_options, 'code',
-              _run=self._run, _prog='eggs')
-
-        assert self.argv == _get_args('--file', filename, ptvsd_extras=['--client', LOCALHOST])
-        assert self.addr == Address.as_client(None, port)
-        assert self.kwargs == {'singlesession': True}
-
-    def test_unsupported(self, setUp):
-        filename = 'spam'
-        port = 8888
-        debug_id = 1
-        debug_options = {'x': 'y'}
-        sys.argv = [filename]
-        debug(filename, port, debug_id, debug_options, '???',
-              _run=self._run, _prog='eggs')
-
-        assert self.argv == _get_args('--file', 'spam', ptvsd_extras=['--client', LOCALHOST])
-        assert self.addr == Address.as_client(None, port)
-        assert self.kwargs == {'singlesession': True}
-
-    def test_extra_sys_argv(self, setUp):
-        filename = 'spam.py'
-        port = 8888
-        debug_id = 1
-        debug_options = {'x': 'y'}
-        sys.argv = [filename, '--abc', 'xyz', '42']
-        debug(filename, port, debug_id, debug_options, 'script',
-              _run=self._run, _prog='eggs')
-
-        assert self.argv == _get_args('--file', 'spam.py', '--abc', 'xyz', '42', ptvsd_extras=['--client', LOCALHOST])
-        assert self.addr == Address.as_client(None, port)
-        assert self.kwargs == {'singlesession': True}