Repository: microsoft/ptvsd

Issue:
Investigate: Unable to find threadStateIndex for the current thread. curPyThread: 0x7fd48ec095c0
@int19h @karthiknadig

I've seen this fail a couple of times already.

The effect is that we can't set the tracing to threads which aren't current when this happens.

The related code is in:
https://github.com/microsoft/ptvsd/blob/master/src/ptvsd/_vendored/pydevd/pydevd_attach_to_process/common/py_settrace.hpp#L115

and it's based on

https://github.com/Microsoft/PTVS/blob/master/Python/Product/PyDebugAttach/PyDebugAttach.cpp#L1152

Now, there's a difference from the implementation I chose vs the one in `PyDebugAttach` in that I actually try to validate that the threadStateIndex was found/valid, whereas `PyDebugAttach` doesn't validate  (and "restores" the threadStateIndex to `-1` in case it didn't match). so, I'd like to check if it's really ok to proceed in this case (if it is I'll just remove that validation from `pydevd`).

Test where this happened:
- test_error_in_condition[module-attach_socket_cmdline-NameError] (MacOS, Python 3.7)


Discussion:
Per pythread.h, this function has been deprecated in Python 3.7 – perhaps that is why?

```c
/* Thread Local Storage (TLS) API
   TLS API is DEPRECATED.  Use Thread Specific Storage (TSS) API.

   The existing TLS API has used int to represent TLS keys across all
   platforms, but it is not POSIX-compliant.  Therefore, the new TSS API uses
   opaque data type to represent TSS keys to be compatible (see PEP 539).
*/

PyAPI_FUNC(int) PyThread_create_key(void) Py_DEPRECATED(3.7);
PyAPI_FUNC(void) PyThread_delete_key(int key) Py_DEPRECATED(3.7);
PyAPI_FUNC(int) PyThread_set_key_value(int key, void *value) Py_DEPRECATED(3.7);
PyAPI_FUNC(void *) PyThread_get_key_value(int key) Py_DEPRECATED(3.7);
PyAPI_FUNC(void) PyThread_delete_key_value(int key) Py_DEPRECATED(3.7);
```

It looks like on Linux and Mac specifically, it might return NULL now:

```c
void *
PyThread_get_key_value(int key)
{
#ifdef PTHREAD_KEY_T_IS_COMPATIBLE_WITH_INT
    return pthread_getspecific(key);
#else
    return NULL;
#endif
}
```

Whereas on Windows it just goes to TLS:

```c
void *
PyThread_get_key_value(int key)
{
    /* because TLS is used in the Py_END_ALLOW_THREAD macro,
     * it is necessary to preserve the windows error state, because
     * it is assumed to be preserved across the call to the macro.
     * Ideally, the macro should be fixed, but it is simpler to
     * do it here.
     */

    DWORD error = GetLastError();
    void *result = TlsGetValue(key);
    SetLastError(error);
    return result;
}
```
**Suggestion:**
* Please fix and release a point release so we can get our UI Tests running without any issues.
* If we do for any unforeseen reason decide to ship a point release of VS Code, we might release this new version of PTVSD (rationale - If debugger breaks consistently on CI, then it could fail for users as well).
    * I.e. its best to be ready with a fix

**Information:**
* We're running some UI Tests on Mac+Win+Linux, Python 2.7 and Python 3.7 on CI
* Debugging always fails on Mac Python 3.7
* Reverting to older version of PTVSD fixes this issue.
* Debugger seems to hang immediately after this message is displayed in terminal (see below image).
* Debugger hangs for over 20 seconds (i.e. it doesn't continue after the `Unable to find threa...` message is printed in the terminal.

**Attachments:**
* Image (evidence of debugger hanging after error printed in terminal)
* Logs (PTVSD logs attached)

[ptvsd-1653.log](https://github.com/microsoft/ptvsd/files/3478325/ptvsd-1653.log)
![000 After_1565192269244_2019-08-07T15_37_49_244Z](https://user-images.githubusercontent.com/1948812/62646233-b0df2d00-b902-11e9-9840-2822082615e7.png)

I'm currently working on this issue.
getting the same error.
Meeting the same error today when I use the latest VSCode 1.37.0.

Resolving patch:
diff --git a/src/ptvsd/_vendored/pydevd/_pydevd_bundle/pydevd_constants.py b/src/ptvsd/_vendored/pydevd/_pydevd_bundle/pydevd_constants.py
--- a/src/ptvsd/_vendored/pydevd/_pydevd_bundle/pydevd_constants.py
+++ b/src/ptvsd/_vendored/pydevd/_pydevd_bundle/pydevd_constants.py
@@ -103,6 +103,7 @@ def get_frame():
 IS_PY3K = False
 IS_PY34_OR_GREATER = False
 IS_PY36_OR_GREATER = False
+IS_PY37_OR_GREATER = False
 IS_PY2 = True
 IS_PY27 = False
 IS_PY24 = False
@@ -112,6 +113,7 @@ def get_frame():
         IS_PY2 = False
         IS_PY34_OR_GREATER = sys.version_info >= (3, 4)
         IS_PY36_OR_GREATER = sys.version_info >= (3, 6)
+        IS_PY37_OR_GREATER = sys.version_info >= (3, 7)
     elif sys.version_info[0] == 2 and sys.version_info[1] == 7:
         IS_PY27 = True
     elif sys.version_info[0] == 2 and sys.version_info[1] == 4:
diff --git a/src/ptvsd/_vendored/pydevd/pydevd_tracing.py b/src/ptvsd/_vendored/pydevd/pydevd_tracing.py
--- a/src/ptvsd/_vendored/pydevd/pydevd_tracing.py
+++ b/src/ptvsd/_vendored/pydevd/pydevd_tracing.py
@@ -1,8 +1,8 @@
 
 from _pydevd_bundle.pydevd_constants import get_frame, IS_CPYTHON, IS_64BIT_PROCESS, IS_WINDOWS, \
-    IS_LINUX, IS_MAC, IS_PY2, DebugInfoHolder
+    IS_LINUX, IS_MAC, IS_PY2, IS_PY37_OR_GREATER, DebugInfoHolder
 from _pydev_imps._pydev_saved_modules import thread, threading
-from _pydev_bundle import pydev_log
+from _pydev_bundle import pydev_log, pydev_monkey
 from os.path import os
 try:
     import ctypes
@@ -146,7 +146,10 @@ def load_python_helper_lib():
         lib = ctypes.pydll.LoadLibrary(filename)
         return lib
     except:
-        pydev_log.exception('Error loading: %s', filename)
+        if DebugInfoHolder.DEBUG_TRACE_LEVEL >= 1:
+            # Only show message if tracing is on (we don't have pre-compiled
+            # binaries for all architectures -- i.e.: ARM).
+            pydev_log.exception('Error loading: %s', filename)
         return None
 
 
@@ -163,9 +166,12 @@ def set_trace_to_threads(tracing_func):
     prev_value = get_interval()
     ret = 0
     try:
-        # Prevent going to any other thread... if we switch the thread during this operation we
-        # could potentially corrupt the interpreter.
-        set_interval(2 ** 15)
+        if not IS_PY37_OR_GREATER:
+            # Prevent going to any other thread... if we switch the thread during this operation we
+            # could potentially corrupt the interpreter.
+            # Note: on CPython 3.7 onwards this is not needed (we have a different implementation
+            # for setting the tracing for other threads in this case).
+            set_interval(2 ** 15)
 
         set_trace_func = TracingFunctionHolder._original_tracing or sys.settrace
 
@@ -216,12 +222,37 @@ def _set_ident(self):
             # show_debug_info = 1 if DebugInfoHolder.DEBUG_TRACE_LEVEL >= 1 else 0
             show_debug_info = 0
 
-            result = lib.AttachDebuggerTracing(ctypes.c_int(show_debug_info), ctypes.py_object(set_trace_func), ctypes.py_object(tracing_func), ctypes.c_uint(thread_ident))
+            if IS_PY37_OR_GREATER:
+                # Hack to increase _Py_TracingPossible.
+                # See comments on py_settrace_37.hpp
+                proceed = thread.allocate_lock()
+                proceed.acquire()
+
+                def dummy_trace_on_py37(frame, event, arg):
+                    return dummy_trace_on_py37
+
+                def increase_tracing_count_on_py37():
+                    SetTrace(dummy_trace_on_py37)
+                    proceed.release()
+
+                start_new_thread = pydev_monkey.get_original_start_new_thread(thread)
+                start_new_thread(increase_tracing_count_on_py37, ())
+                proceed.acquire()  # Only proceed after the release() is done.
+                proceed = None
+
+            result = lib.AttachDebuggerTracing(
+                ctypes.c_int(show_debug_info),
+                ctypes.py_object(set_trace_func),
+                ctypes.py_object(tracing_func),
+                ctypes.c_uint(thread_ident),
+                ctypes.py_object(None),
+            )
             if result != 0:
                 pydev_log.info('Unable to set tracing for existing threads. Result: %s', result)
                 ret = result
     finally:
-        set_interval(prev_value)
+        if not IS_PY37_OR_GREATER:
+            set_interval(prev_value)
 
     return ret
 


Test patch:
diff --git a/src/ptvsd/_vendored/pydevd/tests_python/test_debugger_json.py b/src/ptvsd/_vendored/pydevd/tests_python/test_debugger_json.py
--- a/src/ptvsd/_vendored/pydevd/tests_python/test_debugger_json.py
+++ b/src/ptvsd/_vendored/pydevd/tests_python/test_debugger_json.py
@@ -728,7 +728,7 @@ def test_case_skipping_filters(case_setup, custom_setup):
             other_filename = os.path.join(not_my_code_dir, 'other.py')
             response = json_facade.write_set_breakpoints(1, filename=other_filename, verified=False)
             assert response.body.breakpoints == [
-                {'verified': False, 'message': 'Breakpoint in file excluded by filters.', 'source': {}, 'line': 1}]
+                {'verified': False, 'message': 'Breakpoint in file excluded by filters.', 'source': {'path': other_filename}, 'line': 1}]
             # Note: there's actually a use-case where we'd hit that breakpoint even if it was excluded
             # by filters, so, we must actually clear it afterwards (the use-case is that when we're
             # stepping into the context with the breakpoint we wouldn't skip it).
@@ -737,7 +737,7 @@ def test_case_skipping_filters(case_setup, custom_setup):
             other_filename = os.path.join(not_my_code_dir, 'file_that_does_not_exist.py')
             response = json_facade.write_set_breakpoints(1, filename=other_filename, verified=False)
             assert response.body.breakpoints == [
-                {'verified': False, 'message': 'Breakpoint in file that does not exist.', 'source': {}, 'line': 1}]
+                {'verified': False, 'message': 'Breakpoint in file that does not exist.', 'source': {'path': other_filename}, 'line': 1}]
 
         elif custom_setup == 'set_exclude_launch_module_full':
             json_facade.write_launch(
@@ -767,7 +767,7 @@ def test_case_skipping_filters(case_setup, custom_setup):
             assert response.body.breakpoints == [{
                 'verified': False,
                 'message': 'Breakpoint in file excluded by filters.\nNote: may be excluded because of "justMyCode" option (default == true).',
-                'source': {},
+                'source': {'path': other_filename},
                 'line': 14
             }]
         elif custom_setup == 'set_just_my_code_and_include':