Repository: microsoft/ptvsd

Issue:
"pydevd_authorize" request
We currently exchange the authorization tokens in the "initialize" request and response. 

However, our most recent design for subprocess debugging implies that, while the TCP connection between the subprocess server and the adapter is established immediately as soon as subprocess starts, "initialize" is not sent by the adapter immediately. Instead, the adapter sends "pydevd_systemInfo" request to the debuggee as soon as connection is established, to determine PID and parent PID, and waits for IDE to request attach to the process. When IDE connects to adapter and requests attach, it sends an "initialize" request to the adapter, and that request is then relayed to the server.

Thus, we need to move the token exchange out of "initialize". We can do this via debugger properties, as mentioned in #1782, but this would require adding "getDebuggerProperty". However, this seems to be better suited for a dedicated request, since when authorization is in place, no other messages can be handled until the tokens are validated. Rather than special-casing one property like that, let's refactor this into a distinct request, in which the server token (if present) is sent as part of the request, and the IDE token (if present) is sent as part of the response. 

On pydevd side, this should be very similar to the existing implementation, but instead of special-casing "initialize" in `process_net_command_json()`, it would special-case "pydevd_authorize". Property names are the same - "debugServerAccessToken" in request, "ideAccessToken" in response. 



Discussion:
On a second thought, we might also want to rename "ideAccessToken". In our most recent design, the token exchange actually occurs between the adapter and the server, not the IDE and the server, so it's confusing. Let's revert it to "clientAccessToken" - this covers both ptvsd (where adapter is involved), and direct connections from an IDE to pydevd, if somebody wants to do that.

Resolving patch:
diff --git a/src/ptvsd/_vendored/pydevd/_pydevd_bundle/_debug_adapter/pydevd_schema.py b/src/ptvsd/_vendored/pydevd/_pydevd_bundle/_debug_adapter/pydevd_schema.py
--- a/src/ptvsd/_vendored/pydevd/_pydevd_bundle/_debug_adapter/pydevd_schema.py
+++ b/src/ptvsd/_vendored/pydevd/_pydevd_bundle/_debug_adapter/pydevd_schema.py
@@ -13971,6 +13971,206 @@ def to_dict(self, update_ids_to_dap=False):  # noqa (update_ids_to_dap may be un
         return dct
 
 
+@register_request('pydevdAuthorize')
+@register
+class PydevdAuthorizeRequest(BaseSchema):
+    """
+    A request to authorize the ide to start accepting commands.
+
+    Note: automatically generated code. Do not edit manually.
+    """
+
+    __props__ = {
+        "seq": {
+            "type": "integer",
+            "description": "Sequence number."
+        },
+        "type": {
+            "type": "string",
+            "enum": [
+                "request"
+            ]
+        },
+        "command": {
+            "type": "string",
+            "enum": [
+                "pydevdAuthorize"
+            ]
+        },
+        "arguments": {
+            "type": "PydevdAuthorizeArguments"
+        }
+    }
+    __refs__ = set(['arguments'])
+
+    __slots__ = list(__props__.keys()) + ['kwargs']
+
+    def __init__(self, arguments, seq=-1, update_ids_from_dap=False, **kwargs):  # noqa (update_ids_from_dap may be unused)
+        """
+        :param string type: 
+        :param string command: 
+        :param PydevdAuthorizeArguments arguments: 
+        :param integer seq: Sequence number.
+        """
+        self.type = 'request'
+        self.command = 'pydevdAuthorize'
+        if arguments is None:
+            self.arguments = PydevdAuthorizeArguments()
+        else:
+            self.arguments = PydevdAuthorizeArguments(update_ids_from_dap=update_ids_from_dap, **arguments) if arguments.__class__ !=  PydevdAuthorizeArguments else arguments
+        self.seq = seq
+        self.kwargs = kwargs
+
+
+    def to_dict(self, update_ids_to_dap=False):  # noqa (update_ids_to_dap may be unused)
+        type = self.type  # noqa (assign to builtin)
+        command = self.command
+        arguments = self.arguments
+        seq = self.seq
+        dct = {
+            'type': type,
+            'command': command,
+            'arguments': arguments.to_dict(update_ids_to_dap=update_ids_to_dap),
+            'seq': seq,
+        }
+        dct.update(self.kwargs)
+        return dct
+
+
+@register
+class PydevdAuthorizeArguments(BaseSchema):
+    """
+    Arguments for 'pydevdAuthorize' request.
+
+    Note: automatically generated code. Do not edit manually.
+    """
+
+    __props__ = {
+        "debugServerAccessToken": {
+            "type": "string",
+            "description": "The access token to access the debug server."
+        }
+    }
+    __refs__ = set()
+
+    __slots__ = list(__props__.keys()) + ['kwargs']
+
+    def __init__(self, debugServerAccessToken=None, update_ids_from_dap=False, **kwargs):  # noqa (update_ids_from_dap may be unused)
+        """
+        :param string debugServerAccessToken: The access token to access the debug server.
+        """
+        self.debugServerAccessToken = debugServerAccessToken
+        self.kwargs = kwargs
+
+
+    def to_dict(self, update_ids_to_dap=False):  # noqa (update_ids_to_dap may be unused)
+        debugServerAccessToken = self.debugServerAccessToken
+        dct = {
+        }
+        if debugServerAccessToken is not None:
+            dct['debugServerAccessToken'] = debugServerAccessToken
+        dct.update(self.kwargs)
+        return dct
+
+
+@register_response('pydevdAuthorize')
+@register
+class PydevdAuthorizeResponse(BaseSchema):
+    """
+    Response to 'pydevdAuthorize' request.
+
+    Note: automatically generated code. Do not edit manually.
+    """
+
+    __props__ = {
+        "seq": {
+            "type": "integer",
+            "description": "Sequence number."
+        },
+        "type": {
+            "type": "string",
+            "enum": [
+                "response"
+            ]
+        },
+        "request_seq": {
+            "type": "integer",
+            "description": "Sequence number of the corresponding request."
+        },
+        "success": {
+            "type": "boolean",
+            "description": "Outcome of the request."
+        },
+        "command": {
+            "type": "string",
+            "description": "The command requested."
+        },
+        "message": {
+            "type": "string",
+            "description": "Contains error message if success == false."
+        },
+        "body": {
+            "type": "object",
+            "properties": {
+                "clientAccessToken": {
+                    "type": "string",
+                    "description": "The access token to access the client (i.e.: usually the IDE)."
+                }
+            },
+            "required": [
+                "clientAccessToken"
+            ]
+        }
+    }
+    __refs__ = set(['body'])
+
+    __slots__ = list(__props__.keys()) + ['kwargs']
+
+    def __init__(self, request_seq, success, command, body, seq=-1, message=None, update_ids_from_dap=False, **kwargs):  # noqa (update_ids_from_dap may be unused)
+        """
+        :param string type: 
+        :param integer request_seq: Sequence number of the corresponding request.
+        :param boolean success: Outcome of the request.
+        :param string command: The command requested.
+        :param PydevdAuthorizeResponseBody body: 
+        :param integer seq: Sequence number.
+        :param string message: Contains error message if success == false.
+        """
+        self.type = 'response'
+        self.request_seq = request_seq
+        self.success = success
+        self.command = command
+        if body is None:
+            self.body = PydevdAuthorizeResponseBody()
+        else:
+            self.body = PydevdAuthorizeResponseBody(update_ids_from_dap=update_ids_from_dap, **body) if body.__class__ !=  PydevdAuthorizeResponseBody else body
+        self.seq = seq
+        self.message = message
+        self.kwargs = kwargs
+
+
+    def to_dict(self, update_ids_to_dap=False):  # noqa (update_ids_to_dap may be unused)
+        type = self.type  # noqa (assign to builtin)
+        request_seq = self.request_seq
+        success = self.success
+        command = self.command
+        body = self.body
+        seq = self.seq
+        message = self.message
+        dct = {
+            'type': type,
+            'request_seq': request_seq,
+            'success': success,
+            'command': command,
+            'body': body.to_dict(update_ids_to_dap=update_ids_to_dap),
+            'seq': seq,
+        }
+        if message is not None:
+            dct['message'] = message
+        dct.update(self.kwargs)
+        return dct
+
+
 @register
 class ErrorResponseBody(BaseSchema):
     """
@@ -16029,3 +16229,38 @@ def to_dict(self, update_ids_to_dap=False):  # noqa (update_ids_to_dap may be un
         }
         dct.update(self.kwargs)
         return dct
+
+
+@register
+class PydevdAuthorizeResponseBody(BaseSchema):
+    """
+    "body" of PydevdAuthorizeResponse
+
+    Note: automatically generated code. Do not edit manually.
+    """
+
+    __props__ = {
+        "clientAccessToken": {
+            "type": "string",
+            "description": "The access token to access the client (i.e.: usually the IDE)."
+        }
+    }
+    __refs__ = set()
+
+    __slots__ = list(__props__.keys()) + ['kwargs']
+
+    def __init__(self, clientAccessToken, update_ids_from_dap=False, **kwargs):  # noqa (update_ids_from_dap may be unused)
+        """
+        :param string clientAccessToken: The access token to access the client (i.e.: usually the IDE).
+        """
+        self.clientAccessToken = clientAccessToken
+        self.kwargs = kwargs
+
+
+    def to_dict(self, update_ids_to_dap=False):  # noqa (update_ids_to_dap may be unused)
+        clientAccessToken = self.clientAccessToken
+        dct = {
+            'clientAccessToken': clientAccessToken,
+        }
+        dct.update(self.kwargs)
+        return dct
diff --git a/src/ptvsd/_vendored/pydevd/_pydevd_bundle/pydevd_process_net_command_json.py b/src/ptvsd/_vendored/pydevd/_pydevd_bundle/pydevd_process_net_command_json.py
--- a/src/ptvsd/_vendored/pydevd/_pydevd_bundle/pydevd_process_net_command_json.py
+++ b/src/ptvsd/_vendored/pydevd/_pydevd_bundle/pydevd_process_net_command_json.py
@@ -15,7 +15,7 @@
     ProcessEvent, Scope, ScopesResponseBody, SetExpressionResponseBody,
     SetVariableResponseBody, SourceBreakpoint, SourceResponseBody,
     VariablesResponseBody, SetBreakpointsResponseBody, Response, InitializeRequest, InitializeResponse,
-    Capabilities)
+    Capabilities, PydevdAuthorizeRequest)
 from _pydevd_bundle.pydevd_api import PyDevdAPI
 from _pydevd_bundle.pydevd_breakpoints import get_exception_class
 from _pydevd_bundle.pydevd_comm_constants import (
@@ -162,12 +162,10 @@ def on_request(py_db, request):
                 print('Handled in pydevd: %s (in PyDevJsonCommandProcessor).\n' % (method_name,))
 
         with py_db._main_lock:
-            if request.__class__ == InitializeRequest:
-                initialize_request = request  # : :type initialize_request: InitializeRequest
-                pydevd_specific_info = initialize_request.arguments.kwargs.get('pydevd', {})
-                if pydevd_specific_info.__class__ == dict:
-                    access_token = pydevd_specific_info.get('debugServerAccessToken')
-                    py_db.authentication.login(access_token)
+            if request.__class__ == PydevdAuthorizeRequest:
+                authorize_request = request  # : :type authorize_request: PydevdAuthorizeRequest
+                access_token = authorize_request.arguments.debugServerAccessToken
+                py_db.authentication.login(access_token)
 
             if not py_db.authentication.is_authenticated():
                 response = Response(
@@ -180,6 +178,15 @@ def on_request(py_db, request):
             if cmd is not None and send_response:
                 py_db.writer.add_command(cmd)
 
+    def on_pydevdauthorize_request(self, py_db, request):
+        ide_access_token = py_db.authentication.ide_access_token
+        body = {'clientAccessToken': None}
+        if ide_access_token:
+            body['clientAccessToken'] = ide_access_token
+
+        response = pydevd_base_schema.build_response(request, kwargs={'body': body})
+        return NetCommand(CMD_RETURN, 0, response, is_json=True)
+
     def on_initialize_request(self, py_db, request):
         body = Capabilities(
             # Supported.
@@ -224,11 +231,8 @@ def on_initialize_request(self, py_db, request):
         # Non-standard capabilities/info below.
         body['supportsDebuggerProperties'] = True
 
-        ide_access_token = py_db.authentication.ide_access_token
         body['pydevd'] = pydevd_info = {}
         pydevd_info['processId'] = os.getpid()
-        if ide_access_token:
-            pydevd_info['ideAccessToken'] = ide_access_token
         self.api.notify_initialize(py_db)
         response = pydevd_base_schema.build_response(request, kwargs={'body': body})
         return NetCommand(CMD_RETURN, 0, response, is_json=True)


Test patch:
diff --git a/src/ptvsd/_vendored/pydevd/tests_python/test_debugger_json.py b/src/ptvsd/_vendored/pydevd/tests_python/test_debugger_json.py
--- a/src/ptvsd/_vendored/pydevd/tests_python/test_debugger_json.py
+++ b/src/ptvsd/_vendored/pydevd/tests_python/test_debugger_json.py
@@ -409,12 +409,9 @@ def write_set_pydevd_source_map(self, source, pydevd_source_maps, success=True):
         assert response.success == success
         return response
 
-    def write_initialize(self, access_token, success=True):
+    def write_initialize(self, success=True):
         arguments = InitializeRequestArguments(
             adapterID='pydevd_test_case',
-            pydevd=dict(
-                debugServerAccessToken=access_token,
-            )
         )
         response = self.wait_for_response(self.write_request(InitializeRequest(arguments)))
         assert response.success == success
@@ -423,6 +420,16 @@ def write_initialize(self, access_token, success=True):
             assert isinstance(process_id, int)
         return response
 
+    def write_authorize(self, access_token, success=True):
+        from _pydevd_bundle._debug_adapter.pydevd_schema import PydevdAuthorizeArguments
+        from _pydevd_bundle._debug_adapter.pydevd_schema import PydevdAuthorizeRequest
+        arguments = PydevdAuthorizeArguments(
+            debugServerAccessToken=access_token,
+        )
+        response = self.wait_for_response(self.write_request(PydevdAuthorizeRequest(arguments)))
+        assert response.success == success
+        return response
+
     def evaluate(self, expression, frameId=None, context=None, fmt=None, success=True):
         eval_request = self.write_request(
             pydevd_schema.EvaluateRequest(pydevd_schema.EvaluateArguments(
@@ -3036,7 +3043,7 @@ def update_command_line_args(writer, args):
             json_facade.write_launch(terminateChildProcesses=False)
 
         json_facade.write_make_initial_run()
-        response = json_facade.write_initialize(None)
+        response = json_facade.write_initialize()
         pid = response.to_dict()['body']['pydevd']['processId']
 
         if check_subprocesses in ('kill_subprocesses', 'dont_kill_subprocesses', 'kill_subprocesses_ignore_pid'):
@@ -3141,17 +3148,15 @@ def update_command_line_args(self, args):
         response = json_facade.write_set_debugger_property(multi_threads_single_notification=True, success=False)
         assert response.message == "Client not authenticated."
 
-        response = json_facade.write_initialize(access_token='wrong', success=False)
+        response = json_facade.write_authorize(access_token='wrong', success=False)
         assert response.message == "Client not authenticated."
 
         response = json_facade.write_set_debugger_property(multi_threads_single_notification=True, success=False)
         assert response.message == "Client not authenticated."
 
-        response = json_facade.write_initialize(access_token='bar123', success=True)
-        # : :type response:InitializeResponse
-        initialize_response_body = response.body
-        # : :type initialize_response_body:Capabilities
-        assert initialize_response_body.kwargs['pydevd']['ideAccessToken'] == 'foo321'
+        authorize_response = json_facade.write_authorize(access_token='bar123', success=True)
+        # : :type authorize_response:PydevdAuthorizeResponse
+        assert authorize_response.body.clientAccessToken == 'foo321'
 
         json_facade.write_set_debugger_property(multi_threads_single_notification=True)
         json_facade.write_launch()
@@ -3170,11 +3175,11 @@ def update_command_line_args(self, args):
 
         json_facade.write_disconnect()
 
-        response = json_facade.write_initialize(access_token='wrong', success=False)
+        response = json_facade.write_authorize(access_token='wrong', success=False)
         assert response.message == "Client not authenticated."
 
-        response = json_facade.write_initialize(access_token='bar123')
-        assert initialize_response_body.kwargs['pydevd']['ideAccessToken'] == 'foo321'
+        authorize_response = json_facade.write_authorize(access_token='bar123')
+        assert authorize_response.body.clientAccessToken == 'foo321'
 
         json_facade.write_set_breakpoints(break_line)
         json_hit = json_facade.wait_for_thread_stopped(line=break_line)