Repository: spyder-ide/spyder

Issue:
Errors when debugging with remote kernel
Hi,

I am working on a similar problem as this post(https://github.com/spyder-ide/spyder/issues/3587#issue-184727853) and all the comments are helpful. But for my use case I would like to debug my python script locally with a python kernel hosted on a cloud server, and I get similar issues:

This command failed to be executed because an error occurred while trying to get the file code from Spyder's editor. The error was:


An exception has occurred, use %tb to see the full traceback.

OSError: [Errno 22] Invalid argument: '....'

For reference, I am using windows with spyder 5.3.1 and I am connecting to a remote linux kernel having python 3.7. The steps I followed to set it up using Mobaxterm:

login to your remote server, create your python environment there and start your python kernel using the following line:
python -m spyder_kernels.console --ip=127.0.0.1 -f=./remotemachine.json
(Note: ip address should can also be localhost)

Open another local terminal in mobaxterm and enter:
ssh -L ::<control-port from remotemachine.json @

Copy the generated remotemachine.json file to your local system, open spyder, click on connect to existing kernel:
enter the remotemachine.json path in Connection File,
in hostname put the ,
in username enter the for your cloud server,
in port put 22(default ssh port on mobaxterm),
and enter the password for remote server login.

And it should connect then. After these steps I am able to send simple command line functions, but not able to debug the file. Is there any new support added for this already?

Thanks.


Discussion:
Hi @shankyemcee thank you for the feedback! I think this is not possible at the moment :/ but maybe @ccordoba12 or @impact27 could have some ideas?
It should work now as you describe, but you have to make sure the version of spyder-kernels matches the version on spyder. I am working on a pr to use the control channel for debugging which should be more reliable, but because this won’t work with python 2.7, it won’t be available before spyder 6. Can you confirm the version on spyder you are using and the version of spyder-kernels on the remote server?
I am using spyder 5.3.1 on my system and spyder_kernels 2.3.1 on both the remote server and my own system. To me the error looks like its trying to look for the file in the remote system, but I am trying to debug it on my own system. Basically I just want to use the compute resources and packages on the remote server, but all file access and debugging should happen on my own system.
Si the way it works is that spyder sends the content of the editor to the remote kerne. I’ll have a look to see if I can reproduce 
@shankyemcee, you posted above:

> An exception has occurred, use %tb to see the full traceback.

Please run `%tb` in the IPython console and post the contents shown afterwards here so we can take a look at them.
Output of %tb:

Traceback (most recent call last):

  File "/project/6027213/shankar5/envs/lit/lib/python3.8/site-packages/spyder_kernels/customize/spydercustomize.py", line 512, in get_file_code
    file_code = frontend_request(blocking=True).get_file_code(

  File "/project/6027213/shankar5/envs/lit/lib/python3.8/site-packages/spyder_kernels/comms/commbase.py", line 557, in __call__
    return self._comms_wrapper._get_call_return_value(

  File "/project/6027213/shankar5/envs/lit/lib/python3.8/site-packages/spyder_kernels/comms/commbase.py", line 440, in _get_call_return_value
    return self._sync_error(reply['value'])

  File "/project/6027213/shankar5/envs/lit/lib/python3.8/site-packages/spyder_kernels/comms/commbase.py", line 496, in _sync_error
    error_wrapper.raise_error()

  File "/project/6027213/shankar5/envs/lit/lib/python3.8/site-packages/spyder_kernels/comms/commbase.py", line 90, in raise_error
    raise self.etype(self)

OSError: [Errno 22] Invalid argument: '/project/6027213/shankar5/envs/lit/C:/Users/Dell G7 User 2/Documents/PROJECTS/vision_transformer/vision_transformer/vit_jax/main.py'

Here '/project/6027213/shankar5/envs/lit/' is where I hosted the remote kernel and 'C:/Users/Dell G7 User 2/Documents/PROJECTS/vision_transformer/vision_transformer/vit_jax/main.py' is the file on my system I am trying to debug
When you press debug, what exactly appears in your console? do you have `debugfile("/project/6027213/shankar5/envs/lit/C:/Users/Dell G7 User 2/Documents/PROJECTS/vision_transformer/vision_transformer/vit_jax/main.py", ...)`?
I have tried:
 - install spyder-kernels 2.3.1 on a remote machine
 - install spyder 5.3.1 on my machine
 - launch spyder-kernels with `python3 -m spyder_kernels.console`
 - copy the connection file at `/home/debian/.local/share/jupyter/runtime`
 - Connect on my local machine with "Connect to an existing kernel"
 - create a file on my desktop containing `print(0)` and a breakpoint
 - press debug on spyder

With that, the bebugging starts normally.
I think I found it, it is a problem of windows vs linux paths:

```
IPdb [5]: os.path.abspath("C:/Users")
Out  [5]: '/home/debian/spyder-kernels/C:/Users'
```
@shankyemcee Could you check if https://github.com/spyder-ide/spyder-kernels/pull/389 solves the issue?
Hi, Can I please know how I can access this update? Is it through updating spyder_kernels or spyder?
On the remote machine:
`git clone https://github.com/spyder-ide/spyder-kernels.git`
`cd spyder-kernels`
`git fetch origin pull/389/head:remote_path_fix`
`git checkout remote_path_fix`
`pip install .`
Hi,
 I followed your steps and re hosted the kernel from the remote server, but I am no longer able to connect my spyder ide to the Kernel on remote server.
That is strange, could you give me the steps you follow to launch and connect, and the exact error you see?
Hi,

I refreshed the connections, and it appears to be working. But one problem is that I don't know which line is executing. For example here is the screenshot of when i used debugging with remote kernel:

![remote_kernel](https://user-images.githubusercontent.com/30503832/174857787-1dfecf6b-8dc8-4052-a66f-e96b2c8a784f.JPG)

and here is a screenshot with debugging in local kernel:

![image](https://user-images.githubusercontent.com/30503832/174857906-6f197527-722a-4658-8a08-688470020228.png)

The arrow is missing in remote kernel connection. Can this be added? Thanks.

spyder-ide/spyder-kernels#389 has been updated, can you try again?
Thankyou, it's working brilliantly! Would this update be directly available through the next spyder_kernels release through conda or pip?
Yes it should if we have time to merge it, otherwise the next one
Sounds good, Thanks a lot for all your help!
We need this open so we don't forget to address it. 

Resolving patch:
diff --git a/external-deps/spyder-kernels/spyder_kernels/customize/spydercustomize.py b/external-deps/spyder-kernels/spyder_kernels/customize/spydercustomize.py
--- a/external-deps/spyder-kernels/spyder_kernels/customize/spydercustomize.py
+++ b/external-deps/spyder-kernels/spyder_kernels/customize/spydercustomize.py
@@ -23,17 +23,15 @@
 import time
 import warnings
 
-from IPython import __version__ as ipy_version
 from IPython.core.getipython import get_ipython
 
-from spyder_kernels.comms.frontendcomm import CommError, frontend_request
+from spyder_kernels.comms.frontendcomm import frontend_request
 from spyder_kernels.customize.namespace_manager import NamespaceManager
 from spyder_kernels.customize.spyderpdb import SpyderPdb, get_new_debugger
 from spyder_kernels.customize.umr import UserModuleReloader
 from spyder_kernels.py3compat import (
-    TimeoutError, PY2, _print, encode, compat_exec, FileNotFoundError)
-from spyder_kernels.customize.utils import (
-    capture_last_Expr, normalise_filename)
+    PY2, _print, encode, compat_exec, FileNotFoundError)
+from spyder_kernels.customize.utils import capture_last_Expr, canonic
 
 if not PY2:
     from IPython.core.inputtransformer2 import (
@@ -505,18 +503,23 @@ def exec_code(code, filename, ns_globals, ns_locals=None, post_mortem=False,
         __tracebackhide__ = "__pdb_exit__"
 
 
-def get_file_code(filename, save_all=True):
-    """Retrive the content of a file."""
+def get_file_code(filename, save_all=True, raise_exception=False):
+    """Retrieve the content of a file."""
     # Get code from spyder
     try:
-        file_code = frontend_request(blocking=True).get_file_code(
+        return frontend_request(blocking=True).get_file_code(
             filename, save_all=save_all)
-    except (CommError, TimeoutError, RuntimeError, FileNotFoundError):
-        file_code = None
-    if file_code is None:
-        with open(filename, 'r') as f:
-            return f.read()
-    return file_code
+    except Exception:
+        # Maybe this is a local file
+        try:
+            with open(filename, 'r') as f:
+                return f.read()
+        except FileNotFoundError:
+            pass
+        if raise_exception:
+            raise
+        # Else return None
+        return None
 
 
 def runfile(filename=None, args=None, wdir=None, namespace=None,
@@ -536,7 +539,7 @@ def runfile(filename=None, args=None, wdir=None, namespace=None,
 
 def _exec_file(filename=None, args=None, wdir=None, namespace=None,
                post_mortem=False, current_namespace=False, stack_depth=0,
-               exec_fun=None):
+               exec_fun=None, canonic_filename=None):
     # Tell IPython to hide this frame (>7.16)
     __tracebackhide__ = True
     ipython_shell = get_ipython()
@@ -544,11 +547,6 @@ def _exec_file(filename=None, args=None, wdir=None, namespace=None,
         filename = get_current_file_name()
         if filename is None:
             return
-    else:
-        # get_debugger replaces \\ by / so we must undo that here
-        # Otherwise code caching doesn't work
-        if os.name == 'nt':
-            filename = filename.replace('/', '\\')
 
     try:
         filename = filename.decode('utf-8')
@@ -562,22 +560,23 @@ def _exec_file(filename=None, args=None, wdir=None, namespace=None,
         __umr__.run()
     if args is not None and not isinstance(args, basestring):
         raise TypeError("expected a character buffer object")
+
     try:
-        file_code = get_file_code(filename)
+        file_code = get_file_code(filename, raise_exception=True)
     except Exception:
+        # Show an error and return None
         _print(
             "This command failed to be executed because an error occurred"
             " while trying to get the file code from Spyder's"
             " editor. The error was:\n\n")
         get_ipython().showtraceback(exception_only=True)
         return
-    if file_code is None:
-        _print("Could not get code from editor.\n")
-        return
 
-    # Normalise the filename
-    filename = os.path.abspath(filename)
-    filename = os.path.normcase(filename)
+    # Here the remote filename has been used. It must now be valid locally.
+    if canonic_filename is not None:
+        filename = canonic_filename
+    else:
+        filename = canonic(filename)
 
     with NamespaceManager(filename, namespace, current_namespace,
                           file_code=file_code, stack_depth=stack_depth + 1
@@ -655,7 +654,7 @@ def debugfile(filename=None, args=None, wdir=None, post_mortem=False,
     if shell.is_debugging():
         # Recursive
         code = (
-            "runfile({}".format(repr(normalise_filename(filename))) +
+            "runfile({}".format(repr(filename)) +
             ", args=%r, wdir=%r, current_namespace=%r)" % (
                 args, wdir, current_namespace)
         )
@@ -666,11 +665,13 @@ def debugfile(filename=None, args=None, wdir=None, post_mortem=False,
     else:
         debugger = get_new_debugger(filename, True)
         _exec_file(
-            filename=debugger.canonic(filename),
-            args=args, wdir=wdir,
+            filename=filename,
+            canonic_filename=debugger.canonic(filename),
+            args=args,
+            wdir=wdir,
             current_namespace=current_namespace,
             exec_fun=debugger.run,
-            stack_depth=1
+            stack_depth=1,
         )
 
 
@@ -696,7 +697,7 @@ def runcell(cellname, filename=None, post_mortem=False):
 
 
 def _exec_cell(cellname, filename=None, post_mortem=False, stack_depth=0,
-               exec_fun=None):
+               exec_fun=None, canonic_filename=None):
     """
     Execute a code cell with a given exec function.
     """
@@ -706,11 +707,6 @@ def _exec_cell(cellname, filename=None, post_mortem=False, stack_depth=0,
         filename = get_current_file_name()
         if filename is None:
             return
-    else:
-        # get_debugger replaces \\ by / so we must undo that here
-        # Otherwise code caching doesn't work
-        if os.name == 'nt':
-            filename = filename.replace('/', '\\')
     try:
         filename = filename.decode('utf-8')
     except (UnicodeError, TypeError, AttributeError):
@@ -736,14 +732,14 @@ def _exec_cell(cellname, filename=None, post_mortem=False, stack_depth=0,
     # Trigger `post_execute` to exit the additional pre-execution.
     # See Spyder PR #7310.
     ipython_shell.events.trigger('post_execute')
-    try:
-        file_code = get_file_code(filename, save_all=False)
-    except Exception:
-        file_code = None
+    file_code = get_file_code(filename, save_all=False)
 
-    # Normalise the filename
-    filename = os.path.abspath(filename)
-    filename = os.path.normcase(filename)
+    # Here the remote filename has been used. It must now be valid locally.
+    if canonic_filename is not None:
+        filename = canonic_filename
+    else:
+        # Normalise the filename
+        filename = canonic(filename)
 
     with NamespaceManager(filename, current_namespace=True,
                           file_code=file_code, stack_depth=stack_depth + 1
@@ -770,7 +766,7 @@ def debugcell(cellname, filename=None, post_mortem=False):
         # Recursive
         code = (
             "runcell({}, ".format(repr(cellname)) +
-            "{})".format(repr(normalise_filename(filename)))
+            "{})".format(repr(filename))
         )
         shell.pdb_session.enter_recursive_debugger(
             code, filename, False,
@@ -779,7 +775,8 @@ def debugcell(cellname, filename=None, post_mortem=False):
         debugger = get_new_debugger(filename, False)
         _exec_cell(
             cellname=cellname,
-            filename=debugger.canonic(filename),
+            filename=filename,
+            canonic_filename=debugger.canonic(filename),
             exec_fun=debugger.run,
             stack_depth=1
         )
diff --git a/external-deps/spyder-kernels/spyder_kernels/customize/spyderpdb.py b/external-deps/spyder-kernels/spyder_kernels/customize/spyderpdb.py
--- a/external-deps/spyder-kernels/spyder_kernels/customize/spyderpdb.py
+++ b/external-deps/spyder-kernels/spyder_kernels/customize/spyderpdb.py
@@ -10,7 +10,6 @@
 import ast
 import bdb
 import logging
-import os
 import sys
 import traceback
 from collections import namedtuple
@@ -100,6 +99,9 @@ def __init__(self, completekey='tab', stdin=None, stdout=None,
         # has no effect in previous versions.
         self.report_skipped = False
 
+        # Keep track of remote filename
+        self.remote_filename = None
+
     # --- Methods overriden for code execution
     def print_exclamation_warning(self):
         """Print pdb warning for exclamation mark."""
@@ -774,6 +776,8 @@ def publish_pdb_state(self):
                 fname = unicode(fname, "utf-8")
             except TypeError:
                 pass
+        if fname == self.mainpyfile and self.remote_filename is not None:
+            fname = self.remote_filename
         lineno = frame.f_lineno
 
         if self._previous_step == (fname, lineno):
@@ -833,11 +837,8 @@ def enter_recursive_debugger(self, code, filename,
         debugger.use_rawinput = self.use_rawinput
         debugger.prompt = "(%s) " % self.prompt.strip()
 
-        filename = debugger.canonic(filename)
-        debugger._wait_for_mainpyfile = True
-        debugger.mainpyfile = filename
+        debugger.set_remote_filename(filename)
         debugger.continue_if_has_breakpoints = continue_if_has_breakpoints
-        debugger._user_requested_quit = False
 
         # Enter recursive debugger
         sys.call_tracing(debugger.run, (code, globals, locals))
@@ -852,14 +853,16 @@ def enter_recursive_debugger(self, code, filename,
         # but the parent debugger (self) is not aware of this.
         self._previous_step = None
 
+    def set_remote_filename(self, filename):
+        """Set remote filename to signal Spyder on mainpyfile."""
+        self.remote_filename = filename
+        self.mainpyfile = self.canonic(filename)
+        self._wait_for_mainpyfile = True
+
 
 def get_new_debugger(filename, continue_if_has_breakpoints):
     """Get a new debugger."""
     debugger = SpyderPdb()
-
-    filename = debugger.canonic(filename)
-    debugger._wait_for_mainpyfile = True
-    debugger.mainpyfile = filename
+    debugger.set_remote_filename(filename)
     debugger.continue_if_has_breakpoints = continue_if_has_breakpoints
-    debugger._user_requested_quit = False
     return debugger
diff --git a/external-deps/spyder-kernels/spyder_kernels/customize/utils.py b/external-deps/spyder-kernels/spyder_kernels/customize/utils.py
--- a/external-deps/spyder-kernels/spyder_kernels/customize/utils.py
+++ b/external-deps/spyder-kernels/spyder_kernels/customize/utils.py
@@ -117,9 +117,15 @@ def capture_last_Expr(code_ast, out_varname):
     return code_ast, capture_last_expression
 
 
-def normalise_filename(filename):
-    """Normalise path for window."""
-    # Recursive
-    if os.name == 'nt':
-        return filename.replace('\\', '/')
-    return filename
+def canonic(filename):
+    """
+    Return canonical form of filename.
+
+    This is a copy of bdb.canonic, so that the debugger will process 
+    filenames in the same way
+    """
+    if filename == "<" + filename[1:-1] + ">":
+        return filename
+    canonic = os.path.abspath(filename)
+    canonic = os.path.normcase(canonic)
+    return canonic


Test patch: