Repository: spyder-ide/spyder

Issue:
Variable explorer doesn't show variables on remote kernel
<!--- **PLEASE READ:** When submitting here, please ensure you've completed the following checklist and checked the boxes to confirm. Issue reports without it may be closed. Thanks! --->

### Issue Report Checklist

- [x] Searched the [issues page](https://github.com/spyder-ide/spyder/issues?q=is%3Aissue) for similar reports
> did not solve the issue
- [x] Read the relevant sections of the [Spyder Troubleshooting Guide](https://github.com/spyder-ide/spyder/wiki/Troubleshooting-Guide-and-FAQ) and followed its advice
- [x] Reproduced the issue after updating with ``conda update spyder`` (or ``pip``, if not using Anaconda)
> Spyder was update from 3.3.6 to 4.0.1
* [ ] Could not reproduce inside ``jupyter qtconsole`` (if console-related)
> Not console-related
* [x] Tried basic troubleshooting (if a bug/error)
    - [ ] Restarted Spyder
    - [x] Reset preferences with ``spyder --reset``
    - [x] Reinstalled the latest version of [Anaconda](https://www.anaconda.com/download/)
    - [x] Tried the other applicable steps from the Troubleshooting Guide
* [x] Completed the **Problem Description**, **Steps to Reproduce** and **Version** sections below


## Problem Description
Variable explorer works for a local kernel.

When connected to a remote kernel, the variables only show up in the variable explorer after I shutdown the remote kernel and connect to it again. Clicking the 'Refresh variables' did not solve the issue.

The proposed solutions described [here](https://github.com/spyder-ide/spyder/issues/7850) and [here](https://stackoverflow.com/questions/55354020/spyder-variable-explorer-does-not-show-variable-for-connected-remote-kernel) did not solve the problem



### What steps reproduce the problem?

1. Install `spyder-kernels` on both local machine and remote host
2. `python -m spyder_kernels.console` on the remote host
3. launch spyder on the local machine
1. connect to the remote host following the information from the [official documentation](python -m spyder_kernels.console)
    1. The connection works i.e. I can run and assign variables on from within the remote host using the spyder installed locally. Variables will only be displayed in the variable explorer if I restart the remote kernel
1. close the remote kernel session
1. open it again, as described above

### What is the expected output? What do you see instead?
Produced variables should be displayed on the variable explorer, on the same remote kernel session, not after restarting the remote kernel



### Paste Traceback/Error Below (if applicable)
<!--- Copy from error dialog or View > Panes > Internal Console --->

```python-traceback

NA - no error message displayed

```

## Versions
<!--- You can get this information from Help > About Spyder...
or (if Spyder won't launch) the "conda list" command
from the Anaconda Prompt/Terminal/command line. --->

* Spyder version: 4.0.1 (local)
* Python version: 3.7.6 (remote host) and 3.7.3 (local)
* Qt version: 5.9.7 (local and remote host)
* PyQt version: 5.9.2 (local and remote host)
* Operating System name/version: 
Ubuntu 18.04 LTS

### Dependencies
<!--- Please go to the menu entry Help > Dependencies,
press the Copy to clipboard button and paste below --->

```
atomicwrites >=1.2.0         :  1.3.0 (OK)
chardet >=2.0.0              :  3.0.4 (OK)
cloudpickle >=0.5.0          :  1.2.2 (OK)
diff_match_patch >=20181111  :  20181111 (OK)
intervaltree                 :  None (OK)
IPython >=4.0                :  7.12.0 (OK)
jedi =0.14.1                 :  0.14.1 (OK)
keyring                      :  None (OK)
nbconvert >=4.0              :  5.6.1 (OK)
numpydoc >=0.6.0             :  0.9.2 (OK)
pexpect >=4.4.0              :  4.8.0 (OK)
pickleshare >=0.4            :  0.7.5 (OK)
psutil >=0.3                 :  5.6.7 (OK)
pygments >=2.0               :  2.5.2 (OK)
pylint >=0.25                :  2.4.4 (OK)
pyls >=0.31.2;<0.32.0        :  0.31.7 (OK)
xdg >=0.26                   :  0.26 (OK)
zmq >=17                     :  18.1.1 (OK)
qdarkstyle >=2.7             :  2.8 (OK)
qtawesome >=0.5.7            :  0.6.1 (OK)
qtconsole >=4.6.0            :  4.6.0 (OK)
qtpy >=1.5.0                 :  1.9.0 (OK)
rtree >=0.8.3                :  0.9.4 (OK)
sphinx >=0.6.6               :  2.4.0 (OK)
spyder_kernels >=1.8.1;<2.0.0:  1.8.1 (OK)
watchdog                     :  None (OK)
cython >=0.21                :  None (NOK)
matplotlib >=2.0.0           :  3.1.0 (OK)
numpy >=1.7                  :  1.18.1 (OK)
pandas >=0.13.1              :  1.0.1 (OK)
scipy >=0.17.0               :  1.4.1 (OK)
sympy >=0.7.3                :  None (NOK)
```



Discussion:
What operating system is your local version of Spyder running?
Ubuntu 18.04.04 LTS
And are you using spyder-kernels 1.8.1 in your remote server?
precisely
This is not so easy to test for us, but we'll try to take a look at it as soon as we can.
great thanks for the feedback
I'm facing a similar issue. I'm using the similar versions of packages as OP but can't see to get the remote kernel vars in the variable explorer. Would be great if someone can take a look at replicating this. Perhaps a sandbox local dockerized env which has kernel (remote) and connecting to that from spyder should replicate this issue. 
Same here. 
I updated to latest conda and latest spyder (spyder-4.1.4). I can run local files onto a remote kernel, here a docker container (NVIDIA PyTorch). 
The Variables Explorer shows the variables when using the local IPython console, but not when using the remote console on the docker container. Real bummer, that functionality is really useful. Thanks. 
Hey @BCArg , can you explain exactly what steps you performed to restart the remote kernel that got the variable explorer workingl? Did you run a particular command in Spyder? Ctrl-C the kernel on the remote and start enough one? We had another user who experienced this, who tried

```python
import os; os._exit(00)
```

which restarts the kernel when run locally in Spyder, and instead it just made the console hang. Thanks!

@impact27 Any insight?
@CAM-Gerlach it was simply CTRL-C on the terminal where the kernel had been started
Thanks @BCArg And then you started another kernel in the same terminal, reconnected and the Variable Explorer worked? Or are you saying the variable explorer only worked after the kernel was already dead?
the variable explorer worked after a new kernel was started, but as far as I recall, it was displaying only the variables assigned in the previous session. I am not sure whether the variables assigned in that new session were being displayed on the variable explorer
Okay, thanks @BCArg . If it was displayed the previous variables, then my guess is most likely it wasn't displaying the new ones since the previous ones wouldn't have existed anymore once the kernel was killed, but I can't be sure. We appreciate your help!
I am having this issue too with Spyder 4. When I connect to a remote kernel, the variable explorer is empty. When connected to a local kernel, the variable explorer is fine

Running the same code with access to the same data:
Local
![image](https://user-images.githubusercontent.com/43831513/90320570-6a0e1500-df10-11ea-9ffd-89db12014ed4.png)

Remote
![image](https://user-images.githubusercontent.com/43831513/90320576-75f9d700-df10-11ea-82ad-0f40b75a3c85.png)

Yep, thanks @andyrevell . As I understand it, we've tracked down the cause and its in the queue to be fixed. @ccordoba12 or @impact27 would know more specifics on that.
Do I understand correctly that "restarting the session" is NOT a solution to this problem?
(I'm pretty this is true)
Does anyone know a version of spyder/spyder-kernels where the variable explorer worked with remote kernels?  I hope to downgrade to that combo until this issue is fixed. 
> Does anyone know a version of spyder/spyder-kernels where the variable explorer worked with remote kernels? I hope to downgrade to that combo until this issue is fixed.

The variable explorer for remote kernels is broken for me from spyder version 4.0.0 and upwards. The last working version for me is spyder 3.3.6 with spyder-kernels 0.5.2. 

Any news on this bug?
We'll take a look at it in our 4.2.1 version, to be released just before Christmas.
Keep paying attention because I have the same problem.
Please make it work. It did failed for me also. Spyder 4.2.0 running on Windows10, `python2.7 -m spyder_kernels.console --matplotlib='inline' -f=./kernel.json` running on remote CentOS7
Maybe the comm port is blocked by the firewall? You can get it by "get_ipython().kernel.frontend_comm.comm_port"

Resolving patch:
diff --git a/spyder/plugins/ipythonconsole/comms/kernelcomm.py b/spyder/plugins/ipythonconsole/comms/kernelcomm.py
--- a/spyder/plugins/ipythonconsole/comms/kernelcomm.py
+++ b/spyder/plugins/ipythonconsole/comms/kernelcomm.py
@@ -10,14 +10,17 @@
 """
 from contextlib import contextmanager
 import logging
+import os
 import pickle
 
 import jupyter_client
 from qtpy.QtCore import QEventLoop, QObject, QTimer, Signal
 import zmq
+from zmq.ssh import tunnel as zmqtunnel
 
 from spyder_kernels.comms.commbase import CommBase, CommError
 from spyder.py3compat import TimeoutError
+from spyder.plugins.ipythonconsole.utils.ssh import openssh_tunnel
 
 logger = logging.getLogger(__name__)
 
@@ -37,18 +40,38 @@ class KernelComm(CommBase, QObject):
 
     def __init__(self):
         super(KernelComm, self).__init__()
-        self.comm_port = None
+        self.remote_comm_port = None
         self.kernel_client = None
 
         # Register handlers
         self.register_call_handler('_async_error', self._async_error)
         self.register_call_handler('_set_comm_port', self._set_comm_port)
 
+    def ssh_tunnel(self, *args, **kwargs):
+        if os.name == 'nt':
+            return zmqtunnel.paramiko_tunnel(*args, **kwargs)
+        else:
+            return openssh_tunnel(None, *args, **kwargs)
+
     def _set_comm_port(self, port):
         """Set comm port."""
-        if port is None:
+        if port is None or port == self.remote_comm_port:
             return
+        self.remote_comm_port = port
+
         client = self.kernel_client
+
+        if hasattr(client, 'ssh_parameters'):
+            # Need to tunnel port
+            hostname, sshkey, password = client.ssh_parameters
+            local_port = zmqtunnel.select_random_ports(1)[0]
+            remote_port = port
+            remote_ip = client.ip
+            self.ssh_tunnel(
+                local_port, remote_port, hostname, remote_ip, sshkey,
+                password, timeout=10)
+            port = local_port
+
         if not (hasattr(client, 'comm_port') and client.comm_port == port):
             client.comm_port = port
             identity = client.session.bsession
diff --git a/spyder/plugins/ipythonconsole/plugin.py b/spyder/plugins/ipythonconsole/plugin.py
--- a/spyder/plugins/ipythonconsole/plugin.py
+++ b/spyder/plugins/ipythonconsole/plugin.py
@@ -1757,6 +1757,8 @@ def _create_client_for_kernel(self, connection_file, hostname, sshkey,
                  kernel_client.iopub_port,
                  kernel_client.stdin_port,
                  kernel_client.hb_port) = newports
+                # Save parameters to connect comm later
+                kernel_client.ssh_parameters = (hostname, sshkey, password)
             except Exception as e:
                 QMessageBox.critical(self, _('Connection error'),
                                    _("Could not open ssh tunnel. The "


Test patch: