Repository: spyder-ide/spyder

Issue:
Variables not defined error while in debug mode when running a list comprehension
### Issue Report Checklist

* [x] Searched the [issues page](https://github.com/spyder-ide/spyder/issues?q=is%3Aissue) for similar reports
* [x] Read the relevant sections of the [Spyder Troubleshooting Guide](https://github.com/spyder-ide/spyder/wiki/Troubleshooting-Guide-and-FAQ) and followed its advice
* [x] Reproduced the issue after updating with ``conda update spyder`` (or ``pip``, if not using Anaconda)
* [x] Could not reproduce inside ``jupyter qtconsole`` (if console-related)
* [x] Tried basic troubleshooting (if a bug/error)
    * [x] Restarted Spyder
    * [x] Reset preferences with ``spyder --reset``
    * [x] Reinstalled the latest version of [Anaconda](https://www.anaconda.com/download/)
    * [x] Tried the other applicable steps from the Troubleshooting Guide
* [x] Completed the **Problem Description**, **Steps to Reproduce** and **Version** sections below


## Problem Description
I'm getting a bizarre variable not defined error while in debug mode. My breakpoint is inside a class and the variable that is marked as not defined is a pandas Timedelta.

I'm using spyder from the base environment but pointing to a python.exe located in a virtual environment for "mymodule". Inside the virtual environment I used ```conda install spyder-kernels```. 

Here is my code snippet to reproduce the problem.

First my main script:

```
from mymodule import myclass
import pandas as pd, numpy as np

n_sample = 1000
n_feature = 10
X = np.random.normal(0, 1, (n_sample, n_feature))
X = pd.DataFrame(X, index=pd.date_range("2000", "2001", periods=n_sample))

mc = myclass(X)
mc.myfun()
```

And the code of mymodule:

```
from abc import ABC, abstractmethod
import pandas as pd, numpy as np


class myclass(ABC):

    def __init__(self, X):
        self.X = X
    
    def myfun(self):
        min_dt = pd.to_timedelta(np.median(np.diff(self.X.index)))
        win_size_range = pd.to_timedelta([min_dt*n for n in np.logspace(1, 3, 5)])
```

I'm putting a breakpoint at the last line to mannualy compute "win_size_range". When I try to compute this variable I'm getting: `*** NameError: name 'min_dt' is not defined`

### What steps reproduce the problem?

1. Launch the script in debug mode
2. Try to manually compute the "win_size_range" in the console

### What is the expected output? What do you see instead?
The variable "win_size_range" computed as a list. But instead getting the error below.


### Paste Traceback/Error Below (if applicable)

```python-traceback

!win_size_range = pd.to_timedelta([min_dt*n for n in np.logspace(1, 3, 5)])
*** NameError: name 'min_dt' is not defined

```

## Versions

* Spyder version: 4.1.3
* Python version: 3.7.6
* Qt version: 5.12.5
* PyQt version: 5.12.3
* Operating System name/version: Windows 7

### Dependencies

```

# Mandatory:
atomicwrites >=1.2.0           :  1.4.0 (OK)
chardet >=2.0.0                :  3.0.4 (OK)
cloudpickle >=0.5.0            :  1.5.0 (OK)
diff_match_patch >=20181111    :  20200713 (OK)
intervaltree                   :  None (OK)
IPython >=4.0                  :  7.16.1 (OK)
jedi =0.15.2                   :  0.15.2 (OK)
nbconvert >=4.0                :  5.6.1 (OK)
numpydoc >=0.6.0               :  1.1.0 (OK)
paramiko >=2.4.0               :  2.7.1 (OK)
parso =0.5.2                   :  0.5.2 (OK)
pexpect >=4.4.0                :  4.8.0 (OK)
pickleshare >=0.4              :  0.7.5 (OK)
psutil >=5.3                   :  5.7.2 (OK)
pygments >=2.0                 :  2.6.1 (OK)
pylint >=0.25                  :  2.4.4 (OK)
pyls >=0.31.9;<0.32.0          :  0.31.9 (OK)
qdarkstyle >=2.8               :  2.8.1 (OK)
qtawesome >=0.5.7              :  0.7.2 (OK)
qtconsole >=4.6.0              :  4.7.5 (OK)
qtpy >=1.5.0                   :  1.9.0 (OK)
rtree >=0.8.3                  :  0.9.4 (OK)
sphinx >=0.6.6                 :  3.1.2 (OK)
spyder_kernels >=1.9.1;<1.10.0 :  1.9.2 (OK)
watchdog                       :  None (OK)
zmq >=17                       :  19.0.1 (OK)

# Optional:
cython >=0.21                  :  0.29.21 (OK)
matplotlib >=2.0.0             :  3.3.0 (OK)
numpy >=1.7                    :  1.19.0 (OK)
pandas >=0.13.1                :  1.0.5 (OK)
scipy >=0.17.0                 :  1.5.0 (OK)
sympy >=0.7.3                  :  1.6.1 (OK)

```



Discussion:
Hi @davidusb-geek. Thanks for reporting this. It seems like there is an issue with our debugger.
@impact27 can you take a look at this one?

I can successfully reproduce this bug in 4.1.5 and in 4.x dev versions with macOS Catalina. As you can see, the variable min_dt is defined in the variable explorer and printing the value in the console (in debug mode) also gives its value so not sure why this is happening.

<img width="1440" alt="Screenshot 2020-10-02 at 1 45 31 PM" src="https://user-images.githubusercontent.com/18587879/94958666-8c0a2980-04b5-11eb-80f8-37cb66f7f2c2.png">

I also tried several operations using this variable and independently running each one and they work, but running this particular code doesn't seem to work. 

<img width="642" alt="Screenshot 2020-10-02 at 1 47 34 PM" src="https://user-images.githubusercontent.com/18587879/94958863-d68ba600-04b5-11eb-959a-6356782d9468.png">

I also updated ipython to 7.18.1 and I have the same issue.

Maybe it is an issue with ipykernel. Could you try to update it?
I tried this with `ipykernel` 5.3.4 (its latest version) on Linux and the problem persists.

However, I traced down this problem to the following Python bug (closed as won't fix):

https://bugs.python.org/issue21161

There's a patch there that will evaluate comprehensions (which is the real issue here) in the right namespace:

https://bugs.python.org/file35305/default.patch

It seems pretty simple but you're the expert in our debugger @impact27. So do you think we should apply it?
A PR with that patch was created last year:

https://github.com/python/cpython/pull/15194

and all Python tests passed with it. So it seems to safe to add it.
I tried:
```
In [3]: exec(compile('[min_dt*n for n in np.logspace(1, 3, 5)]\n', '<stdin>', 'single'), globals(), {"min_dt": 0})
Traceback (most recent call last):

  File "<ipython-input-3-eac8e4ae0979>", line 1, in <module>
    exec(compile('[min_dt*n for n in np.logspace(1, 3, 5)]\n', '<stdin>', 'single'), globals(), {"min_dt": 0})

  File "<stdin>", line 1, in <module>

  File "<stdin>", line 1, in <listcomp>

NameError: name 'min_dt' is not defined
```
So I think the fix looks good. I can't think of a likely situation where this is a problem. (Unless the user does something with `globals()` and `locals()`)

If the user types in `globals()`, he won't see the correct globals variable with this fix. So the questions is which is more likely:
 - The user wants to use a list comprehension
 - The user want to check if something is not in globals
I opened https://bugs.python.org/issue41918

Resolving patch:
diff --git a/external-deps/spyder-kernels/spyder_kernels/customize/spyderpdb.py b/external-deps/spyder-kernels/spyder_kernels/customize/spyderpdb.py
--- a/external-deps/spyder-kernels/spyder_kernels/customize/spyderpdb.py
+++ b/external-deps/spyder-kernels/spyder_kernels/customize/spyderpdb.py
@@ -74,7 +74,17 @@ def default(self, line):
                        " command instead")
             return
         locals = self.curframe_locals
-        globals = self.curframe.f_globals
+
+        # This is necessary to allow running comprehensions with the
+        # frame locals. It also fallbacks to the right globals if the
+        # user wants to work with them instead.
+        # See spyder-ide/spyder#13909.
+        if not 'globals()' in line:
+            ns = self.curframe.f_globals.copy()
+            ns.update(locals)
+        else:
+            ns = self.curframe.f_globals
+
         try:
             line = TransformerManager().transform_cell(line)
             try:
@@ -91,7 +101,7 @@ def default(self, line):
                 sys.displayhook = self.displayhook
                 if execute_events:
                      get_ipython().events.trigger('pre_execute')
-                exec(code, globals, locals)
+                exec(code, ns, locals)
                 if execute_events:
                      get_ipython().events.trigger('post_execute')
             finally:


Test patch:
diff --git a/external-deps/spyder-kernels/spyder_kernels/console/tests/test_console_kernel.py b/external-deps/spyder-kernels/spyder_kernels/console/tests/test_console_kernel.py
--- a/external-deps/spyder-kernels/spyder_kernels/console/tests/test_console_kernel.py
+++ b/external-deps/spyder-kernels/spyder_kernels/console/tests/test_console_kernel.py
@@ -739,5 +739,32 @@ def test_callables_and_modules(kernel, exclude_callables_and_modules,
     settings['exclude_unsupported'] = False
 
 
+def test_comprehensions_with_locals_in_pdb(kernel):
+    """
+    Test that evaluating comprehensions with locals works in Pdb.
+
+    Also test that we use the right frame globals, in case the user
+    wants to work with them.
+
+    This is a regression test for spyder-ide/spyder#13909.
+    """
+    pdb_obj = SpyderPdb()
+    pdb_obj.curframe = inspect.currentframe()
+    pdb_obj.curframe_locals = pdb_obj.curframe.f_locals
+    kernel._pdb_obj = pdb_obj
+
+    # Create a local variable.
+    kernel._pdb_obj.default('zz = 10')
+    assert kernel.get_value('zz') == 10
+
+    # Run a list comprehension with this variable.
+    kernel._pdb_obj.default("compr = [zz * i for i in [1, 2, 3]]")
+    assert kernel.get_value('compr') == [10, 20, 30]
+
+    # Check that the variable is not reported as being part of globals.
+    kernel._pdb_obj.default("in_globals = 'zz' in globals()")
+    assert kernel.get_value('in_globals') == False
+
+
 if __name__ == "__main__":
     pytest.main()