Repository: spyder-ide/spyder

Issue:
Error debugging Plotly code in Spyder 5.4.1
F10 debugging throws NoneType Errors in latest Spyder 5.4.1:
Python 3.10.9 64-bit | Qt 5.15.2 | PyQt5 5.15.7 | Windows 10 | Anaconda

Spyder Run works fine.
Cut n Paste this code into Spyder Console works fine.
Debug with F10 get the nonetype error below.
Please suggest workaround or fix.

```python
import pandas as pd
import plotly.express as px
import plotly.io as pio
pio.renderers.default='svg'  # this line required for plots to show in Spyder Plots panel

df = pd.DataFrame([
    dict(Task="Job A", Start='2009-01-01', Finish='2009-02-28'),
    dict(Task="Job B", Start='2009-03-05', Finish='2009-04-15'),
    dict(Task="Job C", Start='2009-02-20', Finish='2009-05-30')
])

fig = px.timeline(df, x_start="Start", x_end="Finish", y="Task")
fig.update_yaxes(autorange="reversed") # otherwise tasks are listed from the bottom up
fig.show()

```
And the error text:

Traceback (most recent call last):

  File ~\.conda\envs\python3_10\lib\site-packages\spyder_kernels\customize\spyderpdb.py:889 in run
    super(SpyderPdb, self).run(cmd, globals, locals)

  File ~\.conda\envs\python3_10\lib\bdb.py:597 in run
    exec(cmd, globals, locals)

  File e:\onedrive\onedrive - siemens healthineers\_projects\atlas (atellica refresh)\scheduling\untitled0.py:30
    fig = px.timeline(df, x_start="Start", x_end="Finish", y="Task")

  File ~\.conda\envs\python3_10\lib\site-packages\plotly\express\_chart_types.py:423 in timeline
    return make_figure(

  File ~\.conda\envs\python3_10\lib\site-packages\plotly\express\_core.py:1988 in make_figure
    apply_default_cascade(args)

  File ~\.conda\envs\python3_10\lib\site-packages\plotly\express\_core.py:931 in apply_default_cascade
    x[1] for x in args["template"].layout.colorscale.sequential

AttributeError: 'NoneType' object has no attribute 'layout'


Discussion:
Hey @Medwar19, thanks for reporting. At which exact point during the debugging the error occurs?
Set a breakpoint on px.timeline and the line below, then Cnt+F5, Cnt+F10, Cnt+F12 for the NoneType error.

updating to Spyder 5.41 created the error in existing code. 
downgrading to Spyder 5.2.2 (that came with current Anaconda release) works fine.

Thanks @Medwar19 for the additional instructions. I can confirm the bug with our latest version.

Could you update to Spyder 5.3.3 and try again? That would help us to narrow down the version that introduced this bug.
Updated to Spyder 5.3.3 - Works fine.
Great! Thanks for the confirmation.
@impact27, could you take a look at this problem? I can confirm @Medwar19's findings: if you put a breakpoint in

```python
fig = px.timeline(df, x_start="Start", x_end="Finish", y="Task")
```

and then click `Run current line` or run `!next`, Spyder throws the error above in Spyder 5.4+ but works fine in 5.3.3 or less.
I can reproduce in ipython so this is not a spyder specific bug. basically args is locals() if a frame because `plotly` uses `locals()` ipython uses `frame.f_locals` which resets the locals.
Maybe we should stop using the iPython debugger and base our debugger on Pdb?
The IPython debugger is a bit too liberal with the use of `frame.f_locals`
> Maybe we should stop using the iPython debugger and base our debugger on Pdb?
The IPython debugger is a bit too liberal with the use of frame.f_locals

Well, I'd prefer to share our debugger work with the IPython team because it'd benefit both Spyder and IPython users. In addition, this seems a regression in IPython because it was working fine with IPython 7 (used by Spyder 5.3.3).

But if you think it'd be better for us to have our own subclass of Pdb, so that we can avoid this kind of regressions in the future and better control the evolution of own our debugger, then please go ahead and do it.
The problem here is that solving this issue would require dropping a lot of IPython features (Basically the whole frame hiding logic). The current approach of the ipython debugger is:

For every executed line, pdb calls `stop_here`:

https://github.com/ipython/ipython/blob/3339fad91de3e3da1f02b0b209afef20f021ca33/IPython/core/debugger.py#L840

The IPython debugger calls `self._is_in_decorator_internal_and_should_skip` which does that:

```
cframe = frame
while getattr(cframe, "f_back", None):
    cframe = cframe.f_back
    if self._get_frame_locals(cframe).get(DEBUGGERSKIP):
        return True
```

This is a loop on every single frame of the stack. It check for the existence of `__debuggerskip__` in the locals.

There are two issues with this approach:
 1 - It calls `frame.f_locals` on every single frame (except for the current frame) which will reset any dict the user got with `locals()` and then modified, which causes the current issue.
 2 - This is a lot of computation to do on every single frame! This causes https://github.com/spyder-ide/spyder/issues/20571

I am not sure what the solution here is. Solving the first issue would require dropping a lot of feature the IPython team has been working on during the last year. The second issue could be mitigated by optimising the code I think.


I opened https://github.com/ipython/ipython/issues/13972
Ok, thanks for the thorough analysis @impact27! The situation looks not really promising for us to keep using the IPython debugger.

Let's wait for a week or so to see what we hear from them. If they don't reply back, I think we should switch to use our own subclass of Pdb, as you suggested.
I got prompted to install 5.4.3
Any update on this issue included in the new release?
Thanks
Hi, i was searching through the issue and just filed a new bug report here. This seems like it might be related. 

https://github.com/spyder-ide/spyder/issues/20949
ok yes i have 5.4.3 and i see a similar error,so no fix for this yet. 
A couple of additional comments on this problem:

* It's an issue with IPython 8, so a workaround for now is to downgrade to IPython 7.34.
* @impact27, I think we can solve it on our side by making `debuggerskip = False` in the `self._predicates` dict of IPython's Pdb class. That way `_is_in_decorator_internal_and_should_skip` won't try to access locals on every frame. What do you think about that?

Resolving patch:
diff --git a/external-deps/spyder-kernels/setup.py b/external-deps/spyder-kernels/setup.py
--- a/external-deps/spyder-kernels/setup.py
+++ b/external-deps/spyder-kernels/setup.py
@@ -51,7 +51,6 @@ def get_version(module='spyder_kernels'):
 ]
 
 TEST_REQUIREMENTS = [
-    'codecov',
     'cython',
     'dask[distributed]',
     'flaky',
diff --git a/external-deps/spyder-kernels/spyder_kernels/customize/spyderpdb.py b/external-deps/spyder-kernels/spyder_kernels/customize/spyderpdb.py
--- a/external-deps/spyder-kernels/spyder_kernels/customize/spyderpdb.py
+++ b/external-deps/spyder-kernels/spyder_kernels/customize/spyderpdb.py
@@ -109,6 +109,17 @@ def __init__(self, completekey='tab', stdin=None, stdout=None,
         # Line received from the frontend
         self._cmd_input_line = None
 
+        # This is not available in IPython 5
+        if hasattr(self, '_predicates'):
+            # Turn off IPython's debugger skip funcionality by default because
+            # it makes our debugger quite slow. It's also important to remark
+            # that this functionality doesn't do anything on its own. Users
+            # need to mark what frames they want to skip for it to be useful.
+            # So, we hope that knowledgeable users will find that they need to
+            # enable it in Spyder.
+            # Fixes spyder-ide/spyder#20639.
+            self._predicates["debuggerskip"] = False
+
     # --- Methods overriden for code execution
     def print_exclamation_warning(self):
         """Print pdb warning for exclamation mark."""


Test patch:
diff --git a/external-deps/spyder-kernels/requirements/tests.txt b/external-deps/spyder-kernels/requirements/tests.txt
--- a/external-deps/spyder-kernels/requirements/tests.txt
+++ b/external-deps/spyder-kernels/requirements/tests.txt
@@ -1,4 +1,3 @@
-codecov
 cython
 dask
 flaky
diff --git a/external-deps/spyder-kernels/spyder_kernels/console/tests/test_console_kernel.py b/external-deps/spyder-kernels/spyder_kernels/console/tests/test_console_kernel.py
--- a/external-deps/spyder-kernels/spyder_kernels/console/tests/test_console_kernel.py
+++ b/external-deps/spyder-kernels/spyder_kernels/console/tests/test_console_kernel.py
@@ -541,11 +541,13 @@ def myFunc(i):
 
 
 @flaky(max_runs=3)
-@pytest.mark.skipif(
-    not PY3, reason="Only meant for Python 3")
+@pytest.mark.skipif(not PY3, reason="Only meant for Python 3")
 @pytest.mark.skipif(
     sys.platform == 'darwin' and sys.version_info[:2] == (3, 8),
     reason="Fails on Mac with Python 3.8")
+@pytest.mark.skipif(
+    os.environ.get('USE_CONDA') != 'true',
+    reason="Doesn't work with pip packages")
 def test_dask_multiprocessing(tmpdir):
     """
     Test that dask multiprocessing works on Python 3.
@@ -1107,7 +1109,7 @@ def test_locals_globals_in_pdb(kernel):
     not sys.platform.startswith('linux'),
     reason="Doesn't work reliably on Windows and Mac")
 @pytest.mark.skipif(
-    not bool(os.environ.get('USE_CONDA')),
+    os.environ.get('USE_CONDA') != 'true',
     reason="Doesn't work with pip packages")
 @pytest.mark.skipif(
     sys.version_info[:2] < (3, 8),
diff --git a/external-deps/spyder-kernels/spyder_kernels/utils/tests/test_nsview.py b/external-deps/spyder-kernels/spyder_kernels/utils/tests/test_nsview.py
--- a/external-deps/spyder-kernels/spyder_kernels/utils/tests/test_nsview.py
+++ b/external-deps/spyder-kernels/spyder_kernels/utils/tests/test_nsview.py
@@ -353,7 +353,7 @@ def test_get_type_string():
     assert get_type_string(series) == 'Series'
 
     index = pd.Index([1, 2, 3])
-    assert get_type_string(index) == 'Int64Index'
+    assert get_type_string(index) in ['Int64Index', 'Index']
 
     # PIL images
     img = PIL.Image.new('RGB', (256,256))