Repository: ipython/ipython

Issue:
%debug magic "up" fails to pass through generator stack
Consider the following example "pdbtest.py":

```
def f(x):
    raise Exception

gen = (f(x) for x in [0])

for x in gen:
    pass
```

If I run this with `python -i pdbtest.py`, when the exception is thrown I can go up the stack all the way to the module-level with pdb:

```
Traceback (most recent call last):
  File "pdbtest.py", line 6, in <module>
    for x in gen:
  File "pdbtest.py", line 4, in <genexpr>
    gen = (f(x) for x in [0])
  File "pdbtest.py", line 2, in f
    raise Exception
Exception
>>> import pdb; pdb.pm()
> /home/antony/tests/pdbtest.py(2)f()
-> raise Exception
(Pdb) u
> /home/antony/tests/pdbtest.py(4)<genexpr>()
-> gen = (f(x) for x in [0])
(Pdb) u
> /home/antony/tests/pdbtest.py(6)<module>()
-> for x in gen:
(Pdb) u
*** Oldest frame
(Pdb) 
>>> 
```

However, if I try to do the same with ipython's %debug magic, I cannot go beyond the generator frame -- which is particularly annoying when you think the actual bug happens earlier:

```
In [1]: %debug
> /path/to/pdbtest.py(2)f()
      1 def f(x):
----> 2     raise Exception
      3 

ipdb> u
> /path/to/pdbtest.py(4)<genexpr>()
      3 
----> 4 gen = (f(x) for x in [0])
      5 

ipdb> u
*** Oldest frame
ipdb>
```

This happens with ipython 2.1.0 and python 3.4.



Discussion:
I agree, this is one of the biggest problems for actually using ipython to run and debug code.

I would guess that this is a general limitation of pdb. Does anyone with more interpreter expertise (@takluyver?) think this can be addressed in IPython?

From the original post, it can't be a general limitation of pdb, because it works in vanilla pdb. It must be something we break. @xapple and @anntzer , if you want to try to dig in and work out what causes this, go for it.

A quick session of print-debugging revealed that `%debug` and `pdb.pm()` use different entry points to `Pdb.interaction`, namely `%debug` fills in both the frame and the traceback arguments, whereas `pdb.pm()` leaves the frame argument empty.  As a consequence, `Pdb.get_frame` later returns different results.  That's all I have so far.

Not sure how much this will do but... *bump*

This is not specific to %debug. I get the same thing with the auto-call pdb setting.
It's a pretty serious bug for a debugger not to be able to travel up the call stack... astonishing that is untouched since it appeared in 2012.
It also affects Python 3.6 (both IPython 6.4.0 and Jupyter Notebook with IPython 5.3.0).

Resolving patch:
diff --git a/IPython/core/ultratb.py b/IPython/core/ultratb.py
--- a/IPython/core/ultratb.py
+++ b/IPython/core/ultratb.py
@@ -1203,7 +1203,7 @@ def debugger(self, force=False):
                 if etb and etb.tb_next:
                     etb = etb.tb_next
                 self.pdb.botframe = etb.tb_frame
-                self.pdb.interaction(self.tb.tb_frame, self.tb)
+                self.pdb.interaction(None, etb)
 
         if hasattr(self, 'tb'):
             del self.tb


Test patch:
diff --git a/IPython/terminal/tests/test_debug_magic.py b/IPython/terminal/tests/test_debug_magic.py
new file mode 100644
--- /dev/null
+++ b/IPython/terminal/tests/test_debug_magic.py
@@ -0,0 +1,74 @@
+"""Test embedding of IPython"""
+
+#-----------------------------------------------------------------------------
+#  Copyright (C) 2013 The IPython Development Team
+#
+#  Distributed under the terms of the BSD License.  The full license is in
+#  the file COPYING, distributed as part of this software.
+#-----------------------------------------------------------------------------
+
+#-----------------------------------------------------------------------------
+# Imports
+#-----------------------------------------------------------------------------
+
+import os
+import sys
+from IPython.testing.decorators import skip_win32
+
+#-----------------------------------------------------------------------------
+# Tests
+#-----------------------------------------------------------------------------
+
+@skip_win32
+def test_debug_magic_passes_through_generators():
+    """
+    This test that we can correctly pass through frames of a generator post-mortem.
+    """
+    import pexpect
+    import re
+    in_prompt = re.compile(b'In ?\[\\d+\]:')
+    ipdb_prompt = 'ipdb>'
+    env = os.environ.copy()
+    child = pexpect.spawn(sys.executable, ['-m', 'IPython', '--colors=nocolor', '--simple-prompt'],
+                          env=env)
+    child.timeout = 2
+
+    child.expect(in_prompt)
+    child.sendline("def f(x):")
+    child.sendline("    raise Exception")
+    child.sendline("")
+
+    child.expect(in_prompt)
+    child.sendline("gen = (f(x) for x in [0])")
+    child.sendline("")
+
+    child.expect(in_prompt)
+    child.sendline("for x in gen:")
+    child.sendline("    pass")
+    child.sendline("")
+
+    child.expect('Exception:')
+
+    child.expect(in_prompt)
+    child.sendline(r'%debug')
+    child.expect('----> 2     raise Exception')
+
+    child.expect(ipdb_prompt)
+    child.sendline('u')
+    child.expect_exact(r'----> 1 gen = (f(x) for x in [0])')
+
+    child.expect(ipdb_prompt)
+    child.sendline('u')
+    child.expect_exact('----> 1 for x in gen:')
+
+    child.expect(ipdb_prompt)
+    child.sendline('u')
+    child.expect_exact('*** Oldest frame')
+
+    child.expect(ipdb_prompt)
+    child.sendline('exit')
+
+    child.expect(in_prompt)
+    child.sendline('exit')
+
+    child.close()