Repository: ipython/ipython

Issue:
ERROR: Internal Python error in the inspect module.
Looks like this is related to issue #53. I just got the following in the notebook:

```
In [33]:

# Switch to NumPy
pw.prtparam = p
```

```
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\ipython-0.13.dev-py2.7.egg\IPython\core\ultratb.py", line 756, in structured_traceback
    records = _fixed_getinnerframes(etb, context, tb_offset)
  File "C:\Python27\lib\site-packages\ipython-0.13.dev-py2.7.egg\IPython\core\ultratb.py", line 242, in _fixed_getinnerframes
    records  = fix_frame_records_filenames(inspect.getinnerframes(etb, context))
  File "C:\Python27\lib\inspect.py", line 1041, in getinnerframes
    framelist.append((tb.tb_frame,) + getframeinfo(tb, context))
  File "C:\Python27\lib\inspect.py", line 1005, in getframeinfo
    lines, lnum = findsource(frame)
  File "C:\Python27\lib\inspect.py", line 578, in findsource
    if pat.match(lines[lnum]): break
IndexError: list index out of range
ERROR: Internal Python error in the inspect module.
Below is the traceback from this internal error.

Unfortunately, your original traceback can not be constructed.
```



Discussion:
Aha, this rings a bell...does re-enabling the monkeypatch that was disabled in 1cd5a5815064ce992f200c103108f2470dde9508 fix it?

Can you provide steps to reproduce this?

It happened in a context with a lot of Qt, OpenGL, and PyCUDA going on. There was a bug in my code causing an exception inside the prtparam setter, which was defined like this:

```

    @property
    def prtparam(self):
        return self._prtparam

    @prtparam.setter
    def prtparam(self, newprtparam):
        if self._prtparam != None:
            self._prtparam.close()
        self._prtparam = newprtparam
        self.prt_view_widget.prtparam = newprtparam
```

Likely the error was raised from the close() call, which was calling both OpenGL and PyCUDA functions. I was able to repeat it in a few fresh runs, but if I run it with the bug in there now, it produces the perfectly reasonable

```

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
D:\Develop\notebooks\<ipython-input-15-dec90c4997b5> in <module>()
      1 # Switch to NumPy
----> 2 pw.prtparam = p

D:\Develop\scripts\particle_window.py in prtparam(self, newprtparam)
     53     def prtparam(self, newprtparam):
     54         if self._prtparam != None:
---> 55             self._prtparam.close()
     56         self._prtparam = newprtparam

D:\Develop\scripts\cuda_prt.pyc in close(self)
    142     def close(self):
    143         # Free the particle VBO
--> 144         self.prt = None
    145         # Free the CUDA context
    146         if self._context != None:

D:\Develop\scripts\cuda_prt.pyc in prt(self, newprt)
     59         self.selected_particles = None
     60         # Make sure the GL context is set
---> 61         self._gl_context_fn()
     62 
     63         # Free the existing particle VBO

TypeError: 'NoneType' object is not callable
```

Hmmm, so you can no longer reproduce it? Looking at the traceback when it was failing, though, it is failing at the point that our monkeypatch was intended to resolve.

The error is that it's getting a line number beyond the end of the file. I think that might occur if you're changing the file  while the module is loaded, so that the loaded line numbers don't match those on disk.

No, it's not happening now, but your thought about changing the file makes good sense. The close() function is at the very end of cuda_prt.py, and I've done 

```
%load_ext autoreload
%autoreload 2
```

so that it automatically updates when I change stuff.

OK, perhaps we have to re-enable the monkeypatch. I'd prefer to catch the error when it reaches our own code, so we benefit from upstream changes to `inspect`. I'll play around and see if I can reproduce it by editing files.

I got the same error in Ubuntu 12.04 (precise) with ipython 0.12.1+dfsg-0ubuntu1  when repeatedly editing a python file and running it in ipython sort of like this:

In [423]: run aiclass22_nlp.py

In [424]: p = main()

# edit... edit...

In [425]: run aiclass22_nlp.py

In [426]: p = main()
ERROR: Internal Python error in the inspect module.
Below is the traceback from this internal error.
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/IPython/core/ultratb.py", line 756, in structured_traceback
    records = _fixed_getinnerframes(etb, context, tb_offset)
  File "/usr/lib/python2.7/dist-packages/IPython/core/ultratb.py", line 242, in _fixed_getinnerframes
    records  = fix_frame_records_filenames(inspect.getinnerframes(etb, context))
  File "/usr/lib/python2.7/inspect.py", line 1043, in getinnerframes
    framelist.append((tb.tb_frame,) + getframeinfo(tb, context))
  File "/usr/lib/python2.7/inspect.py", line 1007, in getframeinfo
    lines, lnum = findsource(frame)
  File "/usr/lib/python2.7/inspect.py", line 580, in findsource
    if pat.match(lines[lnum]): break
IndexError: list index out of range

Unfortunately, your original traceback can not be constructed.

I've assigned this to myself to try to reproduce the error.

OK, I've got a reproducible case - Create a file test.py with:

```
1
2
3
def f():
    1/0
```

Then %run or import it in IPython, and call `f()` to get the ZeroDivisionError.

Now edit the file to remove the 1 2 3 lines, and without re-running/re-importing, call `f()` _twice_ more in IPython, which results in this:

```
In [5]: f()
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-5-0ec059b9bfe1> in <module>()
----> 1 f()

/home/thomas/scratch/test.py in f()
      3 

ZeroDivisionError: integer division or modulo by zero

In [6]: f()
ERROR: Internal Python error in the inspect module.
Below is the traceback from this internal error.
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/IPython/core/ultratb.py", line 756, in structured_traceback
    records = _fixed_getinnerframes(etb, context, tb_offset)
  File "/usr/lib/python2.7/dist-packages/IPython/core/ultratb.py", line 242, in _fixed_getinnerframes
    records  = fix_frame_records_filenames(inspect.getinnerframes(etb, context))
  File "/usr/lib/python2.7/inspect.py", line 1043, in getinnerframes
    framelist.append((tb.tb_frame,) + getframeinfo(tb, context))
  File "/usr/lib/python2.7/inspect.py", line 1007, in getframeinfo
    lines, lnum = findsource(frame)
  File "/usr/lib/python2.7/inspect.py", line 580, in findsource
    if pat.match(lines[lnum]): break
IndexError: list index out of range

Unfortunately, your original traceback can not be constructed.
```

Why it's only the second call that fails, I don't know. I'll try to work up a proper test case for this, and then a fix.


Resolving patch:
diff --git a/IPython/core/ultratb.py b/IPython/core/ultratb.py
--- a/IPython/core/ultratb.py
+++ b/IPython/core/ultratb.py
@@ -126,15 +126,10 @@ def inspect_error():
     error('Internal Python error in the inspect module.\n'
           'Below is the traceback from this internal error.\n')
 
-
-# N.B. This function is a monkeypatch we are currently not applying.
-# It was written some time ago, to fix an apparent Python bug with
-# codeobj.co_firstlineno . Unfortunately, we don't know under what conditions
-# the bug occurred, so we can't tell if it has been fixed. If it reappears, we
-# will apply the monkeypatch again. Also, note that findsource() is not called
-# by our code at this time - we don't know if it was when the monkeypatch was
-# written, or if the monkeypatch is needed for some other code (like a debugger).
-# For the discussion about not applying it, see gh-1229. TK, Jan 2011.
+# This function is a monkeypatch we apply to the Python inspect module. We have
+# now found when it's needed (see discussion on issue gh-1456), and we have a
+# test case (IPython.core.tests.test_ultratb.ChangedPyFileTest) that fails if
+# the monkeypatch is not applied. TK, Aug 2012.
 def findsource(object):
     """Return the entire source file and starting line number for an object.
 
@@ -210,10 +205,8 @@ def findsource(object):
         return lines, lnum
     raise IOError('could not find code object')
 
-# Not applying the monkeypatch - see above the function for details. TK, Jan 2012
-# Monkeypatch inspect to apply our bugfix.  This code only works with py25
-#if sys.version_info[:2] >= (2,5):
-#    inspect.findsource = findsource
+# Monkeypatch inspect to apply our bugfix.  This code only works with Python >= 2.5
+inspect.findsource = findsource
 
 def fix_frame_records_filenames(records):
     """Try to fix the filenames in each record from inspect.getinnerframes().


Test patch:
diff --git a/IPython/core/tests/test_ultratb.py b/IPython/core/tests/test_ultratb.py
new file mode 100644
--- /dev/null
+++ b/IPython/core/tests/test_ultratb.py
@@ -0,0 +1,51 @@
+"""Tests for IPython.core.ultratb
+"""
+
+import os.path
+import unittest
+
+from IPython.testing import tools as tt
+from IPython.utils.syspathcontext import prepended_to_syspath
+from IPython.utils.tempdir import TemporaryDirectory
+
+ip = get_ipython()
+
+file_1 = """1
+2
+3
+def f():
+  1/0
+"""
+
+file_2 = """def f():
+  1/0
+"""
+
+class ChangedPyFileTest(unittest.TestCase):
+    def test_changing_py_file(self):
+        """Traceback produced if the line where the error occurred is missing?
+        
+        https://github.com/ipython/ipython/issues/1456
+        """
+        with TemporaryDirectory() as td:
+            fname = os.path.join(td, "foo.py")
+            with open(fname, "w") as f:
+                f.write(file_1)
+            
+            with prepended_to_syspath(td):
+                ip.run_cell("import foo")
+            
+            with tt.AssertPrints("ZeroDivisionError"):
+                ip.run_cell("foo.f()")
+            
+            # Make the file shorter, so the line of the error is missing.
+            with open(fname, "w") as f:
+                f.write(file_2)
+            
+            # For some reason, this was failing on the *second* call after
+            # changing the file, so we call f() twice.
+            with tt.AssertNotPrints("Internal Python error", channel='stderr'):
+                with tt.AssertPrints("ZeroDivisionError"):
+                    ip.run_cell("foo.f()")
+                with tt.AssertPrints("ZeroDivisionError"):
+                    ip.run_cell("foo.f()")
diff --git a/IPython/testing/tools.py b/IPython/testing/tools.py
--- a/IPython/testing/tools.py
+++ b/IPython/testing/tools.py
@@ -306,7 +306,10 @@ def write(self, s):
             super(MyStringIO, self).write(s)
 
 notprinted_msg = """Did not find {0!r} in printed output (on {1}):
-{2!r}"""
+-------
+{2!s}
+-------
+"""
 
 class AssertPrints(object):
     """Context manager for testing that code prints certain text.
@@ -337,7 +340,13 @@ def __exit__(self, etype, value, traceback):
         printed = self.buffer.getvalue()
         assert self.s in printed, notprinted_msg.format(self.s, self.channel, printed)
         return False
-    
+
+printed_msg = """Found {0!r} in printed output (on {1}):
+-------
+{2!s}
+-------
+"""
+
 class AssertNotPrints(AssertPrints):
     """Context manager for checking that certain output *isn't* produced.
     
@@ -346,7 +355,7 @@ def __exit__(self, etype, value, traceback):
         self.tee.flush()
         setattr(sys, self.channel, self.orig_stream)
         printed = self.buffer.getvalue()
-        assert self.s not in printed, notprinted_msg.format(self.s, self.channel, printed)
+        assert self.s not in printed, printed_msg.format(self.s, self.channel, printed)
         return False
 
 @contextmanager