Repository: ipython/ipython

Issue:
We should clear interactive namespace's __warningregistry__ between executing each cell
A very very common source of confusion when using python interactively is that warnings disappear. Specifically, they show up the first time they might appear, but then never again after that...:

```
In [14]: np.zeros(10) / 0
/usr/bin/ipython3:1: RuntimeWarning: invalid value encountered in true_divide
  #!/usr/bin/env python3
Out[14]: array([ nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan])

In [15]: 0 / np.zeros(5)
Out[15]: array([ nan,  nan,  nan,  nan,  nan])
```

This is because by default, the `warnings` module attempts to only show each warning once per  "source"; the idea is that if you write something like

```
for i in xrange(10000): 
    np.log(0)  # **
```

then you really only need to hear about the problem on line `**` once, not 10,000 times.

But this interacts badly with interactive usage, because the `warnings` module has a bug: it thinks that the first line of one interactive cell is the same line as the first line of another interactive cell, so if `np.zeros(10) / 0` has raised a warning then it thinks `0 / np.zeros(5)` has as well. Because they are the same line. Oops.

Specifically, the way the `warnings` module keeps track of these things is with a really weird hack: whenever it issues a warning, it _reaches into the execution environment of the code issuing the warning_, and creates a global variable called `__warningregistry__`. For example:

```
In [1]: __warningregistry__
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-1-41b71685b3a2> in <module>()
----> 1 __warningregistry__

NameError: name '__warningregistry__' is not defined

In [2]: np.zeros(10) / 0
/home/njs/.user-python2.7-64bit-3/bin/ipython:1: RuntimeWarning: invalid value encountered in divide
  #!/home/njs/.user-python2.7-64bit-3/bin/python
Out[2]: array([ nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan])

In [3]: __warningregistry__
Out[3]: {('invalid value encountered in divide', RuntimeWarning, 1): True}
```

This `__warningregistry__` value says that line 1 of the code in this namespace has already issued the given warning, so if you ever see another piece of code whose line number is 1, and which is executing in this namespace, try to issue this warning, then that warning should not be displayed.

However, in the interactive case, we know that every time we execute a new cell, the line numbers are effectively reset -- line 1 of this cell is totally different from line 1 of the previous cell. Ipython should take care of informing the `warnings` module of this fact.

This can be done very simply: just delete the `__warningregistry__` variable from the interactive namespace after each time a cell finishes executing. This will cause the `warnings` module to treat each new block of code as a new block of code.



Discussion:
You can do this by registering a post_execute hook. I don't agree that it should be the default behavior.

What possible advantage is there to making this user-configurable? In
circumstance would I ever want valid warnings to be randomly suppressed
based on unrelated code that I just happen to have run at another time?

On Sat, Oct 4, 2014 at 9:53 PM, Min RK notifications@github.com wrote:

> You can do this by registering a post_execute hook. I don't agree that it
> should be the default behavior.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/ipython/ipython/issues/6611#issuecomment-57918168.

## 

Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org

(Also, it's new users who are constantly bitten by this -- we average
something like 1 post per week on the numpy list where this is a
contributing factor in the confusion. Telling them that they ought to have
known to install a post_execute hook is not going to really help ;-).)

On Sat, Oct 4, 2014 at 9:56 PM, Nathaniel Smith njs@pobox.com wrote:

> What possible advantage is there to making this user-configurable? In
> circumstance would I ever want valid warnings to be randomly suppressed
> based on unrelated code that I just happen to have run at another time?
> 
> On Sat, Oct 4, 2014 at 9:53 PM, Min RK notifications@github.com wrote:
> 
> > You can do this by registering a post_execute hook. I don't agree that it
> > should be the default behavior.
> > 
> > —
> > Reply to this email directly or view it on GitHub
> > https://github.com/ipython/ipython/issues/6611#issuecomment-57918168.
> 
> ## 
> 
> Nathaniel J. Smith
> Postdoctoral researcher - Informatics - University of Edinburgh
> http://vorpus.org

## 

Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org

The vast majority of warnings that I see are the result of code I don't control. I don't want to see those warnings every time I use somebody else's code that I can't fix. There's a reason Python only shows warnings the first time they happen, and I don't think we should break that because the result would be super annoying.

The warnings that you don't control will not be affected. Only the warnings caused directly by the code you typed in would be affected by Nathaniel's suggestion.

```
[~/scratch]
|1> import warnings

[~/scratch]
|2> warnings.warn('bar')
/Users/rkern/Library/Enthought/Canopy_64bit/User/bin/ipython:1: UserWarning: bar
  #!/Users/rkern/Library/Enthought/Canopy_64bit/User/bin/python

[~/scratch]
|3> warnings.warn('bar')
/Users/rkern/Library/Enthought/Canopy_64bit/User/bin/ipython:1: UserWarning: bar
  #!/Users/rkern/Library/Enthought/Canopy_64bit/User/bin/python

[~/scratch]
|4> !cat foo.py
import warnings

def f():
    warnings.warn('foo')

[~/scratch]
|5> import foo

[~/scratch]
|6> foo.f()
foo.py:4: UserWarning: foo
  warnings.warn('foo')

[~/scratch]
|7> foo.f()

[~/scratch]
|8> foo.f()

[~/scratch]
|9> !cat /Users/rkern/.ipython/extensions/clear_warnings.py

class WarningClearer(object):
    def __init__(self, ip):
        self.ip = ip

    def clear_warnings(self):
        self.ip.user_ns.pop('__warningregistry__', None)


def load_ipython_extension(ip):
    wc = WarningClearer(ip)
    ip.events.register('post_execute', wc.clear_warnings)
```

Thanks for the explanation, I didn't realize that there are many warnings registries. In that case, I think this proposal seems reasonable.

Yes, sorry for not making that clear -- each execution environment has its
own **warningregistry**, which only tracks the warnings that warnings.py
thinks were _directly_ caused by code running in that environment.

On Sat, Oct 4, 2014 at 11:07 PM, Min RK notifications@github.com wrote:

> Thanks for the explanation, I didn't realize that there are many warnings
> registries. In that case, I think this proposal seems reasonable.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/ipython/ipython/issues/6611#issuecomment-57920008.

## 

Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org


Resolving patch:
diff --git a/IPython/core/events.py b/IPython/core/events.py
--- a/IPython/core/events.py
+++ b/IPython/core/events.py
@@ -63,14 +63,6 @@ def unregister(self, event, function):
         """Remove a callback from the given event."""
         self.callbacks[event].remove(function)
     
-    def reset(self, event):
-        """Clear all callbacks for the given event."""
-        self.callbacks[event] = []
-    
-    def reset_all(self):
-        """Clear all callbacks for all events."""
-        self.callbacks = {n:[] for n in self.callbacks}
-    
     def trigger(self, event, *args, **kwargs):
         """Call callbacks for ``event``.
         
diff --git a/IPython/core/interactiveshell.py b/IPython/core/interactiveshell.py
--- a/IPython/core/interactiveshell.py
+++ b/IPython/core/interactiveshell.py
@@ -874,6 +874,8 @@ def set_hook(self,name,hook, priority=50, str_key=None, re_key=None,
     def init_events(self):
         self.events = EventManager(self, available_events)
 
+        self.events.register("pre_execute", self._clear_warning_registry)
+
     def register_post_execute(self, func):
         """DEPRECATED: Use ip.events.register('post_run_cell', func)
         
@@ -883,6 +885,13 @@ def register_post_execute(self, func):
              "ip.events.register('post_run_cell', func) instead.")
         self.events.register('post_run_cell', func)
     
+    def _clear_warning_registry(self):
+        # clear the warning registry, so that different code blocks with
+        # overlapping line number ranges don't cause spurious suppression of
+        # warnings (see gh-6611 for details)
+        if "__warningregistry__" in self.user_global_ns:
+            del self.user_global_ns["__warningregistry__"]
+
     #-------------------------------------------------------------------------
     # Things related to the "main" module
     #-------------------------------------------------------------------------


Test patch:
diff --git a/IPython/core/tests/test_events.py b/IPython/core/tests/test_events.py
--- a/IPython/core/tests/test_events.py
+++ b/IPython/core/tests/test_events.py
@@ -25,22 +25,8 @@ def test_register_unregister(self):
         self.em.trigger('ping_received')
         self.assertEqual(cb.call_count, 1)
     
-    def test_reset(self):
-        cb = Mock()
-        self.em.register('ping_received', cb)
-        self.em.reset('ping_received')
-        self.em.trigger('ping_received')
-        assert not cb.called
-    
-    def test_reset_all(self):
-        cb = Mock()
-        self.em.register('ping_received', cb)
-        self.em.reset_all()
-        self.em.trigger('ping_received')
-        assert not cb.called
-    
     def test_cb_error(self):
         cb = Mock(side_effect=ValueError)
         self.em.register('ping_received', cb)
         with tt.AssertPrints("Error in callback"):
-            self.em.trigger('ping_received')
\ No newline at end of file
+            self.em.trigger('ping_received')
diff --git a/IPython/core/tests/test_interactiveshell.py b/IPython/core/tests/test_interactiveshell.py
--- a/IPython/core/tests/test_interactiveshell.py
+++ b/IPython/core/tests/test_interactiveshell.py
@@ -301,7 +301,10 @@ def test_silent_postexec(self):
             assert post_explicit.called
         finally:
             # remove post-exec
-            ip.events.reset_all()
+            ip.events.unregister('pre_run_cell', pre_explicit)
+            ip.events.unregister('pre_execute', pre_always)
+            ip.events.unregister('post_run_cell', post_explicit)
+            ip.events.unregister('post_execute', post_always)
     
     def test_silent_noadvance(self):
         """run_cell(silent=True) doesn't advance execution_count"""
@@ -840,3 +843,17 @@ def test_syntaxerror_input_transformer(self):
 
 
 
+def test_warning_suppression():
+    ip.run_cell("import warnings")
+    try:
+        with tt.AssertPrints("UserWarning: asdf", channel="stderr"):
+            ip.run_cell("warnings.warn('asdf')")
+        # Here's the real test -- if we run that again, we should get the
+        # warning again. Traditionally, each warning was only issued once per
+        # IPython session (approximately), even if the user typed in new and
+        # different code that should have also triggered the warning, leading
+        # to much confusion.
+        with tt.AssertPrints("UserWarning: asdf", channel="stderr"):
+            ip.run_cell("warnings.warn('asdf')")
+    finally:
+        ip.run_cell("del warnings")