Repository: numpy/numpy

Issue:
reload(numpy);numpy.max([]) raises TypeError
Using `maintenance/1.11.x` on Windows (msvc builds), the following code fails:

``` Python
Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:18:55) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from importlib import reload
>>> import numpy
>>> reload(numpy)
<module 'numpy' from 'X:\\Python35\\lib\\site-packages\\numpy\\__init__.py'>
>>> numpy.max([])
Traceback (most recent call last):
  File "X:\Python35\lib\site-packages\numpy\core\fromnumeric.py", line 2290, in amax
    amax = a.max
AttributeError: 'list' object has no attribute 'max'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "X:\Python35\lib\site-packages\numpy\core\fromnumeric.py", line 2293, in amax
    out=out, **kwargs)
  File "X:\Python35\lib\site-packages\numpy\core\_methods.py", line 26, in _amax
    return umr_maximum(a, axis, None, out, keepdims)
TypeError: an integer is required (got type type)
```

I traced this to #7736, backport of #4619 



Discussion:
The _NoValue sentinel changes on reload, but not its values captured in 
function default arguments. The problem is referring to it via `import 
numpy as np; np._NoValue` instead of `from numpy import _NoValue` in 
fromnumeric.py, which is not reload-safe.

@pv Good catch.

I wonder if it would make more sense for  `_NoValue` to be locally defined in the modules? It is a private class in any case.

That said, I thought C modules could not be reloaded?

Curiously, there is no error with 2.7 or 3.5 in Fedora 24.

Yes, I think the reload-safe solution is to use module-specific private 
sentinels. Otherwise, different modules end up with different singletons 
on reload. Or maybe just put the sentinel in `numpy.core` etc. private 
place, which people are unlikely to reload.

I can confirm that using `from numpy import _NoValue; _NoValue` instead of `import numpy as np; np._NoValue` fixes this issue.

Looks like we need a global, reload safe, `_NoValue`, as calls from other modules pass through `fromnumeric` which checks the values. The quick work around is probably to define it somewhere more reload safe than `numpy/__init__.py`.

Possible alternative approach: add something like this to `numpy/__init__.py`:

``` python
if "_already_loaded" in globals():
    raise RuntimeError("Reloading numpy isn't supported")
_already_loaded = True
```

Probably the explicit import works also, looks like the local binding hangs onto the instance in the original import of Numpy before any reloads.

This all makes me wonder if the various warnings defined in `__init__.py` are also safe if used in `catch_warnings`. We don't do that, just wondering.

@njsmith Yeah, reloading numpy alone doesn't look to achieve much except mess up the local stuff defined in the `__init__.py`.

I note that `isinstance` also fails for reloaded classes. I'm inclined to follow @njsmith and just disallow reloading numpy, it isn't clear to me that reloading does anything useful. Thoughts?


Resolving patch:
diff --git a/numpy/__init__.py b/numpy/__init__.py
--- a/numpy/__init__.py
+++ b/numpy/__init__.py
@@ -109,56 +109,8 @@
 import sys
 import warnings
 
-# Disallow reloading numpy. Doing that does nothing to change previously
-# loaded modules, which would need to be reloaded separately, but it does
-# change the identity of the warnings and sentinal classes defined below
-# with dire consequences when checking for identity.
-if '_is_loaded' in globals():
-    raise RuntimeError('Reloading numpy is not supported')
-_is_loaded = True
-
-
-# Define some global warnings and the _NoValue sentinal. Defining them here
-# means that their identity will change if numpy is reloaded, hence if that is
-# to be allowed they should be moved into their own, non-reloadable module.
-# Note that these should be defined (or imported) before the other imports.
-class ModuleDeprecationWarning(DeprecationWarning):
-    """Module deprecation warning.
-
-    The nose tester turns ordinary Deprecation warnings into test failures.
-    That makes it hard to deprecate whole modules, because they get
-    imported by default. So this is a special Deprecation warning that the
-    nose tester will let pass without making tests fail.
-
-    """
-    pass
-
-
-class VisibleDeprecationWarning(UserWarning):
-    """Visible deprecation warning.
-
-    By default, python will not show deprecation warnings, so this class
-    can be used when a very visible warning is helpful, for example because
-    the usage is most likely a user bug.
-
-    """
-    pass
-
-
-class _NoValue:
-    """Special keyword value.
-
-    This class may be used as the default value assigned to a deprecated
-    keyword in order to check if it has been given a user defined value.
-    """
-    pass
-
-
-# oldnumeric and numarray were removed in 1.9. In case some packages import
-# but do not use them, we define them here for backward compatibility.
-oldnumeric = 'removed'
-numarray = 'removed'
-
+from ._globals import ModuleDeprecationWarning, VisibleDeprecationWarning
+from ._globals import _NoValue
 
 # We first need to detect if we're being called as part of the numpy setup
 # procedure itself in a reliable manner.
@@ -177,6 +129,7 @@ class _NoValue:
         its source directory; please exit the numpy source tree, and relaunch
         your python interpreter from there."""
         raise ImportError(msg)
+
     from .version import git_revision as __git_revision__
     from .version import version as __version__
 
@@ -239,3 +192,8 @@ def pkgload(*packages, **options):
     warnings.filterwarnings("ignore", message="numpy.dtype size changed")
     warnings.filterwarnings("ignore", message="numpy.ufunc size changed")
     warnings.filterwarnings("ignore", message="numpy.ndarray size changed")
+
+    # oldnumeric and numarray were removed in 1.9. In case some packages import
+    # but do not use them, we define them here for backward compatibility.
+    oldnumeric = 'removed'
+    numarray = 'removed'
diff --git a/numpy/_globals.py b/numpy/_globals.py
new file mode 100644
--- /dev/null
+++ b/numpy/_globals.py
@@ -0,0 +1,62 @@
+"""
+Module defining global singleton classes.
+
+This module raises a RuntimeError if an attempt to reload it is made. In that
+way the identities of the classes defined here are fixed and will remain so
+even if numpy itself is reloaded. In particular, a function like the following
+will still work correctly after numpy is reloaded::
+
+    def foo(arg=np._NoValue):
+        if arg is np._NoValue:
+            ...
+
+That was not the case when the singleton classes were defined in the numpy
+``__init__.py`` file. See gh-7844 for a discussion of the reload problem that
+motivated this module.
+
+"""
+from __future__ import division, absolute_import, print_function
+
+
+__ALL__ = [
+    'ModuleDeprecationWarning', 'VisibleDeprecationWarning', '_NoValue'
+    ]
+
+
+# Disallow reloading this module so as to preserve the identities of the
+# classes defined here.
+if '_is_loaded' in globals():
+    raise RuntimeError('Reloading numpy._globals is not allowed')
+_is_loaded = True
+
+
+class ModuleDeprecationWarning(DeprecationWarning):
+    """Module deprecation warning.
+
+    The nose tester turns ordinary Deprecation warnings into test failures.
+    That makes it hard to deprecate whole modules, because they get
+    imported by default. So this is a special Deprecation warning that the
+    nose tester will let pass without making tests fail.
+
+    """
+    pass
+
+
+class VisibleDeprecationWarning(UserWarning):
+    """Visible deprecation warning.
+
+    By default, python will not show deprecation warnings, so this class
+    can be used when a very visible warning is helpful, for example because
+    the usage is most likely a user bug.
+
+    """
+    pass
+
+
+class _NoValue:
+    """Special keyword value.
+
+    This class may be used as the default value assigned to a deprecated
+    keyword in order to check if it has been given a user defined value.
+    """
+    pass


Test patch:
diff --git a/numpy/tests/test_reloading.py b/numpy/tests/test_reloading.py
--- a/numpy/tests/test_reloading.py
+++ b/numpy/tests/test_reloading.py
@@ -2,7 +2,6 @@
 
 import sys
 
-import numpy as np
 from numpy.testing import assert_raises, assert_, run_module_suite
 
 if sys.version_info[:2] >= (3, 4):
@@ -10,13 +9,22 @@
 else:
     from imp import reload
 
-def test_reloading_exception():
+def test_numpy_reloading():
     # gh-7844. Also check that relevant globals retain their identity.
+    import numpy as np
+    import numpy._globals
+
     _NoValue = np._NoValue
     VisibleDeprecationWarning = np.VisibleDeprecationWarning
     ModuleDeprecationWarning = np.ModuleDeprecationWarning
 
-    assert_raises(RuntimeError, reload, np)
+    reload(np)
+    assert_(_NoValue is np._NoValue)
+    assert_(ModuleDeprecationWarning is np.ModuleDeprecationWarning)
+    assert_(VisibleDeprecationWarning is np.VisibleDeprecationWarning)
+
+    assert_raises(RuntimeError, reload, numpy._globals)
+    reload(np)
     assert_(_NoValue is np._NoValue)
     assert_(ModuleDeprecationWarning is np.ModuleDeprecationWarning)
     assert_(VisibleDeprecationWarning is np.VisibleDeprecationWarning)