Repository: numba/numba

Issue:
Numba `__init__.py` not catching incompatible NumPy version early enough
## Reporting a bug

- [ ] I have tried using the latest released version of Numba (most recent is
 visible in the change log (https://github.com/numba/numba/blob/main/CHANGE_LOG).
- [ ] I have included a self contained code sample to reproduce the problem.
  i.e. it's possible to run as 'python bug.py'.

Following on from numerous issues reported against #8615, no released version of Numba is compatible with NumPy 1.24, this is stated in the docs e.g.:
https://github.com/numba/numba/blob/288a38bbd5a15418a211bf067878dfdf3c139509/docs/source/user/installing.rst?plain=1#L281

The problem in practice seems to be as follows:
1. As of the version tags on the `release0.56` branch of Numba (though this also probably impacts all Numba versions), all builds made against these tags are not compatible with NumPy 1.24 due to some changes in NumPy `ufunc` objects, see #8691 for necessary adjustments. Essentially the `resolve_dtypes` method was added which tripped a piece of guard code in Numba that returns `-1` without an `Exception` being set, this manifests as a `SystemError` on import of the C extension containing this code.
2. Tools installing Numba wheels e.g. `pip` are not constrained by the maximum NumPy version in metadata as they are in `conda` packages. The mechanism to prevent use of incompatible installations in this case is to raise an `Exception` at runtime from Numba's `__init__.py`: https://github.com/numba/numba/blob/288a38bbd5a15418a211bf067878dfdf3c139509/numba/__init__.py#L133-L155 called here: https://github.com/numba/numba/blob/288a38bbd5a15418a211bf067878dfdf3c139509/numba/__init__.py#L206

I think the issue is that the mechanism in 2. fails if a C extension with the problem in 1. is imported as part of Numba's `__init__` ahead of the call to ` _ensure_critical_deps`. i.e. the check for compatibility of dependencies is too late.



Discussion:

Resolving patch:
diff --git a/numba/__init__.py b/numba/__init__.py
--- a/numba/__init__.py
+++ b/numba/__init__.py
@@ -7,6 +7,55 @@
 import sys
 import warnings
 
+
+# ---------------------- WARNING WARNING WARNING ----------------------------
+# THIS MUST RUN FIRST, DO NOT MOVE... SEE DOCSTRING IN _ensure_critical_deps
+def _ensure_critical_deps():
+    """
+    Make sure the Python, NumPy and SciPy present are supported versions.
+    This has to be done _before_ importing anything from Numba such that
+    incompatible versions can be reported to the user. If this occurs _after_
+    importing things from Numba and there's an issue in e.g. a Numba c-ext, a
+    SystemError might have occurred which prevents reporting the likely cause of
+    the problem (incompatible versions of critical dependencies).
+    """
+    #NOTE THIS CODE SHOULD NOT IMPORT ANYTHING FROM NUMBA!
+
+    def extract_version(mod):
+        return tuple(map(int, mod.__version__.split('.')[:2]))
+
+    PYVERSION = sys.version_info[:2]
+
+    if PYVERSION < (3, 8):
+        msg = ("Numba needs Python 3.8 or greater. Got Python "
+               f"{PYVERSION[0]}.{PYVERSION[1]}.")
+        raise ImportError(msg)
+
+    import numpy as np
+    numpy_version = extract_version(np)
+
+    if numpy_version < (1, 18):
+        msg = (f"Numba needs NumPy 1.18 or greater. Got NumPy "
+               f"{numpy_version[0]}.{numpy_version[1]}.")
+        raise ImportError(msg)
+
+    try:
+        import scipy
+    except ImportError:
+        pass
+    else:
+        sp_version = extract_version(scipy)
+        if sp_version < (1, 0):
+            msg = ("Numba requires SciPy version 1.0 or greater. Got SciPy "
+                   f"{scipy.__version__}.")
+            raise ImportError(msg)
+
+
+_ensure_critical_deps()
+# END DO NOT MOVE
+# ---------------------- WARNING WARNING WARNING ----------------------------
+
+
 from ._version import get_versions
 from numba.misc.init_utils import generate_version_info
 
@@ -130,34 +179,6 @@ def _ensure_llvm():
 
     check_jit_execution()
 
-def _ensure_critical_deps():
-    """
-    Make sure Python, NumPy and SciPy have supported versions.
-    """
-    from numba.np.numpy_support import numpy_version
-    from numba.core.utils import PYVERSION
-
-    if PYVERSION < (3, 8):
-        msg = ("Numba needs Python 3.8 or greater. Got Python "
-               f"{PYVERSION[0]}.{PYVERSION[1]}.")
-        raise ImportError(msg)
-
-    if numpy_version < (1, 18):
-        msg = (f"Numba needs NumPy 1.18 or greater. Got NumPy "
-               f"{numpy_version[0]}.{numpy_version[1]}.")
-        raise ImportError(msg)
-
-    try:
-        import scipy
-    except ImportError:
-        pass
-    else:
-        sp_version = tuple(map(int, scipy.__version__.split('.')[:2]))
-        if sp_version < (1, 0):
-            msg = ("Numba requires SciPy version 1.0 or greater. Got SciPy "
-                   f"{scipy.__version__}.")
-            raise ImportError(msg)
-
 
 def _try_enable_svml():
     """
@@ -207,7 +228,6 @@ def _try_enable_svml():
     return False
 
 _ensure_llvm()
-_ensure_critical_deps()
 
 # we know llvmlite is working as the above tests passed, import it now as SVML
 # needs to mutate runtime options (sets the `-vector-library`).


Test patch: