Repository: numba/numba

Issue:
User defined exceptions in numba 0.41
I apologize that I cannot give a simple reproducible version of the issue.  It happens in a complex code base that I can't expect you to install. I have not seen the error in other contexts, for example when I just run snippet below by itself, outside of the application.  This code works in 0.40, and this code has in fact been running in production for quite some time.  I've tested using python 3.6 and 3.7

The overall context is a run of scipy `leastsq`.  An exception is raised by numba, and this occurs in one of the function evaluations.  It does not occur the first time this function is called, it occurs after a few hundred or thousand calls.  The input to the function is the same type every time and there is nothing special that I can see about the situation when the exception is raised.

The function takes a numpy array with fields as an argument and writes some values into it.

When the exception is raised occurs, it is giving this error
```
Untyped global name 'gauss2d_set_norm': cannot determine Numba type of <class 'function'>

File "../../../../../../../gpfs01/astro/workarea/esheldon/anaconda-dirs/miniconda3/envs/cosmos/lib/python3.7/site-packages/ngmix/gmix_nb.py", line 183:
def gmix_set_norms(gmix):
    <source elided>
    for gauss in gmix:
        gauss2d_set_norm(gauss)
```
It seems that perhaps numba has forgotten that it has already compiled this function, and is trying to determine its type again.

Here is the function. 
```python
@njit
def gauss2d_set_norm(gauss):
    """
    set the normalization, and nromalized variances

    A GMixRangeError is raised if the determinant is too small

    parameters
    ----------
    gauss: a 2-d gaussian structure
        See gmix.py
    """

    if gauss['det'] < GMIX_LOW_DETVAL:
        raise GMixRangeError("det too low")

    T=gauss['irr']+gauss['icc']
    if T <= GMIX_LOW_DETVAL:
        raise GMixRangeError("T too low")

    idet=1.0/gauss['det']
    gauss['drr'] = gauss['irr']*idet
    gauss['drc'] = gauss['irc']*idet
    gauss['dcc'] = gauss['icc']*idet
    gauss['norm'] = 1./(2*numpy.pi*numpy.sqrt(gauss['det']))

    gauss['pnorm'] = gauss['p']*gauss['norm']

    gauss['norm_set']=1
```
 The input is an element of an array, e.g. I create an array and am sending it an element, such as follows:

```python
gmix=np.zeros(n, dtype=dtype)
# do some initialization of the data

# call the function on each element
for gauss in gmix:
    gauss2d_set_norm(gauss)
```



Discussion:
> Untyped global name 'gauss2d_set_norm': cannot determine Numba type of <class 'function'>

That is a compile error at `gmix_set_norms`.  It is as if `gauss2d_set_norm` is not decorated with `@njit`.        You may want to go into the debugger (i.e. pdb) when that error occurs and poke around the global dictionary to see what `gauss2d_set_norm` is at that point.
I hacked on the code to try to get a more clear error message indicating that the problem is associated with numba, I might have done too much hacking.

If I run my code without changes I see this error.
```
TypeError: __init__() missing 1 required positional argument: 'value'
```
However, this error message makes no sense when examined at face value.  It  occurs calling a numba defined function (there are no constructors called in the function where the error occurs).  In fact the error is coming from numba,  deeper in the code when it tries to call the function I mentioned above, but somehow the error is not propagated as expected.

@esheldon Any chance you could please try and get some back traces from the locations the errors are coming from to help diagnose what's going on? Many thanks.
Here is the full Traceback back. It does not show that the error is bubbling up from numba
```
Traceback (most recent call last):
  File "/home/esheldon/anaconda3/envs/numbabug/bin/fitcosmos", line 30, in <module>
    main()
  File "/home/esheldon/anaconda3/envs/numbabug/bin/fitcosmos", line 27, in main
    processor.go()
  File "/home/esheldon/anaconda3/envs/numbabug/lib/python3.7/site-packages/fitcosmos/process.py", line 51, in go
    output, epochs_data = self._process_fof(fofid)
  File "/home/esheldon/anaconda3/envs/numbabug/lib/python3.7/site-packages/fitcosmos/process.py", line 104, in _process_fof
    output, epochs_data = self.fitter.go(mbobs_list)
  File "/home/esheldon/anaconda3/envs/numbabug/lib/python3.7/site-packages/fitcosmos/fitting.py", line 174, in go
    _fit_all_psfs(mbobs_list, self['mof']['psf'])
  File "/home/esheldon/anaconda3/envs/numbabug/lib/python3.7/site-packages/fitcosmos/fitting.py", line 392, in _fit_all_psfs
    fitter.go()
  File "/home/esheldon/anaconda3/envs/numbabug/lib/python3.7/site-packages/fitcosmos/fitting.py", line 412, in go
    _fit_one_psf(psf_obs, self.psf_conf)
  File "/home/esheldon/anaconda3/envs/numbabug/lib/python3.7/site-packages/fitcosmos/fitting.py", line 435, in _fit_one_psf
    runner.go(ntry=pconf['ntry'])
  File "/home/esheldon/anaconda3/envs/numbabug/lib/python3.7/site-packages/ngmix/bootstrap.py", line 2282, in go
    fitter.go(guess)
  File "/home/esheldon/anaconda3/envs/numbabug/lib/python3.7/site-packages/ngmix/fitting.py", line 1681, in go
    **self.lm_pars
  File "/home/esheldon/anaconda3/envs/numbabug/lib/python3.7/site-packages/ngmix/fitting.py", line 1989, in run_leastsq
    lm_tup = leastsq(func, guess, full_output=1, **keys)
  File "/home/esheldon/anaconda3/envs/numbabug/lib/python3.7/site-packages/scipy/optimize/minpack.py", line 394, in leastsq
    gtol, maxfev, epsfcn, factor, diag)
  File "/home/esheldon/anaconda3/envs/numbabug/lib/python3.7/site-packages/ngmix/fitting.py", line 1805, in _calc_fdiff
    start,
TypeError: __init__() missing 1 required positional argument: 'value'
```
That error at line 1805 is a call to a numba compiled function.  Again, the function gets called hundreds or thousands of times before the exception is raised.  The types entered to the functions are the same every time.
OK, I've traced it to raising an exception in the numba compiled function.  If I comment out the `raise GMixRangeError` from below the error does not occur.

Note the error above is saying `TypeError: __init__() missing 1 required positional argument: 'value'`. Is it possible the raise of that exception is being rewritten by numba?

I also tried raising `ValueError` instead of my user defined exception and that propagates up properly.

```python
@njit
def gauss2d_set_norm(gauss):
    """
    set the normalization, and nromalized variances

    A GMixRangeError is raised if the determinant is too small

    parameters
    ----------
    gauss: a 2-d gaussian structure
        See gmix.py
    """

    if gauss['det'] < GMIX_LOW_DETVAL:
        raise GMixRangeError("det too low")

    T=gauss['irr']+gauss['icc']
    if T <= GMIX_LOW_DETVAL:
        raise GMixRangeError("T too low")

    idet=1.0/gauss['det']
    gauss['drr'] = gauss['irr']*idet
    gauss['drc'] = gauss['irc']*idet
    gauss['dcc'] = gauss['icc']*idet
    gauss['norm'] = 1./(2*numpy.pi*numpy.sqrt(gauss['det']))

    gauss['pnorm'] = gauss['p']*gauss['norm']

    gauss['norm_set']=1
```
Definition of my  exception class
```python
class GMixRangeError(Exception):
    """
    Some number was out of range
    """
    def __init__(self, value):
        super(GMixRangeError, self).__init__(value)
        self.value = value

    def __str__(self):
        return repr(self.value)
```
@esheldon thanks for doing that. The Numba exception code was rewritten in 0.41 to include synthetic stack frame injection, expect it's related to that. Will take a look.
ok, that's the problem. Reproducer:
```python
from numba import njit

class UDE(Exception):
    def __init__(self, value):
        super(UDE, self).__init__(value)
        self.value = value

@njit
def foo(x):
    if x < 0:
        raise UDE("error message")


foo(10)
foo(-1)
```
Numba 0.38 (because I have that to hand) works ok, but it's broken on 0.41.
Thanks. Sorry for my initial specification of this issue, which was I think misleading.
No problem, thanks for reporting, it wasn't clear at the outset what the cause was. The problem is this line:
https://github.com/numba/numba/blob/6c890ead1b9dc4501dbbb057fff46748115a2b0d/numba/_helperlib.c#L922
at this point in the code we've unpacked the exception info tuple, figured out that it's a UDE class that needs to be instantiated, and then it ends up being called without the unpacked arguments, hence error.


Resolving patch:
diff --git a/numba/consts.py b/numba/consts.py
--- a/numba/consts.py
+++ b/numba/consts.py
@@ -82,10 +82,21 @@ def _infer_call(self, func, expr):
         if expr.kws or expr.vararg:
             self._fail(expr)
         # Check supported callables
-        if (func in (slice,) or
-            (isinstance(func, type) and issubclass(func, BaseException))):
+        _slice = func in (slice,)
+        _exc = isinstance(func, type) and issubclass(func, BaseException)
+        if _slice or _exc:
             args = [self.infer_constant(a.name) for a in expr.args]
-            return func(*args)
+            if _slice:
+                return func(*args)
+            elif _exc:
+                # If the exception class is user defined it may implement a ctor
+                # that does not pass the args to the super. Therefore return the
+                # raw class and the args so this can be instantiated at the call
+                # site in the way the user source expects it to be.
+                return func, args
+            else:
+                assert 0, 'Unreachable'
+
         self._fail(expr)
 
     def _infer_getattr(self, value, expr):
diff --git a/numba/rewrites/static_raise.py b/numba/rewrites/static_raise.py
--- a/numba/rewrites/static_raise.py
+++ b/numba/rewrites/static_raise.py
@@ -21,8 +21,11 @@ def _break_constant(self, const):
         """
         Break down constant exception.
         """
-        if isinstance(const, BaseException):
-            return const.__class__, const.args
+        if isinstance(const, tuple): # it's a tuple(exception class, args)
+            if not self._is_exception_type(const[0]):
+                raise NotImplementedError("unsupported exception constant %r"
+                                          % (const[0],))
+            return const[0], tuple(const[1])
         elif self._is_exception_type(const):
             return const, None
         else:
diff --git a/numba/targets/callconv.py b/numba/targets/callconv.py
--- a/numba/targets/callconv.py
+++ b/numba/targets/callconv.py
@@ -360,6 +360,11 @@ def return_user_exc(self, builder, exc, exc_args=None, loc=None,
         if exc_args is not None and not isinstance(exc_args, tuple):
             raise TypeError("exc_args should be None or tuple, got %r"
                             % (exc_args,))
+        # None is indicative of no args, set the exc_args to an empty tuple
+        # as PyObject_CallObject(exc, exc_args) requires the second argument to
+        # be a tuple (or nullptr, but doing this makes it consistent)
+        if exc_args is None:
+            exc_args = tuple()
 
         pyapi = self.context.get_python_api(builder)
         # Build excinfo struct


Test patch:
diff --git a/numba/tests/test_exceptions.py b/numba/tests/test_exceptions.py
--- a/numba/tests/test_exceptions.py
+++ b/numba/tests/test_exceptions.py
@@ -19,6 +19,43 @@ class MyError(Exception):
 class OtherError(Exception):
     pass
 
+class UDEArgsToSuper(Exception):
+    def __init__(self, arg, value0):
+        super(UDEArgsToSuper, self).__init__(arg)
+        self.value0 = value0
+
+    def __eq__(self, other):
+        if not isinstance(other, self.__class__):
+            return False
+        same = True
+        same |= self.args == other.args
+        same |= self.value0 == other.value0
+        return same
+
+    def __hash__(self):
+        return hash((super(UDEArgsToSuper).__hash__(), self.value0))
+
+
+class UDENoArgSuper(Exception):
+    def __init__(self, arg, value0):
+        super(UDENoArgSuper, self).__init__()
+        self.deferarg = arg
+        self.value0 = value0
+
+    def __eq__(self, other):
+        if not isinstance(other, self.__class__):
+            return False
+        same = True
+        same |= self.args == other.args
+        same |= self.deferarg == other.deferarg
+        same |= self.value0 == other.value0
+        return same
+
+    def __hash__(self):
+        return hash((super(UDENoArgSuper).__hash__(), self.deferarg,
+                    self.value0))
+
+
 def raise_class(exc):
     def raiser(i):
         if i == 1:
@@ -55,6 +92,9 @@ def outer(i):
 def assert_usecase(i):
     assert i == 1, "bar"
 
+def ude_bug_usecase():
+    raise UDEArgsToSuper() # oops user forgot args to exception ctor
+
 
 class TestRaising(TestCase):
 
@@ -86,6 +126,10 @@ def check_against_python(self, exec_mode, pyfunc, cfunc,
             cfunc(*args)
         self.assertEqual(pyerr.exception.args, jiterr.exception.args)
 
+        # special equality check for UDEs
+        if isinstance(pyerr.exception, (UDEArgsToSuper, UDENoArgSuper)):
+            self.assertTrue(pyerr.exception == jiterr.exception)
+
         # in npm check bottom of traceback matches as frame injection with
         # location info should ensure this
         if exec_mode is no_pyobj_flags:
@@ -127,15 +171,17 @@ def test_raise_class_objmode(self):
         self.check_raise_class(flags=force_pyobj_flags)
 
     def check_raise_instance(self, flags):
-        pyfunc = raise_instance(MyError, "some message")
-        cres = compile_isolated(pyfunc, (types.int32,), flags=flags)
-        cfunc = cres.entry_point
-
-        self.assertEqual(cfunc(0), 0)
-        self.check_against_python(flags, pyfunc, cfunc, MyError, 1)
-        self.check_against_python(flags, pyfunc, cfunc, ValueError, 2)
-        self.check_against_python(flags, pyfunc, cfunc,
-                                  np.linalg.linalg.LinAlgError, 3)
+        for clazz in [MyError, UDEArgsToSuper,
+                      UDENoArgSuper]:
+            pyfunc = raise_instance(clazz, "some message")
+            cres = compile_isolated(pyfunc, (types.int32,), flags=flags)
+            cfunc = cres.entry_point
+
+            self.assertEqual(cfunc(0), 0)
+            self.check_against_python(flags, pyfunc, cfunc, clazz, 1)
+            self.check_against_python(flags, pyfunc, cfunc, ValueError, 2)
+            self.check_against_python(flags, pyfunc, cfunc,
+                                      np.linalg.linalg.LinAlgError, 3)
 
     def test_raise_instance_objmode(self):
         self.check_raise_instance(flags=force_pyobj_flags)
@@ -148,14 +194,16 @@ def check_raise_nested(self, flags, **jit_args):
         """
         Check exception propagation from nested functions.
         """
-        inner_pyfunc = raise_instance(MyError, "some message")
-        pyfunc = outer_function(inner_pyfunc)
-        inner_cfunc = jit(**jit_args)(inner_pyfunc)
-        cfunc = jit(**jit_args)(outer_function(inner_cfunc))
+        for clazz in [MyError, UDEArgsToSuper,
+                      UDENoArgSuper]:
+            inner_pyfunc = raise_instance(clazz, "some message")
+            pyfunc = outer_function(inner_pyfunc)
+            inner_cfunc = jit(**jit_args)(inner_pyfunc)
+            cfunc = jit(**jit_args)(outer_function(inner_cfunc))
 
-        self.check_against_python(flags, pyfunc, cfunc, MyError, 1)
-        self.check_against_python(flags, pyfunc, cfunc, ValueError, 2)
-        self.check_against_python(flags, pyfunc, cfunc, OtherError, 3)
+            self.check_against_python(flags, pyfunc, cfunc, clazz, 1)
+            self.check_against_python(flags, pyfunc, cfunc, ValueError, 2)
+            self.check_against_python(flags, pyfunc, cfunc, OtherError, 3)
 
     def test_raise_nested_objmode(self):
         self.check_raise_nested(force_pyobj_flags, forceobj=True)
@@ -165,19 +213,27 @@ def test_raise_nested_nopython(self):
         self.check_raise_nested(no_pyobj_flags, nopython=True)
 
     def check_reraise(self, flags):
+        def raise_exc(exc):
+            raise exc
         pyfunc = reraise
         cres = compile_isolated(pyfunc, (), flags=flags)
         cfunc = cres.entry_point
-        def gen_impl(fn):
-            def impl():
-                try:
-                    1/0
-                except ZeroDivisionError as e:
-                    fn()
-            return impl
-        pybased = gen_impl(pyfunc)
-        cbased = gen_impl(cfunc)
-        self.check_against_python(flags, pybased, cbased, ZeroDivisionError,)
+        for op, err in [(lambda : raise_exc(ZeroDivisionError),
+                         ZeroDivisionError),
+                        (lambda : raise_exc(UDEArgsToSuper("msg", 1)),
+                         UDEArgsToSuper),
+                        (lambda : raise_exc(UDENoArgSuper("msg", 1)),
+                         UDENoArgSuper)]:
+            def gen_impl(fn):
+                def impl():
+                    try:
+                        op()
+                    except err as e:
+                        fn()
+                return impl
+            pybased = gen_impl(pyfunc)
+            cbased = gen_impl(cfunc)
+            self.check_against_python(flags, pybased, cbased, err,)
 
     def test_reraise_objmode(self):
         self.check_reraise(flags=force_pyobj_flags)
@@ -220,14 +276,17 @@ def test_assert_statement_nopython(self):
 
     def check_raise_from_exec_string(self, flags):
         # issue #3428
-        f_text = "def f(a):\n  assert a != 1"
-        loc = {}
-        exec(f_text, {}, loc)
-        pyfunc = loc['f']
-        cres = compile_isolated(pyfunc, (types.int32,), flags=flags)
-        cfunc = cres.entry_point
-        cfunc(2)
-        self.check_against_python(flags, pyfunc, cfunc, AssertionError, 1)
+        simple_raise = "def f(a):\n  raise exc('msg', 10)"
+        assert_raise = "def f(a):\n  assert a != 1"
+        for f_text, exc in [(assert_raise, AssertionError),
+                            (simple_raise, UDEArgsToSuper),
+                            (simple_raise,UDENoArgSuper)]:
+            loc = {}
+            exec(f_text, {'exc': exc}, loc)
+            pyfunc = loc['f']
+            cres = compile_isolated(pyfunc, (types.int32,), flags=flags)
+            cfunc = cres.entry_point
+            self.check_against_python(flags, pyfunc, cfunc, exc, 1)
 
     def test_assert_from_exec_string_objmode(self):
         self.check_raise_from_exec_string(flags=force_pyobj_flags)
@@ -235,7 +294,20 @@ def test_assert_from_exec_string_objmode(self):
     def test_assert_from_exec_string_nopython(self):
         self.check_raise_from_exec_string(flags=no_pyobj_flags)
 
+    def check_user_code_error_traceback(self, flags):
+        # this test checks that if a user tries to compile code that contains
+        # a bug in exception initialisation (e.g. missing arg) then this also
+        # has a frame injected with the location information.
+        pyfunc = ude_bug_usecase
+        cres = compile_isolated(pyfunc, (), flags=flags)
+        cfunc = cres.entry_point
+        self.check_against_python(flags, pyfunc, cfunc, TypeError)
+
+    def test_user_code_error_traceback(self):
+        self.check_user_code_error_traceback(flags=force_pyobj_flags)
 
+    def test_user_code_error_traceback(self):
+        self.check_user_code_error_traceback(flags=no_pyobj_flags)
 
 if __name__ == '__main__':
     unittest.main()