Repository: numba/numba

Issue:
Cache does not work when passing numpy arrays to objmode inside overload
<!--

Thanks for opening an issue! To help the Numba team handle your information
efficiently, please first ensure that there is no other issue present that
already describes the issue you have
(search at https://github.com/numba/numba/issues?&q=is%3Aissue).

-->

## Reporting a bug

<!--

Before submitting a bug report please ensure that you can check off these boxes:

-->

- [x] I have tried using the latest released version of Numba (most recent is
 visible in the change log (https://github.com/numba/numba/blob/master/CHANGE_LOG).
- [x] I have included below a minimal working reproducer (if you are unsure how
 to write one see http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports).

<!--

Please include details of the bug here, including, if applicable, what you
expected to happen!

-->

Hello, I found this bug with `cache=True` when using overloads. Replicator is below:

```python
import numpy as np
import numba
from numba.extending import overload


def print_this(a, b):
    print(a)
    print(b)


def do_something(a, b):
    print(a)
    print(b)


@overload(do_something)
def overload_do_something(a, b):
    def _do_something_impl(a, b):
        with numba.objmode:
            print_this(a, b)

    return _do_something_impl


@numba.njit(cache=True)
def test_caching():
    a = np.arange(20)
    b = np.arange(50)
    #a = 1  # works if pass this instead of numpy array
    #b = 3
    do_something(a, b)


test_caching()

```

Thanks!
Cache does not work when passing numpy arrays to objmode inside overload
<!--

Thanks for opening an issue! To help the Numba team handle your information
efficiently, please first ensure that there is no other issue present that
already describes the issue you have
(search at https://github.com/numba/numba/issues?&q=is%3Aissue).

-->

## Reporting a bug

<!--

Before submitting a bug report please ensure that you can check off these boxes:

-->

- [x] I have tried using the latest released version of Numba (most recent is
 visible in the change log (https://github.com/numba/numba/blob/master/CHANGE_LOG).
- [x] I have included below a minimal working reproducer (if you are unsure how
 to write one see http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports).

<!--

Please include details of the bug here, including, if applicable, what you
expected to happen!

-->

Hello, I found this bug with `cache=True` when using overloads. Replicator is below:

```python
import numpy as np
import numba
from numba.extending import overload


def print_this(a, b):
    print(a)
    print(b)


def do_something(a, b):
    print(a)
    print(b)


@overload(do_something)
def overload_do_something(a, b):
    def _do_something_impl(a, b):
        with numba.objmode:
            print_this(a, b)

    return _do_something_impl


@numba.njit(cache=True)
def test_caching():
    a = np.arange(20)
    b = np.arange(50)
    #a = 1  # works if pass this instead of numpy array
    #b = 3
    do_something(a, b)


test_caching()

```

Thanks!


Discussion:
This is segfaulting when the cache is loaded.
I get:
```
$ NUMBA_DEBUGINFO=1 gdb python -ex 'r issue6130.py' -ex bt
<snip>

Program received signal SIGSEGV, Segmentation fault.
PyList_GetItem () at /tmp/build/80754af9/python_1585235023510/work/Objects/listobject.c:199
199     /tmp/build/80754af9/python_1585235023510/work/Objects/listobject.c: No such file or directory.
#0  PyList_GetItem () at /tmp/build/80754af9/python_1585235023510/work/Objects/listobject.c:199
#1  0x00007fffeeb91632 in __main__::overload_do_something::$3clocals$3e::_do_something_impl$245(Array<long long, 1, C, mutable, aligned>, Array<long long, 1, C, mutable, aligned>) () at issue6130.py:19
#2  0x00007fffeeb911b0 in __main__::test_caching$241 () at issue6130.py:31
#3  0x00007fffeeb9126e in cpython::__main__::test_caching$241 ()
#4  0x00007fffeeb066bc in call_cfunc ()
   from <numba path>/numba/_dispatcher.cpython-37m-x86_64-linux-gnu.so
#5  0x00007fffeeb06af9 in compile_and_invoke ()
   from <numba path>/numba/_dispatcher.cpython-37m-x86_64-linux-gnu.so
#6  0x00007fffeeb06c90 in Dispatcher_call ()
   from <numba path>/numba/_dispatcher.cpython-37m-x86_64-linux-gnu.so
<snip>
```

also, run `NUMBA_DEBUG_JIT=1 python iss6130.py` twice so the compiler dumps out each IR node before executing:

```
DEBUGJIT: $2load_global.0 = global(numba: <module 'numba' from '/Users/siu/dev/numba/numba/__init__.py'>)
DEBUGJIT: $4load_attr.1 = getattr(value=$2load_global.0, attr=objmode)
DEBUGJIT: del $2load_global.0
DEBUGJIT: $phi6.0 = $4load_attr.1
DEBUGJIT: del $phi6.0
DEBUGJIT: del $4load_attr.1
DEBUGJIT: jump 6
DEBUGJIT: $13 = const(ObjModeLiftedWith, ObjModeLiftedWith(<function overload_do_something.<locals>._do_something_impl at 0x614546c20>))
DEBUGJIT: $14 = call $13(a, b, func=$13, args=[Var(a, iss6130.py:19), Var(b, iss6130.py:19)], kws=(), vararg=None)
DEBUGJIT: # lower_call: expr = call $13(a, b, func=$13, args=[Var(a, iss6130.py:19), Var(b, iss6130.py:19)], kws=(), vararg=None)
[1]    53204 segmentation fault  NUMBA_DEBUG_JIT=1 python iss6130.py
```
that indicates it's something wrong in calling the new function for the `with objmode`
Thanks @stuartarchibald and @sklam 

I looked at `_lower_call_ObjModeDispatcher` as suggested to me by Siu. The crash seems to happen in the LLVM IR generated by this line: https://github.com/numba/numba/blob/b9efc8323c4b5915be55b7a375c4b0b8fdd84fef/numba/core/lowering.py#L795

(Reached that conclusion by setting cgutil.printfs).

Some kind of boxing issue with the numpy arrays?
What I found so far is that the call to `PyList_GetItem` that is made as part of `argobjs` is passing a null pointer for the list argument of `PyList_GetItem` when loading from cache.

This is the argobj for the first numpy array:
```
<ir.CallInstr '.151' of type 'i8*', opname 'call', operands [<ir.Function 'NRT_adapt_ndarray_to_python' of type 'i8* (i8*, i32, i32, i8*)*'>, <ir.CastInstr '.150' of type 'i8*', opname 'bitcast', operands [<ir.AllocaInstr '.145' of type '{i8*, i8*, i64, i64, i64*, [1 x i64], [1 x i64]}*', opname 'alloca', operands ()>]>, <ir.Constant type='i32' value=1>, <ir.Constant type='i32' value=1>, <ir.CallInstr '.132' of type 'i8*', opname 'call', operands [<ir.Function 'PyList_GetItem' of type 'i8* (i8*, i64)*'>, <ir.LoadInstr '.129' of type 'i8*', opname 'load', operands [<ir.GEPInstr '.128' of type 'i8**', opname 'getelementptr', operands [<ir.CastInstr '.88' of type '{i8*, i8*}*', opname 'bitcast', operands [<ir.CastInstr '.87' of type 'i8*', opname 'inttoptr', operands [<ir.Instruction '.86' of type 'i64', opname 'add', operands (<ir.CastInstr '.85' of type 'i64', opname 'ptrtoint', operands [<ir.LoadInstr '.75' of type 'i8*', opname 'load', operands [<ir.GlobalVariable '_ZN08NumbaEnv8__main__21overload_do_something12$3clocals$3e22_do_something_impl$244E5ArrayIxLi1E1C7mutable7alignedE5ArrayIxLi1E1C7mutable7alignedE' of type 'i8**'>]>]>, <ir.Constant type='i64' value=16>)>]>]>, <ir.Constant type='i32' value=0>, <ir.Constant type='i32' value=1>]>]>, <ir.Constant type='i64' value=0>]>]>
```

But right now I don't know why `PyList_GetItem` is used in this context, what that list is or why the list is null when loading from cache.
Thanks @juanjgalvez for debugging it. A null pointer in Python C-API can be caused by a previous C-API failed and raised an exception. The code may have forgotten to check for the error.
Think it comes from here:
https://github.com/numba/numba/blob/ebbae83720e6c29ef6fbb756298211d24869279f/numba/core/pythonapi.py#L137
Think further it's because the environments in question are being garbage collected, so there's no live object at the address referred to by the env ptr.
Holding a reference to `env` https://github.com/numba/numba/blob/ebbae83720e6c29ef6fbb756298211d24869279f/numba/core/environment.py#L57 stops the segfault.
I have guessed it's `env` related. Great that you find it so quickly.
Thanks @stuartarchibald and @sklam . I don't know where the reference should be kept in a proper way, but I can confirm that holding a reference stops the segfault on my end.
I have gotten to the bottom of the missing references to `env`. At rebuilding of a `CompileResult`, the `referenced_envs` in https://github.com/numba/numba/blob/b9efc8323c4b5915be55b7a375c4b0b8fdd84fef/numba/core/compiler.py#L164 needs to be stored into the `CompileResult`.
This is segfaulting when the cache is loaded.
I get:
```
$ NUMBA_DEBUGINFO=1 gdb python -ex 'r issue6130.py' -ex bt
<snip>

Program received signal SIGSEGV, Segmentation fault.
PyList_GetItem () at /tmp/build/80754af9/python_1585235023510/work/Objects/listobject.c:199
199     /tmp/build/80754af9/python_1585235023510/work/Objects/listobject.c: No such file or directory.
#0  PyList_GetItem () at /tmp/build/80754af9/python_1585235023510/work/Objects/listobject.c:199
#1  0x00007fffeeb91632 in __main__::overload_do_something::$3clocals$3e::_do_something_impl$245(Array<long long, 1, C, mutable, aligned>, Array<long long, 1, C, mutable, aligned>) () at issue6130.py:19
#2  0x00007fffeeb911b0 in __main__::test_caching$241 () at issue6130.py:31
#3  0x00007fffeeb9126e in cpython::__main__::test_caching$241 ()
#4  0x00007fffeeb066bc in call_cfunc ()
   from <numba path>/numba/_dispatcher.cpython-37m-x86_64-linux-gnu.so
#5  0x00007fffeeb06af9 in compile_and_invoke ()
   from <numba path>/numba/_dispatcher.cpython-37m-x86_64-linux-gnu.so
#6  0x00007fffeeb06c90 in Dispatcher_call ()
   from <numba path>/numba/_dispatcher.cpython-37m-x86_64-linux-gnu.so
<snip>
```

also, run `NUMBA_DEBUG_JIT=1 python iss6130.py` twice so the compiler dumps out each IR node before executing:

```
DEBUGJIT: $2load_global.0 = global(numba: <module 'numba' from '/Users/siu/dev/numba/numba/__init__.py'>)
DEBUGJIT: $4load_attr.1 = getattr(value=$2load_global.0, attr=objmode)
DEBUGJIT: del $2load_global.0
DEBUGJIT: $phi6.0 = $4load_attr.1
DEBUGJIT: del $phi6.0
DEBUGJIT: del $4load_attr.1
DEBUGJIT: jump 6
DEBUGJIT: $13 = const(ObjModeLiftedWith, ObjModeLiftedWith(<function overload_do_something.<locals>._do_something_impl at 0x614546c20>))
DEBUGJIT: $14 = call $13(a, b, func=$13, args=[Var(a, iss6130.py:19), Var(b, iss6130.py:19)], kws=(), vararg=None)
DEBUGJIT: # lower_call: expr = call $13(a, b, func=$13, args=[Var(a, iss6130.py:19), Var(b, iss6130.py:19)], kws=(), vararg=None)
[1]    53204 segmentation fault  NUMBA_DEBUG_JIT=1 python iss6130.py
```
that indicates it's something wrong in calling the new function for the `with objmode`
Thanks @stuartarchibald and @sklam 

I looked at `_lower_call_ObjModeDispatcher` as suggested to me by Siu. The crash seems to happen in the LLVM IR generated by this line: https://github.com/numba/numba/blob/b9efc8323c4b5915be55b7a375c4b0b8fdd84fef/numba/core/lowering.py#L795

(Reached that conclusion by setting cgutil.printfs).

Some kind of boxing issue with the numpy arrays?
What I found so far is that the call to `PyList_GetItem` that is made as part of `argobjs` is passing a null pointer for the list argument of `PyList_GetItem` when loading from cache.

This is the argobj for the first numpy array:
```
<ir.CallInstr '.151' of type 'i8*', opname 'call', operands [<ir.Function 'NRT_adapt_ndarray_to_python' of type 'i8* (i8*, i32, i32, i8*)*'>, <ir.CastInstr '.150' of type 'i8*', opname 'bitcast', operands [<ir.AllocaInstr '.145' of type '{i8*, i8*, i64, i64, i64*, [1 x i64], [1 x i64]}*', opname 'alloca', operands ()>]>, <ir.Constant type='i32' value=1>, <ir.Constant type='i32' value=1>, <ir.CallInstr '.132' of type 'i8*', opname 'call', operands [<ir.Function 'PyList_GetItem' of type 'i8* (i8*, i64)*'>, <ir.LoadInstr '.129' of type 'i8*', opname 'load', operands [<ir.GEPInstr '.128' of type 'i8**', opname 'getelementptr', operands [<ir.CastInstr '.88' of type '{i8*, i8*}*', opname 'bitcast', operands [<ir.CastInstr '.87' of type 'i8*', opname 'inttoptr', operands [<ir.Instruction '.86' of type 'i64', opname 'add', operands (<ir.CastInstr '.85' of type 'i64', opname 'ptrtoint', operands [<ir.LoadInstr '.75' of type 'i8*', opname 'load', operands [<ir.GlobalVariable '_ZN08NumbaEnv8__main__21overload_do_something12$3clocals$3e22_do_something_impl$244E5ArrayIxLi1E1C7mutable7alignedE5ArrayIxLi1E1C7mutable7alignedE' of type 'i8**'>]>]>, <ir.Constant type='i64' value=16>)>]>]>, <ir.Constant type='i32' value=0>, <ir.Constant type='i32' value=1>]>]>, <ir.Constant type='i64' value=0>]>]>
```

But right now I don't know why `PyList_GetItem` is used in this context, what that list is or why the list is null when loading from cache.
Thanks @juanjgalvez for debugging it. A null pointer in Python C-API can be caused by a previous C-API failed and raised an exception. The code may have forgotten to check for the error.
Think it comes from here:
https://github.com/numba/numba/blob/ebbae83720e6c29ef6fbb756298211d24869279f/numba/core/pythonapi.py#L137
Think further it's because the environments in question are being garbage collected, so there's no live object at the address referred to by the env ptr.
Holding a reference to `env` https://github.com/numba/numba/blob/ebbae83720e6c29ef6fbb756298211d24869279f/numba/core/environment.py#L57 stops the segfault.
I have guessed it's `env` related. Great that you find it so quickly.
Thanks @stuartarchibald and @sklam . I don't know where the reference should be kept in a proper way, but I can confirm that holding a reference stops the segfault on my end.
I have gotten to the bottom of the missing references to `env`. At rebuilding of a `CompileResult`, the `referenced_envs` in https://github.com/numba/numba/blob/b9efc8323c4b5915be55b7a375c4b0b8fdd84fef/numba/core/compiler.py#L164 needs to be stored into the `CompileResult`.

Resolving patch:
diff --git a/numba/core/compiler.py b/numba/core/compiler.py
--- a/numba/core/compiler.py
+++ b/numba/core/compiler.py
@@ -91,6 +91,7 @@ class Flags(utils.ConfigOptions):
              # List of functions to call to initialize on unserialization
              # (i.e cache load).
              "reload_init",
+             "referenced_envs",
              ]
 
 
@@ -158,6 +159,7 @@ def _rebuild(cls, target_context, libdata, fndesc, env,
                  call_helper=None,
                  metadata=None,  # Do not store, arbitrary & potentially large!
                  reload_init=reload_init,
+                 referenced_envs=referenced_envs,
                  )
 
         # Load Environments
diff --git a/numba/core/pythonapi.py b/numba/core/pythonapi.py
--- a/numba/core/pythonapi.py
+++ b/numba/core/pythonapi.py
@@ -131,10 +131,28 @@ def read_const(self, index):
         """
         Look up constant number *index* inside the environment body.
         A borrowed reference is returned.
+
+        The returned LLVM value may have NULL value at runtime which indicates
+        an error at runtime.
         """
         assert index < len(self.env.consts)
 
-        return self.pyapi.list_getitem(self.env_body.consts, index)
+        builder = self.pyapi.builder
+        consts = self.env_body.consts
+        ret = cgutils.alloca_once(builder, self.pyapi.pyobj, zfill=True)
+        with builder.if_else(cgutils.is_not_null(builder, consts)) as \
+                (br_not_null, br_null):
+            with br_not_null:
+                getitem = self.pyapi.list_getitem(consts, index)
+                builder.store(getitem, ret)
+            with br_null:
+                # This can happen when the Environment is accidentally released
+                # and has subsequently been garbage collected.
+                self.pyapi.err_set_string(
+                    "PyExc_RuntimeError",
+                    "`env.consts` is NULL in `read_const`",
+                )
+        return builder.load(ret)
 
 
 _IteratorLoop = namedtuple('_IteratorLoop', ('value', 'do_break'))


Test patch:
diff --git a/numba/tests/support.py b/numba/tests/support.py
--- a/numba/tests/support.py
+++ b/numba/tests/support.py
@@ -19,6 +19,7 @@
 import ctypes
 import multiprocessing as mp
 import warnings
+import traceback
 from contextlib import contextmanager
 
 import numpy as np
@@ -779,6 +780,26 @@ def run_in_new_process_caching(func, cache_dir_prefix=__name__, verbose=True):
     The childprocess's stdout and stderr will be captured and redirected to
     the current process's stdout and stderr.
 
+    Returns
+    -------
+    ret : dict
+        exitcode: 0 for success. 1 for exception-raised.
+        stdout: str
+        stderr: str
+    """
+    cache_dir = temp_directory(cache_dir_prefix)
+    return run_in_new_process_in_cache_dir(func, cache_dir, verbose=verbose)
+
+
+def run_in_new_process_in_cache_dir(func, cache_dir, verbose=True):
+    """Spawn a new process to run `func` with a temporary cache directory.
+
+    The childprocess's stdout and stderr will be captured and redirected to
+    the current process's stdout and stderr.
+
+    Similar to ``run_in_new_process_caching()`` but the ``cache_dir`` is a
+    directory path instead of a name prefix for the directory path.
+
     Returns
     -------
     ret : dict
@@ -788,7 +809,6 @@ def run_in_new_process_caching(func, cache_dir_prefix=__name__, verbose=True):
     """
     ctx = mp.get_context('spawn')
     qout = ctx.Queue()
-    cache_dir = temp_directory(cache_dir_prefix)
     with override_env_config('NUMBA_CACHE_DIR', cache_dir):
         proc = ctx.Process(target=_remote_runner, args=[func, qout])
         proc.start()
diff --git a/numba/tests/test_extending.py b/numba/tests/test_extending.py
--- a/numba/tests/test_extending.py
+++ b/numba/tests/test_extending.py
@@ -20,6 +20,7 @@
     captured_stdout,
     temp_directory,
     override_config,
+    run_in_new_process_in_cache_dir,
 )
 from numba.core.errors import LoweringError
 import unittest
@@ -1748,12 +1749,56 @@ def testcase(x):
             got = testcase_cached(123)
             self.assertEqual(got, expect)
 
-            self.assertEqual(
-                testcase_cached._cache_hits[testcase.signatures[0]], 1,
-            )
-            self.assertEqual(
-                testcase_cached._cache_misses[testcase.signatures[0]], 0,
+    @classmethod
+    def check_objmode_cache_ndarray(cls):
+        def do_this(a, b):
+            return np.sum(a + b)
+
+        def do_something(a, b):
+            return np.sum(a + b)
+
+        @overload(do_something)
+        def overload_do_something(a, b):
+            def _do_something_impl(a, b):
+                with objmode(y='float64'):
+                    y = do_this(a, b)
+                return y
+            return _do_something_impl
+
+        @njit(cache=True)
+        def test_caching():
+            a = np.arange(20)
+            b = np.arange(20)
+            return do_something(a, b)
+
+        got = test_caching()
+        expect = test_caching.py_func()
+
+        # Check result
+        if got != expect:
+            raise AssertionError("incorrect result")
+        return test_caching
+
+    @classmethod
+    def check_objmode_cache_ndarray_check_cache(cls):
+        disp = cls.check_objmode_cache_ndarray()
+        if len(disp.stats.cache_misses) != 0:
+            raise AssertionError('unexpected cache miss')
+        if len(disp.stats.cache_hits) <= 0:
+            raise AssertionError("unexpected missing cache hit")
+
+    def test_check_objmode_cache_ndarray(self):
+        # See issue #6130.
+        # Env is missing after cache load.
+        cache_dir = temp_directory(self.__class__.__name__)
+        with override_config("CACHE_DIR", cache_dir):
+            # Test in local process to populate the cache.
+            self.check_objmode_cache_ndarray()
+            # Run in new process to use the cache in a fresh process.
+            res = run_in_new_process_in_cache_dir(
+                self.check_objmode_cache_ndarray_check_cache, cache_dir
             )
+        self.assertEqual(res['exitcode'], 0)
 
 
 class TestMisc(TestCase):