Repository: numba/numba

Issue:
Memory System left uninitialized in several tests
While investigating segfaults in #2451 on PPC64LE, I noticed that several tests, among them

    numba.tests.test_nrt.TestNrtMemInfo.test_buffer
    numba.tests.test_nrt.TestNrtMemInfo.test_meminfo_refct_1
    numba.tests.test_nrt.TestNrtMemInfo.test_meminfo_refct_2

do not manage to initialize the C global structure `TheMSys` before attempting to use `rtsys.meminfo_new(addr, d)`. Since that method eventually leads to an invocation of `TheMSys.atomic_inc(&TheMSys.stats_mi_alloc);` and `TheMSys.atomic_inc` remains a `NULL` function pointer until initialized, this causes a segfault.

This problem is highly likely to also occur on x86-64 or any other platform, and can be reproduced, for instance, with

```shell
./runtests.py numba.tests.test_nrt.TestNrtMemInfo.test_meminfo_refct_1
```

The problem disappears if the ugly hack

```python
import numba; numba.targets.cpu.CPUContext(numba.typing.Context())
```

is added before the beginning of the afore-mentioned tests, thus triggering the initialization of `TheMSys`. This solves the segfaults for all three tests and allows them to pass. It may well be the case that similar initialization problems also exists in other (sub)tests as well.

*A probable reason for this not having been noticed earlier is that tests are not run in isolation, but instead sequentially within the same process. Since previous tests will have triggered initialization of* `TheMSys`, *the failure to do the same here escaped notice.*



Discussion:

Resolving patch:
diff --git a/numba/runtime/nrt.py b/numba/runtime/nrt.py
--- a/numba/runtime/nrt.py
+++ b/numba/runtime/nrt.py
@@ -47,6 +47,11 @@ def initialize(self, ctx):
 
             self._init = True
 
+    def _init_guard(self):
+        if not self._init:
+            msg = "Runtime must be initialized before use."
+            raise RuntimeError(msg)
+
     @staticmethod
     def shutdown():
         """
@@ -60,6 +65,7 @@ def library(self):
         """
         Return the Library object containing the various NRT functions.
         """
+        self._init_guard()
         return self._library
 
     def meminfo_new(self, data, pyobj):
@@ -68,6 +74,7 @@ def meminfo_new(self, data, pyobj):
         MemInfo will acquire a reference on `pyobj`.
         The release of MemInfo will release a reference on `pyobj`.
         """
+        self._init_guard()
         mi = _nrt.meminfo_new(data, pyobj)
         return MemInfo(mi)
 
@@ -81,6 +88,7 @@ def meminfo_alloc(self, size, safe=False):
         This is used for debugging and testing purposes.
         See `NRT_MemInfo_alloc_safe()` in "nrt.h" for details.
         """
+        self._init_guard()
         if safe:
             mi = _nrt.meminfo_alloc_safe(size)
         else:
@@ -92,6 +100,7 @@ def get_allocation_stats(self):
         Returns a namedtuple of (alloc, free, mi_alloc, mi_free) for count of
         each memory operations.
         """
+        # No init guard needed to access stats members
         return _nrt_mstats(alloc=_nrt.memsys_get_stats_alloc(),
                            free=_nrt.memsys_get_stats_free(),
                            mi_alloc=_nrt.memsys_get_stats_mi_alloc(),


Test patch:
diff --git a/numba/tests/test_nrt.py b/numba/tests/test_nrt.py
--- a/numba/tests/test_nrt.py
+++ b/numba/tests/test_nrt.py
@@ -8,7 +8,7 @@
 import numpy as np
 
 from numba import unittest_support as unittest
-from numba import njit
+from numba import njit, targets, typing
 from numba.compiler import compile_isolated, Flags, types
 from numba.runtime import rtsys
 from numba.runtime import nrtopt
@@ -28,14 +28,42 @@ def __del__(self):
         type(self).alive -= 1
 
 
+class TestNrtMemInfoNotInitialized(unittest.TestCase):
+    """
+    Unit test for checking the use of the NRT fails if the
+    initialization sequence has not been run.
+    """
+    _numba_parallel_test_ = False
+
+    def test_init_fail(self):
+        methods = {'library': (),
+                   'meminfo_new': ((), ()),
+                   'meminfo_alloc': ((),),
+                   }
+
+        for meth, args in methods.items():
+            try:
+                with self.assertRaises(RuntimeError) as raises:
+                    rtsys._init = False
+                    fn = getattr(rtsys, meth)
+                    fn(*args)
+
+                msg = "Runtime must be initialized before use."
+                self.assertIn(msg, str(raises.exception))
+            finally:
+                rtsys._init = True
+
+
 class TestNrtMemInfo(unittest.TestCase):
     """
-    Unitest for core MemInfo functionality
+    Unit test for core MemInfo functionality
     """
 
     def setUp(self):
         # Reset the Dummy class
         Dummy.alive = 0
+        # initialize the NRT (in case the tests are run in isolation)
+        targets.cpu.CPUContext(typing.Context())
 
     def test_meminfo_refct_1(self):
         d = Dummy()