Repository: numba/numba

Issue:
SegFault using threads, nogil=True and many (re-) allocations
System: x86_64 GNU/Linux, LLVM 3.6.1, Python 3.4.3, llvmlite latest git, Numba latest git
(also reproducible with llvmlite 0.6.0, Numba 0.20.0)

Given the script:

``` python
from sys import argv
from numpy import zeros
from numba import jit
from concurrent.futures import ThreadPoolExecutor
from multiprocessing import cpu_count


@jit(nopython=True, nogil=True)
def loop(loops, size):
    for _ in range(loops):
        zeros(size).sum()


if __name__ == "__main__":
    size = 1000 if len(argv) <= 1 else int(argv[1])
    loops = 1000 if len(argv) <= 2 else int(argv[2])
    jobs = 1000 if len(argv) <= 3 else int(argv[3])
    with ThreadPoolExecutor(cpu_count()) as executor:
        for _ in range(jobs):
            executor.submit(loop, loops, size)
```

I receive a SegFault with the following `gdb` output (on a machine with 24 HT cores) :

```
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffed5c7700 (LWP 27511)]
[New Thread 0x7fffecdc6700 (LWP 27512)]
[New Thread 0x7fffe7fff700 (LWP 27513)]
[New Thread 0x7fffe77fe700 (LWP 27514)]
[New Thread 0x7fffe6ffd700 (LWP 27515)]
[New Thread 0x7fffe67fc700 (LWP 27516)]
[New Thread 0x7fffe5ffb700 (LWP 27517)]
[New Thread 0x7fffe57fa700 (LWP 27518)]
[New Thread 0x7fffe4ff9700 (LWP 27519)]
[New Thread 0x7fffc3fff700 (LWP 27520)]
[New Thread 0x7fffc37fe700 (LWP 27521)]
[New Thread 0x7fffc2ffd700 (LWP 27522)]
[New Thread 0x7fffc27fc700 (LWP 27523)]
[New Thread 0x7fffc1ffb700 (LWP 27524)]
[New Thread 0x7fffc17fa700 (LWP 27525)]
[New Thread 0x7fffc0ff9700 (LWP 27526)]
[New Thread 0x7fffa3fff700 (LWP 27527)]
[New Thread 0x7fffa37fe700 (LWP 27528)]
[New Thread 0x7fffa2ffd700 (LWP 27529)]
[New Thread 0x7fffa27fc700 (LWP 27530)]
[New Thread 0x7fffa1ffb700 (LWP 27531)]
[New Thread 0x7fffa17fa700 (LWP 27532)]
[New Thread 0x7fffa0ff9700 (LWP 27533)]
[New Thread 0x7fff83fff700 (LWP 27534)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffa27fc700 (LWP 27530)]
nrt_pop_meminfo_list (list=0x7fffeeb0d7c0 <TheMSys>) at numba/runtime/nrt.c:63
```

If you have trouble reproducing the error, try changing the number of jobs, loops, threads or the array size. Whether and how fast I get the segmentation fault depends on these values.
The script runs fine when not using `nogil=True` or using processes instead of threads via `ProcessPoolExecutor`.

Fix #1330: Solve ABA issue in the nrt_pop_meminfo_list.
Fixes #1330.
Revert back to lock base implementation but a smarter one.
Allow concurrent pop, concurrent push, but not concurrent push & pop.



Discussion:
problem seems to come from the freelist for MemInfo

Confirmed ABA problem in NRT freelist pop routine.

By the way, if we are actually using locking, it may be better to bite the bullet and use system APIs (pthread mutexes under POSIX, critical sections under Windows).

Now that I think of it, yet another option is to remove the freelists and to allocate the data inline with the meminfo structure.

We don't always allocate the data.  MemInfo may reference borrowed pointer from numpy.

Yes, but the only case where allocation performance may matter is when we allocate the data ourselves...

That's true.  We can then avoid the freelist and just rely on malloc() and free().


Resolving patch:
diff --git a/numba/runtime/atomicops.py b/numba/runtime/atomicops.py
--- a/numba/runtime/atomicops.py
+++ b/numba/runtime/atomicops.py
@@ -37,7 +37,7 @@ def _define_decref(module, atomic_decr):
     fn_decref = module.get_global_variable_named("NRT_decref")
     fn_decref.linkage = 'linkonce_odr'
     calldtor = module.add_function(ir.FunctionType(ir.VoidType(),
-        [ir.IntType(8).as_pointer(), ir.IntType(32)]),
+        [ir.IntType(8).as_pointer()]),
         name="NRT_MemInfo_call_dtor")
 
     builder = ir.IRBuilder(fn_decref.append_basic_block())
@@ -51,8 +51,7 @@ def _define_decref(module, atomic_decr):
     refct_eq_0 = builder.icmp_unsigned("==", newrefct,
                                        ir.Constant(newrefct.type, 0))
     with cgutils.if_unlikely(builder, refct_eq_0):
-        do_defer = ir.Constant(ir.IntType(32), 0)
-        builder.call(calldtor, [ptr, do_defer])
+        builder.call(calldtor, [ptr])
     builder.ret_void()
 
 
diff --git a/numba/runtime/nrt.py b/numba/runtime/nrt.py
--- a/numba/runtime/nrt.py
+++ b/numba/runtime/nrt.py
@@ -73,11 +73,6 @@ def meminfo_alloc(self, size, safe=False):
             mi = _nrt.meminfo_alloc(size)
         return MemInfo(mi)
 
-    def process_defer_dtor(self):
-        """Process all deferred dtors.
-        """
-        _nrt.memsys_process_defer_dtor()
-
     def get_allocation_stats(self):
         """
         Returns a namedtuple of (alloc, free, mi_alloc, mi_free) for count of


Test patch:
diff --git a/numba/tests/test_nrt.py b/numba/tests/test_nrt.py
--- a/numba/tests/test_nrt.py
+++ b/numba/tests/test_nrt.py
@@ -62,26 +62,6 @@ def test_meminfo_refct_2(self):
         del mi
         self.assertEqual(Dummy.alive, 0)
 
-    def test_defer_dtor(self):
-        d = Dummy()
-        self.assertEqual(Dummy.alive, 1)
-        addr = 0xdeadcafe  # some made up location
-
-        mi = rtsys.meminfo_new(addr, d)
-        self.assertEqual(mi.refcount, 1)
-        # Set defer flag
-        mi.defer = True
-        del d
-        self.assertEqual(Dummy.alive, 1)
-        mi.acquire()
-        self.assertEqual(Dummy.alive, 1)
-        mi.release()
-        del mi
-        # mi refct is zero but not yet removed due to deferring
-        self.assertEqual(Dummy.alive, 1)
-        rtsys.process_defer_dtor()
-        self.assertEqual(Dummy.alive, 0)
-
     @unittest.skipIf(PYVERSION <= (2, 7), "memoryview not supported")
     def test_fake_memoryview(self):
         d = Dummy()