Repository: numba/numba

Issue:
ipython kernel dies when concatenating tuples
Something strange seems to happen when I run the following in ipython/jupyter: 

``` python
import numba as nb
print nb.__version__
#0.27.0+0.g6ca78ce.dirty

import numpy as np
print np.__version__
#1.11.0

@nb.njit
def test_tuple():
    a = (2,3.0)
    b = np.zeros(3,dtype=np.int32)
    return (b,) + a

test_tuple()
# (array([-265424792,      32610, -265417008], dtype=int32), 2, 3.0)
test_tuple()
#The kernel appears to have died
```

The first time I call the function `test_tuple` it returns a tuple with the correct datatypes, but the array in the tuple has incorrect values. 
The second time I call the function the kernel just dies and tries to restart. No other error message is given. 

Is this expected behaviour? 

ipython kernel dies when concatenating tuples
Something strange seems to happen when I run the following in ipython/jupyter: 

``` python
import numba as nb
print nb.__version__
#0.27.0+0.g6ca78ce.dirty

import numpy as np
print np.__version__
#1.11.0

@nb.njit
def test_tuple():
    a = (2,3.0)
    b = np.zeros(3,dtype=np.int32)
    return (b,) + a

test_tuple()
# (array([-265424792,      32610, -265417008], dtype=int32), 2, 3.0)
test_tuple()
#The kernel appears to have died
```

The first time I call the function `test_tuple` it returns a tuple with the correct datatypes, but the array in the tuple has incorrect values. 
The second time I call the function the kernel just dies and tries to restart. No other error message is given. 

Is this expected behaviour? 



Discussion:
Thanks for the report. This is not expected. The problem is `ipython` invariant. 

@pitrou @sklam 
Built 10661a7 With `NRT_Debug`:

```
NRT_Allocate bytes=116 ptr=0x2bffdf0
NRT_MemInfo_alloc_safe_aligned 0x2bffe20 12
nrt_meminfo_call_dtor 0x2bffdf0
nrt_internal_dtor_safe 0x2bffe20, 0xc
NRT_Free 0x2bffdf0
NRT_release 0x2bffdf0 refct=0
python: numba/runtime/nrt.c:316: NRT_MemInfo_release: Assertion `mi->refct > 0 && "RefCt cannot be 0"' failed.
Aborted (core dumped)
```

Valgrind:

```
NRT_Allocate bytes=116 ptr=0xcc4aa00
NRT_MemInfo_alloc_safe_aligned 0xcc4aa40 12
nrt_meminfo_call_dtor 0xcc4aa00
nrt_internal_dtor_safe 0xcc4aa40, 0xc
NRT_Free 0xcc4aa00
==23487== Invalid read of size 8
==23487==    at 0x1881F501: NRT_MemInfo_release (nrt.c:314)
==23487==    by 0x1881D95C: MemInfo_dealloc (_nrt_python.c:146)
==23487==    by 0x105CF7F6: array_dealloc (arrayobject.c:411)
==23487==    by 0x4F01B6D: tupledealloc (tupleobject.c:236)
==23487==    by 0x4F7707D: PyEval_EvalFrameEx (ceval.c:1396)
==23487==    by 0x4F79B48: _PyEval_EvalCodeWithName (ceval.c:4018)
==23487==    by 0x4F79CD7: PyEval_EvalCodeEx (ceval.c:4039)
==23487==    by 0x4F79D1A: PyEval_EvalCode (ceval.c:777)
==23487==    by 0x4F9F01F: run_mod (pythonrun.c:976)
==23487==    by 0x4F9F01F: PyRun_FileExFlags (pythonrun.c:929)
==23487==    by 0x4FA0622: PyRun_SimpleFileExFlags (pythonrun.c:396)
==23487==    by 0x4FBB8C6: run_file (main.c:318)
==23487==    by 0x4FBB8C6: Py_Main (main.c:769)
==23487==    by 0x400ADC: main (python.c:65)
==23487==  Address 0xcc4aa00 is 0 bytes inside a block of size 116 free'd
==23487==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==23487==    by 0x1881EF8F: NRT_Free (nrt.c:418)
==23487==    by 0x1881EF8F: NRT_MemInfo_destroy (nrt.c:293)
==23487==    by 0x1881EF8F: NRT_MemInfo_call_dtor (nrt.c:310)
==23487==    by 0x402E26B: ???
==23487==    by 0x52BCEDF: ??? (in <path>/lib/libpython3.5m.so.1.0)
```

looks like a use after free?

The problem seems to come from a no-op cast that is not incref'ing. 

Actually, it is coming from `tuple_add` that didn't incref'ed.

Thanks for the report. This is not expected. The problem is `ipython` invariant. 

@pitrou @sklam 
Built 10661a7 With `NRT_Debug`:

```
NRT_Allocate bytes=116 ptr=0x2bffdf0
NRT_MemInfo_alloc_safe_aligned 0x2bffe20 12
nrt_meminfo_call_dtor 0x2bffdf0
nrt_internal_dtor_safe 0x2bffe20, 0xc
NRT_Free 0x2bffdf0
NRT_release 0x2bffdf0 refct=0
python: numba/runtime/nrt.c:316: NRT_MemInfo_release: Assertion `mi->refct > 0 && "RefCt cannot be 0"' failed.
Aborted (core dumped)
```

Valgrind:

```
NRT_Allocate bytes=116 ptr=0xcc4aa00
NRT_MemInfo_alloc_safe_aligned 0xcc4aa40 12
nrt_meminfo_call_dtor 0xcc4aa00
nrt_internal_dtor_safe 0xcc4aa40, 0xc
NRT_Free 0xcc4aa00
==23487== Invalid read of size 8
==23487==    at 0x1881F501: NRT_MemInfo_release (nrt.c:314)
==23487==    by 0x1881D95C: MemInfo_dealloc (_nrt_python.c:146)
==23487==    by 0x105CF7F6: array_dealloc (arrayobject.c:411)
==23487==    by 0x4F01B6D: tupledealloc (tupleobject.c:236)
==23487==    by 0x4F7707D: PyEval_EvalFrameEx (ceval.c:1396)
==23487==    by 0x4F79B48: _PyEval_EvalCodeWithName (ceval.c:4018)
==23487==    by 0x4F79CD7: PyEval_EvalCodeEx (ceval.c:4039)
==23487==    by 0x4F79D1A: PyEval_EvalCode (ceval.c:777)
==23487==    by 0x4F9F01F: run_mod (pythonrun.c:976)
==23487==    by 0x4F9F01F: PyRun_FileExFlags (pythonrun.c:929)
==23487==    by 0x4FA0622: PyRun_SimpleFileExFlags (pythonrun.c:396)
==23487==    by 0x4FBB8C6: run_file (main.c:318)
==23487==    by 0x4FBB8C6: Py_Main (main.c:769)
==23487==    by 0x400ADC: main (python.c:65)
==23487==  Address 0xcc4aa00 is 0 bytes inside a block of size 116 free'd
==23487==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==23487==    by 0x1881EF8F: NRT_Free (nrt.c:418)
==23487==    by 0x1881EF8F: NRT_MemInfo_destroy (nrt.c:293)
==23487==    by 0x1881EF8F: NRT_MemInfo_call_dtor (nrt.c:310)
==23487==    by 0x402E26B: ???
==23487==    by 0x52BCEDF: ??? (in <path>/lib/libpython3.5m.so.1.0)
```

looks like a use after free?

The problem seems to come from a no-op cast that is not incref'ing. 

Actually, it is coming from `tuple_add` that didn't incref'ed.


Resolving patch:
diff --git a/numba/targets/tupleobj.py b/numba/targets/tupleobj.py
--- a/numba/targets/tupleobj.py
+++ b/numba/targets/tupleobj.py
@@ -21,7 +21,9 @@ def namedtuple_constructor(context, builder, sig, args):
 @lower_builtin('+', types.BaseTuple, types.BaseTuple)
 def tuple_add(context, builder, sig, args):
     left, right = [cgutils.unpack_tuple(builder, x) for x in args]
-    return context.make_tuple(builder, sig.return_type, left + right)
+    res = context.make_tuple(builder, sig.return_type, left + right)
+    # The tuple's contents are borrowed
+    return impl_ret_borrowed(context, builder, sig.return_type, res)
 
 def tuple_cmp_ordered(context, builder, op, sig, args):
     tu, tv = sig.args


Test patch:
diff --git a/numba/tests/test_tuples.py b/numba/tests/test_tuples.py
--- a/numba/tests/test_tuples.py
+++ b/numba/tests/test_tuples.py
@@ -420,6 +420,20 @@ def test_type(self):
             self.assertPreciseEqual(got, expected)
 
 
+class TestTupleNRT(TestCase, MemoryLeakMixin):
+    def test_tuple_add(self):
+        def pyfunc(x):
+            a = np.arange(3)
+            return (a,) + (x,)
+
+        cfunc = jit(nopython=True)(pyfunc)
+        x = 123
+        expect_a, expect_x = pyfunc(x)
+        got_a, got_x = cfunc(x)
+        np.testing.assert_equal(got_a, expect_a)
+        self.assertEqual(got_x, expect_x)
+
+
 class TestNamedTupleNRT(TestCase, MemoryLeakMixin):
 
     def test_return(self):