Repository: numpy/numpy

Issue:
Segfault when using numpy.searchsorted with recarrays in multithreaded environment
recarrays have a dtype.flags of 0 which allows numpy to release the GIL when using the NPY_BEGIN_THREADS_DESCR macro.
We suspect that PyArray_SearchSorted in item_selection.c uses VOID_compare for comparison of recarray elements which requires Python API functions.

Python code to reproduce under Windows 7 x64 using Python 2.7.5 x64 and Numpy 1.8.1 (or 1.7.1)

``` python
import numpy
import threading

def index(a,b,notfound=-1):
    """ find index of b in a """
    index = numpy.argsort(a)
    sorted_a = a[index]
    sorted_index = numpy.searchsorted(sorted_a,b)

    bindex = numpy.take(index,sorted_index,mode="clip")
    bindex[a[bindex] != b] = notfound

    return bindex

def index2dim(a,b,notfound=-1):
    """ like .index, but for n-dim arrays a and b along axis 0 """
    a = a.view([('',a.dtype)] * a.shape[1]).ravel()
    b = b.view([('',b.dtype)] * b.shape[1]).ravel()
    print str(threading.currentThread().ident),type(a),type(b),a.dtype,b.dtype,repr(a),repr(b)
    x = index(a,b,notfound=notfound)
    return x

def thread(a,b):
    while True:
        index2dim(a,b)

a = numpy.arange(200)
b = numpy.arange(10)

a = a.reshape((100,2))
b = b.reshape((5,2))

for i in xrange(10):
    t = threading.Thread(target=thread,args=(a,b))
    t.start()
```

Thanks and best regards
Martin Gfeller (martin.gfeller@swisscom.com) and Fabio Scala (fabio.scala@swisscom.com)

Division by zero when creating array from N-dimensional new-style-buffer with zero length and no strides (Trac #2197)
_Original ticket http://projects.scipy.org/numpy/ticket/2197 on 2012-08-08 by trac user batavus, assigned to unknown._

I believe there's a bug in function _array_from_buffer_3118 in file numpy/numpy/core/src/multiarray/ctors.c

Consider the code which currently starts at line 1228

```
    if (view->strides != NULL) {
        for (k = 0; k < nd; ++k) {
            strides[k] = view->strides[k];
        }
    }
    else {
        d = view->len;
        for (k = 0; k < nd; ++k) {
            d /= view->shape[k];
            strides[k] = d;
        }
    }
```

if strides is NULL and shape is something containing at least on zero (e.g. [0] or [1,0,2]) such that view->len is zero, there will be a division 0 by 0 in "d /= view->shape[k];"

I suggest replacing the line

d /= view->shape[k];

by

if (view->shape[k]) d /= view->shape[k];

This should resolve the problem and be consistent with how strides is defined in numpy for such cases.  Also, it's probably a good idea to add a test for this issue.



Discussion:
Looks like  `repr(a),repr(b)` is what done it. If those are omitted, there is no segfault.

repr triggers it as it runs lots of python code, but the issue is indeed the gil being released in searchsorted as void arrays do not specify that they need the api.
are there any operations of void arrays where the gil would be a problem? because adding the flag seems like a simple fix.

If the void arrays contain objects, then it will be a problem. For sorting itself, IIRC, the GIL is held for void arrays.

Which is to say, I think the GIL needs to be held. If that causes problems, then we should fix them.

I mean which operations exist on void arrays where the GIL is not needed?
sorting takes it, but not searchsorted
I'm wondering whether to fix searchsorted or to add the api flag to the dtype which will also fix it.

I don't know the answer to that, but I can't think of any off hand. The possibility of a void type containing objects makes problems with comparisons or anything else that calls a python function on objects. Setting the API flag is probably the safe way to go. 

In other places we check explicitly if the datatype "HasRefs" or so. So you can check if the void type actually does contain objects, just to note in case, not sure if it applies here.

The GIL is only needed for `VOID_compare` when the type has fields. You can also get the above code to work if you replace the two structured array views in `index2dim` by:

```
a = a.view(numpy.dtype((numpy.void, a.dtype.itemsize * a.shape[1]))).ravel()
b = b.view(numpy.dtype((numpy.void, b.dtype.itemsize * b.shape[1]))).ravel()
```

So ideally the `NPY_NEEDS_PYAPI` flag would only be set for types wth fields, not sure if that is actually possible, if not it should always be set.

Do note that it makes no difference if the void type contains objects, the above example has none and fails miserably. I think it is the calls to `PyTuple_xxx`  starting [here](https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/arraytypes.c.src#L2636) that unleash hell.

@charris : The print/repr most likely isn't the only thing that can make it crash. It was just inserted to produce some Python calls other than numpy in order to reproduce the crash with a small script. 

It crashed for us in a larger software system (without repr), and stopped crashing once we bound  the process to a single core, suggesting a GIL problem. 

Heh, now I remember seeing that code before ;). As the comment already in that code @jaimefrio linked says, this whole comparison function has a pretty bad smell to it and it can't possibly run in parallel unless it gets rewritten first.

I have submitted a PR that fixes the issue for the code presented, and doesn't penalize the fieldless void types. Not sure if I have caught all the possible entry points to create a type with fields. Also not sure how to test this, comments welcome.

@jaimefrio: Thank you, Jaime, very much for your reformulation of the original expression (which I got off the web somewhere) using numpy.void - this works around the problem for us and is also a bit faster.

All the quick responses and help much appreciated! Martin

Looks reasonable. Code currently starts at line 1307.

How can I assign this bug to me?  I would like to fix it.  Thanks!

No need for assignment, just fix it and make a PR. In the commit message include `Closes #650`. You should also take a look at `doc/source/dev/gitwash/development_workflow.rst` to see how to do the commit message.

Ok.  Thanks for the quick reply.

This issue can be tested with the following code which currently crashes python (before this fix):

``` python
>>> import ctypes, numpy
>>> numpy.array((ctypes.c_double*0*3)())
Floating point exception: 8
```


Resolving patch:
diff --git a/numpy/fft/fftpack.py b/numpy/fft/fftpack.py
--- a/numpy/fft/fftpack.py
+++ b/numpy/fft/fftpack.py
@@ -53,10 +53,12 @@ def _raw_fft(a, n=None, axis=-1, init_function=fftpack.cffti,
         raise ValueError("Invalid number of FFT data points (%d) specified." % n)
 
     try:
-        wsave = fft_cache[n]
-    except(KeyError):
+        # Thread-safety note: We rely on list.pop() here to atomically
+        # retrieve-and-remove a wsave from the cache.  This ensures that no
+        # other thread can get the same wsave while we're using it.
+        wsave = fft_cache.setdefault(n, []).pop()
+    except (IndexError):
         wsave = init_function(n)
-        fft_cache[n] = wsave
 
     if a.shape[axis] != n:
         s = list(a.shape)
@@ -77,6 +79,12 @@ def _raw_fft(a, n=None, axis=-1, init_function=fftpack.cffti,
     r = work_function(a, wsave)
     if axis != -1:
         r = swapaxes(r, axis, -1)
+
+    # As soon as we put wsave back into the cache, another thread could pick it
+    # up and start using it, so we must not do this until after we're
+    # completely done using it ourselves.
+    fft_cache[n].append(wsave)
+
     return r
 
 
diff --git a/numpy/lib/nanfunctions.py b/numpy/lib/nanfunctions.py
--- a/numpy/lib/nanfunctions.py
+++ b/numpy/lib/nanfunctions.py
@@ -228,7 +228,7 @@ def nanmin(a, axis=None, out=None, keepdims=False):
         # Check for all-NaN axis
         mask = np.all(mask, axis=axis, keepdims=keepdims)
         if np.any(mask):
-            res = _copyto(res, mask, np.nan)
+            res = _copyto(res, np.nan, mask)
             warnings.warn("All-NaN axis encountered", RuntimeWarning)
     return res
 
@@ -327,7 +327,7 @@ def nanmax(a, axis=None, out=None, keepdims=False):
         # Check for all-NaN axis
         mask = np.all(mask, axis=axis, keepdims=keepdims)
         if np.any(mask):
-            res = _copyto(res, mask, np.nan)
+            res = _copyto(res, np.nan, mask)
             warnings.warn("All-NaN axis encountered", RuntimeWarning)
     return res
 
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -49,9 +49,9 @@
 
 MAJOR               = 1
 MINOR               = 8
-MICRO               = 1
-ISRELEASED          = True
-VERSION             = '%d.%d.%d' % (MAJOR, MINOR, MICRO)
+MICRO               = 2
+ISRELEASED          = False
+VERSION             = '%d.%d.%drc1' % (MAJOR, MINOR, MICRO)
 
 
 # Return the git revision as a string


Test patch:
diff --git a/numpy/core/tests/test_multiarray.py b/numpy/core/tests/test_multiarray.py
--- a/numpy/core/tests/test_multiarray.py
+++ b/numpy/core/tests/test_multiarray.py
@@ -1143,6 +1143,12 @@ def test_partition(self):
                 d[i:].partition(0, kind=k)
             assert_array_equal(d, tgt)
 
+            d = np.array([0, 1, 2, 3, 4, 5, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,
+                          7, 7, 7, 7, 7, 9])
+            kth = [0, 3, 19, 20]
+            assert_equal(np.partition(d, kth, kind=k)[kth], (0, 3, 7, 7))
+            assert_equal(d[np.argpartition(d, kth, kind=k)][kth], (0, 3, 7, 7))
+
             d = np.array([2, 1])
             d.partition(0, kind=k)
             assert_raises(ValueError, d.partition, 2)
@@ -1332,6 +1338,18 @@ def test_partition_cdtype(self):
             assert_equal(np.partition(d, k)[k], tgt[k])
             assert_equal(d[np.argpartition(d, k)][k], tgt[k])
 
+    def test_partition_fuzz(self):
+        # a few rounds of random data testing
+       for j in range(10, 30):
+           for i in range(1, j - 2):
+               d = np.arange(j)
+               np.random.shuffle(d)
+               d = d % np.random.randint(2, 30)
+               idx = np.random.randint(d.size)
+               kth = [0, idx, i, i + 1]
+               tgt = np.sort(d)[kth]
+               assert_array_equal(np.partition(d, kth)[kth], tgt,
+                                  err_msg="data: %r\n kth: %r" % (d, kth))
 
     def test_flatten(self):
         x0 = np.array([[1, 2, 3], [4, 5, 6]], np.int32)
diff --git a/numpy/core/tests/test_regression.py b/numpy/core/tests/test_regression.py
--- a/numpy/core/tests/test_regression.py
+++ b/numpy/core/tests/test_regression.py
@@ -1945,6 +1945,26 @@ def test_format_on_flex_array_element(self):
         formatted = '{0}'.format(arr[0])
         assert_equal(formatted, str(arr[0]))
 
+    def test_richcompare_crash(self):
+        # gh-4613
+        import operator as op
+
+        # dummy class where __array__ throws exception
+        class Foo(object):
+            __array_priority__ = 1002
+            def __array__(self,*args,**kwargs):
+                raise Exception()
+
+        rhs = Foo()
+        lhs = np.array(1)
+        for f in [op.lt, op.le, op.gt, op.ge]:
+            if sys.version_info[0] >= 3:
+                assert_raises(TypeError, f, lhs, rhs)
+            else:
+                f(lhs, rhs)
+        assert_(not op.eq(lhs, rhs))
+        assert_(op.ne(lhs, rhs))
+
 
 if __name__ == "__main__":
     run_module_suite()
diff --git a/numpy/core/tests/test_scalarmath.py b/numpy/core/tests/test_scalarmath.py
--- a/numpy/core/tests/test_scalarmath.py
+++ b/numpy/core/tests/test_scalarmath.py
@@ -75,6 +75,18 @@ def test_blocked(self):
                 np.add(1, inp2, out=out)
                 assert_almost_equal(out, exp1, err_msg=msg)
 
+    def test_lower_align(self):
+        # check data that is not aligned to element size
+        # i.e doubles are aligned to 4 bytes on i386
+        d = np.zeros(23 * 8, dtype=np.int8)[4:-4].view(np.float64)
+        o = np.zeros(23 * 8, dtype=np.int8)[4:-4].view(np.float64)
+        assert_almost_equal(d + d, d * 2)
+        np.add(d, d, out=o)
+        np.add(np.ones_like(d), d, out=o)
+        np.add(d, np.ones_like(d), out=o)
+        np.add(np.ones_like(d), d)
+        np.add(d, np.ones_like(d))
+
 
 class TestPower(TestCase):
     def test_small_types(self):
diff --git a/numpy/core/tests/test_umath.py b/numpy/core/tests/test_umath.py
--- a/numpy/core/tests/test_umath.py
+++ b/numpy/core/tests/test_umath.py
@@ -711,6 +711,12 @@ def test_minmax_blocked(self):
                     inp[i] = -1e10
                     assert_equal(inp.min(), -1e10, err_msg=msg)
 
+    def test_lower_align(self):
+        # check data that is not aligned to element size
+        # i.e doubles are aligned to 4 bytes on i386
+        d = np.zeros(23 * 8, dtype=np.int8)[4:-4].view(np.float64)
+        assert_equal(d.max(), d[0])
+        assert_equal(d.min(), d[0])
 
 
 class TestAbsolute(TestCase):
@@ -736,6 +742,21 @@ def test_abs_blocked(self):
                             np.abs(inp, out=out)
                             assert_array_equal(out, d, err_msg=msg)
 
+                            assert_array_equal(-inp, -1*inp, err_msg=msg)
+                            np.negative(inp, out=out)
+                            assert_array_equal(out, -1*inp, err_msg=msg)
+
+    def test_lower_align(self):
+        # check data that is not aligned to element size
+        # i.e doubles are aligned to 4 bytes on i386
+        d = np.zeros(23 * 8, dtype=np.int8)[4:-4].view(np.float64)
+        assert_equal(np.abs(d), d)
+        assert_equal(np.negative(d), -d)
+        np.negative(d, out=d)
+        np.negative(np.ones_like(d), out=d)
+        np.abs(d, out=d)
+        np.abs(np.ones_like(d), out=d)
+
 
 class TestSpecialMethods(TestCase):
     def test_wrap(self):
diff --git a/numpy/fft/tests/test_fftpack.py b/numpy/fft/tests/test_fftpack.py
--- a/numpy/fft/tests/test_fftpack.py
+++ b/numpy/fft/tests/test_fftpack.py
@@ -2,6 +2,14 @@
 
 import numpy as np
 from numpy.testing import TestCase, run_module_suite, assert_array_almost_equal
+from numpy.testing import assert_array_equal
+import threading
+import sys
+if sys.version_info[0] >= 3:
+    import queue
+else:
+    import Queue as queue
+
 
 def fft1(x):
     L = len(x)
@@ -9,6 +17,7 @@ def fft1(x):
     phase = np.arange(L).reshape(-1, 1) * phase
     return np.sum(x*np.exp(phase), axis=1)
 
+
 class TestFFTShift(TestCase):
 
     def test_fft_n(self):
@@ -23,5 +32,44 @@ def test_basic(self):
         assert_array_almost_equal(fft1(x), np.fft.fft(x))
 
 
+class TestFFTThreadSafe(TestCase):
+    threads = 16
+    input_shape = (800, 200)
+
+    def _test_mtsame(self, func, *args):
+        def worker(args, q):
+            q.put(func(*args))
+
+        q = queue.Queue()
+        expected = func(*args)
+
+        # Spin off a bunch of threads to call the same function simultaneously
+        t = [threading.Thread(target=worker, args=(args, q))
+             for i in range(self.threads)]
+        [x.start() for x in t]
+
+        # Make sure all threads returned the correct value
+        for i in range(self.threads):
+            assert_array_equal(q.get(timeout=5), expected,
+                'Function returned wrong value in multithreaded context')
+        [x.join() for x in t]
+
+    def test_fft(self):
+        a = np.ones(self.input_shape) * 1+0j
+        self._test_mtsame(np.fft.fft, a)
+
+    def test_ifft(self):
+        a = np.ones(self.input_shape) * 1+0j
+        self._test_mtsame(np.fft.ifft, a)
+
+    def test_rfft(self):
+        a = np.ones(self.input_shape)
+        self._test_mtsame(np.fft.rfft, a)
+
+    def test_irfft(self):
+        a = np.ones(self.input_shape) * 1+0j
+        self._test_mtsame(np.fft.irfft, a)
+
+
 if __name__ == "__main__":
     run_module_suite()
diff --git a/numpy/lib/tests/test_nanfunctions.py b/numpy/lib/tests/test_nanfunctions.py
--- a/numpy/lib/tests/test_nanfunctions.py
+++ b/numpy/lib/tests/test_nanfunctions.py
@@ -114,6 +114,31 @@ def test_matrices(self):
             assert_(res.shape == (3, 1))
             res = f(mat)
             assert_(np.isscalar(res))
+        # check that rows of nan are dealt with for subclasses (#4628)
+        mat[1] = np.nan
+        for f in self.nanfuncs:
+            with warnings.catch_warnings(record=True) as w:
+                warnings.simplefilter('always')
+                res = f(mat, axis=0)
+                assert_(isinstance(res, np.matrix))
+                assert_(not np.any(np.isnan(res)))
+                assert_(len(w) == 0)
+
+            with warnings.catch_warnings(record=True) as w:
+                warnings.simplefilter('always')
+                res = f(mat, axis=1)
+                assert_(isinstance(res, np.matrix))
+                assert_(np.isnan(res[1, 0]) and not np.isnan(res[0, 0])
+                        and not np.isnan(res[2, 0]))
+                assert_(len(w) == 1, 'no warning raised')
+                assert_(issubclass(w[0].category, RuntimeWarning))
+
+            with warnings.catch_warnings(record=True) as w:
+                warnings.simplefilter('always')
+                res = f(mat)
+                assert_(np.isscalar(res))
+                assert_(res != np.nan)
+                assert_(len(w) == 0)
 
 
 class TestNanFunctions_ArgminArgmax(TestCase):
diff --git a/numpy/linalg/tests/test_regression.py b/numpy/linalg/tests/test_regression.py
--- a/numpy/linalg/tests/test_regression.py
+++ b/numpy/linalg/tests/test_regression.py
@@ -73,7 +73,15 @@ def test_large_svd_32bit(self):
         # See gh-4442, 64bit would require very large/slow matrices.
         x = np.eye(1000, 66)
         np.linalg.svd(x)
-        
+
+    def test_svd_no_uv(self):
+        # gh-4733
+        for shape in (3, 4), (4, 4), (4, 3):
+            for t in float, complex:
+                a = np.ones(shape, dtype=t)
+                w = linalg.svd(a, compute_uv=False)
+                c = np.count_nonzero(np.absolute(w) > 0.5)
+                assert_equal(c, 1)
 
 if __name__ == '__main__':
     run_module_suite()