kwgoodman/bottleneck — Issue #35: Possible problem when working with very large arrays
Author: kwgoodman
Created: 2012-02-24 20:22:08+00:00
State: open

As discussed recently (January) on the numpy list, be careful when using C ints for indexing into arrays. A C int can overflow for very large but-some-people-use-them arrays.

I think Bottleneck uses:

cdef Py_ssize_t idx

for indexing. So that should be safe for large arrays. BUT it does use ints for the size of the array:

cdef np.npy_intp *dim
dim = PyArray_DIMS(a)
cdef int n = dim[0]
for idx in range(n):
...

That could be a problem since n could overflow. So switch to:

cdef Py_ssize_t n = dim[0]

But first try to confirm there is a problem by working on very large arrays. And that the issue is fixed by the above or similar.