Repository: Theano/Theano

Issue:
Segmentation Fault When training with a big dataset
I'm running a Custom MLP with Theano (using Lasagne) and I can run it over a small (input shape 10kx10k) and medium (input shape 400k_200k) but when I run it over a larger dataset (input shape 1.3Mk_100k) it gives me the segmentation fault error when it starts training. The memory starts to grow and then it crashes. I have 500GB of memory and when it gets to less than 20% of memory it throws the segmentation fault error. I'm using cpu and my .theanorc looks like this:

```
[global]
OMP_NUM_THREADS=4
openmp=True
floatX = float64
#profile_memory=True
#profile=True
device=cpu
[blas] 
ldflags=-L/usr/lib/  -lopenblas  -lblas 
```



Note that the input matrices are sparse and I can't do mini-batch and have to do **full batch**.

I used gdb to find where the error occurs and the output is:

```
Thread 17 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff3d31700 (LWP 27105)]
(anonymous namespace)::__struct_compiled_op_1b42432866be1c904afdc4503a8345e1::run ()
    at /home/user1/.theano/compiledir_Linux-4.4--generic-x86_64-with-Ubuntu-16.04-xenial-x86_64-2.7.12-64/tmpB1iBsD/mod.cpp:518
518                 V1_i = V3_i + V5_i;

#0  (anonymous namespace)::__struct_compiled_op_5c38a1b25524822864d2313292bdfdc6::run ()
    at /home/arahimi/.theano/compiledir_Linux-4.4--generic-x86_64-with-Ubuntu-16.04-xenial-x86_64-2.7.12-64/tmpLIG_F8/mod.cpp:518
#1  0x00007fffce85d43e in ?? () from /usr/lib/x86_64-linux-gnu/libgomp.so.1
#2  0x00007ffff7bc16ba in start_thread (arg=0x7fffeed2f700) at pthread_create.c:333
#3  0x00007ffff78f782d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

```
 
Theano, Scipy and Numpy versions:
```
t>>> theano.__version__
'0.9.0beta1.dev-RELEASE'
>>> scipy.__version__
'0.18.1'
>>> numpy.__version__
'1.12.0'
```

I tried with older versions and then I updated to the dev versions but the problem exists.

Theano config is attached.
Also the code that builds the model is attached.

I commented all other operations in the custom layer and left only S.dot(input, w) but the error still exists:
`activation = S.dot(input, self.W)`

where input is a csr sparse matrix 1386766 X 138640 and W is hidden layer parameters with size 138640 X 3000 where 3000 is hidden layer size.
**When I decrease the hidden layer size from 3000 to 1000 it works.**

Given that I still have enough memory, why I can't use higher hidden layer sizes?


[mlpconv_python.txt](https://github.com/Theano/Theano/files/765365/mlpconv_python.txt)



[theano_config.txt](https://github.com/Theano/Theano/files/765364/theano_config.txt)



Discussion:
What happens if you disable openmp? Does it still crash?
Also, what if you run the minimal example (with only `S.dot(...)`) disabling the compilation of C code (`gxx = ""` in the config flags)?
Also, can you try with the fix in #5553?
I tried the suggestions:
set openmp=False in .theanorc
set gxx="" in .theanorc
commented other computations except S.dot(...)).
and also applied fix #5553.

It still crashes with segmentation fault.
Does a sparse product this size work in scipy?

Is the problem the number of elements in W, or the number of elements in the output? (try reducing the first dimension of the input)
To help us identify the op that cause the problem, can you use MonitorMode
in the function that segfault? Use this as an example:

http://deeplearning.net/software/theano/tutorial/debug_faq.html#how-do-i-step-through-a-compiled-function

You can use just the inspect_inputs. Modify it to only print the shape of
the value, not the value themself. Something like:

def inspect_inputs(i, node, fn):
    print(i, node, "input(s) value(s):", [input[0].shape for input in
fn.inputs],
          end='')



On Fri, Feb 17, 2017 at 2:37 AM rebecca-palmer <notifications@github.com>
wrote:

> Does a sparse product this size work in scipy?
>
> Is the problem the number of elements in W, or the number of elements in
> the output? (try reducing the first dimension of the input)
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/Theano/Theano/issues/5525#issuecomment-280576996>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AALC-_F0gbxpt8-TN3e-S-lUZlr5-P5Gks5rdU5UgaJpZM4L82OW>
> .
>

I can confirm that I multiplied the exact same matrix X (input) to a random w with the same size in scipy and it worked without a problem (X.dot(w)).

I will try the MonitorMode and report the results asap.
MonitorMode logs: (3000 is hidden layer size, 138640 is input feature size and 1386766 is the number of samples in full batch).

```
0 InplaceDimShuffle{x}(TensorConstant{0.0}) input(s) shape(s): [()] output(s) shape(s): [(1,)]
1 InplaceDimShuffle{x,0}(b) input(s) shape(s): [(930,)] output(s) shape(s): [(1, 930)]
2 Elemwise{sgn}(W) input(s) shape(s): [(3000, 930)] output(s) shape(s): [(3000, 930)]
3 Elemwise{abs_,no_inplace}(W) input(s) shape(s): [(3000, 930)] output(s) shape(s): [(3000, 930)]
4 InplaceDimShuffle{1,0}(W) input(s) shape(s): [(3000, 930)] output(s) shape(s): [(930, 3000)]
5 InplaceDimShuffle{x,0}(b) input(s) shape(s): [(3000,)] output(s) shape(s): [(1, 3000)]
6 Elemwise{sgn}(W) input(s) shape(s): [(138640, 3000)] output(s) shape(s): [(138640, 3000)]
7 Elemwise{abs_,no_inplace}(W) input(s) shape(s): [(138640, 3000)] output(s) shape(s): [(138640, 3000)]
8 SparseDot(SparseVariable{csr,float32}, W) input(s) shape(s): [(1386766, 138640), (138640, 3000)] output(s) shape(s): [(1386766, 3000)]
9 SparseTranspose(SparseVariable{csr,float32}) input(s) shape(s): [(1386766, 138640)] output(s) shape(s): [(138640, 1386766)]
10 SparseTranspose(SparseVariable{csr,float32}) input(s) shape(s): [(1386766, 1386766)] output(s) shape(s): [(1386766, 1386766)]
11 Elemwise{mul,no_inplace}(TensorConstant{(1,) of 0.999}, <TensorType(float32, vector)>) input(s) shape(s): [(1,), (930,)] output(s) shape(s): [(930,)]
12 Elemwise{mul,no_inplace}(TensorConstant{(1,) of 0.9}, <TensorType(float32, vector)>) input(s) shape(s): [(1,), (930,)] output(s) shape(s): [(930,)]
13 Elemwise{pow,no_inplace}(W, TensorConstant{(1, 1) of 2}) input(s) shape(s): [(3000, 930), (1, 1)] output(s) shape(s): [(3000, 930)]
14 Elemwise{pow,no_inplace}(W, TensorConstant{(1, 1) of 2}) input(s) shape(s): [(138640, 3000), (1, 1)] output(s) shape(s): [(138640, 3000)]
15 Elemwise{pow}(W, TensorConstant{(1, 1) of 1}) input(s) shape(s): [(3000, 930), (1, 1)] output(s) shape(s): [(3000, 930)]
16 Elemwise{pow}(W, TensorConstant{(1, 1) of 1}) input(s) shape(s): [(138640, 3000), (1, 1)] output(s) shape(s): [(138640, 3000)]
17 Elemwise{mul,no_inplace}(TensorConstant{(1, 1) of 0.999}, <TensorType(float32, matrix)>) input(s) shape(s): [(1, 1), (3000, 930)] output(s) shape(s): [(3000, 930)]
18 Elemwise{mul,no_inplace}(TensorConstant{(1, 1) of 0.9}, <TensorType(float32, matrix)>) input(s) shape(s): [(1, 1), (3000, 930)] output(s) shape(s): [(3000, 930)]
19 Elemwise{mul,no_inplace}(TensorConstant{(1,) of 0.999}, <TensorType(float32, vector)>) input(s) shape(s): [(1,), (3000,)] output(s) shape(s): [(3000,)]
20 Elemwise{mul,no_inplace}(TensorConstant{(1,) of 0.9}, <TensorType(float32, vector)>) input(s) shape(s): [(1,), (3000,)] output(s) shape(s): [(3000,)]
21 Elemwise{add,no_inplace}(<TensorType(float32, scalar)>, TensorConstant{1}) input(s) shape(s): [(), ()] output(s) shape(s): [()]
22 Elemwise{mul,no_inplace}(TensorConstant{(1, 1) of 0.999}, <TensorType(float32, matrix)>) input(s) shape(s): [(1, 1), (138640, 3000)] output(s) shape(s): [(138640, 3000)]
23 Elemwise{mul,no_inplace}(TensorConstant{(1, 1) of 0.9}, <TensorType(float32, matrix)>) input(s) shape(s): [(1, 1), (138640, 3000)] output(s) shape(s): [(138640, 3000)]
24 Elemwise{second}(Elemwise{abs_,no_inplace}.0, TensorConstant{(1, 1) of 2.5e-06}) input(s) shape(s): [(3000, 930), (1, 1)] output(s) shape(s): [(3000, 930)]
25 Sum{acc_dtype=float64}(Elemwise{abs_,no_inplace}.0) input(s) shape(s): [(3000, 930)] output(s) shape(s): [()]
26 Elemwise{second}(Elemwise{abs_,no_inplace}.0, TensorConstant{(1, 1) of 2.5e-06}) input(s) shape(s): [(138640, 3000), (1, 1)] output(s) shape(s): [(138640, 3000)]
27 Sum{acc_dtype=float64}(Elemwise{abs_,no_inplace}.0) input(s) shape(s): [(138640, 3000)] output(s) shape(s): [()]
28 SparseDot(SparseVariable{csr,float32}, SparseDot.0) input(s) shape(s): [(1386766, 1386766), (1386766, 3000)] output(s) shape(s): [(1386766, 3000)]
29 Elemwise{second}(Elemwise{pow,no_inplace}.0, TensorConstant{(1, 1) of 2.5e-06}) input(s) shape(s): [(3000, 930), (1, 1)] output(s) shape(s): [(3000, 930)]
30 Sum{acc_dtype=float64}(Elemwise{pow,no_inplace}.0) input(s) shape(s): [(3000, 930)] output(s) shape(s): [()]
31 Elemwise{second}(Elemwise{pow,no_inplace}.0, TensorConstant{(1, 1) of 2.5e-06}) input(s) shape(s): [(138640, 3000), (1, 1)] output(s) shape(s): [(138640, 3000)]
32 Sum{acc_dtype=float64}(Elemwise{pow,no_inplace}.0) input(s) shape(s): [(138640, 3000)] output(s) shape(s): [()]
33 Elemwise{pow,no_inplace}(TensorConstant{0.899999976158}, Elemwise{add,no_inplace}.0) input(s) shape(s): [(), ()] output(s) shape(s): [()]
34 Elemwise{pow,no_inplace}(TensorConstant{0.999000012875}, Elemwise{add,no_inplace}.0) input(s) shape(s): [(), ()] output(s) shape(s): [()]
35 Elemwise{mul}(Elemwise{second}.0, Elemwise{sgn}.0) input(s) shape(s): [(3000, 930), (3000, 930)] output(s) shape(s): [(3000, 930)]
36 Elemwise{add,no_inplace}(TensorConstant{0}, Sum{acc_dtype=float64}.0) input(s) shape(s): [(), ()] output(s) shape(s): [()]
37 Elemwise{mul}(Elemwise{second}.0, Elemwise{sgn}.0) input(s) shape(s): [(138640, 3000), (138640, 3000)] output(s) shape(s): [(138640, 3000)]
38 Elemwise{add,no_inplace}(TensorConstant{0}, Sum{acc_dtype=float64}.0) input(s) shape(s): [(), ()] output(s) shape(s): [()]
39 Elemwise{add,no_inplace}(SparseDot.0, InplaceDimShuffle{x,0}.0) input(s) shape(s): [(1386766, 3000), (1, 3000)]Segmentation fault (core dumped)
```

seems like the error is happening when adding the bias term (with no_inplace)?

The two layers are:

```
#hidden layer
class SparseConvolutionDenseLayer(DenseLayer):
    def __init__(self, incoming, H=None, **kwargs):
        super(SparseConvolutionDenseLayer, self).__init__(incoming, **kwargs)
        #self.H = H
        
    def get_output_for(self, input, **kwargs):
        if not isinstance(input, (S.SparseVariable, S.SparseConstant,
                                  S.sharedvar.SparseTensorSharedVariable)):
            raise ValueError("Input for this layer must be sparse")
        
        activation = S.dot(input, self.W)
        #do the convolution
        H = kwargs.get('H')
        activation = S.dot(H, activation)

        if self.b is not None:
            activation = activation + self.b.dimshuffle('x', 0)
        return self.nonlinearity(activation)

#output layer
class ConvolutionDenseLayer(DenseLayer):

    def __init__(self, incoming, H=None, **kwargs):
        super(ConvolutionDenseLayer, self).__init__(incoming, **kwargs)
        #self.H = H
    
    def get_output_for(self, input, **kwargs):
        target_indices = kwargs.get('target_indices') 
        activation = T.dot(input, self.W)
        #do the convolution
        H = kwargs.get('H')
        activation = S.dot(H, activation)

        if self.b is not None:
            activation = activation + self.b.dimshuffle('x', 0)
        activation = activation[target_indices, :]
        return self.nonlinearity(activation)
```
It again happens when I commented the custom part of the layers `actication = S.dot(H, activation) ` and again the error is when adding the bias term:

```
0 InplaceDimShuffle{x}(TensorConstant{0.0}) input(s) shape(s): [()] output(s) shape(s): [(1,)]
1 InplaceDimShuffle{x,0}(b) input(s) shape(s): [(930,)] output(s) shape(s): [(1, 930)]
2 Elemwise{sgn}(W) input(s) shape(s): [(3000, 930)] output(s) shape(s): [(3000, 930)]
3 Elemwise{abs_,no_inplace}(W) input(s) shape(s): [(3000, 930)] output(s) shape(s): [(3000, 930)]
4 InplaceDimShuffle{1,0}(W) input(s) shape(s): [(3000, 930)] output(s) shape(s): [(930, 3000)]
5 InplaceDimShuffle{x,0}(b) input(s) shape(s): [(3000,)] output(s) shape(s): [(1, 3000)]
6 Elemwise{sgn}(W) input(s) shape(s): [(138640, 3000)] output(s) shape(s): [(138640, 3000)]
7 Elemwise{abs_,no_inplace}(W) input(s) shape(s): [(138640, 3000)] output(s) shape(s): [(138640, 3000)]
8 SparseDot(SparseVariable{csr,float32}, W) input(s) shape(s): [(1386766, 138640), (138640, 3000)] output(s) shape(s): [(1386766, 3000)]
9 SparseTranspose(SparseVariable{csr,float32}) input(s) shape(s): [(1386766, 138640)] output(s) shape(s): [(138640, 1386766)]
10 Elemwise{mul,no_inplace}(TensorConstant{(1,) of 0.999}, <TensorType(float32, vector)>) input(s) shape(s): [(1,), (930,)] output(s) shape(s): [(930,)]
11 Elemwise{mul,no_inplace}(TensorConstant{(1,) of 0.9}, <TensorType(float32, vector)>) input(s) shape(s): [(1,), (930,)] output(s) shape(s): [(930,)]
12 Elemwise{pow,no_inplace}(W, TensorConstant{(1, 1) of 2}) input(s) shape(s): [(3000, 930), (1, 1)] output(s) shape(s): [(3000, 930)]
13 Elemwise{pow,no_inplace}(W, TensorConstant{(1, 1) of 2}) input(s) shape(s): [(138640, 3000), (1, 1)] output(s) shape(s): [(138640, 3000)]
14 Elemwise{pow}(W, TensorConstant{(1, 1) of 1}) input(s) shape(s): [(3000, 930), (1, 1)] output(s) shape(s): [(3000, 930)]
15 Elemwise{pow}(W, TensorConstant{(1, 1) of 1}) input(s) shape(s): [(138640, 3000), (1, 1)] output(s) shape(s): [(138640, 3000)]
16 Elemwise{mul,no_inplace}(TensorConstant{(1, 1) of 0.999}, <TensorType(float32, matrix)>) input(s) shape(s): [(1, 1), (3000, 930)] output(s) shape(s): [(3000, 930)]
17 Elemwise{mul,no_inplace}(TensorConstant{(1, 1) of 0.9}, <TensorType(float32, matrix)>) input(s) shape(s): [(1, 1), (3000, 930)] output(s) shape(s): [(3000, 930)]
18 Elemwise{mul,no_inplace}(TensorConstant{(1,) of 0.999}, <TensorType(float32, vector)>) input(s) shape(s): [(1,), (3000,)] output(s) shape(s): [(3000,)]
19 Elemwise{mul,no_inplace}(TensorConstant{(1,) of 0.9}, <TensorType(float32, vector)>) input(s) shape(s): [(1,), (3000,)] output(s) shape(s): [(3000,)]
20 Elemwise{add,no_inplace}(<TensorType(float32, scalar)>, TensorConstant{1}) input(s) shape(s): [(), ()] output(s) shape(s): [()]
21 Elemwise{mul,no_inplace}(TensorConstant{(1, 1) of 0.999}, <TensorType(float32, matrix)>) input(s) shape(s): [(1, 1), (138640, 3000)] output(s) shape(s): [(138640, 3000)]
22 Elemwise{mul,no_inplace}(TensorConstant{(1, 1) of 0.9}, <TensorType(float32, matrix)>) input(s) shape(s): [(1, 1), (138640, 3000)] output(s) shape(s): [(138640, 3000)]
23 Elemwise{second}(Elemwise{abs_,no_inplace}.0, TensorConstant{(1, 1) of 2.5e-06}) input(s) shape(s): [(3000, 930), (1, 1)] output(s) shape(s): [(3000, 930)]
24 Sum{acc_dtype=float64}(Elemwise{abs_,no_inplace}.0) input(s) shape(s): [(3000, 930)] output(s) shape(s): [()]
25 Elemwise{second}(Elemwise{abs_,no_inplace}.0, TensorConstant{(1, 1) of 2.5e-06}) input(s) shape(s): [(138640, 3000), (1, 1)] output(s) shape(s): [(138640, 3000)]
26 Sum{acc_dtype=float64}(Elemwise{abs_,no_inplace}.0) input(s) shape(s): [(138640, 3000)] output(s) shape(s): [()]
27 Elemwise{add,no_inplace}(SparseDot.0, InplaceDimShuffle{x,0}.0) input(s) shape(s): [(1386766, 3000), (1, 3000)]Segmentation fault (core dumped)
```

That's sparse + dense with >2**31 elements, which is likely to trigger an integer overflow at [pos = row * Yi + col * Yj;](https://github.com/Theano/Theano/blob/master/theano/sparse/opt.py#L154) (which uses 32-bit signed integers, and where Yi, Yj are strides in elements, i.e. typically num_cols, 1).

Though that does leave the question of why gxx="" didn't help...

Resolving patch:
diff --git a/theano/sparse/opt.py b/theano/sparse/opt.py
--- a/theano/sparse/opt.py
+++ b/theano/sparse/opt.py
@@ -137,28 +137,29 @@ def c_code(self, node, name, inputs, outputs, sub):
                 }
 
                 npy_intp N =  PyArray_DIMS(%(_indptr)s)[0]-1;
-                const npy_int32 * __restrict__ indptr = (npy_int32 *)PyArray_DATA(%(_indptr)s);
-                const npy_int32 * __restrict__ indices = (npy_int32*)PyArray_DATA(%(_indices)s);
+
+                const dtype_%(_indptr)s* __restrict__ indptr = (dtype_%(_indptr)s*)PyArray_DATA(%(_indptr)s);
+                const dtype_%(_indices)s* __restrict__ indices = (dtype_%(_indices)s*)PyArray_DATA(%(_indices)s);
                 const dtype_%(_data)s* __restrict__ data = (dtype_%(_data)s*)PyArray_DATA(%(_data)s);
 
                 dtype_%(y)s* ydata = (dtype_%(y)s*)PyArray_DATA(%(y)s);
                 dtype_%(z)s* zdata = (dtype_%(z)s*)PyArray_DATA(%(z)s);
-                int Yi = PyArray_STRIDES(%(y)s)[0]/PyArray_DESCR(%(y)s)->elsize;
-                int Yj = PyArray_STRIDES(%(y)s)[1]/PyArray_DESCR(%(y)s)->elsize;
+                npy_intp Yi = PyArray_STRIDES(%(y)s)[0]/PyArray_DESCR(%(y)s)->elsize;
+                npy_intp Yj = PyArray_STRIDES(%(y)s)[1]/PyArray_DESCR(%(y)s)->elsize;
 
-                npy_int32 pos;
+                npy_intp pos;
                 if (%(format)s == 0){
-                for (npy_int32 col = 0; col < N; ++col){
-                  for (npy_int32 ind = indptr[col]; ind < indptr[col+1]; ++ind){
-                    npy_int32 row = indices[ind];
+                for (npy_intp col = 0; col < N; ++col){
+                  for (dtype_%(_indptr)s ind = indptr[col]; ind < indptr[col+1]; ++ind){
+                    npy_intp row = indices[ind];
                     pos = row * Yi + col * Yj;
                     zdata[pos] = ydata[pos] + data[ind];
                   }
                 }
                 }else{
-                for (npy_int32 row = 0; row < N; ++row){
-                  for (npy_int32 ind = indptr[row]; ind < indptr[row+1]; ++ind){
-                    npy_int32 col = indices[ind];
+                for (npy_intp row = 0; row < N; ++row){
+                  for (dtype_%(_indptr)s ind = indptr[row]; ind < indptr[row+1]; ++ind){
+                    npy_intp col = indices[ind];
                     pos = row * Yi + col * Yj;
                     zdata[pos] = ydata[pos] + data[ind];
                   }
@@ -171,7 +172,7 @@ def infer_shape(self, node, shapes):
         return [shapes[3]]
 
     def c_code_cache_version(self):
-        return (1,)
+        return (2,)
 
 
 @gof.local_optimizer([sparse.AddSD])
@@ -336,6 +337,8 @@ def c_code(self, node, name, inputs, outputs, sub):
             npy_intp M = PyArray_DIMS(%(z)s)[0];
             npy_intp N = PyArray_DIMS(%(z)s)[1];
             npy_intp K = PyArray_DIMS(%(b)s)[0];
+            if (N > 0x7fffffffL)
+            {PyErr_SetString(PyExc_NotImplementedError, "array too big (overflows int32 index)"); %(fail)s;}
 
             // strides tell you how many bytes to skip to go to next column/row entry
             npy_intp Szm = PyArray_STRIDES(%(z)s)[0] / PyArray_DESCR(%(z)s)->elsize;
@@ -413,7 +416,7 @@ def c_code(self, node, name, inputs, outputs, sub):
         return rval
 
     def c_code_cache_version(self):
-        return (2,)
+        return (3,)
 sd_csc = StructuredDotCSC()
 
 
@@ -529,6 +532,8 @@ def c_code(self, node, name, inputs, outputs, sub):
             npy_intp M = PyArray_DIMS(%(z)s)[0];
             npy_intp N = PyArray_DIMS(%(z)s)[1];
             npy_intp K = PyArray_DIMS(%(b)s)[0];
+            if (N > 0x7fffffffL)
+            {PyErr_SetString(PyExc_NotImplementedError, "array too big (overflows int32 index)"); %(fail)s;}
 
             // strides tell you how many bytes to skip to go to next column/row entry
             npy_intp Szm = PyArray_STRIDES(%(z)s)[0] / PyArray_DESCR(%(z)s)->elsize;
@@ -590,7 +595,7 @@ def c_code(self, node, name, inputs, outputs, sub):
         """ % dict(locals(), **sub)
 
     def c_code_cache_version(self):
-        return (1,)
+        return (2,)
 sd_csr = StructuredDotCSR()
 
 
@@ -832,6 +837,8 @@ def c_code(self, node, name, inputs, outputs, sub):
             npy_intp Sy = PyArray_STRIDES(%(y)s)[1] / PyArray_DESCR(%(y)s)->elsize;
 
             // blas expects ints; convert here (rather than just making N etc ints) to avoid potential overflow in the negative-stride correction
+            if ((N > 0x7fffffffL)||(Sy > 0x7fffffffL)||(Szn > 0x7fffffffL)||(Sy < -0x7fffffffL)||(Szn < -0x7fffffffL))
+            {PyErr_SetString(PyExc_NotImplementedError, "array too big for BLAS (overflows int32 index)"); %(fail)s;}
             int N32 = N;
             int Sy32 = Sy;
             int Szn32 = Szn;
@@ -845,7 +852,7 @@ def c_code(self, node, name, inputs, outputs, sub):
                 }
             }
 
-            for (npy_int32 k = 0; k < K; ++k)
+            for (npy_intp k = 0; k < K; ++k)
             {
                 for (npy_int32 m_idx = Dptr[k * Sptr]; m_idx < Dptr[(k+1)*Sptr]; ++m_idx)
                 {
@@ -873,7 +880,7 @@ def c_code(self, node, name, inputs, outputs, sub):
         return rval
 
     def c_code_cache_version(self):
-        return (2, blas.blas_header_version())
+        return (3, blas.blas_header_version())
 usmm_csc_dense = UsmmCscDense(inplace=False)
 usmm_csc_dense_inplace = UsmmCscDense(inplace=True)
 
@@ -1106,7 +1113,7 @@ def make_node(self, a_data, a_indices, a_indptr, b):
                                [tensor.tensor(b.dtype, (False,))])
 
     def c_code_cache_version(self):
-        return (2,)
+        return (3,)
 
     # def perform(self, node, (a_data, a_indices, a_indptr, b), (out,)):
     #    return NotImplementedError()
@@ -1169,7 +1176,7 @@ def c_code(self, node, name, inputs, outputs, sub):
             const npy_intp Sb = PyArray_STRIDES(%(_b)s)[0];
 
             // loop over columns
-            for (npy_int32 j = 0; j < N; ++j)
+            for (npy_intp j = 0; j < N; ++j)
             {
                 // for each non-null value in the sparse column
                 for (npy_int32 i_idx = indptr[j]; i_idx < indptr[j+1]; ++i_idx)
@@ -1232,7 +1239,7 @@ def make_node(self, a_data, a_indices, a_indptr, b):
                                [tensor.tensor(b.dtype, (False,))])
 
     def c_code_cache_version(self):
-        return (2,)
+        return (3,)
 
     # def perform(self, node, (a_data, a_indices, a_indptr, b), (out,)):
     #    return NotImplemented()
@@ -1295,7 +1302,7 @@ def c_code(self, node, name, inputs, outputs, sub):
             const npy_intp Sb = PyArray_STRIDES(%(_b)s)[0];
 
             // loop over columns
-            for (npy_int32 j = 0; j < N; ++j)
+            for (npy_intp j = 0; j < N; ++j)
             {
                 // extract i-th row of dense matrix
                 const dtype_%(_b)s* __restrict__ b_row = (dtype_%(_b)s*)(PyArray_BYTES(%(_b)s) + Sb * j);
@@ -1400,7 +1407,7 @@ def make_node(self, a_data, a_indices, a_indptr, b):
                                [tensor.tensor(b.dtype, (False,))])
 
     def c_code_cache_version(self):
-        return (1,)
+        return (2,)
 
     def c_code(self, node, name, inputs, outputs, sub):
         _data, _indices, _indptr, _b, = inputs
@@ -1459,7 +1466,7 @@ def c_code(self, node, name, inputs, outputs, sub):
             const npy_intp Sb = PyArray_STRIDES(%(_b)s)[0] / PyArray_DESCR(%(_b)s)->elsize;
 
             // loop over rows
-            for (npy_int32 j = 0; j < N; ++j)
+            for (npy_intp j = 0; j < N; ++j)
             {
                 // for each non-null value in the sparse column
                 for (npy_int32 i_idx = indptr[j]; i_idx < indptr[j+1]; ++i_idx)
@@ -1558,7 +1565,7 @@ def make_node(self, a_data, a_indices, a_indptr, b):
                                [tensor.tensor(b.dtype, (False,))])
 
     def c_code_cache_version(self):
-        return (2,)
+        return (3,)
 
     def c_code(self, node, name, inputs, outputs, sub):
         _data, _indices, _indptr, _b, = inputs
@@ -1623,7 +1630,7 @@ def c_code(self, node, name, inputs, outputs, sub):
             const npy_intp Sb = PyArray_STRIDES(%(_b)s)[0] / PyArray_DESCR(%(_b)s)->elsize;
 
             // loop over columns
-            for (npy_int32 j = 0; j < N; ++j)
+            for (npy_intp j = 0; j < N; ++j)
             {
                 // for each non-null value in the sparse column
                 for (npy_int32 i_idx = indptr[j]; i_idx < indptr[j+1]; ++i_idx)
@@ -1753,7 +1760,7 @@ def make_node(self, x, y, p_data, p_ind, p_ptr, p_ncols):
         ])
 
     def c_code_cache_version(self):
-        return (3, blas.blas_header_version())
+        return (4, blas.blas_header_version())
 
     def c_support_code(self):
         return blas.blas_header_text()
@@ -1897,11 +1904,13 @@ def c_code(self, node, name, inputs, outputs, sub):
             memcpy(Dzp, Dpp, PyArray_DIMS(%(p_ptr)s)[0]*sizeof(dtype_%(p_ptr)s));
 
             // blas expects ints; convert here (rather than just making K etc ints) to avoid potential overflow in the negative-stride correction
+            if ((K > 0x7fffffffL)||(Sdx > 0x7fffffffL)||(Sdy > 0x7fffffffL)||(Sdx < -0x7fffffffL)||(Sdy < -0x7fffffffL))
+            {PyErr_SetString(PyExc_NotImplementedError, "array too big for BLAS (overflows int32 index)"); %(fail)s;}
             int K32 = K;
             int Sdx32 = Sdx;
             int Sdy32 = Sdy;
 
-            for (npy_int32 m = 0; m < M; ++m) {
+            for (npy_intp m = 0; m < M; ++m) {
                 for (npy_int32 n_idx = Dpp[m * Sdpp]; n_idx < Dpp[(m+1)*Sdpp]; ++n_idx) {
                     const npy_int32 n = Dpi[n_idx * Sdpi]; // row index of non-null value for column K
 


Test patch: