Repository: scipy/scipy

Issue #11255: Inefficient outer indexing on sparse matrices when using indices of shape `(N, 1)`
Created: 2019-12-20T13:58:28Z
A typical way to perform outer indexing can be unnecessarily and surprisingly super slow. It is illustrated in the in/out below.
I think the issue could be fixed by adding another fast path for input as in my In [8] to scipy._index.IndexMixin, in a similar manner to how it's done when using the method of In [10].
Reproducing code example:
In [1]: import scipy.sparse as ss, numpy as np                                                                                       

In [2]: matrix = ss.random(2000, 50000, format="csr")                                                                                

In [3]: idx0 = np.arange(1000)[:, None]                                                                                              

In [4]: idx1 = np.arange(1000, 2000)                                                                                                 

In [5]: imod = np.arange(25)[None, :]                                                                                                

In [6]: idx1 = (idx1[:, None] + 1000 * imod).T.reshape(-1, 1)                                                                        

In [8]: # The 'easy' way to index: also really slow 
   ...: %timeit out0 = matrix[idx0, idx1.T]                                                                                          
743 ms � 48 ms per loop (mean � std. dev. of 7 runs, 1 loop each)

In [9]: # Equivalent submatrix, but not so easy to write: Much faster 
   ...: %timeit out1 = matrix[idx0.ravel(), :][:, idx1.ravel()]                                                                      
3.57 ms � 89.6 �s per loop (mean � std. dev. of 7 runs, 100 loops each)

In [10]: # There is an existing fast path when the second index is 1D 
    ...: %timeit out2 = matrix[idx0, idx1.ravel()]                                                                                   
3.47 ms � 113 �s per loop (mean � std. dev. of 7 runs, 100 loops each)
Scipy/Numpy/Python version information:
In [40]: import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info)                                     
1.3.0 1.16.4 sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0)

Duplicate issue #12015: Reordering of CSC matrix breaks when you go above int32 limits
Created: 2020-05-04T03:07:24Z
I am trying to solve a linear system as effeciently as possible, and one of the methods I have been trying is by reordering the matrix beforehand with reverse_cuthill_mckee.
This works well for small matrices, but when I move to a large size this breaks. (60000x60000)
The code to reproduce this is really simple:
perm = scipy.sparse.csgraph.reverse_cuthill_mckee(H, symmetric_mode=True)
I,J = np.ix_(perm,perm)
H_reordered = H[I,J]

Where H is a symmetric 60000x60000 sparse csc matrix with some nonzero elements. In my case it has about 7 million nonzero elements
When I run the above I get:
Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2020.1\plugins\python-ce\helpers\pydev\pydevd.py", line 1438, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2020.1\plugins\python-ce\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "D:/Dropbox/ActiveLearning/Pytorch/Active-Learning/pcg.py", line 37, in <module>
    H_ref = H[I,J]
  File "D:\Dropbox\ActiveLearning\Pytorch\Active-Learning\venv\lib\site-packages\scipy\sparse\_index.py", line 75, in __getitem__
    return self._get_arrayXarray(row, col)
  File "D:\Dropbox\ActiveLearning\Pytorch\Active-Learning\venv\lib\site-packages\scipy\sparse\compressed.py", line 664, in _get_arrayXarray
    csr_sample_values(M, N, self.indptr, self.indices, self.data,
ValueError: could not convert integer scalar

And the reason as far as I can see is that the reordering involves making a 60000x60000 ndarray, which has about 3.6*10^9 elements and hence is too big int32.
Does anyone know of a workaround for this problem? or a fix?

Closing comment (2020-05-20T16:03:13Z): This is a duplicate of #11255 and is fixed in master.
On master:
In [1]: import numpy as np

In [2]: from scipy import sparse

In [3]: %load_ext memory_profiler

In [4]: s = sparse.random(10000, 10000, format='csr')

In [5]: idx = np.arange(10000)

In [6]: %memit s[idx.reshape(-1, 1), idx]
peak memory: 100.10 MiB, increment: 11.83 MiB

In [7]: %memit s[np.ix_(idx, idx)]
peak memory: 100.10 MiB, increment: 0.00 MiB

In [8]: s[idx.reshape(-1, 1), idx] == s[np.ix_(idx, idx)]
/home/ewm/venv/scipy/lib/python3.8/site-packages/IPython/core/interactiveshell.py:3331: SparseEfficiencyWarning: Comparing sparse matrices using == is inefficient, try using != instead.
  exec(code_obj, self.user_global_ns, self.user_ns)
Out[8]:
<10000x10000 sparse matrix of type '<class 'numpy.bool_'>'
        with 100000000 stored elements in Compressed Sparse Row format>

In [9]: np.all(_8.data)
Out[9]: True

The fix was #11335.