Repository: scipy/scipy

Issue:
Scipy sparse performance
Hi, 

I am the developer of GridCal, a program that makes extensive use of the sparse library of scipy.

There is a feature in the program called time-series power flow, which performs a series of calls to a Newton-Raphson based solver to simulate electrical circuits.

The profiling of this feature showed that the scipy function function [`get_index_dtype`](https://github.com/scipy/scipy/blob/master/scipy/sparse/sputils.py) was using the 9% of the simulation resources. This was way ahead of the resources taken by the linear system solver or the total amount of sparse matrix multiplications which are a lot.

This arose when testing different linear algebra libraries (Pardiso, KLU, SuperLU, etc...) 
The differences were negligible in my program, when in standalone benchmarks the libraries do have a noticeable performance difference. So the conclusion is that the bottleneck should be somewhere else.

I think there is room for improvement here.


Discussion:
Can you share a self-contained script that exhibits this behavior?
Well, the best I can do is to publish the test script:

```
from GridCal.Engine import *


def test():
    fname = os.path.join('..', '..', 'Grids_and_profiles', 'grids', 'IEEE39.gridcal')
    print('Reading...')
    main_circuit = FileOpen(fname).open()
    options = PowerFlowOptions(SolverType.NR, verbose=False,
                               initialize_with_existing_solution=False,
                               multi_core=False, dispatch_storage=True,
                               control_q=ReactivePowerControlMode.NoControl,
                               control_p=True)

    ############################################################
    # Time Series
    ############################################################
    print('Running TS...', '')
    start = time.time()

    ts = TimeSeries(grid=main_circuit, options=options)
    ts.run()

    end = time.time()
    dt = end - start
    print('  total', dt, 's')

if __name__ == '__main__':
    import cProfile
    cProfile.runctx('test()', None, locals())
```

You'll need to install GridCal `pip install GridCal`, and then you'll need [the test file](https://github.com/SanPen/GridCal/blob/master/Grids_and_profiles/grids/IEEE39.gridcal).


I seems to me that it could be beneficial to have a safety override. 

Once I know that the dimensions of my matrices are correct I can safely disable the dimensionality checks and alike.
Hi, 

I don't know if this is being reviewed, but I'll leave here the performance test from scipy. [Scipy_performance_test.csv.txt](https://github.com/scipy/scipy/files/3643210/Scipy_performance_test.csv.txt)

These are the top routines called by the process, sorted by "own time":


Name | Call count | Time(ms) | Own time(ms)
-- | -- | -- | --
check_format | 1034808 | 26615 | 9758
get_index_dtype | 3312174 | 20398 | 8845
method 'reduce' of 'numpy.ufunc' objects | 2618590 | 7158 | 7158
built-in method numpy.array | 16754393 | 6999 | 6963
_check | 519565 | 18529 | 6461
__init__ | 6624370 | 6256 | 6256
__init__ | 1034808 | 60367 | 4293
prune | 1080000 | 7514 | 3425
check_shape | 1667363 | 5202 | 3346
bmat | 67767 | 23876 | 2840
tocoo | 112963 | 10123 | 2489




Resolving patch:
diff --git a/scipy/sparse/_bsr.py b/scipy/sparse/_bsr.py
--- a/scipy/sparse/_bsr.py
+++ b/scipy/sparse/_bsr.py
@@ -228,13 +228,16 @@ def __init__(self, arg1, shape=None, dtype=None, copy=False, blocksize=None):
         self.check_format(full_check=False)
 
     def check_format(self, full_check=True):
-        """check whether the matrix format is valid
-
-            *Parameters*:
-                full_check:
-                    True  - rigorous check, O(N) operations : default
-                    False - basic check, O(1) operations
-
+        """Check whether the matrix respects the BSR format.
+
+        Parameters
+        ----------
+        full_check : bool, optional
+            If `True`, run rigorous check, scanning arrays for valid values.
+            Note that activating those check might copy arrays for casting,
+            modifying indices and index pointers' inplace.
+            If `False`, run basic checks on attributes. O(1) operations.
+            Default is `True`.
         """
         M,N = self.shape
         R,C = self.blocksize
@@ -247,11 +250,6 @@ def check_format(self, full_check=True):
             warn("indices array has non-integer dtype (%s)"
                     % self.indices.dtype.name)
 
-        idx_dtype = self._get_index_dtype((self.indices, self.indptr))
-        self.indptr = np.asarray(self.indptr, dtype=idx_dtype)
-        self.indices = np.asarray(self.indices, dtype=idx_dtype)
-        self.data = to_native(self.data)
-
         # check array shapes
         if self.indices.ndim != 1 or self.indptr.ndim != 1:
             raise ValueError("indices, and indptr should be 1-D")
@@ -285,6 +283,10 @@ def check_format(self, full_check=True):
                     raise ValueError("index pointer values must form a "
                                         "non-decreasing sequence")
 
+            idx_dtype = self._get_index_dtype((self.indices, self.indptr))
+            self.indptr = np.asarray(self.indptr, dtype=idx_dtype)
+            self.indices = np.asarray(self.indices, dtype=idx_dtype)
+            self.data = to_native(self.data)
         # if not self.has_sorted_indices():
         #    warn('Indices were not in sorted order. Sorting indices.')
         #    self.sort_indices(check_first=False)
diff --git a/scipy/sparse/_compressed.py b/scipy/sparse/_compressed.py
--- a/scipy/sparse/_compressed.py
+++ b/scipy/sparse/_compressed.py
@@ -133,13 +133,16 @@ def _set_self(self, other, copy=False):
         self._shape = check_shape(other.shape)
 
     def check_format(self, full_check=True):
-        """check whether the matrix format is valid
+        """Check whether the matrix respects the CSR or CSC format.
 
         Parameters
         ----------
         full_check : bool, optional
-            If `True`, rigorous check, O(N) operations. Otherwise
-            basic check, O(1) operations (default True).
+            If `True`, run rigorous check, scanning arrays for valid values.
+            Note that activating those check might copy arrays for casting,
+            modifying indices and index pointers' inplace.
+            If `False`, run basic checks on attributes. O(1) operations.
+            Default is `True`.
         """
         # use _swap to determine proper bounds
         major_name, minor_name = self._swap(('row', 'column'))
@@ -153,11 +156,6 @@ def check_format(self, full_check=True):
             warn("indices array has non-integer dtype ({})"
                  "".format(self.indices.dtype.name), stacklevel=3)
 
-        idx_dtype = self._get_index_dtype((self.indptr, self.indices))
-        self.indptr = np.asarray(self.indptr, dtype=idx_dtype)
-        self.indices = np.asarray(self.indices, dtype=idx_dtype)
-        self.data = to_native(self.data)
-
         # check array shapes
         for x in [self.data.ndim, self.indices.ndim, self.indptr.ndim]:
             if x != 1:
@@ -192,6 +190,11 @@ def check_format(self, full_check=True):
                     raise ValueError("index pointer values must form a "
                                      "non-decreasing sequence")
 
+            idx_dtype = self._get_index_dtype((self.indptr, self.indices))
+            self.indptr = np.asarray(self.indptr, dtype=idx_dtype)
+            self.indices = np.asarray(self.indices, dtype=idx_dtype)
+            self.data = to_native(self.data)
+
         # if not self.has_sorted_indices():
         #    warn('Indices were not in sorted order.  Sorting indices.')
         #    self.sort_indices()


Test patch: