Repository: scipy/scipy

Issue:
High overhead in lu_solve due to get_lapack_funcs
I'm using Scipy 0.17.0 and Numpy 1.11.0 on Python 3.4.3 on a 64 bit Gentoo system.

I need to apply the inverse of several moderately sized dense matrices repeatedly and thought that LU factorization is perfect for the job. However, the implementation in Scipy seems to suffer from significant overhead. The following benchmark demonstrates my issue:

```python
import numpy as np
import scipy.linalg
import cProfile

def bench_lu(n=100):
    A = np.random.rand(n,n)
    A_lu = scipy.linalg.lu_factor(A, check_finite=False)

    f = np.random.rand(n)
    for i in range(50000):
        f = scipy.linalg.lu_solve(A_lu, f, check_finite=False)

cProfile.run('bench_lu(n=100)', sort='tottime')
```

On my box, it produces the following output:

```
         2450029 function calls in 2.487 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   100002    0.760    0.000    1.000    0.000 numerictypes.py:942(_can_coerce_all)
    50000    0.692    0.000    2.418    0.000 decomp_lu.py:75(lu_solve)
    50001    0.243    0.000    1.642    0.000 blas.py:223(_get_funcs)
   700000    0.184    0.000    0.184    0.000 numerictypes.py:951(<listcomp>)
    50001    0.161    0.000    1.327    0.000 blas.py:177(find_best_blas_type)
    50001    0.082    0.000    1.119    0.000 numerictypes.py:964(find_common_type)
        1    0.067    0.067    2.487    2.487 numpy-benchmark.py:5(bench_lu)
   800003    0.056    0.000    0.056    0.000 {built-in method len}
    50001    0.035    0.000    0.050    0.000 numeric.py:414(asarray)
    50001    0.031    0.000    0.031    0.000 {built-in method isinstance}
    50001    0.031    0.000    0.031    0.000 numerictypes.py:1015(<listcomp>)
    50001    0.027    0.000    1.669    0.000 lapack.py:405(get_lapack_funcs)
    50001    0.026    0.000    0.026    0.000 blas.py:206(<listcomp>)
   100002    0.020    0.000    0.020    0.000 {method 'get' of 'dict' objects}
    50002    0.019    0.000    0.019    0.000 {built-in method getattr}
    50001    0.014    0.000    0.014    0.000 {built-in method array}
    50001    0.013    0.000    0.013    0.000 {method 'index' of 'list' objects}
    50001    0.008    0.000    0.008    0.000 {method 'append' of 'list' objects}
    50001    0.007    0.000    0.007    0.000 misc.py:169(_datacopied)
    50001    0.007    0.000    0.007    0.000 numerictypes.py:1016(<listcomp>)
        1    0.001    0.001    0.001    0.001 decomp_lu.py:17(lu_factor)
        2    0.000    0.000    0.000    0.000 {method 'rand' of 'mtrand.RandomState' objects}
        1    0.000    0.000    2.487    2.487 {built-in method exec}
        1    0.000    0.000    2.487    2.487 <string>:1(<module>)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
```

As you can see, more than half of the total time is spent in `get_lapack_funcs`, which is called by `lu_solve`. Of course, this is a per-call overhead for `lu_solve` and becomes less significant for larger matrices, but to me it seems excessively high for my test case of 100x100 matrices.


Discussion:
Under the hood `lu_solve` is using `?getrs`, which can be accessed directly using the Cython bindings for Lapack:

https://docs.scipy.org/doc/scipy/reference/linalg.cython_lapack.html

That would eliminate the overhead.

Also, if you are absolutely sure about your data type you can call the corresponding LAPACK routine yourself. 

```
def bench_lu(n=100):
    A = np.random.rand(n,n)
    A_lu, a_piv = scipy.linalg.lu_factor(A, check_finite=False)
    f = np.random.rand(n)
    for i in range(50000):
        x = scipy.linalg.lapack.dgetrs(A_lu, a_piv, f)
```

But this has no finiteness checks and size mismatch detections. So it is pretty fragile to use it like so.

get_lapack_funcs however probably can be sped up, since it's not doing 
anything complicated.

I'm not seeing this at all:

```
   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    50000  116.507    0.002  117.731    0.002 decomp_lu.py:75(lu_solve)
   100002    0.471    0.000    0.573    0.000 numerictypes.py:942(_can_coerce_all)
    50001    0.226    0.000    1.127    0.000 blas.py:226(_get_funcs)
    50001    0.157    0.000    0.830    0.000 blas.py:177(find_best_blas_type)
   800003    0.102    0.000    0.102    0.000 {len}
    50001    0.070    0.000    0.643    0.000 numerictypes.py:964(find_common_type)
        1    0.063    0.063  117.795  117.795 tmp14.py:5(bench_lu)
    50001    0.036    0.000    1.162    0.000 lapack.py:417(get_lapack_funcs)
    50001    0.035    0.000    0.050    0.000 numeric.py:463(asarray)
    50001    0.026    0.000    0.026    0.000 {isinstance}
```

And the line profiler results for 

    import numpy as np
    import scipy.linalg

    n = 100

    A = np.random.rand(n,n)
    A_lu = scipy.linalg.lu_factor(A, check_finite=False)

    f = np.random.rand(n)
    for i in range(1000):
        f = scipy.linalg.lu_solve(A_lu, f, check_finite=False)

are:


    $ kernprof.py -l -v tmp14.py
    Wrote profile results to tmp14.py.lprof
    Timer unit: 1e-06 s

    File: /home/rgommers/Code/scipy/scipy/linalg/decomp_lu.py
    Function: lu_solve at line 75
    Total time: 1.64857 s

    Line #      Hits         Time  Per Hit   % Time  Line Contents
    ==============================================================
        75                                           @profile
        76                                           def lu_solve(lu_and_piv, b, trans=0, overwrite_b=False, check_finite=True):
        77      1000          972      1.0      0.1      (lu, piv) = lu_and_piv
        78      1000          901      0.9      0.1      if check_finite:
        79                                                   b1 = asarray_chkfinite(b)
        80                                               else:
        81      1000         2131      2.1      0.1          b1 = asarray(b)
        82      1000         1941      1.9      0.1      overwrite_b = overwrite_b or _datacopied(b1, b)
        83      1000         1379      1.4      0.1      if lu.shape[0] != b1.shape[0]:
        84                                                   raise ValueError("incompatible dimensions.")
        85                                           
        86      1000        61475     61.5      3.7      getrs, = get_lapack_funcs(('getrs',), (lu, b1))
        87      1000      1577477   1577.5     95.7      x,info = getrs(lu, piv, b1, trans=trans, overwrite_b=overwrite_b)
        88      1000         1400      1.4      0.1      if info == 0:
        89      1000          895      0.9      0.1          return x
        90                                               raise ValueError('illegal value in %d-th argument of internal gesv|posv'
        91                                                                                                               % -info)


@rgommers Why does your program take 117s to execute, when mine ran in 2.4s?
no clue, would need to compare that between builds on that machine and my other one. maybe some lapack related issue. will do that later.
I have similar timings as reported. It is just quite inefficient implementation of `_can_corece_all`, right?
For get_blas_? I think the overhead might be justified but for the lapack funcs I think indeed this is an unnecessary effort to pick the right lapack routine. It can simply check the dtypes instead of jumping all the hoops.

Resolving patch:
diff --git a/benchmarks/benchmarks/blas_lapack.py b/benchmarks/benchmarks/blas_lapack.py
new file mode 100644
--- /dev/null
+++ b/benchmarks/benchmarks/blas_lapack.py
@@ -0,0 +1,38 @@
+from __future__ import division, absolute_import, print_function
+
+import numpy as np
+
+try:
+    import scipy.linalg.lapack as la
+    import scipy.linalg.blas as bla
+except ImportError:
+    pass
+
+from .common import Benchmark
+
+
+class GetBlasLapackFuncs(Benchmark):
+    """
+    Test the speed of grabbing the correct BLAS/LAPACK routine flavor.
+
+    In particular, upon receiving strange dtype arrays the results shouldn't
+    diverge too much. Hence the results here should be comparable
+    """
+
+    param_names = ['dtype1', 'dtype2',
+                   'dtype1_ord', 'dtype2_ord',
+                   'size']
+    params = [
+        ['b', 'G', 'd'],
+        ['d', 'F', '?']
+        ['C', 'F'],
+        ['C', 'F'],
+        [10, 100, 1000]
+    ]
+
+    def setup(self, dtype1, dtype2, dtype1_ord, dtype2_ord, size):
+        self.arr1 = np.empty(size, dtype=dtype1, order=dtype1_ord)
+        self.arr2 = np.empty(size, dtype=dtype2, order=dtype2_ord)
+
+    def time_find_best_blas_type(self, arr1, arr2):
+        prefix, dtype, prefer_fortran = bla.find_best_blas_type((arr1, arr2))
diff --git a/scipy/linalg/blas.py b/scipy/linalg/blas.py
--- a/scipy/linalg/blas.py
+++ b/scipy/linalg/blas.py
@@ -222,8 +222,25 @@
 from scipy.linalg._fblas import *
 del empty_module
 
-# 'd' will be default for 'i',..
-_type_conv = {'f': 's', 'd': 'd', 'F': 'c', 'D': 'z', 'G': 'z'}
+# all numeric dtypes '?bBhHiIlLqQefdgFDGO' that are safe to be converted to
+
+# single precision float   : '?bBhH!!!!!!ef!!!!!!'
+# double precision float   : '?bBhHiIlLqQefdg!!!!'
+# single precision complex : '?bBhH!!!!!!ef!!F!!!'
+# double precision complex : '?bBhHiIlLqQefdgFDG!'
+
+_type_score = {x: 1 for x in '?bBhHef'}
+_type_score.update({x: 2 for x in 'iIlLqQd'})
+
+# Handle float128(g) and complex256(G) separately in case non-windows systems.
+# On windows, the values will be rewritten to the same key with the same value.
+_type_score.update({'F': 3, 'D': 4, 'g': 2, 'G': 4})
+
+# Final mapping to the actual prefixes and dtypes
+_type_conv = {1: ('s', _np.dtype('float32')),
+              2: ('d', _np.dtype('float64')),
+              3: ('c', _np.dtype('complex64')),
+              4: ('z', _np.dtype('complex128'))}
 
 # some convenience alias for complex functions
 _blas_alias = {'cnrm2': 'scnrm2', 'znrm2': 'dznrm2',
@@ -270,26 +287,30 @@ def find_best_blas_type(arrays=(), dtype=None):
 
     """
     dtype = _np.dtype(dtype)
+    max_score = _type_score.get(dtype.char, 5)
     prefer_fortran = False
 
     if arrays:
-        # use the most generic type in arrays
-        dtypes = [ar.dtype for ar in arrays]
-        dtype = _np.find_common_type(dtypes, ())
-        try:
-            index = dtypes.index(dtype)
-        except ValueError:
-            index = 0
-        if arrays[index].flags['FORTRAN']:
-            # prefer Fortran for leading array with column major order
-            prefer_fortran = True
-
-    prefix = _type_conv.get(dtype.char, 'd')
-    if dtype.char == 'G':
-        # complex256 -> complex128 (i.e., C long double -> C double)
-        dtype = _np.dtype('D')
-    elif dtype.char not in 'fdFD':
-        dtype = _np.dtype('d')
+        # In most cases, single element is passed through, quicker route
+        if len(arrays) == 1:
+            max_score = _type_score.get(arrays[0].dtype.char, 5)
+            prefer_fortran = arrays[0].flags['FORTRAN']
+        else:
+            # use the most generic type in arrays
+            scores = [_type_score.get(x.dtype.char, 5) for x in arrays]
+            max_score = max(scores)
+            ind_max_score = scores.index(max_score)
+            # safe upcasting for mix of float64 and complex64 --> prefix 'z'
+            if max_score == 3 and (2 in scores):
+                max_score = 4
+
+            if arrays[ind_max_score].flags['FORTRAN']:
+                # prefer Fortran for leading array with column major order
+                prefer_fortran = True
+
+    # Get the LAPACK prefix and the corresponding dtype if not fall back
+    # to 'd' and double precision float.
+    prefix, dtype = _type_conv.get(max_score, ('d', _np.dtype('float64')))
 
     return prefix, dtype, prefer_fortran
 
@@ -318,7 +339,7 @@ def _get_funcs(names, arrays, dtype,
     if prefer_fortran:
         module1, module2 = module2, module1
 
-    for i, name in enumerate(names):
+    for name in names:
         func_name = prefix + name
         func_name = alias.get(func_name, func_name)
         func = getattr(module1[0], func_name, None)


Test patch: