Repository: scipy/scipy

Issue:
Sparse pivot for linalg.lu
Currently `p, l, u = scipy.linalg.lu(a)` returns a dense permutation matrix `p` for the pivots of shape `(M, M)` for `a` of shape `(M, N)`. If the input matrix is tall and skinny (`M` >> `N`), then the storage costs of `p` are unacceptable. 

A workaround is to make LAPACK calls directly, e.g. use `linalg.lapack.dgetrf(a)`, since the LAPACK implementation of the LU factorization returns a 1-d integer array for the pivots. 

It seems that `linalg.lu` uses `linalg/src/lu.f` rather than the LAPACK implementation. Is this faster than the LAPACK implementation? Either way, I suggest to add an option to return an 1-d integer array for the pivots. If this option is invoked, then the appropriate LAPACK routine is called. This should be very easy to implement, shall I make a PR?
Sparse pivot for linalg.lu
Currently `p, l, u = scipy.linalg.lu(a)` returns a dense permutation matrix `p` for the pivots of shape `(M, M)` for `a` of shape `(M, N)`. If the input matrix is tall and skinny (`M` >> `N`), then the storage costs of `p` are unacceptable. 

A workaround is to make LAPACK calls directly, e.g. use `linalg.lapack.dgetrf(a)`, since the LAPACK implementation of the LU factorization returns a 1-d integer array for the pivots. 

It seems that `linalg.lu` uses `linalg/src/lu.f` rather than the LAPACK implementation. Is this faster than the LAPACK implementation? Either way, I suggest to add an option to return an 1-d integer array for the pivots. If this option is invoked, then the appropriate LAPACK routine is called. This should be very easy to implement, shall I make a PR?


Discussion:
If you only need the permutation you can use `scipy.linalg.lu_factor` which is a bit more labourous to use the L, U matrices but it is meant to be used with `scipy.linalg.lu_solve`
`scipy.linalg.lu_factor` is only for square matrices. My use case is precisely tall skinny matrices.
Also, it seems `dgetrf` is faster than `scipy.linalg.lu`, especially for tall matrices. For square matrices (in simple benchmarks) it is a bit faster, but not by much. 
That is a bug if `lu_factor` only works with square arrays and we have to fix it. dgetrs should require squareness not the trf
Right, `lu_factor` calls `getrf`, but it explicitly checks for a square matrix,
I would suggest removing the the check, and updating the documentation. It should also be a little clearer in the documentation what the difference is between `lu_factor` and `lu`. 
For `lu_factor` one can say something along the likes of
"Unlike `scipy.linalg.lu`, this function returns L and U stored in the same matrix LU, with U the upper triangular part, and L the lower triangular part with ones on the diagonal. Furthermore the permutation matrix P is returned as a 1D integer array instead of a  dense matrix"

Shall I make a PR?
That would much be appreciated if you can indeed. Thanks @RikVoorhaar 
If you only need the permutation you can use `scipy.linalg.lu_factor` which is a bit more labourous to use the L, U matrices but it is meant to be used with `scipy.linalg.lu_solve`
`scipy.linalg.lu_factor` is only for square matrices. My use case is precisely tall skinny matrices.
Also, it seems `dgetrf` is faster than `scipy.linalg.lu`, especially for tall matrices. For square matrices (in simple benchmarks) it is a bit faster, but not by much. 
That is a bug if `lu_factor` only works with square arrays and we have to fix it. dgetrs should require squareness not the trf
Right, `lu_factor` calls `getrf`, but it explicitly checks for a square matrix,
I would suggest removing the the check, and updating the documentation. It should also be a little clearer in the documentation what the difference is between `lu_factor` and `lu`. 
For `lu_factor` one can say something along the likes of
"Unlike `scipy.linalg.lu`, this function returns L and U stored in the same matrix LU, with U the upper triangular part, and L the lower triangular part with ones on the diagonal. Furthermore the permutation matrix P is returned as a 1D integer array instead of a  dense matrix"

Shall I make a PR?
That would much be appreciated if you can indeed. Thanks @RikVoorhaar 

Resolving patch:
diff --git a/scipy/linalg/_decomp_lu.py b/scipy/linalg/_decomp_lu.py
--- a/scipy/linalg/_decomp_lu.py
+++ b/scipy/linalg/_decomp_lu.py
@@ -25,7 +25,7 @@ def lu_factor(a, overwrite_a=False, check_finite=True):
 
     Parameters
     ----------
-    a : (M, M) array_like
+    a : (M, N) array_like
         Matrix to decompose
     overwrite_a : bool, optional
         Whether to overwrite data in A (may increase performance)
@@ -36,7 +36,7 @@ def lu_factor(a, overwrite_a=False, check_finite=True):
 
     Returns
     -------
-    lu : (N, N) ndarray
+    lu : (M, N) ndarray
         Matrix containing U in its upper triangle, and L in its lower triangle.
         The unit diagonal elements of L are not stored.
     piv : (N,) ndarray
@@ -45,11 +45,14 @@ def lu_factor(a, overwrite_a=False, check_finite=True):
 
     See Also
     --------
+    lu : gives lu factorization in more user-friendly format
     lu_solve : solve an equation system using the LU factorization of a matrix
 
     Notes
     -----
-    This is a wrapper to the ``*GETRF`` routines from LAPACK.
+    This is a wrapper to the ``*GETRF`` routines from LAPACK. Unlike
+    :func:`lu`, it outputs the L and U factors into a single array
+    and returns pivot indices instead of a permutation matrix.
 
     Examples
     --------
@@ -71,8 +74,6 @@ def lu_factor(a, overwrite_a=False, check_finite=True):
         a1 = asarray_chkfinite(a)
     else:
         a1 = asarray(a)
-    if len(a1.shape) != 2 or (a1.shape[0] != a1.shape[1]):
-        raise ValueError('expected square matrix')
     overwrite_a = overwrite_a or (_datacopied(a1, a))
     getrf, = get_lapack_funcs(('getrf',), (a1,))
     lu, piv, info = getrf(a1, overwrite_a=overwrite_a)


Test patch:
diff --git a/scipy/linalg/tests/test_decomp.py b/scipy/linalg/tests/test_decomp.py
--- a/scipy/linalg/tests/test_decomp.py
+++ b/scipy/linalg/tests/test_decomp.py
@@ -28,7 +28,8 @@
                           eigh_tridiagonal, null_space, cdf2rdf, LinAlgError)
 
 from scipy.linalg.lapack import (dgbtrf, dgbtrs, zgbtrf, zgbtrs, dsbev,
-                                 dsbevd, dsbevx, zhbevd, zhbevx)
+                                 dsbevd, dsbevx, zhbevd, zhbevx,
+                                 get_lapack_funcs)
 
 from scipy.linalg._misc import norm
 from scipy.linalg._decomp_qz import _select_function
@@ -941,7 +942,15 @@ def _test_common(self, data):
         pl, u = lu(data, permute_l=1)
         assert_array_almost_equal(pl @ u, data)
 
-    # Simple tests
+    def _test_common_lu_factor(self, data):
+        l_and_u1, piv1 = lu_factor(data)
+        (getrf,) = get_lapack_funcs(("getrf",), (data,))
+        l_and_u2, piv2, _ = getrf(data, overwrite_a=False)
+        assert_array_equal(l_and_u1, l_and_u2)
+        assert_array_equal(piv1, piv2)
+
+    # Simple tests.
+    # For lu_factor gives a LinAlgWarning because these matrices are singular
     def test_simple(self):
         self._test_common(self.a)
 
@@ -957,24 +966,30 @@ def test_simple2_complex(self):
     # rectangular matrices tests
     def test_hrectangular(self):
         self._test_common(self.hrect)
+        self._test_common_lu_factor(self.hrect)
 
     def test_vrectangular(self):
         self._test_common(self.vrect)
+        self._test_common_lu_factor(self.vrect)
 
     def test_hrectangular_complex(self):
         self._test_common(self.chrect)
+        self._test_common_lu_factor(self.chrect)
 
     def test_vrectangular_complex(self):
         self._test_common(self.cvrect)
+        self._test_common_lu_factor(self.cvrect)
 
     # Bigger matrices
     def test_medium1(self):
         """Check lu decomposition on medium size, rectangular matrix."""
         self._test_common(self.med)
+        self._test_common_lu_factor(self.med)
 
     def test_medium1_complex(self):
         """Check lu decomposition on medium size, rectangular matrix."""
         self._test_common(self.cmed)
+        self._test_common_lu_factor(self.cmed)
 
     def test_check_finite(self):
         p, l, u = lu(self.a, check_finite=False)