Repository: scipy/scipy

Issue:
Wrap LAPACK.dgesvd for singular value decomposition
Currently, scipy only wraps dgesdd, which does essentially the same thing, but makes very mild assumptions about floating point arithmetic. At least in some cases, this approach fails and dgesvd works more robustly.
Also, it seems that other platforms (e.g. Matlab and Octave) use the dgesvd implementation.

http://www.netlib.org/lapack/explore-html/db/db4/dgesdd_8f.html

http://www.netlib.org/lapack/explore-html/d8/d2d/dgesvd_8f.html



Discussion:
The Cython LAPACK wrappers that are new in 0.16.x do wrap `dgesvd`.

The assumption:

```
The divide and conquer algorithm makes very mild assumptions about
 floating point arithmetic. It will work on machines with a guard
 digit in add/subtract, or on those binary machines without guard
 digits which subtract like the Cray X-MP, Cray Y-MP, Cray C-90, or
 Cray-2. It could conceivably fail on hexadecimal or decimal machines
 without guard digits, but we know of none.
```

Is that really a concern? Do you have an example where the assumption doesn't hold?

This is curious. I was trying to reproduce the problem I had before but it now seems that my results are essentially the same for both implementations. I'm not sure if something changed since I updated all relevant python packages. Sorry about this.

Regardless, when A has redundancy in the singular values, dgesvd and dgesdd return different representations of the subspaces (of course, the SVD solution is not unique in this case, so this is expected). However, it might be important to have dgesvd available in Numpy for cases where one would like to compare results to Matlab/Octave directly.

I did not find the wrapped _dgesvd_ function in 0.17 [SciPy](http://docs.scipy.org/doc/scipy-dev/reference/linalg.lapack.html#module-scipy.linalg.lapack) while _dgesdd_ is still there. Could you point me to the right place?

*gesvd is not wrapped on Python level on Scipy 0.17.dev. An ABI-stable
pointer is however exported to Cython, however:
http://docs.scipy.org/doc/scipy-dev/reference/linalg.cython_lapack.html
.
Adding the routine also to scipy.linalg.lapack would in principle be
straightforward; cf.
https://github.com/scipy/scipy/blob/master/scipy/linalg/flapack.pyf.src#L555
Due to historical reasons, these wrappers are not autogenerated, and do
not consequently cover all of LAPACK.

I'm hitting an issue with my code, which uses lots of SVD calls, that I get a fair number of `numpy.linalg.linalg.LinAlgError: SVD did not converge`'s. I'm using Anaconda with MKL, and have [opened an issue with the Anaconda folks](https://github.com/ContinuumIO/anaconda-issues/issues/695), but I don't know how long it will take them to sort it out.

In the meantime, I'd like to try adding an option to compute SVD using the alternate algorithm. We could add a `kwarg` to `linalg.svd(..., method='dgesdd' | 'dgesvd')` (or really something more human-readable) for it. WDYT? It would be a potential workaround for this issue, and expose the alternative algorithm for people who want to use it.

If that's a no-go, I'd like to at least modify `scipy`, if necessary, to make my own derivative version of `svd` (it's not many lines anyway) that would do it. I take it that adding it to `scipy.linalg.lapack`, as mentioned in @pv's most recent comment, would do it?


Resolving patch:
diff --git a/scipy/linalg/decomp_svd.py b/scipy/linalg/decomp_svd.py
--- a/scipy/linalg/decomp_svd.py
+++ b/scipy/linalg/decomp_svd.py
@@ -8,12 +8,13 @@
 from .misc import LinAlgError, _datacopied
 from .lapack import get_lapack_funcs, _compute_lwork
 from .decomp import _asarray_validated
+from scipy._lib.six import string_types
 
 __all__ = ['svd', 'svdvals', 'diagsvd', 'orth']
 
 
 def svd(a, full_matrices=True, compute_uv=True, overwrite_a=False,
-        check_finite=True):
+        check_finite=True, lapack_driver='gesdd'):
     """
     Singular Value Decomposition.
 
@@ -40,6 +41,13 @@ def svd(a, full_matrices=True, compute_uv=True, overwrite_a=False,
         Whether to check that the input matrix contains only finite numbers.
         Disabling may give a performance gain, but may result in problems
         (crashes, non-termination) if the inputs do contain infinities or NaNs.
+    lapack_driver : {'gesdd', 'gesvd'}, optional
+        Whether to use the more efficient divide-and-conquer approach
+        (``'gesdd'``) or general rectangular approach (``'gesvd'``)
+        to compute the SVD. MATLAB and Octave use the ``'gesvd'`` approach.
+        Default is ``'gesdd'``.
+
+        .. versionadded:: 0.18
 
     Returns
     -------
@@ -53,7 +61,7 @@ def svd(a, full_matrices=True, compute_uv=True, overwrite_a=False,
         Unitary matrix having right singular vectors as rows.
         Of shape ``(N,N)`` or ``(K,N)`` depending on `full_matrices`.
 
-    For ``compute_uv = False``, only `s` is returned.
+    For ``compute_uv=False``, only `s` is returned.
 
     Raises
     ------
@@ -88,24 +96,30 @@ def svd(a, full_matrices=True, compute_uv=True, overwrite_a=False,
     a1 = _asarray_validated(a, check_finite=check_finite)
     if len(a1.shape) != 2:
         raise ValueError('expected matrix')
-    m,n = a1.shape
+    m, n = a1.shape
     overwrite_a = overwrite_a or (_datacopied(a1, a))
 
-    gesdd, gesdd_lwork = get_lapack_funcs(('gesdd', 'gesdd_lwork'), (a1,))
+    if not isinstance(lapack_driver, string_types):
+        raise TypeError('lapack_driver must be a string')
+    if lapack_driver not in ('gesdd', 'gesvd'):
+        raise ValueError('lapack_driver must be "gesdd" or "gesvd", not "%s"'
+                         % (lapack_driver,))
+    funcs = (lapack_driver, lapack_driver + '_lwork')
+    gesXd, gesXd_lwork = get_lapack_funcs(funcs, (a1,))
 
     # compute optimal lwork
-    lwork = _compute_lwork(gesdd_lwork, a1.shape[0], a1.shape[1],
+    lwork = _compute_lwork(gesXd_lwork, a1.shape[0], a1.shape[1],
                            compute_uv=compute_uv, full_matrices=full_matrices)
 
     # perform decomposition
-    u,s,v,info = gesdd(a1, compute_uv=compute_uv, lwork=lwork,
-                       full_matrices=full_matrices, overwrite_a=overwrite_a)
+    u, s, v, info = gesXd(a1, compute_uv=compute_uv, lwork=lwork,
+                          full_matrices=full_matrices, overwrite_a=overwrite_a)
 
     if info > 0:
         raise LinAlgError("SVD did not converge")
     if info < 0:
         raise ValueError('illegal value in %d-th argument of internal gesdd'
-                                                                    % -info)
+                         % -info)
     if compute_uv:
         return u, s, v
     else:
@@ -158,7 +172,7 @@ def svdvals(a, overwrite_a=False, check_finite=True):
     a = _asarray_validated(a, check_finite=check_finite)
     if a.size:
         return svd(a, compute_uv=0, overwrite_a=overwrite_a,
-                check_finite=False)
+                   check_finite=False)
     elif len(a.shape) != 2:
         raise ValueError('expected matrix')
     else:
@@ -190,7 +204,7 @@ def diagsvd(s, M, N):
     if MorN == M:
         return r_['-1', part, zeros((M, N-M), typ)]
     elif MorN == N:
-        return r_[part, zeros((M-N,N), typ)]
+        return r_[part, zeros((M-N, N), typ)]
     else:
         raise ValueError("Length of s must be M or N.")
 
@@ -220,7 +234,7 @@ def orth(A):
     u, s, vh = svd(A, full_matrices=False)
     M, N = A.shape
     eps = numpy.finfo(float).eps
-    tol = max(M,N) * numpy.amax(s) * eps
+    tol = max(M, N) * numpy.amax(s) * eps
     num = numpy.sum(s > tol, dtype=int)
-    Q = u[:,:num]
+    Q = u[:, :num]
     return Q
diff --git a/scipy/linalg/lapack.py b/scipy/linalg/lapack.py
--- a/scipy/linalg/lapack.py
+++ b/scipy/linalg/lapack.py
@@ -135,6 +135,16 @@
    cgesdd_lwork
    zgesdd_lwork
 
+   sgesvd
+   dgesvd
+   cgesvd
+   zgesvd
+
+   sgesvd_lwork
+   dgesvd_lwork
+   cgesvd_lwork
+   zgesvd_lwork
+
    sgesv
    dgesv
    cgesv


Test patch:
diff --git a/scipy/linalg/tests/test_decomp.py b/scipy/linalg/tests/test_decomp.py
--- a/scipy/linalg/tests/test_decomp.py
+++ b/scipy/linalg/tests/test_decomp.py
@@ -787,14 +787,20 @@ def test_check_finite(self):
         assert_array_almost_equal(x1,x2)
 
 
-class TestSVD(TestCase):
+class TestSVD_GESDD(TestCase):
     def setUp(self):
+        self.lapack_driver = 'gesdd'
         seed(1234)
 
+    def test_degenerate(self):
+        assert_raises(TypeError, svd, [[1.]], lapack_driver=1.)
+        assert_raises(ValueError, svd, [[1.]], lapack_driver='foo')
+
     def test_simple(self):
         a = [[1,2,3],[1,20,3],[2,5,6]]
         for full_matrices in (True, False):
-            u,s,vh = svd(a, full_matrices=full_matrices)
+            u,s,vh = svd(a, full_matrices=full_matrices,
+                         lapack_driver=self.lapack_driver)
             assert_array_almost_equal(dot(transpose(u),u),identity(3))
             assert_array_almost_equal(dot(transpose(vh),vh),identity(3))
             sigma = zeros((u.shape[0],vh.shape[0]),s.dtype.char)
@@ -805,7 +811,8 @@ def test_simple(self):
     def test_simple_singular(self):
         a = [[1,2,3],[1,2,3],[2,5,6]]
         for full_matrices in (True, False):
-            u,s,vh = svd(a, full_matrices=full_matrices)
+            u,s,vh = svd(a, full_matrices=full_matrices,
+                         lapack_driver=self.lapack_driver)
             assert_array_almost_equal(dot(transpose(u),u),identity(3))
             assert_array_almost_equal(dot(transpose(vh),vh),identity(3))
             sigma = zeros((u.shape[0],vh.shape[0]),s.dtype.char)
@@ -816,7 +823,8 @@ def test_simple_singular(self):
     def test_simple_underdet(self):
         a = [[1,2,3],[4,5,6]]
         for full_matrices in (True, False):
-            u,s,vh = svd(a, full_matrices=full_matrices)
+            u,s,vh = svd(a, full_matrices=full_matrices,
+                         lapack_driver=self.lapack_driver)
             assert_array_almost_equal(dot(transpose(u),u),identity(u.shape[0]))
             sigma = zeros((u.shape[0],vh.shape[0]),s.dtype.char)
             for i in range(len(s)):
@@ -826,7 +834,8 @@ def test_simple_underdet(self):
     def test_simple_overdet(self):
         a = [[1,2],[4,5],[3,4]]
         for full_matrices in (True, False):
-            u,s,vh = svd(a, full_matrices=full_matrices)
+            u,s,vh = svd(a, full_matrices=full_matrices,
+                         lapack_driver=self.lapack_driver)
             assert_array_almost_equal(dot(transpose(u),u), identity(u.shape[1]))
             assert_array_almost_equal(dot(transpose(vh),vh),identity(2))
             sigma = zeros((u.shape[1],vh.shape[0]),s.dtype.char)
@@ -840,7 +849,8 @@ def test_random(self):
         for i in range(3):
             for a in [random([n,m]),random([m,n])]:
                 for full_matrices in (True, False):
-                    u,s,vh = svd(a, full_matrices=full_matrices)
+                    u,s,vh = svd(a, full_matrices=full_matrices,
+                                 lapack_driver=self.lapack_driver)
                     assert_array_almost_equal(dot(transpose(u),u),identity(u.shape[1]))
                     assert_array_almost_equal(dot(vh, transpose(vh)),identity(vh.shape[0]))
                     sigma = zeros((u.shape[1],vh.shape[0]),s.dtype.char)
@@ -851,7 +861,8 @@ def test_random(self):
     def test_simple_complex(self):
         a = [[1,2,3],[1,2j,3],[2,5,6]]
         for full_matrices in (True, False):
-            u,s,vh = svd(a, full_matrices=full_matrices)
+            u,s,vh = svd(a, full_matrices=full_matrices,
+                         lapack_driver=self.lapack_driver)
             assert_array_almost_equal(dot(conj(transpose(u)),u),identity(u.shape[1]))
             assert_array_almost_equal(dot(conj(transpose(vh)),vh),identity(vh.shape[0]))
             sigma = zeros((u.shape[0],vh.shape[0]),s.dtype.char)
@@ -866,7 +877,8 @@ def test_random_complex(self):
             for full_matrices in (True, False):
                 for a in [random([n,m]),random([m,n])]:
                     a = a + 1j*random(list(a.shape))
-                    u,s,vh = svd(a, full_matrices=full_matrices)
+                    u,s,vh = svd(a, full_matrices=full_matrices,
+                                 lapack_driver=self.lapack_driver)
                     assert_array_almost_equal(dot(conj(transpose(u)),u),identity(u.shape[1]))
                     # This fails when [m,n]
                     # assert_array_almost_equal(dot(conj(transpose(vh)),vh),identity(len(vh),dtype=vh.dtype.char))
@@ -882,11 +894,11 @@ def test_crash_1580(self):
             for dt in [np.float32, np.float64, np.complex64, np.complex128]:
                 a = np.random.rand(*sz).astype(dt)
                 # should not crash
-                svd(a)
+                svd(a, lapack_driver=self.lapack_driver)
 
     def test_check_finite(self):
         a = [[1,2,3],[1,20,3],[2,5,6]]
-        u,s,vh = svd(a, check_finite=False)
+        u,s,vh = svd(a, check_finite=False, lapack_driver=self.lapack_driver)
         assert_array_almost_equal(dot(transpose(u),u),identity(3))
         assert_array_almost_equal(dot(transpose(vh),vh),identity(3))
         sigma = zeros((u.shape[0],vh.shape[0]),s.dtype.char)
@@ -909,7 +921,13 @@ def test_gh_5039(self):
              [0., 0.16666667, 0.66666667, 0.16666667, 0., 0.],
              [0., 0., 0.16666667, 0.66666667, 0.16666667, 0.],
              [0., 0., 0., 0.16666667, 0.66666667, 0.16666667]])
-        svd(b)
+        svd(b, lapack_driver=self.lapack_driver)
+
+
+class TestSVD_GESVD(TestSVD_GESDD):
+    def setUp(self):
+        self.lapack_driver = 'gesvd'
+        seed(1234)
 
 
 class TestSVDVals(TestCase):
@@ -2295,6 +2313,7 @@ def test_schur_complex(self):
 
     def test_svd(self):
         assert_no_overwrite(svd, [(3,3)])
+        assert_no_overwrite(lambda a: svd(a, lapack_driver='gesvd'), [(3,3)])
 
     def test_svdvals(self):
         assert_no_overwrite(svdvals, [(3,3)])