Repository: scipy/scipy

Issue:
new feature: adding LOBPCG solver in svds in addition to ARPACK
### Reproducing code example:
https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.svds.html is a 

> naive implementation using ARPACK as an eigensolver on A.H * A or A * A.H, depending on which one is more efficient.

It is simple and beneficial to add an alternative to ARPACK solver https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.lobpcg.html again just on A.H * A or A * A.H, depending on which one is more efficient. It is demonstrated in MATLAB that LOBPCG may outperform ARPACK for such a use, see comments at https://www.mathworks.com/matlabcentral/fileexchange/48-lobpcg-m

### Error message:
N/A

### Scipy/Numpy/Python version information:
N/A




Discussion:
@rc @pvanmulbregt 
This is now done in scikit-learn and can probably be just mostly copy/pasted from https://github.com/lobpcg/scikit-learn/blob/lobpcg_svd/sklearn/utils/extmath.py to scipy... 
It works well for dense truncated_svd too.

@lobpcg You can also open a PR which might be a bit more focused. 
@lobpcg Let's see how the scikit-learn version review goes. I will not be of much use during the weekend anyway...
@rc would you like to take the lead on this one as PR https://github.com/scipy/scipy/pull/9650 is near completion? 
@lobpcg I will be mostly offline till the end of the week, but afterwards it could be possible. To refresh my memory, it will be mostly copying code from the sklearn version, right?
@rc Yes, copy/past from https://github.com/lobpcg/scikit-learn/blob/lobpcg_svd/sklearn/utils/extmath.py to scipy. But one also needs to add the tests, although again mostly copy/past existing tests for SVD via ARPACK. Not too much work for such a significant contribution.

I could have done the editing myself, but setting up yet another environment for contributing is quite a pain for me. I would be glad to review and help you if any new issues arise in testing. 
PR #9650 is now merged to the master
Thanks @lobpcg, all. Closing.
@rgommers you closed the wrong PR! Please reopen this one and close PR #9650
@rgommers as well as the  [1.3.0](/scipy/scipy/milestone/37) milestone...
Don't think so - this is an issue, not a PR. PR gh-9650 is merged (which is its final status, it doesn't need closing) and will be present in the next release. This issue is about the same thing right, and gh-9650 implements what is suggested in this issue? Hence this one can be closed (or is there anything left to do?).
Sorry for the confusion. PR [gh-9650](https://github.com/scipy/scipy/pull/9650) does not address any open issues - it is just a bug-fix release of lobpcg for eigenvalue problems. There was no issue to close with this PR as the PR is now being added to the master - I was incorrect in this respect.

The present issue is to add lobpcg as a new SVD solver. It depends on   PR [gh-9650](https://github.com/scipy/scipy/pull/9650) but the work is not yet even started. Could you please reopen it and remove it from the [1.3.0](/scipy/scipy/milestone/37) milestone, to play it safe?
Ah okay, got it - thanks @lobpcg, reopening:)
Here is how this was done in https://github.com/scikit-learn/scikit-learn/pull/12319/ :
```
def lobpcg_svd(M, n_components, n_oversamples=10, n_iter='auto',
               transpose='auto', flip_sign=True, random_state=0,
               tol=None, explicitNormalMatrix=None):
    """Computes a truncated SVD using LOBPCG to accelerate the randomized SVD.
    Compared to 'randomised', the 'lobpcg' option gives more accurate
    approximations, with the same n_iter, n_components, and n_oversamples,
    at the slightly increased costs, allows setting the tolerance, and can
    output the accuracy. tol = None or tol = .0 in 'lobpcg' is ignored and
    substituted by a local default in LOBPCG.
    Parameters
    ----------
    M : ndarray or sparse matrix
        Matrix to decompose, real or complex.
    n_components : int
        Number of singular values and vectors to extract.
    n_oversamples : int (default is 10)
        Additional number of random vectors to sample the range of M so as
        to ensure proper conditioning. The total number of random vectors
        used to find the range of M is n_components + n_oversamples. Smaller
        number can improve speed but can negatively impact the quality of
        approximation of singular vectors and singular values.
    n_iter : int or 'auto' (default is 'auto')
        Number of lobpcg iterations. It can be used to deal with very noisy
        problems. When 'auto', it is set to 4, unless `n_components` is small
        (< .1 * min(X.shape)) `n_iter` in which case is set to 7.
        This improves precision with few components.
    transpose : True, False or 'auto' (default)
        Whether the algorithm should be applied to M.T instead of M. The
        result should approximately be the same. The 'auto' mode will
        trigger the transposition if M.shape[0] > M.shape[1] since this
        leads to the normal matrix of the smaller size and thus runs faster.
    flip_sign : boolean, (True by default)
        The output of a singular value decomposition is only unique up to a
        permutation of the signs of the singular vectors. If `flip_sign` is
        set to `True`, the sign ambiguity is resolved by making the largest
        loadings for each component in the left singular vectors positive.
    random_state : int, RandomState instance or None, optional (default=None)
        The seed of the pseudo random number generator to use when shuffling
        the data.  If int, random_state is the seed used by the random number
        generator; If RandomState instance, random_state is the random number
        generator; If None, the random number generator is the RandomState
        instance used by `np.random`.
    tol : scalar, (default=None)
        Optional solver tolerance (stopping criterion), if large enough, may
        overwrite n_iter. If None, lobpcg sets is internally.
    explicitNormalMatrix : boolean, (default=None)
        Optional parameter that determines if the normal matrix used by lobpcg
        is computed explicitly or implicitly via LinearOperator performing
        multiplication of the normal matrix and a vector. The latter may be
        faster for data matrices M of large sizes or sparse.
    Notes
    -----
    LOBPCG solver may become numerically unstable if the requested tolerance
    is unreasonably small and the maximal number of iterations is large.
    References
    ----------
    Toward the Optimal Preconditioned Eigensolver: Locally Optimal Block
    Preconditioned Conjugate Gradient Method, Andrew V. Knyazev (2001)
    https://doi.org/10.1137%2FS1064827500366124
    """
    if isinstance(M, (sparse.lil_matrix, sparse.dok_matrix)):
        warnings.warn("Calculating SVD of a {} is expensive. "
                      "csr_matrix is more efficient.".format(
                          type(M).__name__),
                      sparse.SparseEfficiencyWarning)

    random_state = check_random_state(random_state)
    n_random = n_components + n_oversamples
    n_samples, n_features = M.shape

    if n_iter == 'auto':
        # Checks if the number of iterations is explicitly specified
        # Adjust n_iter. 7 was found a good compromise for PCA. See #5299
        n_iter = 7 if n_components < .1 * min(M.shape) else 4

    if transpose == 'auto':
        # Make the normal matrix A with the smallest size
        transpose = n_samples > n_features
    if transpose:
        M = M.T.conj()

    Q = random_state.normal(size=(M.shape[0], n_random))
    if M.dtype.kind == 'f':
        # Ensure f32 is preserved as f32
        Q = Q.astype(M.dtype, copy=False)

    # The values are chosen experimentally
    if explicitNormalMatrix is None:
        if sparse.issparse(M):
            explicitNormalMatrix = False
        elif min(M.shape) > 4000 or min(M.shape)/max(M.shape) > 0.5:
            explicitNormalMatrix = False
        else:
            # Rectangular and small-size data matrix M
            explicitNormalMatrix = True

    # Determine the normal matrix
    if explicitNormalMatrix:
        A = safe_sparse_dot(M, M.T.conj())
    else:
        MLO = aslinearoperator(M)

        if hasattr(MLO, 'H'):

            def _matvec(V):
                return MLO(MLO.H(V))

        else:  # Old SciPy versions.
            MTLO = aslinearoperator(M.T.conj())

            def _matvec(V):
                return MLO(MTLO(V))

        Ms0 = M.shape[0]
        A = LinearOperator(dtype=M.dtype, shape=(Ms0, Ms0),
                           matvec=_matvec, matmat=_matvec)

    # For lobpcg debugging, use verbosityLevel = 1
    lobpcgVerbosityLevel = 0
    # lobpcg computes largest, be default, eigenvalues of the normal matrix
    # A, given implicitly via LinearOperator or explicitly as dense or sparse
    _, Q = lobpcg(A, Q, maxiter=n_iter,
                  verbosityLevel=lobpcgVerbosityLevel, tol=tol)
    del A

    # Project M to the (k + p) dimensional space using the basis vectors
    B = safe_sparse_dot(Q.T.conj(), M)

    # Compute the SVD on the thin matrix: (k + p) wide
    Uhat, s, V = linalg.svd(B, full_matrices=False)

    del B
    U = np.dot(Q, Uhat)

    if flip_sign:
        if not transpose:
            U, V = svd_flip(U, V)
        else:
            # In case of transpose u_based_decision=false
            # to actually flip based on u and not v.
            U, V = svd_flip(U, V, u_based_decision=False)

    if transpose:
        # Transpose back the results according to the input convention
        return (V[:n_components, :].T.conj(), s[:n_components],
                U[:, :n_components].T.conj())
    else:
        return U[:, :n_components], s[:n_components], V[:n_components, :]
```
> I do not have a setup for contributing to SciPy

Hi @lobpcg, if the above is still the case, I'm interested in trying to add in the new feature! I do have a local scipy setup and am definitely interested in trying this out.


> 
> 
> > I do not have a setup for contributing to SciPy
> 
> Hi @lobpcg, if the above is still the case, I'm interested in trying to add in the new feature! I do have a local scipy setup and am definitely interested in trying this out.

@clementkng Of course, please feel free. It is actually involved, not just copy/paste. 
@lobpcg Has something changed since when you commented it was mainly copy/paste code and tests? What else do we need to consider?
Nothing has changed: copy/paste, tests, and edit to make everything (input/output, tolerance, tests) is compatible with other SVD solvers. There may be many tests for SVD solvers there - I have not checked how many. And update the documentation to reflect the changes.
@lobpcg Is the current Scipy [LOBPCG solver](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.lobpcg.html) reliable to use? I'm confused since you mention I have to copy/paste from the scikit-learn implementation, but if the solver already exists in Scipy, then we would use that, right?
Copy/paste the SVD interface https://github.com/scipy/scipy/issues/9275#issuecomment-504673772 , to the current Scipy LOBPCG eigenvalue solver. 

@lobpcg I've currently pasted the implementation as well as the [test](https://github.com/scikit-learn/scikit-learn/blob/6ef8dadd4cb19c4fd39ae4e98904846528a9a28e/sklearn/utils/tests/test_extmath.py#L88) in scikit-learn that are relevant to the solver. I'm getting a couple of mismatch errors that I'm currently trying to fix, but looking through the version there and the interface you provided, I noticed that all the matrix transposes in the scikit-learn PR additionally have calls to `.conj()` attached to them in this version. Is there a reason for this distinction? 
> 
> 
> @lobpcg I've currently pasted the implementation as well as the [test](https://github.com/scikit-learn/scikit-learn/blob/6ef8dadd4cb19c4fd39ae4e98904846528a9a28e/sklearn/utils/tests/test_extmath.py#L88) in scikit-learn that are relevant to the solver. I'm getting a couple of mismatch errors that I'm currently trying to fix, but looking through the version there and the interface you provided, I noticed that all the matrix transposes in the scikit-learn PR additionally have calls to `.conj()` attached to them in this version. Is there a reason for this distinction?

 `.conj()` is a must to have in this code in scipy to handle complex-valued input, which is necessary in scipy; and should come with a unit test where the input matrix is complex-valued. However, scikit-learn simply does not support complex numbers. For real-valued numbers,  `.conj()` has no effect and does not hurt, but scikit-learn maintainers still have removed it recently for better style and consistency with scikit-learn. 
@lobpcg To make everything work w/ existing SVD solvers, we need to make sure the interface to both functions is the same. To do that, we would need to pick defaults for arguments specific to LOBPCG i.e. `n_oversamples`, and figure out how to add existing arguments not supported in LOBPCG but elsewhere i.e. `ncv` in the `arpack` solver. Does this sound like a good plan? I can push what I have after trying to fiddle w/ the tests if this isn't clear. 

Regarding the first point, is there a good way to determine those argument defaults?    
Consistent doesn't mean identical.

I recommend defaulting n_oversamples = 10 which to my recollection  is the choice in scikit-learn. Rename n_oversamples into ncv, if you want having the same names.

-------- Original message --------
From: Clement Ng <notifications@github.com>
Date: 7/11/19 3:39 PM (GMT-08:00)
To: scipy/scipy <scipy@noreply.github.com>
Cc: Andrew Knyazev <andrew.knyazev@ucdenver.edu>, Mention <mention@noreply.github.com>
Subject: Re: [scipy/scipy] new feature: adding LOBPCG solver in svds in   addition to ARPACK (#9275)


@lobpcg<https://github.com/lobpcg> To make everything work w/ existing SVD solvers, we need to make sure the interface to both functions is the same. To do that, we would need to pick defaults for arguments specific to LOBPCG i.e. n_oversamples, and figure out how to add existing arguments not supported in LOBPCG but elsewhere i.e. ncv in the arpack solver. Does this sound like a good plan? I can push what I have after trying to fiddle w/ the tests if this isn't clear.

Regarding the first point, is there a good way to determine those argument defaults?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/scipy/scipy/issues/9275?email_source=notifications&email_token=AKFMTPPNYWJD6XPPMUXSIDLP66ZDTA5CNFSM4FVJ2D62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZYFUTQ#issuecomment-510679630>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AKFMTPNAOQHRTCOHR7KHAITP66ZDTANCNFSM4FVJ2D6Q>.

How important would you consider adding support for the other arguments currently not supported by the LOBPCG solver i.e. `which`, `v0`, and `return_singular_vectors`? Some of these should be easy to implement, but others will require some thought.
@clementkng Sorry, I do not have time to go other the functions to understand the meaning of the arguments that you mention. May be you can try to see yourself how the corresponding scikit-learn functions are set up and mimic this setup? scikit-learn supports basically the same solvers, e.g., arpack, as scipy. 

This is what I have meant by " actually involved, not just copy/paste."

Resolving patch:
diff --git a/scipy/sparse/linalg/eigen/arpack/arpack.py b/scipy/sparse/linalg/eigen/arpack/arpack.py
--- a/scipy/sparse/linalg/eigen/arpack/arpack.py
+++ b/scipy/sparse/linalg/eigen/arpack/arpack.py
@@ -50,6 +50,7 @@
 from scipy.linalg import eig, eigh, lu_factor, lu_solve
 from scipy.sparse.sputils import isdense
 from scipy.sparse.linalg import gmres, splu
+from scipy.sparse.linalg.eigen.lobpcg import lobpcg
 from scipy._lib._util import _aligned_zeros
 from scipy._lib._threadsafety import ReentrancyLock
 
@@ -1065,13 +1066,13 @@ def get_OPinv_matvec(A, M, sigma, hermitian=False, tol=0):
             return SpLuInv(A).matvec
         else:
             return IterOpInv(_aslinearoperator_with_dtype(A),
-                              M, sigma, tol=tol).matvec
+                             M, sigma, tol=tol).matvec
     else:
         if ((not isdense(A) and not isspmatrix(A)) or
                 (not isdense(M) and not isspmatrix(M))):
             return IterOpInv(_aslinearoperator_with_dtype(A),
-                              _aslinearoperator_with_dtype(M),
-                              sigma, tol=tol).matvec
+                             _aslinearoperator_with_dtype(M),
+                             sigma, tol=tol).matvec
         elif isdense(A) or isdense(M):
             return LuInv(A - sigma * M).matvec
         else:
@@ -1453,21 +1454,26 @@ def eigsh(A, k=6, M=None, sigma=None, which='LM', v0=None,
     OPinv : N x N matrix, array, sparse matrix, or LinearOperator
         See notes in sigma, above.
     return_eigenvectors : bool
-        Return eigenvectors (True) in addition to eigenvalues. This value determines
-        the order in which eigenvalues are sorted. The sort order is also dependent on the `which` variable.
+        Return eigenvectors (True) in addition to eigenvalues.
+        This value determines the order in which eigenvalues are sorted.
+        The sort order is also dependent on the `which` variable.
 
             For which = 'LM' or 'SA':
-                If `return_eigenvectors` is True, eigenvalues are sorted by algebraic value.
+                If `return_eigenvectors` is True, eigenvalues are sorted by
+                algebraic value.
 
-                If `return_eigenvectors` is False, eigenvalues are sorted by absolute value.
+                If `return_eigenvectors` is False, eigenvalues are sorted by
+                absolute value.
 
             For which = 'BE' or 'LA':
                 eigenvalues are always sorted by algebraic value.
 
             For which = 'SM':
-                If `return_eigenvectors` is True, eigenvalues are sorted by algebraic value.
+                If `return_eigenvectors` is True, eigenvalues are sorted by
+                algebraic value.
 
-                If `return_eigenvectors` is False, eigenvalues are sorted by decreasing absolute value.
+                If `return_eigenvectors` is False, eigenvalues are sorted by
+                decreasing absolute value.
 
     mode : string ['normal' | 'buckling' | 'cayley']
         Specify strategy to use for shift-invert mode.  This argument applies
@@ -1708,8 +1714,9 @@ def _herm(x):
 
 
 def svds(A, k=6, ncv=None, tol=0, which='LM', v0=None,
-         maxiter=None, return_singular_vectors=True):
-    """Compute the largest or smallest k singular values/vectors for a sparse matrix.
+         maxiter=None, return_singular_vectors=True,
+         solver='arpack'):
+    """Compute the largest or smallest k singular values/vectors for a sparse matrix. The order of the singular values is not guaranteed.
 
     Parameters
     ----------
@@ -1752,6 +1759,9 @@ def svds(A, k=6, ncv=None, tol=0, which='LM', v0=None,
         - "vh": only return the vh matrix, without computing u (if N <= M).
 
         .. versionadded:: 0.16.0
+    solver : str, optional
+            Eigenvalue solver to use. Should be 'arpack' or 'lobpcg'.
+            Default: 'arpack'
 
     Returns
     -------
@@ -1769,7 +1779,7 @@ def svds(A, k=6, ncv=None, tol=0, which='LM', v0=None,
 
     Notes
     -----
-    This is a naive implementation using ARPACK as an eigensolver
+    This is a naive implementation using ARPACK or LOBPCG as an eigensolver
     on A.H * A or A * A.H, depending on which one is more efficient.
 
     Examples
@@ -1783,6 +1793,13 @@ def svds(A, k=6, ncv=None, tol=0, which='LM', v0=None,
     >>> np.sqrt(eigs(A.dot(A.T), k=2)[0]).real
     array([ 5.6059665 ,  2.75193379])
     """
+    if which == 'LM':
+        largest = True
+    elif which == 'SM':
+        largest = False
+    else:
+        raise ValueError("which must be either 'LM' or 'SM'.")
+
     if not (isinstance(A, LinearOperator) or isspmatrix(A)):
         A = np.asarray(A)
 
@@ -1796,92 +1813,90 @@ def svds(A, k=6, ncv=None, tol=0, which='LM', v0=None,
             X_dot = A.matvec
             X_matmat = A.matmat
             XH_dot = A.rmatvec
+            XH_mat = A.rmatmat
         else:
             X_dot = A.rmatvec
+            X_matmat = A.rmatmat
             XH_dot = A.matvec
+            XH_mat = A.matmat
 
             dtype = getattr(A, 'dtype', None)
             if dtype is None:
-                dtype = A.dot(np.zeros([m,1])).dtype
-
-            # A^H * V; works around lack of LinearOperator.adjoint.
-            # XXX This can be slow!
-            def X_matmat(V):
-                out = np.empty((V.shape[1], m), dtype=dtype)
-                for i, col in enumerate(V.T):
-                    out[i, :] = A.rmatvec(col.reshape(-1, 1)).T
-                return out.T
+                dtype = A.dot(np.zeros([m, 1])).dtype
 
     else:
         if n > m:
             X_dot = X_matmat = A.dot
-            XH_dot = _herm(A).dot
+            XH_dot = XH_mat = _herm(A).dot
         else:
-            XH_dot = A.dot
+            XH_dot = XH_mat = A.dot
             X_dot = X_matmat = _herm(A).dot
 
     def matvec_XH_X(x):
         return XH_dot(X_dot(x))
 
+    def matmat_XH_X(x):
+        return XH_mat(X_matmat(x))
+
     XH_X = LinearOperator(matvec=matvec_XH_X, dtype=A.dtype,
+                          matmat=matmat_XH_X,
                           shape=(min(A.shape), min(A.shape)))
 
     # Get a low rank approximation of the implicitly defined gramian matrix.
     # This is not a stable way to approach the problem.
-    eigvals, eigvec = eigsh(XH_X, k=k, tol=tol ** 2, maxiter=maxiter,
-                                  ncv=ncv, which=which, v0=v0)
-
-    # In 'LM' mode try to be clever about small eigenvalues.
-    # Otherwise in 'SM' mode do not try to be clever.
-    if which == 'LM':
-
-        # Gramian matrices have real non-negative eigenvalues.
-        eigvals = np.maximum(eigvals.real, 0)
-
-        # Use the sophisticated detection of small eigenvalues from pinvh.
-        t = eigvec.dtype.char.lower()
-        factor = {'f': 1E3, 'd': 1E6}
-        cond = factor[t] * np.finfo(t).eps
-        cutoff = cond * np.max(eigvals)
-
-        # Get a mask indicating which eigenpairs are not degenerately tiny,
-        # and create the re-ordered array of thresholded singular values.
-        above_cutoff = (eigvals > cutoff)
-        nlarge = above_cutoff.sum()
-        nsmall = k - nlarge
-        slarge = np.sqrt(eigvals[above_cutoff])
-        s = np.zeros_like(eigvals)
-        s[:nlarge] = slarge
-        if not return_singular_vectors:
-            return s
+    if solver == 'lobpcg':
 
-        if n > m:
-            vlarge = eigvec[:, above_cutoff]
-            ularge = X_matmat(vlarge) / slarge if return_singular_vectors != 'vh' else None
-            vhlarge = _herm(vlarge)
+        if k == 1 and v0 is not None:
+            X = np.reshape(v0, (-1, 1))
         else:
-            ularge = eigvec[:, above_cutoff]
-            vhlarge = _herm(X_matmat(ularge) / slarge) if return_singular_vectors != 'u' else None
-
-        u = _augmented_orthonormal_cols(ularge, nsmall) if ularge is not None else None
-        vh = _augmented_orthonormal_rows(vhlarge, nsmall) if vhlarge is not None else None
+            X = np.random.RandomState(52).randn(min(A.shape), k)
 
-    elif which == 'SM':
+        eigvals, eigvec = lobpcg(XH_X, X, tol=tol ** 2, maxiter=maxiter,
+                                 largest=largest)
 
-        s = np.sqrt(eigvals)
-        if not return_singular_vectors:
-            return s
-
-        if n > m:
-            v = eigvec
-            u = X_matmat(v) / s if return_singular_vectors != 'vh' else None
-            vh = _herm(v)
-        else:
-            u = eigvec
-            vh = _herm(X_matmat(u) / s) if return_singular_vectors != 'u' else None
+    elif solver == 'arpack' or solver is None:
+        eigvals, eigvec = eigsh(XH_X, k=k, tol=tol ** 2, maxiter=maxiter,
+                                ncv=ncv, which=which, v0=v0)
 
     else:
-
-        raise ValueError("which must be either 'LM' or 'SM'.")
+        raise ValueError("solver must be either 'arpack', or 'lobpcg'.")
+
+    # Gramian matrices have real non-negative eigenvalues.
+    eigvals = np.maximum(eigvals.real, 0)
+
+    # Use the sophisticated detection of small eigenvalues from pinvh.
+    t = eigvec.dtype.char.lower()
+    factor = {'f': 1E3, 'd': 1E6}
+    cond = factor[t] * np.finfo(t).eps
+    cutoff = cond * np.max(eigvals)
+
+    # Get a mask indicating which eigenpairs are not degenerately tiny,
+    # and create the re-ordered array of thresholded singular values.
+    above_cutoff = (eigvals > cutoff)
+    nlarge = above_cutoff.sum()
+    nsmall = k - nlarge
+    slarge = np.sqrt(eigvals[above_cutoff])
+    s = np.zeros_like(eigvals)
+    s[:nlarge] = slarge
+    if not return_singular_vectors:
+        return np.sort(s)
+
+    if n > m:
+        vlarge = eigvec[:, above_cutoff]
+        ularge = X_matmat(vlarge) / slarge if return_singular_vectors != 'vh' else None
+        vhlarge = _herm(vlarge)
+    else:
+        ularge = eigvec[:, above_cutoff]
+        vhlarge = _herm(X_matmat(ularge) / slarge) if return_singular_vectors != 'u' else None
+
+    u = _augmented_orthonormal_cols(ularge, nsmall) if ularge is not None else None
+    vh = _augmented_orthonormal_rows(vhlarge, nsmall) if vhlarge is not None else None
+
+    indexes_sorted = np.argsort(s)
+    s = s[indexes_sorted]
+    if u is not None:
+        u = u[:, indexes_sorted]
+    if vh is not None:
+        vh = vh[indexes_sorted]
 
     return u, s, vh


Test patch:
diff --git a/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py b/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py
--- a/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py
+++ b/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py
@@ -72,11 +72,11 @@ def _get_test_tolerance(type_char, mattype=None):
     return tol, rtol, atol
 
 
-def generate_matrix(N, complex=False, hermitian=False,
+def generate_matrix(N, complex_=False, hermitian=False,
                     pos_definite=False, sparse=False):
-    M = np.random.random((N,N))
-    if complex:
-        M = M + 1j * np.random.random((N,N))
+    M = np.random.random((N, N))
+    if complex_:
+        M = M + 1j * np.random.random((N, N))
 
     if hermitian:
         if pos_definite:
@@ -84,7 +84,7 @@ def generate_matrix(N, complex=False, hermitian=False,
                 i = np.arange(N)
                 j = np.random.randint(N, size=N-2)
                 i, j = np.meshgrid(i, j)
-                M[i,j] = 0
+                M[i, j] = 0
             M = np.dot(M.conj(), M.T)
         else:
             M = np.dot(M.conj(), M.T)
@@ -93,13 +93,13 @@ def generate_matrix(N, complex=False, hermitian=False,
                 j = np.random.randint(N, size=N * N // 4)
                 ind = np.nonzero(i == j)
                 j[ind] = (j[ind] + 1) % N
-                M[i,j] = 0
-                M[j,i] = 0
+                M[i, j] = 0
+                M[j, i] = 0
     else:
         if sparse:
             i = np.random.randint(N, size=N * N // 2)
             j = np.random.randint(N, size=N * N // 2)
-            M[i,j] = 0
+            M[i, j] = 0
     return M
 
 
@@ -136,26 +136,26 @@ def assert_allclose_cc(actual, desired, **kw):
         assert_allclose(actual, conj(desired), **kw)
 
 
-def argsort_which(eval, typ, k, which,
+def argsort_which(eigenvalues, typ, k, which,
                   sigma=None, OPpart=None, mode=None):
     """Return sorted indices of eigenvalues using the "which" keyword
     from eigs and eigsh"""
     if sigma is None:
-        reval = np.round(eval, decimals=_ndigits[typ])
+        reval = np.round(eigenvalues, decimals=_ndigits[typ])
     else:
         if mode is None or mode == 'normal':
             if OPpart is None:
-                reval = 1. / (eval - sigma)
+                reval = 1. / (eigenvalues - sigma)
             elif OPpart == 'r':
-                reval = 0.5 * (1. / (eval - sigma)
-                               + 1. / (eval - np.conj(sigma)))
+                reval = 0.5 * (1. / (eigenvalues - sigma)
+                               + 1. / (eigenvalues - np.conj(sigma)))
             elif OPpart == 'i':
-                reval = -0.5j * (1. / (eval - sigma)
-                                 - 1. / (eval - np.conj(sigma)))
+                reval = -0.5j * (1. / (eigenvalues - sigma)
+                                 - 1. / (eigenvalues - np.conj(sigma)))
         elif mode == 'cayley':
-            reval = (eval + sigma) / (eval - sigma)
+            reval = (eigenvalues + sigma) / (eigenvalues - sigma)
         elif mode == 'buckling':
-            reval = eval / (eval - sigma)
+            reval = eigenvalues / (eigenvalues - sigma)
         else:
             raise ValueError("mode='%s' not recognized" % mode)
 
@@ -237,41 +237,41 @@ def eval_evec(symmetric, d, typ, k, which, v0=None, sigma=None,
         # solve
         if general:
             try:
-                eval, evec = eigs_func(ac, k, bc, **kwargs)
+                eigenvalues, evec = eigs_func(ac, k, bc, **kwargs)
             except ArpackNoConvergence:
                 kwargs['maxiter'] = 20*a.shape[0]
-                eval, evec = eigs_func(ac, k, bc, **kwargs)
+                eigenvalues, evec = eigs_func(ac, k, bc, **kwargs)
         else:
             try:
-                eval, evec = eigs_func(ac, k, **kwargs)
+                eigenvalues, evec = eigs_func(ac, k, **kwargs)
             except ArpackNoConvergence:
                 kwargs['maxiter'] = 20*a.shape[0]
-                eval, evec = eigs_func(ac, k, **kwargs)
+                eigenvalues, evec = eigs_func(ac, k, **kwargs)
 
-        ind = argsort_which(eval, typ, k, which,
+        ind = argsort_which(eigenvalues, typ, k, which,
                             sigma, OPpart, mode)
-        eval = eval[ind]
-        evec = evec[:,ind]
+        eigenvalues = eigenvalues[ind]
+        evec = evec[:, ind]
 
         # check eigenvectors
         LHS = np.dot(a, evec)
         if general:
-            RHS = eval * np.dot(b, evec)
+            RHS = eigenvalues * np.dot(b, evec)
         else:
-            RHS = eval * evec
+            RHS = eigenvalues * evec
 
             assert_allclose(LHS, RHS, rtol=rtol, atol=atol, err_msg=err)
 
         try:
             # check eigenvalues
-            assert_allclose_cc(eval, exact_eval, rtol=rtol, atol=atol,
+            assert_allclose_cc(eigenvalues, exact_eval, rtol=rtol, atol=atol,
                                err_msg=err)
             break
         except AssertionError:
             ntries += 1
 
     # check eigenvalues
-    assert_allclose_cc(eval, exact_eval, rtol=rtol, atol=atol, err_msg=err)
+    assert_allclose_cc(eigenvalues, exact_eval, rtol=rtol, atol=atol, err_msg=err)
 
 
 class DictWithRepr(dict):
@@ -300,9 +300,9 @@ def __init__(self):
         M = generate_matrix(N, hermitian=True,
                             pos_definite=True).astype('f').astype('d')
         Ac = generate_matrix(N, hermitian=True, pos_definite=True,
-                             complex=True).astype('F').astype('D')
+                             complex_=True).astype('F').astype('D')
         Mc = generate_matrix(N, hermitian=True, pos_definite=True,
-                             complex=True).astype('F').astype('D')
+                             complex_=True).astype('F').astype('D')
         v0 = np.random.random(N)
 
         # standard symmetric problem
@@ -359,7 +359,7 @@ def __init__(self):
         Ar = generate_matrix(N).astype('f').astype('d')
         M = generate_matrix(N, hermitian=True,
                             pos_definite=True).astype('f').astype('d')
-        Ac = generate_matrix(N, complex=True).astype('F').astype('D')
+        Ac = generate_matrix(N, complex_=True).astype('F').astype('D')
         v0 = np.random.random(N)
 
         # standard real nonsymmetric problem
@@ -501,7 +501,7 @@ def test_general_nonsymmetric_starting_vector():
 
 def test_standard_nonsymmetric_no_convergence():
     np.random.seed(1234)
-    m = generate_matrix(30, complex=True)
+    m = generate_matrix(30, complex_=True)
     tol, rtol, atol = _get_test_tolerance('d')
     try:
         w, v = eigs(m, 4, which='LM', v0=m[:, 0], maxiter=5, tol=tol)
@@ -529,8 +529,8 @@ def test_eigen_bad_kwargs():
 
 def test_ticket_1459_arpack_crash():
     for dtype in [np.float32, np.float64]:
-        # XXX: this test does not seem to catch the issue for float32,
-        #      but we made the same fix there, just to be sure
+        # This test does not seem to catch the issue for float32,
+        # but we made the same fix there, just to be sure
 
         N = 6
         k = 2
@@ -595,15 +595,16 @@ def test_svd_simple_real():
                   [0, 0, 1, 0]], float)
     z = csc_matrix(x)
 
-    for m in [x.T, x, y, z, z.T]:
-        for k in range(1, min(m.shape)):
-            u, s, vh = sorted_svd(m, k)
-            su, ss, svh = svds(m, k)
+    for solver in [None, 'arpack', 'lobpcg']:
+        for m in [x.T, x, y, z, z.T]:
+            for k in range(1, min(m.shape)):
+                u, s, vh = sorted_svd(m, k)
+                su, ss, svh = svds(m, k, solver=solver)
 
-            m_hat = svd_estimate(u, s, vh)
-            sm_hat = svd_estimate(su, ss, svh)
+                m_hat = svd_estimate(u, s, vh)
+                sm_hat = svd_estimate(su, ss, svh)
 
-            assert_array_almost_equal_nulp(m_hat, sm_hat, nulp=1000)
+                assert_array_almost_equal_nulp(m_hat, sm_hat, nulp=1000)
 
 
 def test_svd_simple_complex():
@@ -617,15 +618,16 @@ def test_svd_simple_complex():
                   [0, 0, 1, 0]], complex)
     z = csc_matrix(x)
 
-    for m in [x, x.T.conjugate(), x.T, y, y.conjugate(), z, z.T]:
-        for k in range(1, min(m.shape) - 1):
-            u, s, vh = sorted_svd(m, k)
-            su, ss, svh = svds(m, k)
+    for solver in [None, 'arpack', 'lobpcg']:
+        for m in [x, x.T.conjugate(), x.T, y, y.conjugate(), z, z.T]:
+            for k in range(1, min(m.shape) - 1):
+                u, s, vh = sorted_svd(m, k)
+                su, ss, svh = svds(m, k, solver=solver)
 
-            m_hat = svd_estimate(u, s, vh)
-            sm_hat = svd_estimate(su, ss, svh)
+                m_hat = svd_estimate(u, s, vh)
+                sm_hat = svd_estimate(su, ss, svh)
 
-            assert_array_almost_equal_nulp(m_hat, sm_hat, nulp=1000)
+                assert_array_almost_equal_nulp(m_hat, sm_hat, nulp=1000)
 
 
 def test_svd_maxiter():
@@ -652,19 +654,22 @@ def test_svd_which():
     x = hilbert(6)
     for which in ['LM', 'SM']:
         _, s, _ = sorted_svd(x, 2, which=which)
-        ss = svds(x, 2, which=which, return_singular_vectors=False)
-        ss.sort()
-        assert_allclose(s, ss, atol=np.sqrt(1e-15))
+        for solver in [None, 'arpack', 'lobpcg']:
+            ss = svds(x, 2, which=which, return_singular_vectors=False,
+                      solver=solver)
+            ss.sort()
+            assert_allclose(s, ss, atol=np.sqrt(1e-15))
 
 
 def test_svd_v0():
     # check that the v0 parameter works as expected
     x = np.array([[1, 2, 3, 4], [5, 6, 7, 8]], float)
 
-    u, s, vh = svds(x, 1)
-    u2, s2, vh2 = svds(x, 1, v0=u[:,0])
+    for solver in [None, 'arpack', 'lobpcg']:
+        u, s, vh = svds(x, 1, solver=solver)
+        u2, s2, vh2 = svds(x, 1, v0=u[:, 0], solver=solver)
 
-    assert_allclose(s, s2, atol=np.sqrt(1e-15))
+        assert_allclose(s, s2, atol=np.sqrt(1e-15))
 
 
 def _check_svds(A, k, U, s, VH):
@@ -697,15 +702,16 @@ def test_svd_LM_ones_matrix():
     for n, m in (6, 5), (5, 5), (5, 6):
         for t in float, complex:
             A = np.ones((n, m), dtype=t)
-            U, s, VH = svds(A, k)
+            for solver in [None, 'arpack', 'lobpcg']:
+                U, s, VH = svds(A, k, solver=solver)
 
-            # Check some generic properties of svd.
-            _check_svds(A, k, U, s, VH)
+                # Check some generic properties of svd.
+                _check_svds(A, k, U, s, VH)
 
-            # Check that the largest singular value is near sqrt(n*m)
-            # and the other singular values have been forced to zero.
-            assert_allclose(np.max(s), np.sqrt(n*m))
-            assert_array_equal(sorted(s)[:-1], 0)
+                # Check that the largest singular value is near sqrt(n*m)
+                # and the other singular values have been forced to zero.
+                assert_allclose(np.max(s), np.sqrt(n*m))
+                assert_array_equal(sorted(s)[:-1], 0)
 
 
 def test_svd_LM_zeros_matrix():
@@ -714,13 +720,14 @@ def test_svd_LM_zeros_matrix():
     for n, m in (3, 4), (4, 4), (4, 3):
         for t in float, complex:
             A = np.zeros((n, m), dtype=t)
-            U, s, VH = svds(A, k)
+            for solver in [None, 'arpack', 'lobpcg']:
+                U, s, VH = svds(A, k, solver=solver)
 
-            # Check some generic properties of svd.
-            _check_svds(A, k, U, s, VH)
+                # Check some generic properties of svd.
+                _check_svds(A, k, U, s, VH)
 
-            # Check that the singular values are zero.
-            assert_array_equal(s, 0)
+                # Check that the singular values are zero.
+                assert_array_equal(s, 0)
 
 
 def test_svd_LM_zeros_matrix_gh_3452():
@@ -729,13 +736,14 @@ def test_svd_LM_zeros_matrix_gh_3452():
     # Note that for complex dype the size of this matrix is too small for k=1.
     n, m, k = 4, 2, 1
     A = np.zeros((n, m))
-    U, s, VH = svds(A, k)
+    for solver in [None, 'arpack', 'lobpcg']:
+        U, s, VH = svds(A, k, solver=solver)
 
-    # Check some generic properties of svd.
-    _check_svds(A, k, U, s, VH)
+        # Check some generic properties of svd.
+        _check_svds(A, k, U, s, VH)
 
-    # Check that the singular values are zero.
-    assert_array_equal(s, 0)
+        # Check that the singular values are zero.
+        assert_array_equal(s, 0)
 
 
 class CheckingLinearOperator(LinearOperator):
@@ -761,7 +769,7 @@ def test_svd_linop():
     def reorder(args):
         U, s, VH = args
         j = np.argsort(s)
-        return U[:,j], s[j], VH[j,:]
+        return U[:, j], s[j], VH[j, :]
 
     for n, m, k in nmks:
         # Test svds on a LinearOperator.
@@ -770,27 +778,29 @@ def reorder(args):
 
         v0 = np.ones(min(A.shape))
 
-        U1, s1, VH1 = reorder(svds(A, k, v0=v0))
-        U2, s2, VH2 = reorder(svds(L, k, v0=v0))
+        for solver in [None, 'arpack', 'lobpcg']:
+            U1, s1, VH1 = reorder(svds(A, k, v0=v0, solver=solver))
+            U2, s2, VH2 = reorder(svds(L, k, v0=v0, solver=solver))
 
-        assert_allclose(np.abs(U1), np.abs(U2))
-        assert_allclose(s1, s2)
-        assert_allclose(np.abs(VH1), np.abs(VH2))
-        assert_allclose(np.dot(U1, np.dot(np.diag(s1), VH1)),
-                        np.dot(U2, np.dot(np.diag(s2), VH2)))
+            assert_allclose(np.abs(U1), np.abs(U2))
+            assert_allclose(s1, s2)
+            assert_allclose(np.abs(VH1), np.abs(VH2))
+            assert_allclose(np.dot(U1, np.dot(np.diag(s1), VH1)),
+                            np.dot(U2, np.dot(np.diag(s2), VH2)))
 
         # Try again with which="SM".
         A = np.random.RandomState(1909).randn(n, m)
         L = CheckingLinearOperator(A)
 
-        U1, s1, VH1 = reorder(svds(A, k, which="SM"))
-        U2, s2, VH2 = reorder(svds(L, k, which="SM"))
+        for solver in [None, 'arpack', 'lobpcg']:
+            U1, s1, VH1 = reorder(svds(A, k, which="SM", solver=solver))
+            U2, s2, VH2 = reorder(svds(L, k, which="SM", solver=solver))
 
-        assert_allclose(np.abs(U1), np.abs(U2))
-        assert_allclose(s1, s2)
-        assert_allclose(np.abs(VH1), np.abs(VH2))
-        assert_allclose(np.dot(U1, np.dot(np.diag(s1), VH1)),
-                        np.dot(U2, np.dot(np.diag(s2), VH2)))
+            assert_allclose(np.abs(U1), np.abs(U2))
+            assert_allclose(s1, s2)
+            assert_allclose(np.abs(VH1), np.abs(VH2))
+            assert_allclose(np.dot(U1, np.dot(np.diag(s1), VH1)),
+                            np.dot(U2, np.dot(np.diag(s2), VH2)))
 
         if k < min(n, m) - 1:
             # Complex input and explicit which="LM".
@@ -799,14 +809,16 @@ def reorder(args):
                 A = (rng.randn(n, m) + 1j * rng.randn(n, m)).astype(dt)
                 L = CheckingLinearOperator(A)
 
-                U1, s1, VH1 = reorder(svds(A, k, which="LM"))
-                U2, s2, VH2 = reorder(svds(L, k, which="LM"))
+                for solver in [None, 'arpack', 'lobpcg']:
+                    U1, s1, VH1 = reorder(svds(A, k, which="LM", solver=solver))
+                    U2, s2, VH2 = reorder(svds(L, k, which="LM", solver=solver))
 
-                assert_allclose(np.abs(U1), np.abs(U2), rtol=eps)
-                assert_allclose(s1, s2, rtol=eps)
-                assert_allclose(np.abs(VH1), np.abs(VH2), rtol=eps)
-                assert_allclose(np.dot(U1, np.dot(np.diag(s1), VH1)),
-                                np.dot(U2, np.dot(np.diag(s2), VH2)), rtol=eps)
+                    assert_allclose(np.abs(U1), np.abs(U2), rtol=eps)
+                    assert_allclose(s1, s2, rtol=eps)
+                    assert_allclose(np.abs(VH1), np.abs(VH2), rtol=eps)
+                    assert_allclose(np.dot(U1, np.dot(np.diag(s1), VH1)),
+                                    np.dot(U2, np.dot(np.diag(s2), VH2)),
+                                    rtol=eps)
 
 
 @pytest.mark.skipif(IS_PYPY, reason="Test not meaningful on PyPy")