SPARK-1782: svd for sparse matrix using ARPACK

copy ARPACK dsaupd/dseupd code from latest breeze
change RowMatrix to use sparse SVD
change tests for sparse SVD

All tests passed. I will run it against some large matrices.
hi Reza, you are right. This looks like a magic number. As Xiangrui suggested, we could separate the API for dense and sparse implementations. The magic number would be gone. 

Date: Tue, 17 Jun 2014 23:10:03 -0700
From: notifications@github.com
To: spark@noreply.github.com
CC: li.pu@outlook.com
Subject: Re: [spark] SPARK-1782: svd for sparse matrix using ARPACK (#964)

In mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala:

>     *
> -   * There is no restriction on m, but we require `n^2` doubles to fit in memory.
> -   * Further, n should be less than m.
> +   * @param k number of singular values to keep. We might return less than k if there are
> +   *          numerically zero singular values. See rCond.
> +   * @param computeU whether to compute U
> +   * @param rCond the reciprocal condition number. All singular values smaller than rCond * sigma(0)
> +   *              are treated as zero, where sigma(0) is the largest singular value.
> +   * @return SingularValueDecomposition(U, s, V)
> +   */
> +  def computeSVD(
> +      k: Int,
> +      computeU: Boolean = false,
> +      rCond: Double = 1e-9): SingularValueDecomposition[RowMatrix, Matrix] = {
> +    if (numCols() < 100) {

Please add quick comment on why 100 is used, to an unfamiliar person it looks like a magic number.


—
Reply to this email directly or view it on GitHub.