GitHub PullRequestReviewCommentEvent at 2014-12-16 07:34:39
Repository: spark
Actor: brkyvz
Payload: {"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/21881678","id":21881678,"diff_hunk":"@@ -197,6 +295,171 @@ class SparseMatrix(\n   }\n \n   override def copy = new SparseMatrix(numRows, numCols, colPtrs, rowIndices, values.clone())\n+\n+  private[mllib] def map(f: Double => Double) =\n+    new SparseMatrix(numRows, numCols, colPtrs, rowIndices, values.map(f))\n+\n+  private[mllib] def update(f: Double => Double): SparseMatrix = {\n+    val len = values.length\n+    var i = 0\n+    while (i < len) {\n+      values(i) = f(values(i))\n+      i += 1\n+    }\n+    this\n+  }\n+}\n+\n+/**\n+ * Factory methods for [[org.apache.spark.mllib.linalg.SparseMatrix]].\n+ */\n+object SparseMatrix {\n+\n+  /**\n+   * Generate an Identity Matrix in `SparseMatrix` format.\n+   * @param n number of rows and columns of the matrix\n+   * @return `SparseMatrix` with size `n` x `n` and values of ones on the diagonal\n+   */\n+  def speye(n: Int): SparseMatrix = {\n+    new SparseMatrix(n, n, (0 to n).toArray, (0 until n).toArray, Array.fill(n)(1.0))\n+  }\n+\n+  /** Generates a SparseMatrix given an Array[Double] of size numRows * numCols. The number of\n+    * non-zeros in `raw` is provided for efficiency. */\n+  private def genRand(\n+      numRows: Int,\n+      numCols: Int,\n+      raw: Array[Double],\n+      nonZero: Int): SparseMatrix = {\n+    val sparseA: ArrayBuffer[Double] = new ArrayBuffer(nonZero)\n+    val sCols: ArrayBuffer[Int] = new ArrayBuffer(numCols + 1)\n+    val sRows: ArrayBuffer[Int] = new ArrayBuffer(nonZero)\n+\n+    var i = 0\n+    var nnz = 0\n+    var lastCol = -1\n+    raw.foreach { v =>\n+      val r = i % numRows\n+      val c = (i - r) / numRows\n+      if ( v != 0.0) {","path":"mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala","position":175,"original_position":175,"commit_id":"a8120d2a83720b621b36942add3a98aa4b96bcc3","original_commit_id":"a8120d2a83720b621b36942add3a98aa4b96bcc3","user":{"login":"brkyvz","id":5243515,"avatar_url":"https://avatars.githubusercontent.com/u/5243515?v=3","gravatar_id":"","url":"https://api.github.com/users/brkyvz","html_url":"https://github.com/brkyvz","followers_url":"https://api.github.com/users/brkyvz/followers","following_url":"https://api.github.com/users/brkyvz/following{/other_user}","gists_url":"https://api.github.com/users/brkyvz/gists{/gist_id}","starred_url":"https://api.github.com/users/brkyvz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brkyvz/subscriptions","organizations_url":"https://api.github.com/users/brkyvz/orgs","repos_url":"https://api.github.com/users/brkyvz/repos","events_url":"https://api.github.com/users/brkyvz/events{/privacy}","received_events_url":"https://api.github.com/users/brkyvz/received_events","type":"User","site_admin":false},"body":"Right now, it's not. Currently users can supply zero values during the construction of SparseMatrix. Two things:\r\n1) Should I add a check in the constructor of SparseMatrix?\r\n2) Should I transform genRand into something like .toSparse() inside DenseMatrix, and add a .toDense() inside SparseMatrix? (I actually had these two methods in my multi model training repo)","created_at":"2014-12-16T07:34:39Z","updated_at":"2014-12-16T07:34:39Z","html_url":"https://github.com/apache/spark/pull/3319#discussion_r21881678","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/3319","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/21881678"},"html":{"href":"https://github.com/apache/spark/pull/3319#discussion_r21881678"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/3319"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/3319","id":24558608,"html_url":"https://github.com/apache/spark/pull/3319","diff_url":"https://github.com/apache/spark/pull/3319.diff","patch_url":"https://github.com/apache/spark/pull/3319.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/3319","number":3319,"state":"open","locked":false,"title":"[SPARK-4409][MLlib] Additional Linear Algebra Utils","user":{"login":"brkyvz","id":5243515,"avatar_url":"https://avatars.githubusercontent.com/u/5243515?v=3","gravatar_id":"","url":"https://api.github.com/users/brkyvz","html_url":"https://github.com/brkyvz","followers_url":"https://api.github.com/users/brkyvz/followers","following_url":"https://api.github.com/users/brkyvz/following{/other_user}","gists_url":"https://api.github.com/users/brkyvz/gists{/gist_id}","starred_url":"https://api.github.com/users/brkyvz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brkyvz/subscriptions","organizations_url":"https://api.github.com/users/brkyvz/orgs","repos_url":"https://api.github.com/users/brkyvz/repos","events_url":"https://api.github.com/users/brkyvz/events{/privacy}","received_events_url":"https://api.github.com/users/brkyvz/received_events","type":"User","site_admin":false},"body":"Addition of a very limited number of local matrix manipulation and generation methods that would be helpful in the further development for algorithms on top of BlockMatrix (SPARK-3974), such as Randomized SVD, and Multi Model Training (SPARK-1486).\r\nThe proposed methods for addition are:\r\n\r\nFor `Matrix`\r\n - map: maps the values in the matrix with a given function. Produces a new matrix.\r\n - update: the values in the matrix are updated with a given function. Occurs in place.\r\n\r\nFactory methods for `DenseMatrix`:\r\n - *zeros: Generate a matrix consisting of zeros\r\n - *ones: Generate a matrix consisting of ones\r\n - *eye: Generate an identity matrix\r\n - *rand: Generate a matrix consisting of i.i.d. uniform random numbers\r\n - *randn: Generate a matrix consisting of i.i.d. gaussian random numbers\r\n - *diag: Generate a diagonal matrix from a supplied vector\r\n*These methods already exist in the factory methods for `Matrices`, however for cases where we require a `DenseMatrix`, you constantly have to add `.asInstanceOf[DenseMatrix]` everywhere, which makes the code \"dirtier\". I propose moving these functions to factory methods for `DenseMatrix` where the putput will be a `DenseMatrix` and the factory methods for `Matrices` will call these functions directly and output a generic `Matrix`.\r\n\r\nFactory methods for `SparseMatrix`:\r\n - speye: Identity matrix in sparse format. Saves a ton of memory when dimensions are large, especially in Multi Model Training, where each row requires being multiplied by a scalar.\r\n - sprand: Generate a sparse matrix with a given density consisting of i.i.d. uniform random numbers.\r\n - sprandn: Generate a sparse matrix with a given density consisting of i.i.d. gaussian random numbers.\r\n - diag: Generate a diagonal matrix from a supplied vector, but is memory efficient, because it just stores the diagonal. Again, very helpful in Multi Model Training.\r\n\r\nFactory methods for `Matrices`:\r\n - Include all the factory methods given above, but return a generic `Matrix` rather than `SparseMatrix` or `DenseMatrix`.\r\n - horzCat: Horizontally concatenate matrices to form one larger matrix. Very useful in both Multi Model Training, and for the repartitioning of BlockMatrix.\r\n - vertCat: Vertically concatenate matrices to form one larger matrix. Very useful for the repartitioning of BlockMatrix.\r\n\r\nThe names for these methods were selected from MATLAB","created_at":"2014-11-17T19:07:08Z","updated_at":"2014-12-16T07:34:39Z","closed_at":null,"merged_at":null,"merge_commit_sha":"f25785e97d61498caf4d63d5b9ea5eca7c2812cc","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/3319/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/3319/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments/{number}","comments_url":"https://api.github.com/repos/apache/spark/issues/3319/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/a8120d2a83720b621b36942add3a98aa4b96bcc3","head":{"label":"brkyvz:SPARK-4409","ref":"SPARK-4409","sha":"a8120d2a83720b621b36942add3a98aa4b96bcc3","user":{"login":"brkyvz","id":5243515,"avatar_url":"https://avatars.githubusercontent.com/u/5243515?v=3","gravatar_id":"","url":"https://api.github.com/users/brkyvz","html_url":"https://github.com/brkyvz","followers_url":"https://api.github.com/users/brkyvz/followers","following_url":"https://api.github.com/users/brkyvz/following{/other_user}","gists_url":"https://api.github.com/users/brkyvz/gists{/gist_id}","starred_url":"https://api.github.com/users/brkyvz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brkyvz/subscriptions","organizations_url":"https://api.github.com/users/brkyvz/orgs","repos_url":"https://api.github.com/users/brkyvz/repos","events_url":"https://api.github.com/users/brkyvz/events{/privacy}","received_events_url":"https://api.github.com/users/brkyvz/received_events","type":"User","site_admin":false},"repo":{"id":21919521,"name":"spark","full_name":"brkyvz/spark","owner":{"login":"brkyvz","id":5243515,"avatar_url":"https://avatars.githubusercontent.com/u/5243515?v=3","gravatar_id":"","url":"https://api.github.com/users/brkyvz","html_url":"https://github.com/brkyvz","followers_url":"https://api.github.com/users/brkyvz/followers","following_url":"https://api.github.com/users/brkyvz/following{/other_user}","gists_url":"https://api.github.com/users/brkyvz/gists{/gist_id}","starred_url":"https://api.github.com/users/brkyvz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brkyvz/subscriptions","organizations_url":"https://api.github.com/users/brkyvz/orgs","repos_url":"https://api.github.com/users/brkyvz/repos","events_url":"https://api.github.com/users/brkyvz/events{/privacy}","received_events_url":"https://api.github.com/users/brkyvz/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/brkyvz/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/brkyvz/spark","forks_url":"https://api.github.com/repos/brkyvz/spark/forks","keys_url":"https://api.github.com/repos/brkyvz/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/brkyvz/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/brkyvz/spark/teams","hooks_url":"https://api.github.com/repos/brkyvz/spark/hooks","issue_events_url":"https://api.github.com/repos/brkyvz/spark/issues/events{/number}","events_url":"https://api.github.com/repos/brkyvz/spark/events","assignees_url":"https://api.github.com/repos/brkyvz/spark/assignees{/user}","branches_url":"https://api.github.com/repos/brkyvz/spark/branches{/branch}","tags_url":"https://api.github.com/repos/brkyvz/spark/tags","blobs_url":"https://api.github.com/repos/brkyvz/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/brkyvz/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/brkyvz/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/brkyvz/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/brkyvz/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/brkyvz/spark/languages","stargazers_url":"https://api.github.com/repos/brkyvz/spark/stargazers","contributors_url":"https://api.github.com/repos/brkyvz/spark/contributors","subscribers_url":"https://api.github.com/repos/brkyvz/spark/subscribers","subscription_url":"https://api.github.com/repos/brkyvz/spark/subscription","commits_url":"https://api.github.com/repos/brkyvz/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/brkyvz/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/brkyvz/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/brkyvz/spark/issues/comments/{number}","contents_url":"https://api.github.com/repos/brkyvz/spark/contents/{+path}","compare_url":"https://api.github.com/repos/brkyvz/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/brkyvz/spark/merges","archive_url":"https://api.github.com/repos/brkyvz/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/brkyvz/spark/downloads","issues_url":"https://api.github.com/repos/brkyvz/spark/issues{/number}","pulls_url":"https://api.github.com/repos/brkyvz/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/brkyvz/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/brkyvz/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/brkyvz/spark/labels{/name}","releases_url":"https://api.github.com/repos/brkyvz/spark/releases{/id}","created_at":"2014-07-16T21:51:06Z","updated_at":"2014-08-12T23:17:42Z","pushed_at":"2014-11-26T17:27:03Z","git_url":"git://github.com/brkyvz/spark.git","ssh_url":"git@github.com:brkyvz/spark.git","clone_url":"https://github.com/brkyvz/spark.git","svn_url":"https://github.com/brkyvz/spark","homepage":null,"size":83557,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"561d31d2f13cc7b1112ba9f9aa8f08bcd032aebb","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments/{number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","created_at":"2014-02-25T08:00:08Z","updated_at":"2014-12-16T07:24:13Z","pushed_at":"2014-12-16T06:58:46Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":1035992,"stargazers_count":2328,"watchers_count":2328,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":2072,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":325,"forks":2072,"open_issues":325,"watchers":2328,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/3319"},"html":{"href":"https://github.com/apache/spark/pull/3319"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/3319"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/3319/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/3319/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/{number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/3319/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/a8120d2a83720b621b36942add3a98aa4b96bcc3"}}}}