GitHub PushEvent at 2014-02-21 22:08:20
Repository: incubator-spark
Actor: mengxr
Payload: {"shas":[["3fede4831eeb7d36d4f8fa4aaa02ad0cc8b4b09e","5a7a6a189b47cd31d24667a751e4003cd04eced5@databricks.com","Super minor: Add require for mergeCombiners in combineByKey\n\nWe changed the behavior in 0.9.0 from requiring that mergeCombiners be null when mapSideCombine was false to requiring that mergeCombiners *never* be null, for external sorting. This patch adds a require() to make this behavior change explicitly messaged rather than resulting in a NPE.\n\nAuthor: Aaron Davidson <aaron@databricks.com>\n\nCloses #623 from aarondav/master and squashes the following commits:\n\n520b80c [Aaron Davidson] Super minor: Add require for mergeCombiners in combineByKey","Aaron Davidson",true],["59b1379594360636e97511982c794bcf36225e1a","1c4a85f4cce80a1214226f676424d149720d1452@gmail.com","SPARK-1114: Allow PySpark to use existing JVM and Gateway\n\nPatch to allow PySpark to use existing JVM and Gateway. Changes to PySpark implementation of SparkConf to take existing SparkConf JVM handle. Change to PySpark SparkContext to allow subclass specific context initialization.\n\nAuthor: Ahir Reddy <ahirreddy@gmail.com>\n\nCloses #622 from ahirreddy/pyspark-existing-jvm and squashes the following commits:\n\na86f457 [Ahir Reddy] Patch to allow PySpark to use existing JVM and Gateway. Changes to PySpark implementation of SparkConf to take existing SparkConf JVM handle. Change to PySpark SparkContext to allow subclass specific context initialization.","Ahir Reddy",true],["45b15e27a84527abeaa8588b0eb1ade7e831e6ef","8bcd50e7b1eea79f965dde19d326001805380997@gmail.com","SPARK-1111: URL Validation Throws Error for HDFS URL's\n\nFixes an error where HDFS URL's cause an exception. Should be merged into master and 0.9.\n\nAuthor: Patrick Wendell <pwendell@gmail.com>\n\nCloses #625 from pwendell/url-validation and squashes the following commits:\n\nd14bfe3 [Patrick Wendell] SPARK-1111: URL Validation Throws Error for HDFS URL's","Patrick Wendell",true],["c8a4c9b1f6005815f5a4a331970624d1706b6b13","291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","MLLIB-25: Implicit ALS runs out of memory for moderately large numbers of features\n\nThere's a step in implicit ALS where the matrix `Yt * Y` is computed. It's computed as the sum of matrices; an f x f matrix is created for each of n user/item rows in a partition. In `ALS.scala:214`:\n\n```\n        factors.flatMapValues{ case factorArray =>\n          factorArray.map{ vector =>\n            val x = new DoubleMatrix(vector)\n            x.mmul(x.transpose())\n          }\n        }.reduceByKeyLocally((a, b) => a.addi(b))\n         .values\n         .reduce((a, b) => a.addi(b))\n```\n\nCompletely correct, but there's a subtle but quite large memory problem here. map() is going to create all of these matrices in memory at once, when they don't need to ever all exist at the same time.\nFor example, if a partition has n = 100000 rows, and f = 200, then this intermediate product requires 32GB of heap. The computation will never work unless you can cough up workers with (more than) that much heap.\n\nFortunately there's a trivial change that fixes it; just add `.view` in there.\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #629 from srowen/ALSMatrixAllocationOptimization and squashes the following commits:\n\n062cda9 [Sean Owen] Update style per review comments\ne9a5d63 [Sean Owen] Avoid unnecessary out of memory situation by not simultaneously allocating lots of matrices","Sean Owen",true]],"size":4,"ref":"refs/heads/master","head":"c8a4c9b1f6005815f5a4a331970624d1706b6b13"}