GitHub PushEvent at 2014-06-03 20:20:50
Repository: spark
Actor: srowen
Payload: {"shas":[["7782a304ad105ec95cf62cb799e365e5fb385a69","5617ee1acc0052186605e19fc1926f15c864b625@cloudera.com","[SPARK-1942] Stop clearing spark.driver.port in unit tests\n\nstop resetting spark.driver.port in unit tests (scala, java and python).\n\nAuthor: Syed Hashmi <shashmi@cloudera.com>\nAuthor: CodingCat <zhunansjtu@gmail.com>\n\nCloses #943 from syedhashmi/master and squashes the following commits:\n\n885f210 [Syed Hashmi] Removing unnecessary file (created by mergetool)\nb8bd4b5 [Syed Hashmi] Merge remote-tracking branch 'upstream/master'\nb895e59 [Syed Hashmi] Revert \"[SPARK-1784] Add a new partitioner\"\n57b6587 [Syed Hashmi] Revert \"[SPARK-1784] Add a balanced partitioner\"\n1574769 [Syed Hashmi] [SPARK-1942] Stop clearing spark.driver.port in unit tests\n4354836 [Syed Hashmi] Revert \"SPARK-1686: keep schedule() calling in the main thread\"\nfd36542 [Syed Hashmi] [SPARK-1784] Add a balanced partitioner\n6668015 [CodingCat] SPARK-1686: keep schedule() calling in the main thread\n4ca94cc [Syed Hashmi] [SPARK-1784] Add a new partitioner","Syed Hashmi",true],["6c044ed100081f1fa7c8df4bd4b58b65a61c3360","d26ec9590e27637b988f65231db53fe8dd05feb1@gmail.com","SPARK-2001 : Remove docs/spark-debugger.md from master\n\nPer discussion in dev list:\n\"\nSeemed like the spark-debugger.md is no longer accurate (see\nhttp://spark.apache.org/docs/latest/spark-debugger.html) and since it\nwas originally written Spark has evolved that makes the doc obsolete.\nThere are already work pending for new replay debugging (I could not\nfind the PR links for it) so I\nWith version control we could always reinstate the old doc if needed,\nbut as of today the doc is no longer reflect the current state of\nSpark's RDD.\n\"\n\nAuthor: Henry Saputra <henry.saputra@gmail.com>\n\nCloses #953 from hsaputra/SPARK-2001-hsaputra and squashes the following commits:\n\ndc324aa [Henry Saputra] SPARK-2001 : Remove docs/spark-debugger.md from master since it is obsolete","Henry Saputra",true],["45e9bc85db231e84a23b8d757136023eabcec13e","ab56adc4e61f1b489818a700493ea97fb9f9e75e@gmail.com","[SPARK-1912] fix compress memory issue during reduce\n\nWhen we need to read a compressed block, we will first create a compress stream instance(LZF or Snappy) and use it to wrap that block.\nLet's say a reducer task need to read 1000 local shuffle blocks, it will first prepare to read that 1000 blocks, which means create 1000 compression stream instance to wrap them. But the initialization of compression instance will allocate some memory and when we have many compression instance at the same time, it is a problem.\nActually reducer reads the shuffle blocks one by one, so we can do the compression instance initialization lazily.\n\nAuthor: Wenchen Fan(Cloud) <cloud0fan@gmail.com>\n\nCloses #860 from cloud-fan/fix-compress and squashes the following commits:\n\n0924a6b [Wenchen Fan(Cloud)] rename 'doWork' into 'getIterator'\n07f32c2 [Wenchen Fan(Cloud)] move the LazyProxyIterator to dataDeserialize\nd80c426 [Wenchen Fan(Cloud)] remove empty lines in short class\n2c8adb2 [Wenchen Fan(Cloud)] add inline comment\n8ebff77 [Wenchen Fan(Cloud)] fix compress memory issue during reduce","Wenchen Fan(Cloud)",true]],"size":3,"ref":"refs/heads/master","head":"45e9bc85db231e84a23b8d757136023eabcec13e"}