GitHub PushEvent at 2014-09-05 18:44:20
Repository: incubator-spark
Actor: tsdeng
Payload: {"shas":[["7801d44fd3bcf4d82e6db12574cc42bef15bf0e1","8bcd50e7b1eea79f965dde19d326001805380997@gmail.com","Organize configuration docs\n\nThis PR improves and organizes the config option page\nand makes a few other changes to config docs. See a preview here:\nhttp://people.apache.org/~pwendell/config-improvements/configuration.html\n\nThe biggest changes are:\n1. The configs for the standalone master/workers were moved to the\nstandalone page and out of the general config doc.\n2. SPARK_LOCAL_DIRS was missing from the standalone docs.\n3. Expanded discussion of injecting configs with spark-submit, including an\nexample.\n4. Config options were organized into the following categories:\n- Runtime Environment\n- Shuffle Behavior\n- Spark UI\n- Compression and Serialization\n- Execution Behavior\n- Networking\n- Scheduling\n- Security\n- Spark Streaming\n\nAuthor: Patrick Wendell <pwendell@gmail.com>\n\nCloses #880 from pwendell/config-cleanup and squashes the following commits:\n\n93f56c3 [Patrick Wendell] Feedback from Matei\n6f66efc [Patrick Wendell] More feedback\n16ae776 [Patrick Wendell] Adding back header section\nd9c264f [Patrick Wendell] Small fix\ne0c1728 [Patrick Wendell] Response to Matei's review\n27d57db [Patrick Wendell] Reverting changes to index.html (covered in #896)\ne230ef9 [Patrick Wendell] Merge remote-tracking branch 'apache/master' into config-cleanup\na374369 [Patrick Wendell] Line wrapping fixes\nfdff7fc [Patrick Wendell] Merge remote-tracking branch 'apache/master' into config-cleanup\n3289ea4 [Patrick Wendell] Pulling in changes from #856\n106ee31 [Patrick Wendell] Small link fix\nf7e79bc [Patrick Wendell] Re-organizing config options.\n54b184d [Patrick Wendell] Adding standalone configs to the standalone page\n592e94a [Patrick Wendell] Stash\n29b5446 [Patrick Wendell] Better discussion of spark-submit in configuration docs\n2d719ef [Patrick Wendell] Small fix\n4af9e07 [Patrick Wendell] Adding SPARK_LOCAL_DIRS docs\n204b248 [Patrick Wendell] Small fixes","Patrick Wendell",true],["4312cf0bade82d4b54eef80e637953e1633edc3e","9585396914bf96a14f5c1901ae0dc4a228f856dd@radialpoint.com","Spark 1916\n\nThe changes could be ported back to 0.9 as well.\nChanging in.read to in.readFully to read the whole input stream rather than the first 1020 bytes.\nThis should ok considering that Flume caps the body size to 32K by default.\n\nAuthor: David Lemieux <david.lemieux@radialpoint.com>\n\nCloses #865 from lemieud/SPARK-1916 and squashes the following commits:\n\na265673 [David Lemieux] Updated SparkFlumeEvent to read the whole stream rather than the first X bytes.\n(cherry picked from commit 0b769b73fb7ae314325857138a2d3138ed157908)\n\nSigned-off-by: Patrick Wendell <pwendell@gmail.com>","David Lemieux",true],["4dbb27b0cf4eb67c92aad2c1158616312f5a54e6","48d3a847ef684d87177a99cd6c21cdf466f2e98a@qq.com","[SPARK-1712]: TaskDescription instance is too big causes Spark to hang\n\nAuthor: witgo <witgo@qq.com>\n\nCloses #694 from witgo/SPARK-1712_new and squashes the following commits:\n\n0f52483 [witgo] review commit\n83ce29b [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1712_new\n52e6752 [witgo] reset test SparkContext\n63636b6 [witgo] review commit\n44a59ee [witgo] review commit\n3b6d48c [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1712_new\n926bd6a [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1712_new\n9a5cfad [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1712_new\n03cc562 [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1712_new\nb0930b0 [witgo] review commit\nb1174bd [witgo] merge master\nf76679b [witgo] merge master\n689495d [witgo] fix scala style bug\n1d35c3c [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1712_new\n062c182 [witgo] fix small bug for code style\n0a428cf [witgo] add unit tests\n158b2dc [witgo] review commit\n4afe71d [witgo] review commit\n9e4ffa7 [witgo] review commit\n1d35c7d [witgo] fix hang\n7965580 [witgo] fix Statement order\n0e29eac [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1712_new\n3ea1ca1 [witgo] remove duplicate serialize\n743a7ad [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1712_new\n86e2048 [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1712_new\n2a89adc [witgo] SPARK-1712: TaskDescription instance is too big causes Spark to hang","witgo",true],["9cff1dd25abc5e848720d853172ed42e35376fd0","d31b42b93708ab9012ac483e4845e85ada8c7876@gmail.com","Added doctest and method description in context.py\n\nAdded doctest for method textFile and description for methods _initialize_context and _ensure_initialized in context.py\n\nAuthor: Jyotiska NK <jyotiska123@gmail.com>\n\nCloses #187 from jyotiska/pyspark_context and squashes the following commits:\n\n356f945 [Jyotiska NK] Added doctest for textFile method in context.py\n5b23686 [Jyotiska NK] Updated context.py with method descriptions","Jyotiska NK",true],["60b89fe6b09ff896a30d74204876da883e307de7","83c986bd2081d3fd42a4602027c9d1ec5f2ebb59@cse.ohio-state.edu","SPARK-1935: Explicitly add commons-codec 1.5 as a dependency.\n\nAuthor: Yin Huai <huai@cse.ohio-state.edu>\n\nCloses #889 from yhuai/SPARK-1935 and squashes the following commits:\n\n7d50ef1 [Yin Huai] Explicitly add commons-codec 1.5 as a dependency.","Yin Huai",true],["8f7141fbc015addb314e1d5801085587b5cbb171","a50d2185142f97721f64187b05c8deeec99604b1@gmail.com","[SPARK-1368][SQL] Optimized HiveTableScan\n\nJIRA issue: [SPARK-1368](https://issues.apache.org/jira/browse/SPARK-1368)\n\nThis PR introduces two major updates:\n\n- Replaced FP style code with `while` loop and reusable `GenericMutableRow` object in critical path of `HiveTableScan`.\n- Using `ColumnProjectionUtils` to help optimizing RCFile and ORC column pruning.\n\nMy quick micro benchmark suggests these two optimizations made the optimized version 2x and 2.5x faster when scanning CSV table and RCFile table respectively:\n\n```\nOriginal:\n\n[info] CSV: 27676 ms, RCFile: 26415 ms\n[info] CSV: 27703 ms, RCFile: 26029 ms\n[info] CSV: 27511 ms, RCFile: 25962 ms\n\nOptimized:\n\n[info] CSV: 13820 ms, RCFile: 10402 ms\n[info] CSV: 14158 ms, RCFile: 10691 ms\n[info] CSV: 13606 ms, RCFile: 10346 ms\n```\n\nThe micro benchmark loads a 609MB CVS file (structurally similar to the `src` test table) into a normal Hive table with `LazySimpleSerDe` and a RCFile table, then scans these tables respectively.\n\nPreparation code:\n\n```scala\npackage org.apache.spark.examples.sql.hive\n\nimport org.apache.spark.sql.hive.LocalHiveContext\nimport org.apache.spark.{SparkConf, SparkContext}\n\nobject HiveTableScanPrepare extends App {\n  val sparkContext = new SparkContext(\n    new SparkConf()\n      .setMaster(\"local\")\n      .setAppName(getClass.getSimpleName.stripSuffix(\"$\")))\n\n  val hiveContext = new LocalHiveContext(sparkContext)\n\n  import hiveContext._\n\n  hql(\"drop table scan_csv\")\n  hql(\"drop table scan_rcfile\")\n\n  hql(\"\"\"create table scan_csv (key int, value string)\n        |  row format serde 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'\n        |  with serdeproperties ('field.delim'=',')\n      \"\"\".stripMargin)\n\n  hql(s\"\"\"load data local inpath \"${args(0)}\" into table scan_csv\"\"\")\n\n  hql(\"\"\"create table scan_rcfile (key int, value string)\n        |  row format serde 'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe'\n        |stored as\n        |  inputformat 'org.apache.hadoop.hive.ql.io.RCFileInputFormat'\n        |  outputformat 'org.apache.hadoop.hive.ql.io.RCFileOutputFormat'\n      \"\"\".stripMargin)\n\n  hql(\n    \"\"\"\n      |from scan_csv\n      |insert overwrite table scan_rcfile\n      |select scan_csv.key, scan_csv.value\n    \"\"\".stripMargin)\n}\n```\n\nBenchmark code:\n\n```scala\npackage org.apache.spark.examples.sql.hive\n\nimport org.apache.spark.sql.hive.LocalHiveContext\nimport org.apache.spark.{SparkConf, SparkContext}\n\nobject HiveTableScanBenchmark extends App {\n  val sparkContext = new SparkContext(\n    new SparkConf()\n      .setMaster(\"local\")\n      .setAppName(getClass.getSimpleName.stripSuffix(\"$\")))\n\n  val hiveContext = new LocalHiveContext(sparkContext)\n\n  import hiveContext._\n\n  val scanCsv = hql(\"select key from scan_csv\")\n  val scanRcfile = hql(\"select key from scan_rcfile\")\n\n  val csvDuration = benchmark(scanCsv.count())\n  val rcfileDuration = benchmark(scanRcfile.count())\n\n  println(s\"CSV: $csvDuration ms, RCFile: $rcfileDuration ms\")\n\n  def benchmark(f: => Unit) = {\n    val begin = System.currentTimeMillis()\n    f\n    val end = System.currentTimeMillis()\n    end - begin\n  }\n}\n```\n\n@marmbrus Please help review, thanks!\n\nAuthor: Cheng Lian <lian.cs.zju@gmail.com>\n\nCloses #758 from liancheng/fastHiveTableScan and squashes the following commits:\n\n4241a19 [Cheng Lian] Distinguishes sorted and possibly not sorted operations more accurately in HiveComparisonTest\ncf640d8 [Cheng Lian] More HiveTableScan optimisations:\nbf0e7dc [Cheng Lian] Added SortedOperation pattern to match *some* definitely sorted operations and avoid some sorting cost in HiveComparisonTest.\n6d1c642 [Cheng Lian] Using ColumnProjectionUtils to optimise RCFile and ORC column pruning\neb62fd3 [Cheng Lian] [SPARK-1368] Optimized HiveTableScan","Cheng Lian",true],["b7e28fa451511b3b0f849c3d2919ac9c2e4231a1","f264844b57730cf91a9068607b05f0fbe2cff97a@gmail.com","initial version of LPA\n\nA straightforward implementation of LPA algorithm for detecting graph communities using the Pregel framework.  Amongst the growing literature on community detection algorithms in networks, LPA is perhaps the most elementary, and despite its flaws it remains a nice and simple approach.\n\nAuthor: Ankur Dave <ankurdave@gmail.com>\nAuthor: haroldsultan <haroldsultan@gmail.com>\nAuthor: Harold Sultan <haroldsultan@gmail.com>\n\nCloses #905 from haroldsultan/master and squashes the following commits:\n\n327aee0 [haroldsultan] Merge pull request #2 from ankurdave/label-propagation\n227a4d0 [Ankur Dave] Untabify\n0ac574c [haroldsultan] Merge pull request #1 from ankurdave/label-propagation\n0e24303 [Ankur Dave] Add LabelPropagationSuite\n84aa061 [Ankur Dave] LabelPropagation: Fix compile errors and style; rename from LPA\n9830342 [Harold Sultan] initial version of LPA","Ankur Dave",true],["eeee978a348ec2a35cc27865cea6357f9db75b74","38ba40da102c3b82607690471c403766355c3e07@imaginea.com","[SPARK-1820] Make GenerateMimaIgnore @DeveloperApi annotation aware.\n\nWe add all the classes annotated as `DeveloperApi` to `~/.mima-excludes`.\n\nAuthor: Prashant Sharma <prashant.s@imaginea.com>\nAuthor: nikhil7sh <nikhilsharmalnmiit@gmail.ccom>\n\nCloses #904 from ScrapCodes/SPARK-1820/ignore-Developer-Api and squashes the following commits:\n\nde944f9 [Prashant Sharma] Code review.\ne3c5215 [Prashant Sharma] Incorporated patrick's suggestions and fixed the scalastyle build.\n9983a42 [nikhil7sh] [SPARK-1820] Make GenerateMimaIgnore @DeveloperApi annotation aware","Prashant Sharma",true],["c8bf4131bc2a2e147e977159fc90e94b85738830","c481c5e387de8107aa9fe99c851e11e1ff44d0c3@databricks.com","[SPARK-1566] consolidate programming guide, and general doc updates\n\nThis is a fairly large PR to clean up and update the docs for 1.0. The major changes are:\n\n* A unified programming guide for all languages replaces language-specific ones and shows language-specific info in tabs\n* New programming guide sections on key-value pairs, unit testing, input formats beyond text, migrating from 0.9, and passing functions to Spark\n* Spark-submit guide moved to a separate page and expanded slightly\n* Various cleanups of the menu system, security docs, and others\n* Updated look of title bar to differentiate the docs from previous Spark versions\n\nYou can find the updated docs at http://people.apache.org/~matei/1.0-docs/_site/ and in particular http://people.apache.org/~matei/1.0-docs/_site/programming-guide.html.\n\nAuthor: Matei Zaharia <matei@databricks.com>\n\nCloses #896 from mateiz/1.0-docs and squashes the following commits:\n\n03e6853 [Matei Zaharia] Some tweaks to configuration and YARN docs\n0779508 [Matei Zaharia] tweak\nef671d4 [Matei Zaharia] Keep frames in JavaDoc links, and other small tweaks\n1bf4112 [Matei Zaharia] Review comments\n4414f88 [Matei Zaharia] tweaks\nd04e979 [Matei Zaharia] Fix some old links to Java guide\na34ed33 [Matei Zaharia] tweak\n541bb3b [Matei Zaharia] miscellaneous changes\nfcefdec [Matei Zaharia] Moved submitting apps to separate doc\n61d72b4 [Matei Zaharia] stuff\n181f217 [Matei Zaharia] migration guide, remove old language guides\ne11a0da [Matei Zaharia] Add more API functions\n6a030a9 [Matei Zaharia] tweaks\n8db0ae3 [Matei Zaharia] Added key-value pairs section\n318d2c9 [Matei Zaharia] tweaks\n1c81477 [Matei Zaharia] New section on basics and function syntax\ne38f559 [Matei Zaharia] Actually added programming guide to Git\na33d6fe [Matei Zaharia] First pass at updating programming guide to support all languages, plus other tweaks throughout\n3b6a876 [Matei Zaharia] More CSS tweaks\n01ec8bf [Matei Zaharia] More CSS tweaks\ne6d252e [Matei Zaharia] Change color of doc title bar to differentiate from 0.9.0","Matei Zaharia",true],["79fa8fd4b198904ca9ed78a821fd924bd1a82902","38ba40da102c3b82607690471c403766355c3e07@imaginea.com","[SPARK-1971] Update MIMA to compare against Spark 1.0.0\n\nAuthor: Prashant Sharma <prashant.s@imaginea.com>\n\nCloses #910 from ScrapCodes/enable-mima/spark-core and squashes the following commits:\n\n79f3687 [Prashant Sharma] updated Mima to check against version 1.0\n1e8969c [Prashant Sharma] Spark core missed out on Mima settings. So in effect we never tested spark core for mima related errors.","Prashant Sharma",true],["ff562b2396bb59b8c71d800d37b3a36ddac3488f","ec5ca97253741bf083943def42ae4f7f3f58d859@baidu.com","[SPARK-1901] worker should make sure executor has exited before updating executor's info\n\nhttps://issues.apache.org/jira/browse/SPARK-1901\n\nAuthor: Zhen Peng <zhenpeng01@baidu.com>\n\nCloses #854 from zhpengg/bugfix-worker-kills-executor and squashes the following commits:\n\n21d380b [Zhen Peng] add some error messages\n506cea6 [Zhen Peng] add some docs for killProcess()\na0b9860 [Zhen Peng] [SPARK-1901] worker should make sure executor has exited before updating executor's info","Zhen Peng",true],["9c1f204d807403a7cb4099c29077e1f87f466179","02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@andrewash.com","Typo: and -> an\n\nAuthor: Andrew Ash <andrew@andrewash.com>\n\nCloses #927 from ash211/patch-5 and squashes the following commits:\n\n79b577d [Andrew Ash] Typo: and -> an","Andrew Ash",true],["23ae36630aa5e6ca4a86e32912a99408129dff7b","fbfd45890515f7eeeeabf87d7a80d3449c3bce83@gmail.com","updated link to mailing list\n\nAuthor: nchammas <nicholas.chammas@gmail.com>\n\nCloses #923 from nchammas/patch-1 and squashes the following commits:\n\n65c4d18 [nchammas] updated link to mailing list","nchammas",true],["41bfdda3cc53b44b0fb2105533b7c0e1a69eb3bb","fd726277649d3b2e3ee5d332dca6c89739b68aa5@gmail.com","SPARK-1976: fix the misleading part in  streaming docs\n\nSpark streaming requires at least two working threads, but the document gives the example like\n\nimport org.apache.spark.api.java.function._\nimport org.apache.spark.streaming._\nimport org.apache.spark.streaming.api._\n// Create a StreamingContext with a local master\nval ssc = new StreamingContext(\"local\", \"NetworkWordCount\", Seconds(1))\nhttp://spark.apache.org/docs/latest/streaming-programming-guide.html\n\nAuthor: CodingCat <zhunansjtu@gmail.com>\n\nCloses #924 from CodingCat/master and squashes the following commits:\n\nbb89f20 [CodingCat] update streaming docs","CodingCat",true],["cf989601d0e784e1c3507720e64636891fe28292","a50d2185142f97721f64187b05c8deeec99604b1@gmail.com","[SPARK-1959] String \"NULL\" shouldn't be interpreted as null value\n\nJIRA issue: [SPARK-1959](https://issues.apache.org/jira/browse/SPARK-1959)\n\nAuthor: Cheng Lian <lian.cs.zju@gmail.com>\n\nCloses #909 from liancheng/spark-1959 and squashes the following commits:\n\n306659c [Cheng Lian] [SPARK-1959] String \"NULL\" shouldn't be interpreted as null value","Cheng Lian",true],["9ecc40d3aeff0eb113f16df55f4249d8143f37f1","fbb1b1249c4fec363b436c739458ff446a82c033@gmail.com","correct tiny comment error\n\nAuthor: Chen Chao <crazyjvm@gmail.com>\n\nCloses #928 from CrazyJvm/patch-8 and squashes the following commits:\n\n144328b [Chen Chao] correct tiny comment error","Chen Chao",true],["3ce81494c512bc97979a743ea77ef913315f7fb6","f070f8f0c50caeda5841963fc7b57e73e59dad56@happy-camper.st","[SPARK-1947] [SQL] Child of SumDistinct or Average should be widened to prevent overflows the same as Sum.\n\nChild of `SumDistinct` or `Average` should be widened to prevent overflows the same as `Sum`.\n\nAuthor: Takuya UESHIN <ueshin@happy-camper.st>\n\nCloses #902 from ueshin/issues/SPARK-1947 and squashes the following commits:\n\n99c3dcb [Takuya UESHIN] Insert Cast for SumDistinct and Average.","Takuya UESHIN",true],["7463cd248f81975bce9ff864002932864bd5b649","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","Optionally include Hive as a dependency of the REPL.\n\nDue to the way spark-shell launches from an assembly jar, I don't think this change will affect anyone who isn't trying to launch the shell directly from sbt.  That said, it is kinda nice to be able to launch all things directly from SBT when developing.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #801 from marmbrus/hiveRepl and squashes the following commits:\n\n9570571 [Michael Armbrust] Optionally include Hive as a dependency of the REPL.","Michael Armbrust",true],["1a0da0ec5799f3226ce238cac03f53fa4f7c9326","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SQL] SPARK-1964 Add timestamp to hive metastore type parser.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #913 from marmbrus/timestampMetastore and squashes the following commits:\n\n8e0154f [Michael Armbrust] Add timestamp to hive metastore type parser.","Michael Armbrust",true],["7d52777effd0ff41aed545f53d2ab8de2364a188","5a7a6a189b47cd31d24667a751e4003cd04eced5@databricks.com","Super minor: Close inputStream in SparkSubmitArguments\n\n`Properties#load()` doesn't close the InputStream, but it'd be closed after being GC'd anyway...\n\nAlso changed file.getName to file, because getName only shows the filename. This will show the full (possibly relative) path, which is less confusing if it's not found.\n\nAuthor: Aaron Davidson <aaron@databricks.com>\n\nCloses #914 from aarondav/tiny and squashes the following commits:\n\ndb9d072 [Aaron Davidson] Super minor: Close inputStream in SparkSubmitArguments","Aaron Davidson",true]],"size":1000,"ref":"refs/heads/master","head":"7ff8c45d714e0f2315910838b739c0c034672015"}