GitHub PushEvent at 2014-08-28 17:35:40
Repository: spark
Actor: srowen
Payload: {"shas":[["b92d823ad13f6fcc325eeb99563bea543871c6aa","8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","[SPARK-2933] [yarn] Refactor and cleanup Yarn AM code.\n\nThis change modifies the Yarn module so that all the logic related\nto running the ApplicationMaster is localized. Instead of, previously,\n4 different classes with mostly identical code, now we have:\n\n- A single, shared ApplicationMaster class, which can operate both in\n  client and cluster mode, and substitutes the old ApplicationMaster\n  (for cluster mode) and ExecutorLauncher (for client mode).\n\nThe benefit here is that all different execution modes for all supported\nyarn versions use the same shared code for monitoring executor allocation,\nsetting up configuration, and monitoring the process's lifecycle.\n\n- A new YarnRMClient interface, which defines basic RM functionality needed\n  by the ApplicationMaster. This interface has concrete implementations for\n  each supported Yarn version.\n\n- A new YarnAllocator interface, which just abstracts the existing interface\n  of the YarnAllocationHandler class. This is to avoid having to touch the\n  allocator code too much in this change, although it might benefit from a\n  similar effort in the future.\n\nThe end result is much easier to understand code, with much less duplication,\nmaking it much easier to fix bugs, add features, and test everything knowing\nthat all supported versions will behave the same.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #2020 from vanzin/SPARK-2933 and squashes the following commits:\n\n3bbf3e7 [Marcelo Vanzin] Merge branch 'master' into SPARK-2933\nff389ed [Marcelo Vanzin] Do not interrupt reporter thread from within itself.\n3a8ed37 [Marcelo Vanzin] Remote stale comment.\n0f5142c [Marcelo Vanzin] Review feedback.\n41f8c8a [Marcelo Vanzin] Fix app status reporting.\nc0794be [Marcelo Vanzin] Correctly clean up staging directory.\n92770cc [Marcelo Vanzin] Merge branch 'master' into SPARK-2933\necaf332 [Marcelo Vanzin] Small fix to shutdown code.\nf02d3f8 [Marcelo Vanzin] Merge branch 'master' into SPARK-2933\nf581122 [Marcelo Vanzin] Review feedback.\n557fdeb [Marcelo Vanzin] Cleanup a couple more constants.\nbe6068d [Marcelo Vanzin] Restore shutdown hook to clean up staging dir.\n5150993 [Marcelo Vanzin] Some more cleanup.\nb6289ab [Marcelo Vanzin] Move cluster/client code to separate methods.\necb23cd [Marcelo Vanzin] More trivial cleanup.\n34f1e63 [Marcelo Vanzin] Fix some questionable error handling.\n5657c7d [Marcelo Vanzin] Finish app if SparkContext initialization times out.\n0e4be3d [Marcelo Vanzin] Keep \"ExecutorLauncher\" as the main class for client-mode AM.\n91beabb [Marcelo Vanzin] Fix UI filter registration.\n8c72239 [Marcelo Vanzin] Trivial cleanups.\n99a52d5 [Marcelo Vanzin] Changes to the yarn-alpha project to use common AM code.\n848ca6d [Marcelo Vanzin] [SPARK-2933] [yarn] Refactor and cleanup Yarn AM code.","Marcelo Vanzin",true],["d8298c46b7bf566d1cd2f7ea9b1b2b2722dcfb17","41fe2974ceaf2e86e676b04cd45b814c367d805f@gmail.com","[SPARK-3170][CORE][BUG]:RDD info loss in \"StorageTab\" and \"ExecutorTab\"\n\ncompeleted stage only need to remove its own partitions that are no longer cached. However, \"StorageTab\" may lost some rdds which are cached actually. Not only in \"StorageTab\", \"ExectutorTab\" may also lose some rdd info which have been overwritten by last rdd in a same task.\n1. \"StorageTab\": when multiple stages run simultaneously, completed stage will remove rdd info which belong to other stages that are still running.\n2. \"ExectutorTab\": taskcontext may lose some \"updatedBlocks\" info of  rdds  in a dependency chain. Like the following example:\n         val r1 = sc.paralize(..).cache()\n         val r2 = r1.map(...).cache()\n         val n = r2.count()\n\nWhen count the r2, r1 and r2 will be cached finally. So in CacheManager.getOrCompute, the taskcontext should contain \"updatedBlocks\" of r1 and r2. Currently, the \"updatedBlocks\" only contain the info of r2.\n\nAuthor: uncleGen <hustyugm@gmail.com>\n\nCloses #2131 from uncleGen/master_ui_fix and squashes the following commits:\n\na6a8a0b [uncleGen] fix some coding style\n3a1bc15 [uncleGen] fix some error in unit test\n56ea488 [uncleGen] there's some line too long\nc82ba82 [uncleGen] Bug Fix: RDD info loss in \"StorageTab\" and \"ExecutorTab\"","uncleGen",true],["5ac4093c9fa29a11e38f884eebb3f5db087de76f","3467d2f86b976ae9f474f73a80f3bfe04b85d8af@chaordicsystems.com","SPARK-3259 - User data should be given to the master\n\nAuthor: Allan Douglas R. de Oliveira <allan@chaordicsystems.com>\n\nCloses #2162 from douglaz/user_data_master and squashes the following commits:\n\n10d15f6 [Allan Douglas R. de Oliveira] Give user data also to the master","Allan Douglas R. de Oliveira",true],["3b5eb7083d3e1955de288e4fd365dca6221f32fb","9eee3e63534349a30e59f0f71d1c0ae509ca1e91@gmail.com","[SPARK-3118][SQL]add \"SHOW TBLPROPERTIES tblname;\" and \"SHOW COLUMNS (FROM|IN) table_name [(FROM|IN) db_name]\" support\n\nJIRA issue: [SPARK-3118] https://issues.apache.org/jira/browse/SPARK-3118\n\neg:\n> SHOW TBLPROPERTIES test;\nSHOW TBLPROPERTIES test;\nnumPartitions\t0\nnumFiles\t1\ntransient_lastDdlTime\t1407923642\nnumRows\t0\ntotalSize\t82\nrawDataSize\t0\n\neg:\n> SHOW COLUMNS  in test;\nSHOW COLUMNS  in test;\nOK\nTime taken: 0.304 seconds\nid\nstid\nbo\n\nAuthor: u0jing <u9jing@gmail.com>\n\nCloses #2034 from u0jing/spark-3118 and squashes the following commits:\n\nb231d87 [u0jing] add golden answer files\n35f4885 [u0jing] add 'show columns' and 'show tblproperties' support","u0jing",true],["4238c17dc9e1f2f93cc9e6c768f92bd27bf1df66","50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","[SPARK-3197] [SQL] Reduce the Expression tree object creations for aggregation function (min/max)\n\nAggregation function min/max in catalyst will create expression tree for each single row, however, the expression tree creation is quite expensive in a multithreading env currently. Hence we got a very bad performance for the min/max.\nHere is the benchmark that I've done in my local.\n\nMaster | Previous Result (ms) | Current Result (ms)\n------------ | ------------- | -------------\nlocal | 3645 | 3416\nlocal[6] | 3602 | 1002\n\nThe Benchmark source code.\n```\ncase class Record(key: Int, value: Int)\n\nobject TestHive2 extends HiveContext(new SparkContext(\"local[6]\", \"TestSQLContext\", new SparkConf()))\n\nobject DataPrepare extends App {\n  import TestHive2._\n\n  val rdd = sparkContext.parallelize((1 to 10000000).map(i => Record(i % 3000, i)), 12)\n\n  runSqlHive(\"SHOW TABLES\")\n  runSqlHive(\"DROP TABLE if exists a\")\n  runSqlHive(\"DROP TABLE if exists result\")\n  rdd.registerAsTable(\"records\")\n\n  runSqlHive(\"\"\"CREATE TABLE a (key INT, value INT)\n                 | ROW FORMAT SERDE\n                 | 'org.apache.hadoop.hive.serde2.columnar.LazyBinaryColumnarSerDe'\n                 | STORED AS RCFILE\n               \"\"\".stripMargin)\n  runSqlHive(\"\"\"CREATE TABLE result (key INT, value INT)\n                 | ROW FORMAT SERDE\n                 | 'org.apache.hadoop.hive.serde2.columnar.LazyBinaryColumnarSerDe'\n                 | STORED AS RCFILE\n               \"\"\".stripMargin)\n\n  hql(s\"\"\"from records\n             | insert into table a\n             | select key, value\n           \"\"\".stripMargin)\n}\n\nobject PerformanceTest extends App {\n  import TestHive2._\n\n  hql(\"SHOW TABLES\")\n  hql(\"set spark.sql.shuffle.partitions=12\")\n\n  val cmd = \"select min(value), max(value) from a group by key\"\n\n  val results = (\"Result1\", benchmark(cmd)) ::\n                (\"Result2\", benchmark(cmd)) ::\n                (\"Result3\", benchmark(cmd)) :: Nil\n  results.foreach { case (prompt, result) => {\n      println(s\"$prompt: took ${result._1} ms (${result._2} records)\")\n    }\n  }\n\n  def benchmark(cmd: String) = {\n    val begin = System.currentTimeMillis()\n    val count = hql(cmd).count\n    val end = System.currentTimeMillis()\n    ((end - begin), count)\n  }\n}\n```\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #2113 from chenghao-intel/aggregation_expression_optimization and squashes the following commits:\n\ndb40395 [Cheng Hao] remove the transient and add val for the expression property\nd56167d [Cheng Hao] Reduce the Expressions creation","Cheng Hao",true],["191d7cf2a655d032f160b9fa181730364681d0e7","249a0f16da7cd0c4abac5930812cf4806ea7da81@us.ibm.com","[SPARK-3256] Added support for :cp <jar> that was broken in Scala 2.10.x for REPL\n\nAs seen with [SI-6502](https://issues.scala-lang.org/browse/SI-6502) of Scala, the _:cp_ command was broken in Scala 2.10.x. As the Spark shell is a friendly wrapper on top of the Scala REPL, it is also affected by this problem.\n\nMy solution was to alter the internal classpath and invalidate any new entries. I also had to add the ability to add new entries to the parent classloader of the interpreter (SparkIMain's global).\n\nThe advantage of this versus wiping the interpreter and replaying all of the commands is that you don't have to worry about rerunning heavy Spark-related commands (going to the cluster) or potentially reloading data that might have changed. Instead, you get to work from where you left off.\n\nUntil this is fixed upstream for 2.10.x, I had to use reflection to alter the internal compiler classpath.\n\nThe solution now looks like this:\n![screen shot 2014-08-13 at 3 46 02 pm](https://cloud.githubusercontent.com/assets/2481802/3912625/f02b1440-232c-11e4-9bf6-bafb3e352d14.png)\n\nAuthor: Chip Senkbeil <rcsenkbe@us.ibm.com>\n\nCloses #1929 from rcsenkbeil/FixReplClasspathSupport and squashes the following commits:\n\nf420cbf [Chip Senkbeil] Added SparkContext.addJar calls to support executing code on remote clusters\na826795 [Chip Senkbeil] Updated AddUrlsToClasspath to use 'new Run' suggestion over hackish compiler error\n2ff1d86 [Chip Senkbeil] Added compilation failure on symbols hack to get Scala classes to load correctly\na220639 [Chip Senkbeil] Added support for :cp <jar> that was broken in Scala 2.10.x for REPL","Chip Senkbeil",true],["48f42781dedecd38ddcb2dcf67dead92bb4318f5","dfc272d1e78652e4de5b4e76a0545cef11a46a5c@gmail.com","[SPARK-3138][SQL] sqlContext.parquetFile should be able to take a single file as parameter\n\n```if (!fs.getFileStatus(path).isDir) throw Exception``` make no sense after this commit #1370\n\nbe careful if someone is working on SPARK-2551, make sure the new change passes test case ```test(\"Read a parquet file instead of a directory\")```\n\nAuthor: chutium <teng.qiu@gmail.com>\n\nCloses #2044 from chutium/parquet-singlefile and squashes the following commits:\n\n4ae477f [chutium] [SPARK-3138][SQL] sqlContext.parquetFile should be able to take a single file as parameter","chutium",true],["4fa2fda88fc7beebb579ba808e400113b512533b","399a71a219a2bbf15a9ad52c797d50eb921e7e28@gmail.com","[SPARK-2871] [PySpark] add RDD.lookup(key)\n\nRDD.lookup(key)\n\n        Return the list of values in the RDD for key `key`. This operation\n        is done efficiently if the RDD has a known partitioner by only\n        searching the partition that the key maps to.\n\n        >>> l = range(1000)\n        >>> rdd = sc.parallelize(zip(l, l), 10)\n        >>> rdd.lookup(42)  # slow\n        [42]\n        >>> sorted = rdd.sortByKey()\n        >>> sorted.lookup(42)  # fast\n        [42]\n\nIt also clean up the code in RDD.py, and fix several bugs (related to preservesPartitioning).\n\nAuthor: Davies Liu <davies.liu@gmail.com>\n\nCloses #2093 from davies/lookup and squashes the following commits:\n\n1789cd4 [Davies Liu] `f` in foreach could be generator or not.\n2871b80 [Davies Liu] Merge branch 'master' into lookup\nc6390ea [Davies Liu] address all comments\n0f1bce8 [Davies Liu] add test case for lookup()\nbe0e8ba [Davies Liu] fix preservesPartitioning\neb1305d [Davies Liu] add RDD.lookup(key)","Davies Liu",true],["7faf755ae4f0cf510048e432340260a6e609066d","c303fda042c807757c14303e8b8efcad59173751@databricks.com","Spark-3213 Fixes issue with spark-ec2 not detecting slaves created with \"Launch More like this\"\n\n... copy the spark_cluster_tag from a spot instance requests over to the instances.\n\nAuthor: Vida Ha <vida@databricks.com>\n\nCloses #2163 from vidaha/vida/spark-3213 and squashes the following commits:\n\n5070a70 [Vida Ha] Spark-3214 Fix issue with spark-ec2 not detecting slaves created with 'Launch More Like This' and using Spot Requests","Vida Ha",true],["63a053ab140d7bf605e8c5b7fb5a7bd52aca29b2","a5c478d606458f6c868a5126471748acd4730a6f@gmail.com","[SPARK-3243] Don't use stale spark-driver.* system properties\n\nIf we set both `spark.driver.extraClassPath` and `--driver-class-path`, then the latter correctly overrides the former. However, the value of the system property `spark.driver.extraClassPath` still uses the former, which is actually not added to the class path. This may cause some confusion...\n\nOf course, this also affects other options (i.e. java options, library path, memory...).\n\nAuthor: Andrew Or <andrewor14@gmail.com>\n\nCloses #2154 from andrewor14/driver-submit-configs-fix and squashes the following commits:\n\n17ec6fc [Andrew Or] Fix tests\n0140836 [Andrew Or] Don't forget spark.driver.memory\ne39d20f [Andrew Or] Also set spark.driver.extra* configs in client mode","Andrew Or",true],["28d41d627919fcb196d9d31bad65d664770bee67","d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","[SPARK-3252][SQL] Add missing condition for test\n\nAccording to the text message, both relations should be tested. So add the missing condition.\n\nAuthor: viirya <viirya@gmail.com>\n\nCloses #2159 from viirya/fix_test and squashes the following commits:\n\nb1c0f52 [viirya] add missing condition.","viirya",true],["cc275f4b7910f6d0ad266a43bac2fdae58e9739e","5a7a6a189b47cd31d24667a751e4003cd04eced5@databricks.com","[SQL] [SPARK-3236] Reading Parquet tables from Metastore mangles location\n\nCurrently we do `relation.hiveQlTable.getDataLocation.getPath`, which returns the path-part of the URI (e.g., \"s3n://my-bucket/my-path\" => \"/my-path\"). We should do `relation.hiveQlTable.getDataLocation.toString` instead, as a URI's toString returns a faithful representation of the full URI, which can later be passed into a Hadoop Path.\n\nAuthor: Aaron Davidson <aaron@databricks.com>\n\nCloses #2150 from aarondav/parquet-location and squashes the following commits:\n\n459f72c [Aaron Davidson] [SQL] [SPARK-3236] Reading Parquet tables from Metastore mangles location","Aaron Davidson",true],["65253502b913f390b26b9b631380b2c6cf1ccdf7","ae5b4077bc1239c0f5b8129b32bd3d92ff16670c@gmail.com","[SPARK-3065][SQL] Add locale setting to fix results do not match for udf_unix_timestamp format \"yyyy MMM dd h:mm:ss a\" run with not \"America/Los_Angeles\" TimeZone in HiveCompatibilitySuite\n\nWhen run the udf_unix_timestamp of org.apache.spark.sql.hive.execution.HiveCompatibilitySuite testcase\nwith not \"America/Los_Angeles\" TimeZone throws error. [https://issues.apache.org/jira/browse/SPARK-3065]\nadd locale setting on beforeAll and afterAll method to fix the bug of HiveCompatibilitySuite testcase\n\nAuthor: luogankun <luogankun@gmail.com>\n\nCloses #1968 from luogankun/SPARK-3065 and squashes the following commits:\n\nc167832 [luogankun] [SPARK-3065][SQL] Add Locale setting to HiveCompatibilitySuite\n0a25e3a [luogankun] [SPARK-3065][SQL] Add Locale setting to HiveCompatibilitySuite","luogankun",true],["7d2a7a91f263bb9fbf24dc4dbffde8fe5e2c7442","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SPARK-3235][SQL] Ensure in-memory tables don't always broadcast.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #2147 from marmbrus/inMemDefaultSize and squashes the following commits:\n\n5390360 [Michael Armbrust] Merge remote-tracking branch 'origin/master' into inMemDefaultSize\n14204d3 [Michael Armbrust] Set the context before creating SparkLogicalPlans.\n8da4414 [Michael Armbrust] Make sure we throw errors when leaf nodes fail to provide statistcs\n18ce029 [Michael Armbrust] Ensure in-memory tables don't always broadcast.","Michael Armbrust",true],["8712653f11b9730f6e7ef1c99c8c5850154abc56","8bcd50e7b1eea79f965dde19d326001805380997@gmail.com","HOTFIX: Don't build with YARN support for Mapr3","Patrick Wendell",true],["64d8ecbbe94c47236ff2d8c94d7401636ba6fca4","1fa2ef4755a9226cb9a0a4840bd89b158ac71391@redhat.com","Add line continuation for script to work w/ py2.7.5\n\nError was -\n\n$ SPARK_HOME=$PWD/dist ./dev/create-release/generate-changelist.py\n  File \"./dev/create-release/generate-changelist.py\", line 128\n    if day < SPARK_REPO_CHANGE_DATE1 or\n                                      ^\nSyntaxError: invalid syntax\n\nAuthor: Matthew Farrellee <matt@redhat.com>\n\nCloses #2139 from mattf/master-fix-generate-changelist.py-0 and squashes the following commits:\n\n6b3a900 [Matthew Farrellee] Add line continuation for script to work w/ py2.7.5","Matthew Farrellee",true],["b86277c13232c3e65ce6c6cf7f6ede6a00546485","f98b0bead8ca0151df080a83631a6c2556256922@huawei.com","[SPARK-3271] delete unused methods in Utils\n\ndelete no used method in Utils\n\nAuthor: scwf <wangfei1@huawei.com>\n\nCloses #2160 from scwf/delete-no-use-method and squashes the following commits:\n\nd8f6b0d [scwf] delete no use method in Utils","scwf",true],["f38fab97c7970168f1bd81d4dc202e36322c95e3","422e9eadb3aaafe12bf408af5b8b9c24c196501c@gmail.com","SPARK-3265 Allow using custom ipython executable with pyspark\n\nAlthough you can make pyspark use ipython with `IPYTHON=1`, and also change the python executable with `PYSPARK_PYTHON=...`, you can't use both at the same time because it hardcodes the default ipython script.\n\nThis makes it use the `PYSPARK_PYTHON` variable if present and fall back to default python, similarly to how the default python executable is handled.\n\nSo you can use a custom ipython like so:\n`PYSPARK_PYTHON=./anaconda/bin/ipython IPYTHON_OPTS=\"notebook\" pyspark`\n\nAuthor: Rob O'Dwyer <odwyerrob@gmail.com>\n\nCloses #2167 from robbles/patch-1 and squashes the following commits:\n\nd98e8a9 [Rob O'Dwyer] Allow using custom ipython executable with pyspark","Rob O'Dwyer",true],["dafe343499bbc688e266106e4bb897f9e619834e","a5c478d606458f6c868a5126471748acd4730a6f@gmail.com","[HOTFIX] Wait for EOF only for the PySpark shell\n\nIn `SparkSubmitDriverBootstrapper`, we wait for the parent process to send us an `EOF` before finishing the application. This is applicable for the PySpark shell because we terminate the application the same way. However if we run a python application, for instance, the JVM actually never exits unless it receives a manual EOF from the user. This is causing a few tests to timeout.\n\nWe only need to do this for the PySpark shell because Spark submit runs as a python subprocess only in this case. Thus, the normal Spark shell doesn't need to go through this case even though it is also a REPL.\n\nThanks davies for reporting this.\n\nAuthor: Andrew Or <andrewor14@gmail.com>\n\nCloses #2170 from andrewor14/bootstrap-hotfix and squashes the following commits:\n\n42963f5 [Andrew Or] Do not wait for EOF unless this is the pyspark shell","Andrew Or",true],["024178c57419f915d26414e1b91ea0019c3650db","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[HOTFIX][SQL] Remove cleaning of UDFs\n\nIt is not safe to run the closure cleaner on slaves.  #2153 introduced this which broke all UDF execution on slaves.  Will re-add cleaning of UDF closures in a follow-up PR.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #2174 from marmbrus/fixUdfs and squashes the following commits:\n\n55406de [Michael Armbrust] [HOTFIX] Remove cleaning of UDFs","Michael Armbrust",true]],"size":22,"ref":"refs/heads/master","head":"76e3ba4264c4a0bc2c33ae6ac862fc40bc302d83"}