GitHub PushEvent at 2014-12-11 06:33:12
Repository: spark
Actor: judynash
Payload: {"shas":[["383c5555c9f26c080bc9e3a463aab21dd5b3797f","50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","[SPARK-4785][SQL] Initilize Hive UDFs on the driver and serialize them with a wrapper\n\nDifferent from Hive 0.12.0, in Hive 0.13.1 UDF/UDAF/UDTF (aka Hive function) objects should only be initialized once on the driver side and then serialized to executors. However, not all function objects are serializable (e.g. GenericUDF doesn't implement Serializable). Hive 0.13.1 solves this issue with Kryo or XML serializer. Several utility ser/de methods are provided in class o.a.h.h.q.e.Utilities for this purpose. In this PR we chose Kryo for efficiency. The Kryo serializer used here is created in Hive. Spark Kryo serializer wasn't used because there's no available SparkConf instance.\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #3640 from chenghao-intel/udf_serde and squashes the following commits:\n\n8e13756 [Cheng Hao] Update the comment\n74466a3 [Cheng Hao] refactor as feedbacks\n396c0e1 [Cheng Hao] avoid Simple UDF to be serialized\ne9c3212 [Cheng Hao] update the comment\n19cbd46 [Cheng Hao] support udf instance ser/de after initialization","Cheng Hao",true],["912563aa3553afc0871d5b5858f533aa39cb99e5","5673041ad99da73cfefa170a7081c0daed5d97b6@cloudera.com","SPARK-4338. [YARN] Ditch yarn-alpha.\n\nSorry if this is a little premature with 1.2 still not out the door, but it will make other work like SPARK-4136 and SPARK-2089 a lot easier.\n\nAuthor: Sandy Ryza <sandy@cloudera.com>\n\nCloses #3215 from sryza/sandy-spark-4338 and squashes the following commits:\n\n1c5ac08 [Sandy Ryza] Update building Spark docs and remove unnecessary newline\n9c1421c [Sandy Ryza] SPARK-4338. Ditch yarn-alpha.","Sandy Ryza",true],["61f1a7022767e64ab092aa91c0c5aa1b2fdbef7c","054cbd1708f02aafe84688c98217f6c4f69e823c@gmail.com","[SPARK-874] adding a --wait flag\n\nThis PR adds a --wait flag to the `./sbin/stop-all.sh` script.\n\nAuthor: jbencook <jbenjamincook@gmail.com>\n\nCloses #3567 from jbencook/master and squashes the following commits:\n\nd05c5bb [jbencook] [SPARK-874] adding a --wait flag","jbencook",true],["b31074466a83d3d1387fc1e4337dfab9e164fc04","5cc7173406045a3242d3595588aa610a46f4e47d@asiainfo.com"," [SPARK-4691][shuffle] Restructure a few lines in shuffle code\n\nIn HashShuffleReader.scala and HashShuffleWriter.scala, no need to judge \"dep.aggregator.isEmpty\" again as this is judged by \"dep.aggregator.isDefined\"\n\nIn SortShuffleWriter.scala, \"dep.aggregator.isEmpty\"  is better than \"!dep.aggregator.isDefined\" ?\n\nAuthor: maji2014 <maji3@asiainfo.com>\n\nCloses #3553 from maji2014/spark-4691 and squashes the following commits:\n\nbf7b14d [maji2014] change a elegant way for SortShuffleWriter.scala\n10d0cf0 [maji2014] change a elegant way\nd8f52dc [maji2014] code optimization for judgement","maji2014",true],["1f5110630c1abb13a357b463c805a39772923b82","284a00ea9901227c42561196ff8576aed3ac5631@gmail.com","[SPARK-4765] Make GC time always shown in UI.\n\nThis commit removes the GC time for each task from the set of\noptional, additional metrics, and instead always shows it for\neach task.\n\ncc pwendell\n\nAuthor: Kay Ousterhout <kayousterhout@gmail.com>\n\nCloses #3622 from kayousterhout/gc_time and squashes the following commits:\n\n15ac242 [Kay Ousterhout] Make TaskDetailsClassNames private[spark]\ne71d893 [Kay Ousterhout] [SPARK-4765] Make GC time always shown in UI.","Kay Ousterhout",true],["30dca924df0efbdc1b638fa7c705fe8743570783","f4cfeb6da63cd4b62a6340d67bbf18015ad432f2@xiaomi.com","[SPARK-4714] BlockManager.dropFromMemory() should check whether block has been removed after synchronizing on BlockInfo instance.\n\nAfter synchronizing on the `info` lock in the `removeBlock`/`dropOldBlocks`/`dropFromMemory` methods in BlockManager, the block that `info` represented may have already removed.\n\nThe three methods have the same logic to get the `info` lock:\n```\n   info = blockInfo.get(id)\n   if (info != null) {\n     info.synchronized {\n       // do something\n     }\n   }\n```\n\nSo, there is chance that when a thread enters the `info.synchronized` block, `info` has already been removed from the `blockInfo` map by some other thread who entered `info.synchronized` first.\n\nThe `removeBlock` and `dropOldBlocks` methods are idempotent, so it's safe for them to run on blocks that have already been removed.\nBut in `dropFromMemory` it may be problematic since it may drop block data which already removed into the diskstore, and this calls data store operations that are not designed to handle missing blocks.\n\nThis patch fixes this issue by adding a check to `dropFromMemory` to test whether blocks have been removed by a racing thread.\n\nAuthor: hushan[胡珊] <hushan@xiaomi.com>\n\nCloses #3574 from suyanNone/refine-block-concurrency and squashes the following commits:\n\nedb989d [hushan[胡珊]] Refine code style and comments position\n55fa4ba [hushan[胡珊]] refine code\ne57e270 [hushan[胡珊]] add check info is already remove or not while having gotten info.syn","hushan[胡珊]",true],["5e4c06f8e54265a4024857f5978ec54c936aeea2","5673041ad99da73cfefa170a7081c0daed5d97b6@cloudera.com","SPARK-4567. Make SparkJobInfo and SparkStageInfo serializable\n\nAuthor: Sandy Ryza <sandy@cloudera.com>\n\nCloses #3426 from sryza/sandy-spark-4567 and squashes the following commits:\n\ncb4b8d2 [Sandy Ryza] SPARK-4567. Make SparkJobInfo and SparkStageInfo serializable","Sandy Ryza",true],["d8f84f26e388055ca7459810e001d05ab60af15b","291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","SPARK-4805 [CORE] BlockTransferMessage.toByteArray() trips assertion\n\nAllocate enough room for type byte as well as message, to avoid tripping assertion about capacity of the buffer\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #3650 from srowen/SPARK-4805 and squashes the following commits:\n\n9e1d502 [Sean Owen] Allocate enough room for type byte as well as message, to avoid tripping assertion about capacity of the buffer","Sean Owen",true],["2b9b72682e587909a84d3ace214c22cec830eeaf","c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","[SPARK-4740] Create multiple concurrent connections between two peer nodes in Netty.\n\nIt's been reported that when the number of disks is large and the number of nodes is small, Netty network throughput is low compared with NIO. We suspect the problem is that only a small number of disks are utilized to serve shuffle files at any given point, due to connection reuse. This patch adds a new config parameter to specify the number of concurrent connections between two peer nodes, default to 2.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #3625 from rxin/SPARK-4740 and squashes the following commits:\n\nad4241a [Reynold Xin] Updated javadoc.\nf33c72b [Reynold Xin] Code review feedback.\n0fefabb [Reynold Xin] Use double check in synchronization.\n41dfcb2 [Reynold Xin] Added test case.\n9076b4a [Reynold Xin] Fixed two NPEs.\n3e1306c [Reynold Xin] Minor style fix.\n4f21673 [Reynold Xin] [SPARK-4740] Create multiple concurrent connections between two peer nodes in Netty.","Reynold Xin",true],["9bd9334f588dbb44d01554f9f4ca68a153a48993","c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","Config updates for the new shuffle transport.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #3657 from rxin/conf-update and squashes the following commits:\n\n7370eab [Reynold Xin] Config updates for the new shuffle transport.","Reynold Xin",true],["f79c1cfc997c1a7ddee480ca3d46f5341b69d3b7","e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","[Minor] Use <sup> tag for help icon in web UI page header\n\nThis small commit makes the `(?)` web UI help link into a superscript, which should address feedback that the current design makes it look like an error occurred or like information is missing.\n\nBefore:\n\n![image](https://cloud.githubusercontent.com/assets/50748/5370611/a3ed0034-7fd9-11e4-870f-05bd9faad5b9.png)\n\nAfter:\n\n![image](https://cloud.githubusercontent.com/assets/50748/5370602/6c5ca8d6-7fd9-11e4-8d1a-568d71290aa7.png)\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #3659 from JoshRosen/webui-help-sup and squashes the following commits:\n\nbd72899 [Josh Rosen] Use <sup> tag for help icon in web UI page header.","Josh Rosen",true],["94b377f94487109a1cc3e07dd230b1df7a96e28d","73279e3fb6c013b10d08fa3ac4497d0f034eab9f@oculusinfo.com","[SPARK-4772] Clear local copies of accumulators as soon as we're done with them\n\nAccumulators keep thread-local copies of themselves.  These copies were only cleared at the beginning of a task.  This meant that (a) the memory they used was tied up until the next task ran on that thread, and (b) if a thread died, the memory it had used for accumulators was locked up forever on that worker.\n\nThis PR clears the thread-local copies of accumulators at the end of each task, in the tasks finally block, to make sure they are cleaned up between tasks.  It also stores them in a ThreadLocal object, so that if, for some reason, the thread dies, any memory they are using at the time should be freed up.\n\nAuthor: Nathan Kronenfeld <nkronenfeld@oculusinfo.com>\n\nCloses #3570 from nkronenfeld/Accumulator-Improvements and squashes the following commits:\n\na581f3f [Nathan Kronenfeld] Change Accumulators to private[spark] instead of adding mima exclude to get around false positive in mima tests\nb6c2180 [Nathan Kronenfeld] Include MiMa exclude as per build error instructions - this version incompatibility should be irrelevent, as it will only surface if a master is talking to a worker running a different version of spark.\n537baad [Nathan Kronenfeld] Fuller refactoring as intended, incorporating JR's suggestions for ThreadLocal localAccums, and keeping clear(), but also calling it in tasks' finally block, rather than just at the beginning of the task.\n39a82f2 [Nathan Kronenfeld] Clear local copies of accumulators as soon as we're done with them","Nathan Kronenfeld",true],["742e7093eca8865225c29bacf4344f2e89bfea41","48d3a847ef684d87177a99cd6c21cdf466f2e98a@qq.com","[SPARK-4161]Spark shell class path is not correctly set if \"spark.driver.extraClassPath\" is set in defaults.conf\n\nAuthor: GuoQiang Li <witgo@qq.com>\n\nCloses #3050 from witgo/SPARK-4161 and squashes the following commits:\n\nabb6fa4 [GuoQiang Li] move usejavacp opt to spark-shell\n89e39e7 [GuoQiang Li] review commit\nc2a6f04 [GuoQiang Li] Spark shell class path is not correctly set if \"spark.driver.extraClassPath\" is set in defaults.conf","GuoQiang Li",true],["0fc637b4c27f9afdf5c829d26c7a86efd8681490","7d45abe39fd242ee2f6575da28f13d66568418b3@oss.nttdata.co.jp","[SPARK-4329][WebUI] HistoryPage pagenation\n\nCurrent HistoryPage have links only to previous page or next page.\nI suggest to add index to access history pages easily.\n\nI implemented like following pics.\n\nIf there are many pages, current page +/- N pages, head page and last page are indexed.\n\n![2014-11-10 16 13 25](https://cloud.githubusercontent.com/assets/4736016/4986246/9c7bbac4-6937-11e4-8695-8634d039d5b6.png)\n![2014-11-10 16 03 21](https://cloud.githubusercontent.com/assets/4736016/4986210/3951bb74-6937-11e4-8b4e-9f90d266d736.png)\n![2014-11-10 16 03 39](https://cloud.githubusercontent.com/assets/4736016/4986211/3b196ad8-6937-11e4-9f81-74bc0a6dad5b.png)\n![2014-11-10 16 03 49](https://cloud.githubusercontent.com/assets/4736016/4986213/40686138-6937-11e4-86c0-41100f0404f6.png)\n![2014-11-10 16 04 04](https://cloud.githubusercontent.com/assets/4736016/4986215/4326c9b4-6937-11e4-87ac-0f30c86ec6e3.png)\n\nAuthor: Kousuke Saruta <sarutak@oss.nttdata.co.jp>\n\nCloses #3194 from sarutak/history-page-indexing and squashes the following commits:\n\n15d3d2d [Kousuke Saruta] Simplified code\nc93932e [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into history-page-indexing\n1c2f605 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into history-page-indexing\n76b05e3 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into history-page-indexing\nb2240f8 [Kousuke Saruta] Fixed style\nec7922e [Kousuke Saruta] Simplified code\n755a004 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into history-page-indexing\ncfa242b [Kousuke Saruta] Added index to HistoryPage","Kousuke Saruta",true],["56212831c6436e287a19908e82c26117cbcb16b0","02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","[SPARK-4771][Docs] Document standalone cluster supervise mode\n\ntdas looks like streaming already refers to the supervise mode. The link from there is broken though.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #3627 from andrewor14/document-supervise and squashes the following commits:\n\n9ca0908 [Andrew Or] Wording changes\n2b55ed2 [Andrew Or] Document standalone cluster supervise mode","Andrew Or",true],["faa8fd8178642ef8fce14186abc45a189042efd4","02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","[SPARK-4215] Allow requesting / killing executors only in YARN mode\n\nCurrently this doesn't do anything in other modes, so we might as well just disable it rather than having the user mistakenly rely on it.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #3615 from andrewor14/dynamic-allocation-yarn-only and squashes the following commits:\n\nce6487a [Andrew Or] Allow requesting / killing executors only in YARN mode","Andrew Or",true],["e230da18f8c354b4b80416aa5277420397acf4f2","c2bf8a29ab4e9ca2b842ca45622f95cb9164f566@intel.com","[SPARK-4793] [Deploy] ensure .jar at end of line\n\nsometimes I switch between different version and do not want to rebuild spark, so I rename assembly.jar into .jar.bak, but still caught by `compute-classpath.sh`\n\nAuthor: Daoyuan Wang <daoyuan.wang@intel.com>\n\nCloses #3641 from adrian-wang/jar and squashes the following commits:\n\n45cbfd0 [Daoyuan Wang] ensure .jar at end of line","Daoyuan Wang",true],["447ae2de5d4c2af865fdb63f8b876b865de60f74","17d07ce14ed8a23e11d276b3691423f0ca203e51@capitalone.com","[SPARK-4569] Rename 'externalSorting' in Aggregator\n\nHi all - I've renamed the unhelpfully named variable and added a comment clarifying what's actually happening.\n\nAuthor: Ilya Ganelin <ilya.ganelin@capitalone.com>\n\nCloses #3666 from ilganeli/SPARK-4569B and squashes the following commits:\n\n1810394 [Ilya Ganelin] [SPARK-4569] Rename 'externalSorting' in Aggregator\ne2d2092 [Ilya Ganelin] [SPARK-4569] Rename 'externalSorting' in Aggregator\nd7cefec [Ilya Ganelin] [SPARK-4569] Rename 'externalSorting' in Aggregator\n5b3f39c [Ilya Ganelin] [SPARK-4569] Rename  in Aggregator","Ilya Ganelin",true],["4f93d0cabe5d1fc7c0fd0a33d992fd85df1fecb4","02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","[SPARK-4759] Fix driver hanging from coalescing partitions\n\nThe driver hangs sometimes when we coalesce RDD partitions. See JIRA for more details and reproduction.\n\nThis is because our use of empty string as default preferred location in `CoalescedRDDPartition` causes the `TaskSetManager` to schedule the corresponding task on host `\"\"` (empty string). The intended semantics here, however, is that the partition does not have a preferred location, and the TSM should schedule the corresponding task accordingly.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #3633 from andrewor14/coalesce-preferred-loc and squashes the following commits:\n\ne520d6b [Andrew Or] Oops\n3ebf8bd [Andrew Or] A few comments\nf370a4e [Andrew Or] Fix tests\n2f7dfb6 [Andrew Or] Avoid using empty string as default preferred location","Andrew Or",true],["36bdb5b748ff670a9bafd787e40c9e142c9a85b9","8bcd50e7b1eea79f965dde19d326001805380997@gmail.com","MAINTENANCE: Automated closing of pull requests.\n\nThis commit exists to close the following pull requests on Github:\n\nCloses #2883 (close requested by 'pwendell')\nCloses #3364 (close requested by 'pwendell')\nCloses #4458 (close requested by 'pwendell')\nCloses #1574 (close requested by 'andrewor14')\nCloses #2546 (close requested by 'andrewor14')\nCloses #2516 (close requested by 'andrewor14')\nCloses #154 (close requested by 'andrewor14')","Patrick Wendell",true]],"size":23,"ref":"refs/heads/master","head":"377532cdff819010aef1786f84c987eddb63af45"}