GitHub PushEvent at 2014-12-23 00:34:37
Repository: spark
Actor: alexliu68
Payload: {"shas":[["a7ed6f3cc537f57de87d28e8466ca88fbfff53b5","5bf839335c632898d579a3afca899ab31d08a673@gmail.com","[SPARK-4880] remove spark.locality.wait in Analytics\n\nspark.locality.wait set to 100000 in examples/graphx/Analytics.scala.\nShould be left to the user.\n\nAuthor: Ernest <earneyzxl@gmail.com>\n\nCloses #3730 from Earne/SPARK-4880 and squashes the following commits:\n\nd79ed04 [Ernest] remove spark.locality.wait in Analytics","Ernest",true],["d5a596d4188bfa85ff49ee85039f54255c19a4de","d22477479bbf55ba62bd822f4112c80f292c7805@madhu.com","[SPARK-4884]: Improve Partition docs\n\nRewording was based on this discussion: http://apache-spark-developers-list.1001551.n3.nabble.com/RDD-data-flow-td9804.html\nThis is the associated JIRA ticket: https://issues.apache.org/jira/browse/SPARK-4884\n\nAuthor: Madhu Siddalingaiah <madhu@madhu.com>\n\nCloses #3722 from msiddalingaiah/master and squashes the following commits:\n\n79e679f [Madhu Siddalingaiah] [DOC]: improve documentation\n51d14b9 [Madhu Siddalingaiah] Merge remote-tracking branch 'upstream/master'\n38faca4 [Madhu Siddalingaiah] Merge remote-tracking branch 'upstream/master'\ncbccbfe [Madhu Siddalingaiah] Documentation: replace <b> with <code> (again)\n332f7a2 [Madhu Siddalingaiah] Documentation: replace <b> with <code>\ncd2b05a [Madhu Siddalingaiah] Merge remote-tracking branch 'upstream/master'\n0fc12d7 [Madhu Siddalingaiah] Documentation: add description for repartitionAndSortWithinPartitions","Madhu Siddalingaiah",true],["f9f58b9a01c4c7eaf0ce5055d6870e69a22297e3","a15f8b81a160b4eebe5c84e9e3b65c87b9b2f18e@leanplum.com","SPARK-4743 - Use SparkEnv.serializer instead of closureSerializer in aggregateByKey and foldByKey\n\nAuthor: Ivan Vergiliev <ivan@leanplum.com>\n\nCloses #3605 from IvanVergiliev/change-serializer and squashes the following commits:\n\na49b7cf [Ivan Vergiliev] Use serializer instead of closureSerializer in aggregate/foldByKey.","Ivan Vergiliev",true],["105293a7d06b26e7b179a0447eb802074ee9c218","5a7a6a189b47cd31d24667a751e4003cd04eced5@databricks.com","[SPARK-4837] NettyBlockTransferService should use spark.blockManager.port config\n\nThis is used in NioBlockTransferService here:\nhttps://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/network/nio/NioBlockTransferService.scala#L66\n\nAuthor: Aaron Davidson <aaron@databricks.com>\n\nCloses #3688 from aarondav/SPARK-4837 and squashes the following commits:\n\nebd2007 [Aaron Davidson] [SPARK-4837] NettyBlockTransferService should use spark.blockManager.port config","Aaron Davidson",true],["9804a759b68f56eceb8a2f4ea90f76a92b5f9f67","02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","[SPARK-4754] Refactor SparkContext into ExecutorAllocationClient\n\nThis is such that the `ExecutorAllocationManager` does not take in the `SparkContext` with all of its dependencies as an argument. This prevents future developers of this class to tie down this class further with the `SparkContext`, which has really become quite a monstrous object.\n\ncc'ing pwendell who originally suggested this, and JoshRosen who may have thoughts about the trait mix-in style of `SparkContext`.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #3614 from andrewor14/dynamic-allocation-sc and squashes the following commits:\n\n187070d [Andrew Or] Merge branch 'master' of github.com:apache/spark into dynamic-allocation-sc\n59baf6c [Andrew Or] Merge branch 'master' of github.com:apache/spark into dynamic-allocation-sc\n347a348 [Andrew Or] Refactor SparkContext into ExecutorAllocationClient","Andrew Or",true],["f728e0fe7e860fe6dd3437e248472a67a2d435f8","50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","[SPARK-2663] [SQL] Support the Grouping Set\n\nAdd support for `GROUPING SETS`, `ROLLUP`, `CUBE` and the the virtual column `GROUPING__ID`.\n\nMore details on how to use the `GROUPING SETS\" can be found at: https://cwiki.apache.org/confluence/display/Hive/Enhanced+Aggregation,+Cube,+Grouping+and+Rollup\nhttps://issues.apache.org/jira/secure/attachment/12676811/grouping_set.pdf\n\nThe generic idea of the implementations are :\n1 Replace the `ROLLUP`, `CUBE` with `GROUPING SETS`\n2 Explode each of the input row, and then feed them to `Aggregate`\n  * Each grouping set are represented as the bit mask for the `GroupBy Expression List`, for each bit, `1` means the expression is selected, otherwise `0` (left is the lower bit, and right is the higher bit in the `GroupBy Expression List`)\n  * Several of projections are constructed according to the grouping sets, and within each projection(Seq[Expression), we replace those expressions with `Literal(null)` if it's not selected in the grouping set (based on the bit mask)\n  * Output Schema of `Explode` is `child.output :+ grouping__id`\n  * GroupBy Expressions of `Aggregate` is `GroupBy Expression List :+ grouping__id`\n  * Keep the `Aggregation expressions` the same for the `Aggregate`\n\nThe expressions substitutions happen in Logic Plan analyzing, so we will benefit from the Logical Plan optimization (e.g. expression constant folding, and map side aggregation etc.), Only an `Explosive` operator added for Physical Plan, which will explode the rows according the pre-set projections.\n\nA known issue will be done in the follow up PR:\n* Optimization `ColumnPruning` is not supported yet for `Explosive` node.\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #1567 from chenghao-intel/grouping_sets and squashes the following commits:\n\nfe65fcc [Cheng Hao] Remove the extra space\n3547056 [Cheng Hao] Add more doc and Simplify the Expand\na7c869d [Cheng Hao] update code as feedbacks\nd23c672 [Cheng Hao] Add GroupingExpression to replace the Seq[Expression]\n414b165 [Cheng Hao] revert the unnecessary changes\nec276c6 [Cheng Hao] Support Rollup/Cube/GroupingSets","Cheng Hao",true],["b68bc6d2647f8a5caf8aa558e4115f9cc254f67c","a32c402ad794ba99617f2b75e9fb0337358f8060@gmail.com","[SPARK-3928][SQL] Support wildcard matches on Parquet files.\n\n...arquetFile accept hadoop glob pattern in path.\n\nAuthor: Thu Kyaw <trk007@gmail.com>\n\nCloses #3407 from tkyaw/master and squashes the following commits:\n\n19115ad [Thu Kyaw] Merge https://github.com/apache/spark\nceded32 [Thu Kyaw] [SPARK-3928][SQL] Support wildcard matches on Parquet files.\nd322c28 [Thu Kyaw] [SPARK-3928][SQL] Support wildcard matches on Parquet files.\nce677c6 [Thu Kyaw] [SPARK-3928][SQL] Support wildcard matches on Parquet files.","Thu Kyaw",true],["22ddb6e0338f4d101389a0655424a8fde6c4cff4","c135ccafd9da216e05a17f80c5af6fcd4e4020ba@asiainfo.com","[SPARK-4756][SQL] FIX: sessionToActivePool grow infinitely, even as sessions expire\n\n**sessionToActivePool** in **SparkSQLOperationManager** grow infinitely, even as sessions expire.\nwe should remove the pool value when the session closed, even though **sessionToActivePool** would not exist in all of sessions.\n\nAuthor: guowei2 <guowei2@asiainfo.com>\n\nCloses #3617 from guowei2/SPARK-4756 and squashes the following commits:\n\ne9b97b8 [guowei2] fix compile bug with Shim12\ncf0f521 [guowei2] Merge remote-tracking branch 'apache/master' into SPARK-4756\ne070998 [guowei2] fix: remove active pool of the session when it expired","guowei2",true],["e7de7e5f46821e1ba3b070b21d6bcf6d5ec8a796","0a823debfdf289bcc3de00d787978f1c9da07112@tencent.com","[SPARK-4693] [SQL] PruningPredicates may be wrong if predicates contains an empty AttributeSet() references\n\nThe sql \"select * from spark_test::for_test where abs(20141202) is not null\" has predicates=List(IS NOT NULL HiveSimpleUdf#org.apache.hadoop.hive.ql.udf.UDFAbs(20141202)) and\npartitionKeyIds=AttributeSet(). PruningPredicates is List(IS NOT NULL HiveSimpleUdf#org.apache.hadoop.hive.ql.udf.UDFAbs(20141202)). Then the exception \"java.lang.IllegalArgumentException: requirement failed: Partition pruning predicates only supported for partitioned tables.\" is thrown.\nThe sql \"select * from spark_test::for_test_partitioned_table where abs(20141202) is not null and type_id=11 and platform = 3\" with partitioned key insert_date has predicates=List(IS NOT NULL HiveSimpleUdf#org.apache.hadoop.hive.ql.udf.UDFAbs(20141202), (type_id#12 = 11), (platform#8 = 3)) and partitionKeyIds=AttributeSet(insert_date#24). PruningPredicates is List(IS NOT NULL HiveSimpleUdf#org.apache.hadoop.hive.ql.udf.UDFAbs(20141202)).\n\nAuthor: YanTangZhai <hakeemzhai@tencent.com>\nAuthor: yantangzhai <tyz0303@163.com>\n\nCloses #3556 from YanTangZhai/SPARK-4693 and squashes the following commits:\n\n620ebe3 [yantangzhai] [SPARK-4693] [SQL] PruningPredicates may be wrong if predicates contains an empty AttributeSet() references\n37cfdf5 [yantangzhai] [SPARK-4693] [SQL] PruningPredicates may be wrong if predicates contains an empty AttributeSet() references\n70a3544 [yantangzhai] [SPARK-4693] [SQL] PruningPredicates may be wrong if predicates contains an empty AttributeSet() references\nefa9b03 [YanTangZhai] Update HiveQuerySuite.scala\n72accf1 [YanTangZhai] Update HiveQuerySuite.scala\ne572b9a [YanTangZhai] Update HiveStrategies.scala\n6e643f8 [YanTangZhai] Merge pull request #11 from apache/master\ne249846 [YanTangZhai] Merge pull request #10 from apache/master\nd26d982 [YanTangZhai] Merge pull request #9 from apache/master\n76d4027 [YanTangZhai] Merge pull request #8 from apache/master\n03b62b0 [YanTangZhai] Merge pull request #7 from apache/master\n8a00106 [YanTangZhai] Merge pull request #6 from apache/master\ncbcba66 [YanTangZhai] Merge pull request #3 from apache/master\ncdef539 [YanTangZhai] Merge pull request #1 from apache/master","YanTangZhai",true],["7687415c2578b5bdc79c9646c246e52da9a4dd4a","f4572b456463079b9d9dee14a4b6e053ed064605@huawei.com","[SPARK-2554][SQL] Supporting SumDistinct partial aggregation\n\nAdding support to the partial aggregation of SumDistinct\n\nAuthor: ravipesala <ravindra.pesala@huawei.com>\n\nCloses #3348 from ravipesala/SPARK-2554 and squashes the following commits:\n\nfd28e4d [ravipesala] Fixed review comments\ne60e67f [ravipesala] Fixed test cases and made it as nullable\n32fe234 [ravipesala] Supporting SumDistinct partial aggregation Conflicts: \tsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregates.scala","ravipesala",true],["ae9f128608f67cbee0a2fb24754783ee3b4f3098","50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","[SPARK-4573] [SQL] Add SettableStructObjectInspector support in \"wrap\" function\n\nHive UDAF may create an customized object constructed by SettableStructObjectInspector, this is critical when integrate Hive UDAF with the refactor-ed UDAF interface.\n\nPerformance issue in `wrap/unwrap` since more match cases added, will do it in another PR.\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #3429 from chenghao-intel/settable_oi and squashes the following commits:\n\n9f0aff3 [Cheng Hao] update code style issues as feedbacks\n2b0561d [Cheng Hao] Add more scala doc\nf5a40e8 [Cheng Hao] add scala doc\n2977e9b [Cheng Hao] remove the timezone setting for test suite\n3ed284c [Cheng Hao] fix the date type comparison\nf1b6749 [Cheng Hao] Update the comment\n932940d [Cheng Hao] Add more unit test\n72e4332 [Cheng Hao] Add settable StructObjectInspector support","Cheng Hao",true],["c3d91da5ea8b85ca75444ec606f2e1eae376c4b2","f98b0bead8ca0151df080a83631a6c2556256922@huawei.com","[SPARK-4861][SQL] Refactory command in spark sql\n\nRemove ```Command``` and use ```RunnableCommand``` instead.\n\nAuthor: wangfei <wangfei1@huawei.com>\nAuthor: scwf <wangfei1@huawei.com>\n\nCloses #3712 from scwf/cmd and squashes the following commits:\n\n51a82f2 [wangfei] fix test failure\n0e03be8 [wangfei] address comments\n4033bed [scwf] remove CreateTableAsSelect in hivestrategy\n5d20010 [wangfei] address comments\n125f542 [scwf] factory command in spark sql","wangfei",true],["ee1fb97a97d5ac18cd2ad8028e84ecbd988fb811","3270cab15a4ae4af5d46c40db43f5b35eac294d2@gmail.com","[SPARK-4728][MLLib] Add exponential, gamma, and log normal sampling to MLlib da...\n\n...ta generators\n\nThis patch adds:\n\n* Exponential, gamma, and log normal generators that wrap Apache Commons math3 to the private API\n* Functions for generating exponential, gamma, and log normal RDDs and vector RDDs\n* Tests for the above\n\nAuthor: RJ Nowling <rnowling@gmail.com>\n\nCloses #3680 from rnowling/spark4728 and squashes the following commits:\n\n455f50a [RJ Nowling] Add tests for exponential, gamma, and log normal samplers to JavaRandomRDDsSuite\n3e1134a [RJ Nowling] Fix val/var, unncessary creation of Distribution objects when setting seeds, and import line longer than line wrap limits\n58f5b97 [RJ Nowling] Fix bounds in tests so they scale with variance, not stdev\n84fd98d [RJ Nowling] Add more values for testing distributions.\n9f96232 [RJ Nowling] [SPARK-4728] Add exponential, gamma, and log normal sampling to MLlib data generators","RJ Nowling",true],["d7fc69a8b5c92b2bbb71f95027e5283968cd0679","d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","[SPARK-4674] Refactor getCallSite\n\nThe current version of `getCallSite` visits the collection of `StackTraceElement` twice. However, it is unnecessary since we can perform our work with a single visit. We also do not need to keep filtered `StackTraceElement`.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #3532 from viirya/refactor_getCallSite and squashes the following commits:\n\n62aa124 [Liang-Chi Hsieh] Fix style.\ne741017 [Liang-Chi Hsieh] Refactor getCallSite.","Liang-Chi Hsieh",true],["283263ffaa941e7e9ba147cf0ad377d9202d3761","5673041ad99da73cfefa170a7081c0daed5d97b6@cloudera.com","SPARK-3428. TaskMetrics for running tasks is missing GC time metrics\n\nAuthor: Sandy Ryza <sandy@cloudera.com>\n\nCloses #3684 from sryza/sandy-spark-3428 and squashes the following commits:\n\ncb827fe [Sandy Ryza] SPARK-3428. TaskMetrics for running tasks is missing GC time metrics","Sandy Ryza",true],["5479450c45038ddb4485cd676f945a961f2420d8","50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","[SPARK-4901] [SQL] Hot fix for ByteWritables.copyBytes\n\nHiveInspectors.scala failed in compiling with Hadoop 1, as the BytesWritable.copyBytes is not available in Hadoop 1.\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #3742 from chenghao-intel/settable_oi_hotfix and squashes the following commits:\n\nbb04d1f [Cheng Hao] hot fix for ByteWritables.copyBytes","Cheng Hao",true],["8e253ebbf8c5867bd6b626ca8f20699d71aa6ddb","f98b0bead8ca0151df080a83631a6c2556256922@huawei.com","[Build] Remove spark-staging-1038\n\nAuthor: scwf <wangfei1@huawei.com>\n\nCloses #3743 from scwf/abc and squashes the following commits:\n\n7d98bc8 [scwf] removing spark-staging-1038","scwf",true],["336cd341ee449098a1db594592a44f5ab9200fa0","c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","Small refactoring to pass SparkEnv into Executor rather than creating SparkEnv in Executor.\n\nThis consolidates some code path and makes constructor arguments simpler for a few classes.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #3738 from rxin/sparkEnvDepRefactor and squashes the following commits:\n\n82e02cc [Reynold Xin] Fixed couple bugs.\n217062a [Reynold Xin] Code review feedback.\nbd00af7 [Reynold Xin] Small refactoring to pass SparkEnv into Executor rather than creating SparkEnv in Executor.","Reynold Xin",true],["cdb2c645ab769a8678dd81cff44a809fcfa4420b","45d0cd84ae16b71953c1beb3be9ef57773a10b25@gmail.com","[SPARK-4889] update history server example cmds\n\nAuthor: Ryan Williams <ryan.blake.williams@gmail.com>\n\nCloses #3736 from ryan-williams/hist and squashes the following commits:\n\n421d8ff [Ryan Williams] add another random typo fix\n76d6a4c [Ryan Williams] remove hdfs example\na2d0f82 [Ryan Williams] code review feedback\n9ca7629 [Ryan Williams] [SPARK-4889] update history server example cmds","Ryan Williams",true],["7981f969762e77f1752ef8f86c546d4fc32a1a4f","45d0cd84ae16b71953c1beb3be9ef57773a10b25@gmail.com","[SPARK-4896] don’t redundantly overwrite executor JAR deps\n\nAuthor: Ryan Williams <ryan.blake.williams@gmail.com>\n\nCloses #2848 from ryan-williams/fetch-file and squashes the following commits:\n\nc14daff [Ryan Williams] Fix copy that was changed to a move inadvertently\n8e39c16 [Ryan Williams] code review feedback\n788ed41 [Ryan Williams] don’t redundantly overwrite executor JAR deps","Ryan Williams",true]],"size":42,"ref":"refs/heads/master","head":"c233ab3d8d75a33495298964fe73dbf7dd8fe305"}