GitHub PushEvent at 2014-09-12 00:09:39
Repository: spark
Actor: davies
Payload: {"shas":[["26862337c97ce14794178d6378fb4155dd24acb9","f98b0bead8ca0151df080a83631a6c2556256922@huawei.com","[SPARK-3193]output errer info when Process exit code is not zero in test suite\n\nhttps://issues.apache.org/jira/browse/SPARK-3193\nI noticed that sometimes pr tests failed due to the Process exitcode != 0,refer to\nhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18688/consoleFull\nhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19118/consoleFull\n\n[info] SparkSubmitSuite:\n[info] - prints usage on empty input\n[info] - prints usage with only --help\n[info] - prints error with unrecognized options\n[info] - handle binary specified but not class\n[info] - handles arguments with --key=val\n[info] - handles arguments to user program\n[info] - handles arguments to user program with name collision\n[info] - handles YARN cluster mode\n[info] - handles YARN client mode\n[info] - handles standalone cluster mode\n[info] - handles standalone client mode\n[info] - handles mesos client mode\n[info] - handles confs with flag equivalents\n[info] - launch simple application with spark-submit *** FAILED ***\n[info]   org.apache.spark.SparkException: Process List(./bin/spark-submit, --class, org.apache.spark.deploy.SimpleApplicationTest, --name, testApp, --master, local, file:/tmp/1408854098404-0/testJar-1408854098404.jar) exited with code 1\n[info]   at org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:872)\n[info]   at org.apache.spark.deploy.SparkSubmitSuite.runSparkSubmit(SparkSubmitSuite.scala:311)\n[info]   at org.apache.spark.deploy.SparkSubmitSuite$$anonfun$14.apply$mcV$sp(SparkSubmitSuite.scala:291)\n[info]   at org.apache.spark.deploy.SparkSubmitSuite$$anonfun$14.apply(SparkSubmitSuite.scala:284)\n[info]   at org.apacSpark assembly has been built with Hive, including Datanucleus jars on classpath\n\nthis PR output the process error info when failed, it can be helpful for diagnosis.\n\nAuthor: scwf <wangfei1@huawei.com>\n\nCloses #2108 from scwf/output-test-error-info and squashes the following commits:\n\n0c48082 [scwf] minor fix according to comments\n563fde1 [scwf] output errer info when Process exitcode not zero","scwf",false],["02b5ac7191c66a866ffedde313eb10f2adfc9b58","38ba40da102c3b82607690471c403766355c3e07@imaginea.com","Minor - Fix trivial compilation warnings.\n\nAuthor: Prashant Sharma <prashant.s@imaginea.com>\n\nCloses #2331 from ScrapCodes/compilation-warn and squashes the following commits:\n\n44c1e76 [Prashant Sharma] Minor - Fix trivial compilation warnings.","Prashant Sharma",false],["07ee4a28c3a502121770f301316cb2256e8f0ce2","28f033f5c5a4d4112d63ef49f10e6664fe6605df@huawei.com","[SPARK-3176] Implement 'ABS and 'LAST' for sql\n\nAdd support for the mathematical function\"ABS\" and the analytic function \"last\" to return a subset of the rows satisfying a query within spark sql. Test-cases included.\n\nAuthor: xinyunh <xinyun.huang@huawei.com>\nAuthor: bomeng <golf8lover>\n\nCloses #2099 from xinyunh/sqlTest and squashes the following commits:\n\n71d15e7 [xinyunh] remove POWER part\n8843643 [xinyunh] fix the code style issue\n39f0309 [bomeng] Modify the code of POWER and ABS. Move them to the file arithmetic\nff8e51e [bomeng] add abs() function support\n7f6980a [xinyunh] fix the bug in 'Last' component\nb3df91b [xinyunh] add 'Last' component","xinyunh",false],["c110614b33a690a3db6ccb1a920fb6a3795aa5a0","a50d2185142f97721f64187b05c8deeec99604b1@gmail.com","[SPARK-3448][SQL] Check for null in SpecificMutableRow.update\n\n`SpecificMutableRow.update` doesn't check for null, and breaks existing `MutableRow` contract.\n\nThe tricky part here is that for performance considerations, the `update` method of all subclasses of `MutableValue` doesn't check for null and sets the null bit to false.\n\nAuthor: Cheng Lian <lian.cs.zju@gmail.com>\n\nCloses #2325 from liancheng/check-for-null and squashes the following commits:\n\n9366c44 [Cheng Lian] Check for null in SpecificMutableRow.update","Cheng Lian",false],["25b5b867d5e18bac1c5bcdc6f8c63d97858194c7","1fa2ef4755a9226cb9a0a4840bd89b158ac71391@redhat.com","[SPARK-3458] enable python \"with\" statements for SparkContext\n\nallow for best practice code,\n\n```\ntry:\n  sc = SparkContext()\n  app(sc)\nfinally:\n  sc.stop()\n```\n\nto be written using a \"with\" statement,\n\n```\nwith SparkContext() as sc:\n  app(sc)\n```\n\nAuthor: Matthew Farrellee <matt@redhat.com>\n\nCloses #2335 from mattf/SPARK-3458 and squashes the following commits:\n\n5b4e37c [Matthew Farrellee] [SPARK-3458] enable python \"with\" statements for SparkContext","Matthew Farrellee",false],["b734ed0c229373dbc589b9eca7327537ca458138","cc463b26613f4ba480f0f94ca5cb2cab678c4fcb@google.com","[SPARK-3395] [SQL] DSL sometimes incorrectly reuses attribute ids, breaking queries\n\nThis resolves https://issues.apache.org/jira/browse/SPARK-3395\n\nAuthor: Eric Liang <ekl@google.com>\n\nCloses #2266 from ericl/spark-3395 and squashes the following commits:\n\n7f2b6f0 [Eric Liang] add regression test\n05bd1e4 [Eric Liang] in the dsl, create a new schema instance in each applySchema","Eric Liang",false],["6f7a76838f15687583e3b0ab43309a3c079368c4","61c510dba8e69cb27c62180d482d39850ebde87b@apache.org","[SPARK-3286] - Cannot view ApplicationMaster UI when Yarn’s url scheme i...\n\n...s https\n\nAuthor: Benoy Antony <benoy@apache.org>\n\nCloses #2276 from benoyantony/SPARK-3286 and squashes the following commits:\n\nc3d51ee [Benoy Antony] Use address with scheme, but Allpha version removes the scheme\ne82f94e [Benoy Antony] Use address with scheme, but Allpha version removes the scheme\n92127c9 [Benoy Antony] rebasing from master\n450c536 [Benoy Antony] [SPARK-3286] - Cannot view ApplicationMaster UI when Yarn’s url scheme is https\nf060c02 [Benoy Antony] [SPARK-3286] - Cannot view ApplicationMaster UI when Yarn’s url scheme is https","Benoy Antony",false],["a0283300c4af5e64a1dc06193245daa1e746b5f4","c2bf8a29ab4e9ca2b842ca45622f95cb9164f566@intel.com","[SPARK-3362][SQL] Fix resolution for casewhen with nulls.\n\nCurrent implementation will ignore else val type.\n\nAuthor: Daoyuan Wang <daoyuan.wang@intel.com>\n\nCloses #2245 from adrian-wang/casewhenbug and squashes the following commits:\n\n3332f6e [Daoyuan Wang] remove wrong comment\n83b536c [Daoyuan Wang] a comment to trigger retest\nd7315b3 [Daoyuan Wang] code improve\need35fc [Daoyuan Wang] bug in casewhen resolve","Daoyuan Wang",false],["f0c87dc86ae65a39cd19370d8d960b4a60854517","c2bf8a29ab4e9ca2b842ca45622f95cb9164f566@intel.com","[SPARK-3363][SQL] Type Coercion should promote null to all other types.\n\nType Coercion should support every type to have null value\n\nAuthor: Daoyuan Wang <daoyuan.wang@intel.com>\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #2246 from adrian-wang/spark3363-0 and squashes the following commits:\n\nc6241de [Daoyuan Wang] minor code clean\n595b417 [Daoyuan Wang] Merge pull request #2 from marmbrus/pr/2246\n832e640 [Michael Armbrust] reduce code duplication\nef6f986 [Daoyuan Wang] make double boolean miss in jsonRDD compatibleType\nc619f0a [Daoyuan Wang] Type Coercion should support every type to have null value","Daoyuan Wang",false],["26503fdf20f4181a2b390c88b83f364e6a4ccc21","e5dd8a3899598b2e85930c5ff06b11c053ef654f@apache.org","[HOTFIX] Fix scala style issue introduced by #2276.","Josh Rosen",false],["1f4a648d4e30e837d6cf3ea8de1808e2254ad70b","5673041ad99da73cfefa170a7081c0daed5d97b6@cloudera.com","SPARK-1713. Use a thread pool for launching executors.\n\nThis patch copies the approach used in the MapReduce application master for launching containers.\n\nAuthor: Sandy Ryza <sandy@cloudera.com>\n\nCloses #663 from sryza/sandy-spark-1713 and squashes the following commits:\n\n036550d [Sandy Ryza] SPARK-1713. [YARN] Use a threadpool for launching executor containers","Sandy Ryza",false],["e4f4886d7148bf48f9e3462b83bfb1ecc7edbe31","ab56adc4e61f1b489818a700493ea97fb9f9e75e@163.com","[SPARK-2096][SQL] Correctly parse dot notations\n\nFirst let me write down the current `projections` grammar of spark sql:\n\n    expression                : orExpression\n    orExpression              : andExpression {\"or\" andExpression}\n    andExpression             : comparisonExpression {\"and\" comparisonExpression}\n    comparisonExpression      : termExpression | termExpression \"=\" termExpression | termExpression \">\" termExpression | ...\n    termExpression            : productExpression {\"+\"|\"-\" productExpression}\n    productExpression         : baseExpression {\"*\"|\"/\"|\"%\" baseExpression}\n    baseExpression            : expression \"[\" expression \"]\" | ... | ident | ...\n    ident                     : identChar {identChar | digit} | delimiters | ...\n    identChar                 : letter | \"_\" | \".\"\n    delimiters                : \",\" | \";\" | \"(\" | \")\" | \"[\" | \"]\" | ...\n    projection                : expression [[\"AS\"] ident]\n    projections               : projection { \",\" projection}\n\nFor something like `a.b.c[1]`, it will be parsed as:\n<img src=\"http://img51.imgspice.com/i/03008/4iltjsnqgmtt_t.jpg\" border=0>\nBut for something like `a[1].b`, the current grammar can't parse it correctly.\nA simple solution is written in `ParquetQuerySuite#NestedSqlParser`, changed grammars are:\n\n    delimiters                : \".\" | \",\" | \";\" | \"(\" | \")\" | \"[\" | \"]\" | ...\n    identChar                 : letter | \"_\"\n    baseExpression            : expression \"[\" expression \"]\" | expression \".\" ident | ... | ident | ...\nThis works well, but can't cover some corner case like `select t.a.b from table as t`:\n<img src=\"http://img51.imgspice.com/i/03008/v2iau3hoxoxg_t.jpg\" border=0>\n`t.a.b` parsed as `GetField(GetField(UnResolved(\"t\"), \"a\"), \"b\")` instead of `GetField(UnResolved(\"t.a\"), \"b\")` using this new grammar.\nHowever, we can't resolve `t` as it's not a filed, but the whole table.(if we could do this, then `select t from table as t` is legal, which is unexpected)\nMy solution is:\n\n    dotExpressionHeader       : ident \".\" ident\n    baseExpression            : expression \"[\" expression \"]\" | expression \".\" ident | ... | dotExpressionHeader  | ident | ...\nI passed all test cases under sql locally and add a more complex case.\n\"arrayOfStruct.field1 to access all values of field1\" is not supported yet. Since this PR has changed a lot of code, I will open another PR for it.\nI'm not familiar with the latter optimize phase, please correct me if I missed something.\n\nAuthor: Wenchen Fan <cloud0fan@163.com>\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #2230 from cloud-fan/dot and squashes the following commits:\n\ne1a8898 [Wenchen Fan] remove support for arbitrary nested arrays\nee8a724 [Wenchen Fan] rollback LogicalPlan, support dot operation on nested array type\na58df40 [Michael Armbrust] add regression test for doubly nested data\n16bc4c6 [Wenchen Fan] some enhance\n95d733f [Wenchen Fan] split long line\ndc31698 [Wenchen Fan] SPARK-2096 Correctly parse dot notations","Wenchen Fan",false],["558962a83fb0758ab5c13ff4ea58cc96c29cbbcc","ba60166985ef87d02b3f95e141f5ecd3cb447aa2@aliyun.com","[SPARK-3411] Improve load-balancing of concurrently-submitted drivers across workers\n\nIf the waiting driver array is too big, the drivers in it will be dispatched to the first worker we get(if it has enough resources), with or without the Randomization.\n\nWe should do randomization every time we dispatch a driver, in order to better balance drivers.\n\nAuthor: WangTaoTheTonic <barneystinson@aliyun.com>\nAuthor: WangTao <barneystinson@aliyun.com>\n\nCloses #1106 from WangTaoTheTonic/fixBalanceDrivers and squashes the following commits:\n\nd1a928b [WangTaoTheTonic] Minor adjustment\nb6560cf [WangTaoTheTonic] solve the shuffle problem for HashSet\nf674e59 [WangTaoTheTonic] add comment and minor fix\n2835929 [WangTao] solve the failed test and avoid filtering\n2ca3091 [WangTao] fix checkstyle\nbc91bb1 [WangTao] Avoid shuffle every time we schedule the driver using round robin\nbbc7087 [WangTaoTheTonic] Optimize the schedule in Master","WangTaoTheTonic",false],["79cdb9b64ad2fa3ab7f2c221766d36658b917c40","8f089d343fe1cdbe17f0b9137eda5a792dd45ee0@alibaba-inc.com","[SPARK-2207][SPARK-3272][MLLib]Add minimum information gain and minimum instances per node as training parameters for decision tree.\n\nThese two parameters can act as early stop rules to do pre-pruning. When a split cause cause left or right child to have less than `minInstancesPerNode` or has less information gain than `minInfoGain`, current node will not be split by this split.\n\nWhen there is no possible splits that satisfy requirements, there is no useful information gain stats, but we still need to calculate the predict value for current node. So I separated calculation of predict from calculation of information gain, which can also save computation when the number of possible splits is large. Please see [SPARK-3272](https://issues.apache.org/jira/browse/SPARK-3272) for more details.\n\nCC: mengxr manishamde jkbradley, please help me review this, thanks.\n\nAuthor: qiping.lqp <qiping.lqp@alibaba-inc.com>\nAuthor: chouqin <liqiping1991@gmail.com>\n\nCloses #2332 from chouqin/dt-preprune and squashes the following commits:\n\nf1d11d1 [chouqin] fix typo\nc7ebaf1 [chouqin] fix typo\n39f9b60 [chouqin] change edge `minInstancesPerNode` to 2 and add one more test\n0278a11 [chouqin] remove `noSplit` and set `Predict` private to tree\nd593ec7 [chouqin] fix docs and change minInstancesPerNode to 1\nefcc736 [qiping.lqp] fix bug\n10b8012 [qiping.lqp] fix style\n6728fad [qiping.lqp] minor fix: remove empty lines\nbb465ca [qiping.lqp] Merge branch 'master' of https://github.com/apache/spark into dt-preprune\ncadd569 [qiping.lqp] add api docs\n46b891f [qiping.lqp] fix bug\ne72c7e4 [qiping.lqp] add comments\n845c6fa [qiping.lqp] fix style\nf195e83 [qiping.lqp] fix style\n987cbf4 [qiping.lqp] fix bug\nff34845 [qiping.lqp] separate calculation of predict of node from calculation of info gain\nac42378 [qiping.lqp] add min info gain and min instances per node parameters in decision tree","qiping.lqp",false],["84e2c8bfe41837baf2aeffa9741e4dbd14351981","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SQL] Add test case with workaround for reading partitioned Avro files\n\nIn order to read from partitioned Avro files we need to also set the `SERDEPROPERTIES` since `TBLPROPERTIES` are not passed to the initialization.  This PR simply adds a test to make sure we don't break this workaround.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #2340 from marmbrus/avroPartitioned and squashes the following commits:\n\n6b969d6 [Michael Armbrust] fix style\nfea2124 [Michael Armbrust] Add test case with workaround for reading partitioned avro files.","Michael Armbrust",false],["f92cde24e8f30dddd5bcec71bb3687498c1406da","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SPARK-3447][SQL] Remove explicit conversion with JListWrapper to avoid NPE\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #2323 from marmbrus/kryoJListNPE and squashes the following commits:\n\n9634f11 [Michael Armbrust] Rollback JSON RDD changes\n4d4d93c [Michael Armbrust] Merge remote-tracking branch 'origin/master' into kryoJListNPE\n646976b [Michael Armbrust] Fix JSON RDD Conversion too\n59065bc [Michael Armbrust] Remove explicit conversion to avoid NPE","Michael Armbrust",false],["c27718f376483dbe6290de612094c8d4ce9b16b4","4742b51e194a83e6f9766502877eb76a945257c4@gmail.com","[SPARK-2781][SQL] Check resolution of LogicalPlans in Analyzer.\n\nLogicalPlan contains a ‘resolved’ attribute indicating that all of its execution requirements have been resolved. This attribute is not checked before query execution. The analyzer contains a step to check that all Expressions are resolved, but this is not equivalent to checking all LogicalPlans. In particular, the Union plan’s implementation of ‘resolved’ verifies that the types of its children’s columns are compatible. Because the analyzer does not check that a Union plan is resolved, it is possible to execute a Union plan that outputs different types in the same column.  See SPARK-2781 for an example.\n\nThis patch adds two checks to the analyzer’s CheckResolution rule. First, each logical plan is checked to see if it is not resolved despite its children being resolved. This allows the ‘problem’ unresolved plan to be included in the TreeNodeException for reporting. Then as a backstop the root plan is checked to see if it is resolved, which recursively checks that the entire plan tree is resolved. Note that the resolved attribute is implemented recursively, and this patch also explicitly checks the resolved attribute on each logical plan in the tree. I assume the query plan trees will not be large enough for this redundant checking to meaningfully impact performance.\n\nBecause this patch starts validating that LogicalPlans are resolved before execution, I had to fix some cases where unresolved plans were passing through the analyzer as part of the implementation of the hive query system. In particular, HiveContext applies the CreateTables and PreInsertionCasts, and ExtractPythonUdfs rules manually after the analyzer runs. I moved these rules to the analyzer stage (for hive queries only), in the process completing a code TODO indicating the rules should be moved to the analyzer.\n\nIt’s worth noting that moving the CreateTables rule means introducing an analyzer rule with a significant side effect - in this case the side effect is creating a hive table. The rule will only attempt to create a table once even if its batch is executed multiple times, because it converts the InsertIntoCreatedTable plan it matches against into an InsertIntoTable. Additionally, these hive rules must be added to the Resolution batch rather than as a separate batch because hive rules rules may be needed to resolve non-root nodes, leaving the root to be resolved on a subsequent batch iteration. For example, the hive compatibility test auto_smb_mapjoin_14, and others, make use of a query plan where the root is a Union and its children are each a hive InsertIntoTable.\n\nMixing the custom hive rules with standard analyzer rules initially resulted in an additional failure because of policy differences between spark sql and hive when casting a boolean to a string. Hive casts booleans to strings as “true” / “false” while spark sql casts booleans to strings as “1” / “0” (causing the cast1.q test to fail). This behavior is a result of the BooleanCasts rule in HiveTypeCoercion.scala, and from looking at the implementation of BooleanCasts I think converting to to “1”/“0” is potentially a programming mistake. (If the BooleanCasts rule is disabled, casting produces “true”/“false” instead.) I believe “true” / “false” should be the behavior for spark sql - I changed the behavior so bools are converted to “true”/“false” to be consistent with hive, and none of the existing spark tests failed.\n\nFinally, in some initial testing with hive it appears that an implicit type coercion of boolean to string results in a lowercase string, e.g. CONCAT( TRUE, “” ) -> “true” while an explicit cast produces an all caps string, e.g. CAST( TRUE AS STRING ) -> “TRUE”.  The change I’ve made just converts to lowercase strings in all cases.  I believe it is at least more correct than the existing spark sql implementation where all Cast expressions become “1” / “0”.\n\nAuthor: Aaron Staple <aaron.staple@gmail.com>\n\nCloses #1706 from staple/SPARK-2781 and squashes the following commits:\n\n32683c4 [Aaron Staple] Fix compilation failure due to merge.\n7c77fda [Aaron Staple] Move ExtractPythonUdfs to Analyzer's extendedRules in HiveContext.\nd49bfb3 [Aaron Staple] Address review comments.\n915b690 [Aaron Staple] Fix merge issue causing compilation failure.\n701dcd2 [Aaron Staple] [SPARK-2781][SQL] Check resolution of LogicalPlans in Analyzer.","Aaron Staple",false],["ed1980ffa9ccb87d76694ba910ef22df034bca49","ad52706cda8c48b376ccaaad9010b6b9783dd9eb@resilientscience.com","[SPARK-2140] Updating heap memory calculation for YARN stable and alpha.\n\nUpdated pull request, reflecting YARN stable and alpha states. I am getting intermittent test failures on my own test infrastructure. Is that tracked anywhere yet?\n\nAuthor: Chris Cope <ccope@resilientscience.com>\n\nCloses #2253 from copester/master and squashes the following commits:\n\n5ad89da [Chris Cope] [SPARK-2140] Removing calculateAMMemory functions since they are no longer needed.\n52b4e45 [Chris Cope] [SPARK-2140] Updating heap memory calculation for YARN stable and alpha.","Chris Cope",false],["a5cdb6359d30f020956206b6c5951ecc7ca66f6e","399a71a219a2bbf15a9ad52c797d50eb921e7e28@gmail.com","Merge branch 'master' into fix_metric","Davies Liu",true]],"size":19,"ref":"refs/heads/fix_metric","head":"a5cdb6359d30f020956206b6c5951ecc7ca66f6e"}