GitHub PushEvent at 2014-11-17 19:53:15
Repository: spark
Actor: andrewor14
Payload: {"shas":[["32218307edc6de2b08d5f7a0db6d566081d27197","9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","[SPARK-4372][MLLIB] Make LR and SVM's default parameters consistent in Scala and Python\n\nThe current default regParam is 1.0 and regType is claimed to be none in Python (but actually it is l2), while regParam = 0.0 and regType is L2 in Scala. We should make the default values consistent. This PR sets the default regType to L2 and regParam to 0.01. Note that the default regParam value in LIBLINEAR (and hence scikit-learn) is 1.0. However, we use average loss instead of total loss in our formulation. Hence regParam=1.0 is definitely too heavy.\n\nIn LinearRegression, we set regParam=0.0 and regType=None, because we have separate classes for Lasso and Ridge, both of which use regParam=0.01 as the default.\n\ndavies atalwalkar\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #3232 from mengxr/SPARK-4372 and squashes the following commits:\n\n9979837 [Xiangrui Meng] update Ridge/Lasso to use default regParam 0.01 cast input arguments\nd3ba096 [Xiangrui Meng] change 'none' back to None\n1909a6e [Xiangrui Meng] change default regParam to 0.01 and regType to L2 in LR and SVM","Xiangrui Meng",true],["825709a0b8f9b4bfb2718ecca8efc32be96c5a57","5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","[SPARK-4310][WebUI] Sort 'Submitted' column in Stage page by time\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #3179 from zsxwing/SPARK-4310 and squashes the following commits:\n\nb0d29f5 [zsxwing] Sort 'Submitted' column in Stage page by time","zsxwing",true],["e421072da0ea87e7056cc3f2130ddaafc731530f","ba60166985ef87d02b3f95e141f5ecd3cb447aa2@aliyun.com","[SPARK-3722][Docs]minor improvement and fix in docs\n\nhttps://issues.apache.org/jira/browse/SPARK-3722\n\nAuthor: WangTao <barneystinson@aliyun.com>\n\nCloses #2579 from WangTaoTheTonic/docsWork and squashes the following commits:\n\n6f91cec [WangTao] use more wording express\n29d22fa [WangTao] delete the specified version link\n34cb4ea [WangTao] Update running-on-yarn.md\n4ee1a26 [WangTao] minor improvement and fix in docs","WangTao",true],["77e845ca7726ffee2d6f8e33ea56ec005dde3874","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SPARK-4394][SQL] Data Sources API Improvements\n\nThis PR adds two features to the data sources API:\n - Support for pushing down `IN` filters\n - The ability for relations to optionally provide information about their `sizeInBytes`.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #3260 from marmbrus/sourcesImprovements and squashes the following commits:\n\n9a5e171 [Michael Armbrust] Use method instead of configuration directly\n99c0e6b [Michael Armbrust] Add support for sizeInBytes.\n416f167 [Michael Armbrust] Support for IN in data sources API.\n2a04ab3 [Michael Armbrust] Simplify implementation of InSet.","Michael Armbrust",true],["abd581752f9314791a688690c07ad1bb68cc09fe","9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","[SPARK-4398][PySpark] specialize sc.parallelize(xrange)\n\n`sc.parallelize(range(1 << 20), 1).count()` may take 15 seconds to finish and the rdd object stores the entire list, making task size very large. This PR adds a specialized version for xrange.\n\nJoshRosen davies\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #3264 from mengxr/SPARK-4398 and squashes the following commits:\n\n8953c41 [Xiangrui Meng] follow davies' suggestion\ncbd58e3 [Xiangrui Meng] specialize sc.parallelize(xrange)","Xiangrui Meng",true],["0c56a039a9c5b871422f0fc55ff4394bc077fb34","fa6fdaa41f01a75e5f1a0f7fe39d1224e78e431d@tencent.com","[Spark Core] SPARK-4380 Edit spilling log from MB to B\n\nhttps://issues.apache.org/jira/browse/SPARK-4380\n\nAuthor: Hong Shen <hongshen@tencent.com>\n\nCloses #3243 from shenh062326/spark_change and squashes the following commits:\n\n4653378 [Hong Shen] Edit spilling log from MB to B\n21ee960 [Hong Shen] Edit spilling log from MB to B\ne9145e8 [Hong Shen] Edit spilling log from MB to B\nda761c2 [Hong Shen] Edit spilling log from MB to B\n946351c [Hong Shen] Edit spilling log from MB to B","Hong Shen",true],["5c265ccde0c5594899ec61f9c1ea100ddff52da7","02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@andrewash.com","SPARK-3663 Document SPARK_LOG_DIR and SPARK_PID_DIR\n\nThese descriptions are from the header of spark-daemon.sh\n\nAuthor: Andrew Ash <andrew@andrewash.com>\n\nCloses #2518 from ash211/SPARK-3663 and squashes the following commits:\n\n058b257 [Andrew Ash] Complete hanging clause in SPARK_PID_DIR description\na17cb4b [Andrew Ash] Update docs for default locations per SPARK-4110\naf89096 [Andrew Ash] SPARK-3663 Document SPARK_LOG_DIR and SPARK_PID_DIR","Andrew Ash",true],["156cf3333dcd93304eb5240f5a6466a3a0311957","5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","[SPARK-4313][WebUI][Yarn] Fix link issue of the executor thread dump page in yarn-cluster mode\n\nIn yarn-cluster mode, the Web UI is running behind a yarn proxy server. Some features(or bugs?) of yarn proxy server will break the links for thread dump.\n\n1. Yarn proxy server will do http redirect internally, so if opening `http://example.com:8088/cluster/app/application_1415344371838_0012/executors`, it will fetch `http://example.com:8088/cluster/app/application_1415344371838_0012/executors/` and return the content but won't change the link in the browser. Then when a user clicks `Thread Dump`, it will jump to `http://example.com:8088/proxy/application_1415344371838_0012/threadDump/?executorId=2`. This is a wrong link. The correct link should be `http://example.com:8088/proxy/application_1415344371838_0012/executors/threadDump/?executorId=2`.\n\nAdding \"/\" to the tab links will fix it.\n\n2. Yarn proxy server has a bug about the URL encode/decode. When a user accesses `http://example.com:8088/proxy/application_1415344371838_0006/executors/threadDump/?executorId=%3Cdriver%3E`, the yarn proxy server will require `http://example.com:36429/executors/threadDump/?executorId=%25253Cdriver%25253E`. But Spark web server expects `http://example.com:36429/executors/threadDump/?executorId=%3Cdriver%3E`. Related to [YARN-2844](https://issues.apache.org/jira/browse/YARN-2844).\n\nFor now, it's a tricky approach to bypass the yarn bug.\n\n![threaddump](https://cloud.githubusercontent.com/assets/1000778/4972567/d1ccba64-68ad-11e4-983e-257530cef35a.png)\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #3183 from zsxwing/SPARK-4313 and squashes the following commits:\n\n3379ca8 [zsxwing] Encode the executor id in the thread dump link and update the comment\nabfa063 [zsxwing] Fix link issue of the executor thread dump page in yarn-cluster mode","zsxwing",true],["c258db9ed4104b6eefe9f55f3e3959a3c46c2900","5a4c97c0700b80cc1afd34bbb0ba182b2f96ae39@gmail.com","Update failed assert text to match code in SizeEstimatorSuite\n\nAuthor: Jeff Hammerbacher <jeff.hammerbacher@gmail.com>\n\nCloses #3242 from hammer/patch-1 and squashes the following commits:\n\nf88d635 [Jeff Hammerbacher] Update failed assert text to match code in SizeEstimatorSuite","Jeff Hammerbacher",true],["ade72c436276237f305d6a6aa4b594d43bcc4743","c2bf8a29ab4e9ca2b842ca45622f95cb9164f566@intel.com","[SPARK-4239] [SQL] support view in HiveQl\n\nCurrently still not support view like\n\nCREATE VIEW view3(valoo)\nTBLPROPERTIES (\"fear\" = \"factor\")\nAS SELECT upper(value) FROM src WHERE key=86;\n\nbecause the text in metastore for this view is like\n\nselect \\`_c0\\` as \\`valoo\\` from (select upper(\\`src\\`.\\`value\\`) from \\`default\\`.\\`src\\` where ...) \\`view3\\`\n\nwhile catalyst cannot resolve \\`_c0\\` for this query.\nFor view without colname definition in parentheses, it works fine.\n\nAuthor: Daoyuan Wang <daoyuan.wang@intel.com>\n\nCloses #3131 from adrian-wang/view and squashes the following commits:\n\n8a56fd6 [Daoyuan Wang] michael's comments\ne46c056 [Daoyuan Wang] add some golden file\n079290a [Daoyuan Wang] remove useless import\n88afcad [Daoyuan Wang] support view in HiveQl","Daoyuan Wang",true],["bbd8f5bee81d5788c356977c173dd1edc42c77a3","f070f8f0c50caeda5841963fc7b57e73e59dad56@happy-camper.st","[SPARK-4245][SQL] Fix containsNull of the result ArrayType of CreateArray expression.\n\nThe `containsNull` of the result `ArrayType` of `CreateArray` should be `true` only if the children is empty or there exists nullable child.\n\nAuthor: Takuya UESHIN <ueshin@happy-camper.st>\n\nCloses #3110 from ueshin/issues/SPARK-4245 and squashes the following commits:\n\n6f64746 [Takuya UESHIN] Move equalsIgnoreNullability method into DataType.\n5a90e02 [Takuya UESHIN] Refine InsertIntoHiveType and add some comments.\ncbecba8 [Takuya UESHIN] Fix a test title.\n884ec37 [Takuya UESHIN] Merge branch 'master' into issues/SPARK-4245\n3c5274b [Takuya UESHIN] Add tests to insert data of types ArrayType / MapType / StructType with nullability is false into Hive table.\n41a94a9 [Takuya UESHIN] Replace InsertIntoTable with InsertIntoHiveTable if data types ignoring nullability are same.\n43e6ef5 [Takuya UESHIN] Fix containsNull for empty array.\n778e997 [Takuya UESHIN] Fix containsNull of the result ArrayType of CreateArray expression.","Takuya UESHIN",true],["f5f757e4ed80759dc5668c63d5663651689f8da8","5673041ad99da73cfefa170a7081c0daed5d97b6@cloudera.com","SPARK-4375. no longer require -Pscala-2.10\n\nIt seems like the winds might have moved away from this approach, but wanted to post the PR anyway because I got it working and to show what it would look like.\n\nAuthor: Sandy Ryza <sandy@cloudera.com>\n\nCloses #3239 from sryza/sandy-spark-4375 and squashes the following commits:\n\n0ffbe95 [Sandy Ryza] Enable -Dscala-2.11 in sbt\ncd42d94 [Sandy Ryza] Update doc\nf6644c3 [Sandy Ryza] SPARK-4375 take 2","Sandy Ryza",true],["0cbdb01e1c817e71c4f80de05c4e5bb11510b368","7acf0e49bc9acc50da7e38d41884e81d5f2df305@qq.com","[SPARK-4333][SQL]  Correctly log number of iterations in RuleExecutor\n\nWhen iterator of RuleExecutor breaks, the num of iterator should be (iteration - 1) not (iteration ).Because log looks like \"Fixed point reached for batch ${batch.name} after 3 iterations.\", but it did 2 iterations really!\n\nAuthor: DoingDone9 <799203320@qq.com>\n\nCloses #3180 from DoingDone9/issue_01 and squashes the following commits:\n\n571e2ed [DoingDone9] Update RuleExecutor.scala\n46514b6 [DoingDone9] When iterator of RuleExecutor breaks, the num of iterator should be iteration - 1 not iteration.","DoingDone9",true],["5930f64bf0d2516304b21bd49eac361a54caabdd","79478dbbc8641d1d6545f348f655464aefbda99b@intel.com","[SPARK-4062][Streaming]Add ReliableKafkaReceiver in Spark Streaming Kafka connector\n\nAdd ReliableKafkaReceiver in Kafka connector to prevent data loss if WAL in Spark Streaming is enabled. Details and design doc can be seen in [SPARK-4062](https://issues.apache.org/jira/browse/SPARK-4062).\n\nAuthor: jerryshao <saisai.shao@intel.com>\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\nAuthor: Saisai Shao <saisai.shao@intel.com>\n\nCloses #2991 from jerryshao/kafka-refactor and squashes the following commits:\n\n5461f1c [Saisai Shao] Merge pull request #8 from tdas/kafka-refactor3\neae4ad6 [Tathagata Das] Refectored KafkaStreamSuiteBased to eliminate KafkaTestUtils and made Java more robust.\nfab14c7 [Tathagata Das] minor update.\n149948b [Tathagata Das] Fixed mistake\n14630aa [Tathagata Das] Minor updates.\nd9a452c [Tathagata Das] Minor updates.\nec2e95e [Tathagata Das] Removed the receiver's locks and essentially reverted to Saisai's original design.\n2a20a01 [jerryshao] Address some comments\n9f636b3 [Saisai Shao] Merge pull request #5 from tdas/kafka-refactor\nb2b2f84 [Tathagata Das] Refactored Kafka receiver logic and Kafka testsuites\ne501b3c [jerryshao] Add Mima excludes\nb798535 [jerryshao] Fix the missed issue\ne5e21c1 [jerryshao] Change to while loop\nea873e4 [jerryshao] Further address the comments\n98f3d07 [jerryshao] Fix comment style\n4854ee9 [jerryshao] Address all the comments\n96c7a1d [jerryshao] Update the ReliableKafkaReceiver unit test\n8135d31 [jerryshao] Fix flaky test\na949741 [jerryshao] Address the comments\n16bfe78 [jerryshao] Change the ordering of imports\n0894aef [jerryshao] Add some comments\n77c3e50 [jerryshao] Code refactor and add some unit tests\ndd9aeeb [jerryshao] Initial commit for reliable Kafka receiver","jerryshao",true],["a0300ea32a9d92bd51c72930bc3979087b0082b2","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SPARK-4390][SQL] Handle NaN cast to decimal correctly\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #3256 from marmbrus/NanDecimal and squashes the following commits:\n\n4c3ba46 [Michael Armbrust] fix style\nd360f83 [Michael Armbrust] Handle NaN cast to decimal","Michael Armbrust",true],["e47c38763914aaf89a7a851c5f41b7549a75615b","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SPARK-4391][SQL] Configure parquet filters using SQLConf\n\nThis is more uniform with the rest of SQL configuration and allows it to be turned on and off without restarting the SparkContext.  In this PR I also turn off filter pushdown by default due to a number of outstanding issues (in particular SPARK-4258).  When those are fixed we should turn it back on by default.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #3258 from marmbrus/parquetFilters and squashes the following commits:\n\n5655bfe [Michael Armbrust] Remove extra line.\n15e9a98 [Michael Armbrust] Enable filters for tests\n75afd39 [Michael Armbrust] Fix comments\n78fa02d [Michael Armbrust] off by default\ne7f9e16 [Michael Armbrust] First draft of correctly configuring parquet filter pushdown","Michael Armbrust",true],["f805025e8efe9cd522e8875141ec27df8d16bbe0","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SQL] Minor cleanup of comments, errors and override.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #3257 from marmbrus/minorCleanup and squashes the following commits:\n\nd8b5abc [Michael Armbrust] Use interpolation.\n2fdf903 [Michael Armbrust] Better error message when coalesce can't be resolved.\nf9fa6cf [Michael Armbrust] Methods in a final class do not also need to be final, use override.\n199fd98 [Michael Armbrust] Fix typo","Michael Armbrust",true],["4b4b50c9e596673c1534df97effad50d107a8007","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SQL] Don't shuffle code generated rows\n\nWhen sort based shuffle and code gen are on we were trying to ship the code generated rows during a shuffle.  This doesn't work because the classes don't exist on the other side.  Instead we now copy into a generic row before shipping.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #3263 from marmbrus/aggCodeGen and squashes the following commits:\n\nf6ba8cf [Michael Armbrust] fix and test","Michael Armbrust",true],["0c7b66bd449093bb5d2dafaf91d54e63e601e320","e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","[SPARK-4322][SQL] Enables struct fields as sub expressions of grouping fields\n\nWhile resolving struct fields, the resulted `GetField` expression is wrapped with an `Alias` to make it a named expression. Assume `a` is a struct instance with a field `b`, then `\"a.b\"` will be resolved as `Alias(GetField(a, \"b\"), \"b\")`. Thus, for this following SQL query:\n\n```sql\nSELECT a.b + 1 FROM t GROUP BY a.b + 1\n```\n\nthe grouping expression is\n\n```scala\nAdd(GetField(a, \"b\"), Literal(1, IntegerType))\n```\n\nwhile the aggregation expression is\n\n```scala\nAdd(Alias(GetField(a, \"b\"), \"b\"), Literal(1, IntegerType))\n```\n\nThis mismatch makes the above SQL query fail during the both analysis and execution phases. This PR fixes this issue by removing the alias when substituting aggregation expressions.\n\n<!-- Reviewable:start -->\n[<img src=\"https://reviewable.io/review_button.png\" height=40 alt=\"Review on Reviewable\"/>](https://reviewable.io/reviews/apache/spark/3248)\n<!-- Reviewable:end -->\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #3248 from liancheng/spark-4322 and squashes the following commits:\n\n23a46ea [Cheng Lian] Code simplification\ndd20a79 [Cheng Lian] Should only trim aliases around `GetField`s\n7f46532 [Cheng Lian] Enables struct fields as sub expressions of grouping fields","Cheng Lian",true],["f76b9683706232c3d4e8e6e61627b8188dcb79dc","1cd02e31b43620d7c664e038ca42a060d61727b9@dontcallme.com","[SPARK-4386] Improve performance when writing Parquet files.\n\nIf you profile the writing of a Parquet file, the single worst time consuming call inside of org.apache.spark.sql.parquet.MutableRowWriteSupport.write is actually in the scala.collection.AbstractSequence.size call. This is because the size call actually ends up COUNTING the elements in a scala.collection.LinearSeqOptimized.length (\"optimized?\").\n\nThis doesn't need to be done. \"size\" is called repeatedly where needed rather than called once at the top of the method and stored in a 'val'.\n\nAuthor: Jim Carroll <jim@dontcallme.com>\n\nCloses #3254 from jimfcarroll/parquet-perf and squashes the following commits:\n\n30cc0b5 [Jim Carroll] Improve performance when writing Parquet files.","Jim Carroll",true]],"size":41,"ref":"refs/heads/master","head":"dbb9da5c3d86301eb1b2f06857d6501552503a63"}