GitHub PushEvent at 2014-05-09 18:08:22
Repository: spark
Actor: kanzhang
Payload: {"shas":[["9c65fa76f9d413e311a80f29d35d3ff7722e9476","9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","[SPARK-1212, Part II] Support sparse data in MLlib\n\nIn PR https://github.com/apache/spark/pull/117, we added dense/sparse vector data model and updated KMeans to support sparse input. This PR is to replace all other `Array[Double]` usage by `Vector` in generalized linear models (GLMs) and Naive Bayes. Major changes:\n\n1. `LabeledPoint` becomes `LabeledPoint(Double, Vector)`.\n2. Methods that accept `RDD[Array[Double]]` now accept `RDD[Vector]`. We cannot support both in an elegant way because of type erasure.\n3. Mark 'createModel' and 'predictPoint' protected because they are not for end users.\n4. Add libSVMFile to MLContext.\n5. NaiveBayes can accept arbitrary labels (introducing a breaking change to Python's `NaiveBayesModel`).\n6. Gradient computation no longer creates temp vectors.\n7. Column normalization and centering are removed from Lasso and Ridge because the operation will densify the data. Simple feature transformation can be done before training.\n\nTODO:\n1. ~~Use axpy when possible.~~\n2. ~~Optimize Naive Bayes.~~\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #245 from mengxr/vector and squashes the following commits:\n\neb6e793 [Xiangrui Meng] move libSVMFile to MLUtils and rename to loadLibSVMData\nc26c4fc [Xiangrui Meng] update DecisionTree to use RDD[Vector]\n11999c7 [Xiangrui Meng] Merge branch 'master' into vector\nf7da54b [Xiangrui Meng] add minSplits to libSVMFile\nda25e24 [Xiangrui Meng] revert the change to default addIntercept because it might change the behavior of existing code without warning\n493f26f [Xiangrui Meng] Merge branch 'master' into vector\n7c1bc01 [Xiangrui Meng] add a TODO to NB\nb9b7ef7 [Xiangrui Meng] change default value of addIntercept to false\nb01df54 [Xiangrui Meng] allow to change or clear threshold in LR and SVM\n4addc50 [Xiangrui Meng] merge master\n4ca5b1b [Xiangrui Meng] remove normalization from Lasso and update tests\nf04fe8a [Xiangrui Meng] remove normalization from RidgeRegression and update tests\nd088552 [Xiangrui Meng] use static constructor for MLContext\n6f59eed [Xiangrui Meng] update libSVMFile to determine number of features automatically\n3432e84 [Xiangrui Meng] update NaiveBayes to support sparse data\n0f8759b [Xiangrui Meng] minor updates to NB\nb11659c [Xiangrui Meng] style update\n78c4671 [Xiangrui Meng] add libSVMFile to MLContext\nf0fe616 [Xiangrui Meng] add a test for sparse linear regression\n44733e1 [Xiangrui Meng] use in-place gradient computation\ne981396 [Xiangrui Meng] use axpy in Updater\ndb808a1 [Xiangrui Meng] update JavaLR example\nbefa592 [Xiangrui Meng] passed scala/java tests\n75c83a4 [Xiangrui Meng] passed test compile\n1859701 [Xiangrui Meng] passed compile\n834ada2 [Xiangrui Meng] optimized MLUtils.computeStats update some ml algorithms to use Vector (cont.)\n135ab72 [Xiangrui Meng] merge glm\n0e57aa4 [Xiangrui Meng] update Lasso and RidgeRegression to parse the weights correctly from GLM mark createModel protected mark predictPoint protected\nd7f629f [Xiangrui Meng] fix a bug in GLM when intercept is not used\n3f346ba [Xiangrui Meng] update some ml algorithms to use Vector","Xiangrui Meng",false],["47ebea5468df2e4f94ef493c5403fcdcda8c5eb2","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SQL] SPARK-1364 Improve datatype and test coverage for ScalaReflection schema inference.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #293 from marmbrus/reflectTypes and squashes the following commits:\n\nf54e8e8 [Michael Armbrust] Improve datatype and test coverage for ScalaReflection schema inference.","Michael Armbrust",false],["92a86b285f8a4af1bdf577dd4c4ea0fd5ca8d682","47f947204c0a430d1f3d77ee35599c230fda5709@gmail.com","[SPARK-1398] Removed findbugs jsr305 dependency\n\nShould be a painless upgrade, and does offer some significant advantages should we want to leverage FindBugs more during the 1.0 lifecycle. http://findbugs.sourceforge.net/findbugs2.html\n\nAuthor: Mark Hamstra <markhamstra@gmail.com>\n\nCloses #307 from markhamstra/findbugs and squashes the following commits:\n\n99f2d09 [Mark Hamstra] Removed unnecessary findbugs jsr305 dependency","Mark Hamstra",false],["fbebaedf26286ee8a75065822a3af1148351f828","bc9cb193f5667274b82836d1631850a6eec5a1f5@iki.fi","Spark parquet improvements\n\nA few improvements to the Parquet support for SQL queries:\n- Instead of files a ParquetRelation is now backed by a directory, which simplifies importing data from other\n  sources\n- InsertIntoParquetTable operation now supports switching between overwriting or appending (at least in\n  HiveQL)\n- tests now use the new API\n- Parquet logging can be set to WARNING level (Default)\n- Default compression for Parquet files (GZIP, as in parquet-mr)\n\nAuthor: Andre Schumacher <andre.schumacher@iki.fi>\n\nCloses #195 from AndreSchumacher/spark_parquet_improvements and squashes the following commits:\n\n54df314 [Andre Schumacher] SPARK-1383 [SQL] Improvements to ParquetRelation","Andre Schumacher",false],["5d1feda217d25616d190f9bb369664e57417cd45","50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","[SPARK-1360] Add Timestamp Support for SQL\n\nThis PR includes:\n1) Add new data type Timestamp\n2) Add more data type casting base on Hive's Rule\n3) Fix bug missing data type in both parsers (HiveQl & SQLParser).\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #275 from chenghao-intel/timestamp and squashes the following commits:\n\ndf709e5 [Cheng Hao] Move orc_ends_with_nulls to blacklist\n24b04b0 [Cheng Hao] Put 3 cases into the black lists(describe_pretty,describe_syntax,lateral_view_outer)\nfc512c2 [Cheng Hao] remove the unnecessary data type equality check in data casting\nd0d1919 [Cheng Hao] Add more data type for scala reflection\n3259808 [Cheng Hao] Add the new Golden files\n3823b97 [Cheng Hao] Update the UnitTest cases & add timestamp type for HiveQL\n54a0489 [Cheng Hao] fix bug mapping to 0 (which is supposed to be null) when NumberFormatException occurs\n9cb505c [Cheng Hao] Fix issues according to PR comments\ne529168 [Cheng Hao] Fix bug of converting from String\n6fc8100 [Cheng Hao] Update Unit Test & CodeStyle\n8a1d4d6 [Cheng Hao] Add DataType for SqlParser\nce4385e [Cheng Hao] Add TimestampType Support","Cheng Hao",false],["c1ea3afb516c204925259f0928dfb17d0fa89621","38ba40da102c3b82607690471c403766355c3e07@imaginea.com","Spark 1162 Implemented takeOrdered in pyspark.\n\nSince python does not have a library for max heap and usual tricks like inverting values etc.. does not work for all cases.\n\nWe have our own implementation of max heap.\n\nAuthor: Prashant Sharma <prashant.s@imaginea.com>\n\nCloses #97 from ScrapCodes/SPARK-1162/pyspark-top-takeOrdered2 and squashes the following commits:\n\n35f86ba [Prashant Sharma] code review\n2b1124d [Prashant Sharma] fixed tests\ne8a08e2 [Prashant Sharma] Code review comments.\n49e6ba7 [Prashant Sharma] SPARK-1162 added takeOrdered to pyspark","Prashant Sharma",false],["b8f534196f9a8c99f75728a06e62282d139dee28","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SQL] SPARK-1333 First draft of java API\n\nWIP: Some work remains...\n * [x] Hive support\n * [x] Tests\n * [x] Update docs\n\nFeedback welcome!\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #248 from marmbrus/javaSchemaRDD and squashes the following commits:\n\nb393913 [Michael Armbrust] @srowen 's java style suggestions.\nf531eb1 [Michael Armbrust] Address matei's comments.\n33a1b1a [Michael Armbrust] Ignore JavaHiveSuite.\n822f626 [Michael Armbrust] improve docs.\nab91750 [Michael Armbrust] Improve Java SQL API: * Change JavaRow => Row * Add support for querying RDDs of JavaBeans * Docs * Tests * Hive support\n0b859c8 [Michael Armbrust] First draft of java API.","Michael Armbrust",false],["a599e43d6e0950f6b6b32150ce264a8c2711470c","9e5fb37886959f2d43529d2ae616012951fc2577@cloudera.com","[SPARK-1134] Fix and document passing of arguments to IPython\n\nThis is based on @dianacarroll's previous pull request https://github.com/apache/spark/pull/227, and @joshrosen's comments on https://github.com/apache/spark/pull/38. Since we do want to allow passing arguments to IPython, this does the following:\n* It documents that IPython can't be used with standalone jobs for now. (Later versions of IPython will deal with PYTHONSTARTUP properly and enable this, see https://github.com/ipython/ipython/pull/5226, but no released version has that fix.)\n* If you run `pyspark` with `IPYTHON=1`, it passes your command-line arguments to it. This way you can do stuff like `IPYTHON=1 bin/pyspark notebook`.\n* The old `IPYTHON_OPTS` remains, but I've removed it from the documentation. This is in case people read an old tutorial that uses it.\n\nThis is not a perfect solution and I'd also be okay with keeping things as they are today (ignoring `$@` for IPython and using IPYTHON_OPTS), and only doing the doc change. With this change though, when IPython fixes https://github.com/ipython/ipython/pull/5226, people will immediately be able to do `IPYTHON=1 bin/pyspark myscript.py` to run a standalone script and get all the benefits of running scripts in IPython (presumably better debugging and such). Without it, there will be no way to run scripts in IPython.\n\n@joshrosen you should probably take the final call on this.\n\nAuthor: Diana Carroll <dcarroll@cloudera.com>\n\nCloses #294 from mateiz/spark-1134 and squashes the following commits:\n\n747bb13 [Diana Carroll] SPARK-1134 bug with ipython prevents non-interactive use with spark; only call ipython if no command line arguments were supplied","Diana Carroll",false],["d94826be6d46edf3bc6377d33787df23a6030a6c","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[BUILD FIX] Fix compilation of Spark SQL Java API.\n\nThe JavaAPI and the Parquet improvements PRs didn't conflict, but broke the build.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #316 from marmbrus/hotFixJavaApi and squashes the following commits:\n\n0b84c2d [Michael Armbrust] Fix compilation of Spark SQL Java API.","Michael Armbrust",false],["9231b011a9ba5a2b25bd3d1a68be7d1a7cb735da","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","Fix jenkins from giving the green light to builds that don't compile.\n\n Adding `| grep` swallows the non-zero return code from sbt failures. See [here](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13735/consoleFull) for a Jenkins run that fails to compile, but still gets a green light.\n\nNote the [BUILD FIX] commit isn't actually part of this PR, but github is out of date.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #317 from marmbrus/fixJenkins and squashes the following commits:\n\n7c77ff9 [Michael Armbrust] Remove output filter that was swallowing non-zero exit codes for test failures.","Michael Armbrust",false],["33e63618d061eeaae257a7350ea3287a702fc123","8bcd50e7b1eea79f965dde19d326001805380997@gmail.com","Revert \"[SPARK-1398] Removed findbugs jsr305 dependency\"\n\nThis reverts commit 92a86b285f8a4af1bdf577dd4c4ea0fd5ca8d682.","Patrick Wendell",false],["ee6e9e7d863022304ac9ced405b353b63accb6ab","8bcd50e7b1eea79f965dde19d326001805380997@gmail.com","SPARK-1337: Application web UI garbage collects newest stages\n\nSimple fix...\n\nAuthor: Patrick Wendell <pwendell@gmail.com>\n\nCloses #320 from pwendell/stage-clean-up and squashes the following commits:\n\n29be62e [Patrick Wendell] SPARK-1337: Application web UI garbage collects newest stages instead old ones","Patrick Wendell",false],["7f32fd42aaadcf6626b4d8bcf6295203b43b2037","5673041ad99da73cfefa170a7081c0daed5d97b6@cloudera.com","SPARK-1350. Always use JAVA_HOME to run executor container JVMs.\n\nAuthor: Sandy Ryza <sandy@cloudera.com>\n\nCloses #313 from sryza/sandy-spark-1350 and squashes the following commits:\n\nbb6d187 [Sandy Ryza] SPARK-1350. Always use JAVA_HOME to run executor container JVMs.","Sandy Ryza",false],["01cf4c402b9fda59680e56112bfaa2b748416d0e","5a7a6a189b47cd31d24667a751e4003cd04eced5@databricks.com","SPARK-1404: Always upgrade spark-env.sh vars to environment vars\n\nThis was broken when spark-env.sh was made idempotent, as the idempotence check is an environment variable, but the spark-env.sh variables may not have been.\n\nTested in zsh, bash, and sh.\n\nAuthor: Aaron Davidson <aaron@databricks.com>\n\nCloses #310 from aarondav/SPARK-1404 and squashes the following commits:\n\nc3406a5 [Aaron Davidson] Add extra export in spark-shell\n6a0e340 [Aaron Davidson] SPARK-1404: Always upgrade spark-env.sh vars to environment vars","Aaron Davidson",false],["f1fa617023d30d8cdc5acef0274bad8cc3e89cea","7853928d03425cf3be35be2fa0c5b99641435d26@gmail.com","[SPARK-1133] Add whole text files reader in MLlib\n\nHere is a pointer to the former [PR164](https://github.com/apache/spark/pull/164).\n\nI add the pull request for the JIRA issue [SPARK-1133](https://spark-project.atlassian.net/browse/SPARK-1133), which brings a new files reader API in MLlib.\n\nAuthor: Xusen Yin <yinxusen@gmail.com>\n\nCloses #252 from yinxusen/whole-files-input and squashes the following commits:\n\n7191be6 [Xusen Yin] refine comments\n0af3faf [Xusen Yin] add JavaAPI test\n01745ee [Xusen Yin] fix deletion error\ncc97dca [Xusen Yin] move whole text file API to Spark core\nd792cee [Xusen Yin] remove the typo character \"+\"\n6bdf2c2 [Xusen Yin] test for small local file system block size\na1f1e7e [Xusen Yin] add two extra spaces\n28cb0fe [Xusen Yin] add whole text files reader","Xusen Yin",false],["16b830888734de260f460506b766edab79d30ecd","5673041ad99da73cfefa170a7081c0daed5d97b6@cloudera.com","SPARK-1375. Additional spark-submit cleanup\n\nAuthor: Sandy Ryza <sandy@cloudera.com>\n\nCloses #278 from sryza/sandy-spark-1375 and squashes the following commits:\n\n5fbf1e9 [Sandy Ryza] SPARK-1375. Additional spark-submit cleanup","Sandy Ryza",false],["a02b535d5e18e987a4b9c4c352838d294f9e853b","8bcd50e7b1eea79f965dde19d326001805380997@gmail.com","Don't create SparkContext in JobProgressListenerSuite.\n\nThis reduces the time of the test from 11 seconds to 20 milliseconds.\n\nAuthor: Patrick Wendell <pwendell@gmail.com>\n\nCloses #324 from pwendell/job-test and squashes the following commits:\n\n868d9eb [Patrick Wendell] Don't create SparkContext in JobProgressListenerSuite.","Patrick Wendell",false],["198892fe8d39a2fad585fa2a7579d8b478456c33","a1a405b82800b492a7268cba6bc58270d2b527de@apache.org","[SPARK-1198] Allow pipes tasks to run in different sub-directories\n\nThis works as is on Linux/Mac/etc but doesn't cover working on Windows.  In here I use ln -sf for symlinks. Putting this up for comments on that. Do we want to create perhaps some classes for doing shell commands - Linux vs Windows.  Is there some other way we want to do this?   I assume we are still supporting jdk1.6?\n\nAlso should I update the Java API for pipes to allow this parameter?\n\nAuthor: Thomas Graves <tgraves@apache.org>\n\nCloses #128 from tgravescs/SPARK1198 and squashes the following commits:\n\nabc1289 [Thomas Graves] remove extra tag in pom file\nba23fc0 [Thomas Graves] Add support for symlink on windows, remove commons-io usage\nda4b221 [Thomas Graves] Merge branch 'master' of https://github.com/tgravescs/spark into SPARK1198\n61be271 [Thomas Graves] Fix file name filter\n6b783bd [Thomas Graves] style fixes\n1ab49ca [Thomas Graves] Add support for running pipe tasks is separate directories","Thomas Graves",false],["d956cc251676d67d87bd6dbfa82be864933d8136","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SQL] Minor fixes.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #315 from marmbrus/minorFixes and squashes the following commits:\n\nb23a15d [Michael Armbrust] fix scaladoc\n11062ac [Michael Armbrust] Fix registering \"SELECT *\" queries as tables and caching them.  As some tests for this and self-joins.\n3997dc9 [Michael Armbrust] Move Row extractor to catalyst.\n208bf5e [Michael Armbrust] More idiomatic naming of DSL functions. * subquery => as * for join condition => on, i.e., `r.join(s, condition = 'a == 'b)` =>`r.join(s, on = 'a == 'b)`\n87211ce [Michael Armbrust] Correctly handle self joins of in-memory cached tables.\n69e195e [Michael Armbrust] Change != to !== in the DSL since != will always translate to != on Any.\n01f2dd5 [Michael Armbrust] Correctly assign aliases to tables in SqlParser.","Michael Armbrust",false],["60e18ce7dd1016647b63586520b713efc45494a8","c481c5e387de8107aa9fe99c851e11e1ff44d0c3@databricks.com","SPARK-1414. Python API for SparkContext.wholeTextFiles\n\nAlso clarified comment on each file having to fit in memory\n\nAuthor: Matei Zaharia <matei@databricks.com>\n\nCloses #327 from mateiz/py-whole-files and squashes the following commits:\n\n9ad64a5 [Matei Zaharia] SPARK-1414. Python API for SparkContext.wholeTextFiles","Matei Zaharia",false]],"size":322,"ref":"refs/heads/SPARK-1118","head":"cb0cc8670bedceb79407554b1da136d869b6c3e9"}