GitHub PushEvent at 2014-09-23 07:21:42
Repository: spark
Actor: sarutak
Payload: {"shas":[["61e21fe7f478e7b06b72851f26b87d99cbbdf117","291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","SPARK-3069 [DOCS] Build instructions in README are outdated\n\nHere's my crack at Bertrand's suggestion. The Github `README.md` contains build info that's outdated. It should just point to the current online docs, and reflect that Maven is the primary build now.\n\n(Incidentally, the stanza at the end about contributions of original work should go in https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark too. It won't hurt to be crystal clear about the agreement to license, given that ICLAs are not required of anyone here.)\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #2014 from srowen/SPARK-3069 and squashes the following commits:\n\n501507e [Sean Owen] Note that Zinc is for Maven builds too\ndb2bd97 [Sean Owen] sbt -> sbt/sbt and add note about zinc\nbe82027 [Sean Owen] Fix additional occurrences of building-with-maven -> building-spark\n91c921f [Sean Owen] Move building-with-maven to building-spark and create a redirect. Update doc links to building-spark.html Add jekyll-redirect-from plugin and make associated config changes (including fixing pygments deprecation). Add example of SBT to README.md\n999544e [Sean Owen] Change \"Building Spark with Maven\" title to \"Building Spark\"; reinstate tl;dr info about dev/run-tests in README.md; add brief note about building with SBT\nc18d140 [Sean Owen] Optionally, remove the copy of contributing text from main README.md\n8e83934 [Sean Owen] Add CONTRIBUTING.md to trigger notice on new pull request page\nb1c04a1 [Sean Owen] Refer to current online documentation for building, and remove slightly outdated copy in README.md","Sean Owen",false],["7b8008f5a4d413b61aa88fbc60959e98e59f17dd","38ba40da102c3b82607690471c403766355c3e07@imaginea.com","[SPARK-2182] Scalastyle rule blocking non ascii characters.\n\n...erators.\n\nAuthor: Prashant Sharma <prashant.s@imaginea.com>\n\nCloses #2358 from ScrapCodes/scalastyle-unicode and squashes the following commits:\n\n12a20f2 [Prashant Sharma] [SPARK-2182] Scalastyle rule blocking (non keyboard typeable) unicode operators.","Prashant Sharma",false],["86d253ec4e2ed94c68687d575f9e2dfbb44463e1","50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","[SPARK-3527] [SQL] Strip the string message\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #2392 from chenghao-intel/trim and squashes the following commits:\n\ne52024f [Cheng Hao] trim the string message","Cheng Hao",false],["9d5fa763d8559ac412a18d7a2f43c4368a0af897","1fa2ef4755a9226cb9a0a4840bd89b158ac71391@redhat.com","[SPARK-3519] add distinct(n) to PySpark\n\nAdded missing rdd.distinct(numPartitions) and associated tests\n\nAuthor: Matthew Farrellee <matt@redhat.com>\n\nCloses #2383 from mattf/SPARK-3519 and squashes the following commits:\n\n30b837a [Matthew Farrellee] Combine test cases to save on JVM startups\n6bc4a2c [Matthew Farrellee] [SPARK-3519] add distinct(n) to SchemaRDD in PySpark\n7a17f2b [Matthew Farrellee] [SPARK-3519] add distinct(n) to PySpark","Matthew Farrellee",false],["7583699873fb4f252c6ce65db1096783ef438731","83c986bd2081d3fd42a4602027c9d1ec5f2ebb59@cse.ohio-state.edu","[SPARK-3308][SQL] Ability to read JSON Arrays as tables\n\nThis PR aims to support reading top level JSON arrays and take every element in such an array as a row (an empty array will not generate a row).\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-3308\n\nAuthor: Yin Huai <huai@cse.ohio-state.edu>\n\nCloses #2400 from yhuai/SPARK-3308 and squashes the following commits:\n\n990077a [Yin Huai] Handle top level JSON arrays.","Yin Huai",false],["30f288ae34a67307aa45b7aecbd0d02a0a14fe69","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SPARK-2890][SQL] Allow reading of data when case insensitive resolution could cause possible ambiguity.\n\nThrowing an error in the constructor makes it possible to run queries, even when there is no actual ambiguity.  Remove this check in favor of throwing an error in analysis when they query is actually is ambiguous.\n\nAlso took the opportunity to add test cases that would have caught a subtle bug in my first attempt at fixing this and refactor some other test code.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #2209 from marmbrus/sameNameStruct and squashes the following commits:\n\n729cca4 [Michael Armbrust] Better tests.\na003aeb [Michael Armbrust] Remove error (it'll be caught in analysis).","Michael Armbrust",false],["8e7ae477ba40a064d27cf149aa211ff6108fe239","4742b51e194a83e6f9766502877eb76a945257c4@gmail.com","[SPARK-2314][SQL] Override collect and take in python library, and count in java library, with optimized versions.\n\nSchemaRDD overrides RDD functions, including collect, count, and take, with optimized versions making use of the query optimizer.  The java and python interface classes wrapping SchemaRDD need to ensure the optimized versions are called as well.  This patch overrides relevant calls in the python and java interfaces with optimized versions.\n\nAdds a new Row serialization pathway between python and java, based on JList[Array[Byte]] versus the existing RDD[Array[Byte]]. I wasn’t overjoyed about doing this, but I noticed that some QueryPlans implement optimizations in executeCollect(), which outputs an Array[Row] rather than the typical RDD[Row] that can be shipped to python using the existing serialization code. To me it made sense to ship the Array[Row] over to python directly instead of converting it back to an RDD[Row] just for the purpose of sending the Rows to python using the existing serialization code.\n\nAuthor: Aaron Staple <aaron.staple@gmail.com>\n\nCloses #1592 from staple/SPARK-2314 and squashes the following commits:\n\n89ff550 [Aaron Staple] Merge with master.\n6bb7b6c [Aaron Staple] Fix typo.\nb56d0ac [Aaron Staple] [SPARK-2314][SQL] Override count in JavaSchemaRDD, forwarding to SchemaRDD's count.\n0fc9d40 [Aaron Staple] Fix comment typos.\nf03cdfa [Aaron Staple] [SPARK-2314][SQL] Override collect and take in sql.py, forwarding to SchemaRDD's collect.","Aaron Staple",false],["df90e81fd383c0d89dee6db16d5520def9190c56","fbfd45890515f7eeeeabf87d7a80d3449c3bce83@gmail.com","[Docs] minor punctuation fix\n\nAuthor: Nicholas Chammas <nicholas.chammas@gmail.com>\n\nCloses #2414 from nchammas/patch-1 and squashes the following commits:\n\n14664bf [Nicholas Chammas] [Docs] minor punctuation fix","Nicholas Chammas",false],["84073eb1172dc959936149265378f6e24d303685","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SQL][DOCS] Improve section on thrift-server\n\nTaken from liancheng's updates. Merged conflicts with #2316.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #2384 from marmbrus/sqlDocUpdate and squashes the following commits:\n\n2db6319 [Michael Armbrust] @liancheng's updates","Michael Armbrust",false],["a9e910430fb6bb4ef1f6ae20761c43b96bb018df","7d45abe39fd242ee2f6575da28f13d66568418b3@oss.nttdata.co.jp","[SPARK-3546] InputStream of ManagedBuffer is not closed and causes running out of file descriptor\n\nAuthor: Kousuke Saruta <sarutak@oss.nttdata.co.jp>\n\nCloses #2408 from sarutak/resolve-resource-leak-issue and squashes the following commits:\n\n074781d [Kousuke Saruta] Modified SuffleBlockFetcherIterator\n5f63f67 [Kousuke Saruta] Move metrics increment logic and debug logging outside try block\nb37231a [Kousuke Saruta] Modified FileSegmentManagedBuffer#nioByteBuffer to check null or not before invoking channel.close\nbf29d4a [Kousuke Saruta] Modified FileSegment to close channel","Kousuke Saruta",false],["ec1adecbb72d291d7ef122fb0505bae53116e0e6","399a71a219a2bbf15a9ad52c797d50eb921e7e28@gmail.com","[SPARK-3430] [PySpark] [Doc] generate PySpark API docs using Sphinx\n\nUsing Sphinx to generate API docs for PySpark.\n\nrequirement: Sphinx\n\n```\n$ cd python/docs/\n$ make html\n```\n\nThe generated API docs will be located at python/docs/_build/html/index.html\n\nIt can co-exists with those generated by Epydoc.\n\nThis is the first working version, after merging in, then we can continue to improve it and replace the epydoc finally.\n\nAuthor: Davies Liu <davies.liu@gmail.com>\n\nCloses #2292 from davies/sphinx and squashes the following commits:\n\n425a3b1 [Davies Liu] cleanup\n1573298 [Davies Liu] move docs to python/docs/\n5fe3903 [Davies Liu] Merge branch 'master' into sphinx\n9468ab0 [Davies Liu] fix makefile\nb408f38 [Davies Liu] address all comments\ne2ccb1b [Davies Liu] update name and version\n9081ead [Davies Liu] generate PySpark API docs using Sphinx","Davies Liu",false],["b20171267d610715d5b0a86b474c903e9bc3a1a3","9a3086209f69e133e0801b707495da958fffdb2a@shazam.com","[SPARK-787] Add S3 configuration parameters to the EC2 deploy scripts\n\nWhen deploying to AWS, there is additional configuration that is required to read S3 files. EMR creates it automatically, there is no reason that the Spark EC2 script shouldn't.\n\nThis PR requires a corresponding PR to the mesos/spark-ec2 to be merged, as it gets cloned in the process of setting up machines: https://github.com/mesos/spark-ec2/pull/58\n\nAuthor: Dan Osipov <daniil.osipov@shazam.com>\n\nCloses #1120 from danosipov/s3_credentials and squashes the following commits:\n\n758da8b [Dan Osipov] Modify documentation to include the new parameter\n71fab14 [Dan Osipov] Use a parameter --copy-aws-credentials to enable S3 credential deployment\n7e0da26 [Dan Osipov] Get AWS credentials out of boto connection instance\n39bdf30 [Dan Osipov] Add S3 configuration parameters to the EC2 deploy scripts","Dan Osipov",false],["a6e1712f1e9c36deb24c5073aa8edcfc047d76eb","4ac35f524a89dc243707b3dcc0ea3280d71f8cb2@gmail.com","Add a Community Projects page\n\nThis adds a new page to the docs listing community projects -- those created outside of Apache Spark that are of interest to the community of Spark users.   Anybody can add to it just by submitting a PR.\n\nThere was a discussion thread about alternatives:\n* Creating a Github organization for Spark projects -  we could not find any sponsors for this, and it would be difficult to organize since many folks just create repos in their company organization or personal accounts\n* Apache has some place for storing community projects, but it was deemed difficult to work with, and again would be some permissions issues -- not everyone could update it.\n\nAuthor: Evan Chan <velvia@gmail.com>\n\nCloses #2219 from velvia/community-projects-page and squashes the following commits:\n\n7316822 [Evan Chan] Point to Spark wiki: supplemental projects page\n613b021 [Evan Chan] Add a few more projects\na85eaaf [Evan Chan] Add a Community Projects page","Evan Chan",false],["0a7091e689a4c8b1e7b61e9f0873e6557f40d952","a5c478d606458f6c868a5126471748acd4730a6f@gmail.com","[SPARK-3555] Fix UISuite race condition\n\nThe test \"jetty selects different port under contention\" is flaky.\n\nIf another process binds to 4040 before the test starts, then the first server we start there will fail, and the subsequent servers we start thereafter may successfully bind to 4040 if it was released between the servers starting. Instead, we should just let Java find a random free port for us and hold onto it for the duration of the test.\n\nAuthor: Andrew Or <andrewor14@gmail.com>\n\nCloses #2418 from andrewor14/fix-port-contention and squashes the following commits:\n\n0cd4974 [Andrew Or] Stop them servers\na7071fe [Andrew Or] Pick random port instead of 4040","Andrew Or",false],["008a5ed4808d1467b47c1d6fa4d950cc6c4976b7","f98b0bead8ca0151df080a83631a6c2556256922@huawei.com","[Minor]ignore all config files in conf\n\nSome config files in ```conf``` should ignore, such as\n        conf/fairscheduler.xml\n        conf/hive-log4j.properties\n        conf/metrics.properties\n...\nSo ignore all ```sh```/```properties```/```conf```/```xml``` files\n\nAuthor: wangfei <wangfei1@huawei.com>\n\nCloses #2395 from scwf/patch-2 and squashes the following commits:\n\n3dc53f2 [wangfei] duplicate ```conf/*.conf```\n3c2986f [wangfei] ignore all config files","wangfei",false],["983609a4dd83e25598455bfce93fa1c1fa9f2c51","b4f0379e6b93e1f88e297cf3e1a13351f197b5b9@huawei.com","[Docs] Correct spark.files.fetchTimeout default value\n\nchange the value of spark.files.fetchTimeout\n\nAuthor: viper-kun <xukun.xu@huawei.com>\n\nCloses #2406 from viper-kun/master and squashes the following commits:\n\necb0d46 [viper-kun] [Docs] Correct spark.files.fetchTimeout default value\n7cf4c7a [viper-kun] Update configuration.md","viper-kun",false],["7d1a37239c50394025d9f16acf5dcd05cfbe7250","5c9688a59f3fcbfdbfeea06378a76af06a09aa95@alpinenow.com","SPARK-3177 (on Master Branch)\n\nThe JIRA and PR was original created for branch-1.1, and move to master branch now.\nChester\n\nThe Issue is due to that yarn-alpha and yarn have different APIs for certain class fields. In this particular case,  the ClientBase using reflection to to address this issue, and we need to different way to test the ClientBase's method.  Original ClientBaseSuite using getFieldValue() method to do this. But it doesn't work for yarn-alpha as the API returns an array of String instead of just String (which is the case for Yarn-stable API).\n\n To fix the test, I add a new method\n\n  def getFieldValue2[A: ClassTag, A1: ClassTag, B](clazz: Class[_], field: String,\n                                                      defaults: => B)\n                              (mapTo:  A => B)(mapTo1: A1 => B) : B =\n    Try(clazz.getField(field)).map(_.get(null)).map {\n      case v: A => mapTo(v)\n      case v1: A1 => mapTo1(v1)\n      case _ => defaults\n    }.toOption.getOrElse(defaults)\n\nto handle the cases where the field type can be either type A or A1. In this new method the type A or A1 is pattern matched and corresponding mapTo function (mapTo or mapTo1) is used.\n\nAuthor: chesterxgchen <chester@alpinenow.com>\n\nCloses #2204 from chesterxgchen/SPARK-3177-master and squashes the following commits:\n\ne72a6ea [chesterxgchen]  The Issue is due to that yarn-alpha and yarn have different APIs for certain class fields. In this particular case,  the ClientBase using reflection to to address this issue, and we need to different way to test the ClientBase's method.  Original ClientBaseSuite using getFieldValue() method to do this. But it doesn't work for yarn-alpha as the API returns an array of String instead of just String (which is the case for Yarn-stable API).","chesterxgchen",false],["8fbd5f4a90f92e064aa057adbd3f8c58dd0087fa","fbfd45890515f7eeeeabf87d7a80d3449c3bce83@gmail.com","[Docs] minor grammar fix\n\nAuthor: Nicholas Chammas <nicholas.chammas@gmail.com>\n\nCloses #2430 from nchammas/patch-2 and squashes the following commits:\n\nd476bfb [Nicholas Chammas] [Docs] minor grammar fix","Nicholas Chammas",false],["cbf983bb4a550ff26756ed7308fb03db42cffcff","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SQL][DOCS] Improve table caching section\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #2434 from marmbrus/patch-1 and squashes the following commits:\n\n67215be [Michael Armbrust] [SQL][DOCS] Improve table caching section","Michael Armbrust",false],["5044e4953a1744593d83fe90628fb4893e5463f1","fbfd45890515f7eeeeabf87d7a80d3449c3bce83@gmail.com","[SPARK-1455] [SPARK-3534] [Build] When possible, run SQL tests only.\n\nIf the only files changed are related to SQL, then only run the SQL tests.\n\nThis patch includes some cosmetic/maintainability refactoring. I would be more than happy to undo some of these changes if they are inappropriate.\n\nWe can accept this patch mostly as-is and address the immediate need documented in [SPARK-3534](https://issues.apache.org/jira/browse/SPARK-3534), or we can keep it open until a satisfactory solution along the lines [discussed here](https://issues.apache.org/jira/browse/SPARK-1455?focusedCommentId=14136424&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14136424) is reached.\n\nNote: I had to hack this patch up to test it locally, so what I'm submitting here and what I tested are technically different.\n\nAuthor: Nicholas Chammas <nicholas.chammas@gmail.com>\n\nCloses #2420 from nchammas/selective-testing and squashes the following commits:\n\ndb3fa2d [Nicholas Chammas] diff against master!\nf9e23f6 [Nicholas Chammas] when possible, run SQL tests only","Nicholas Chammas",false]],"size":62,"ref":"refs/heads/remove-useless-resolve-put","head":"be87f0c0c3e1fe8c92f11822fb7c5cd0b6208901"}