GitHub PushEvent at 2014-06-10 13:19:29
Repository: spark
Actor: tgravescs
Payload: {"shas":[["6cf335d79a2f69ecd9a139dd0a03acff60585be4","284a00ea9901227c42561196ff8576aed3ac5631@gmail.com","Added a TaskSetManager unit test.\n\nThis test ensures that when there are no\nalive executors that satisfy a particular locality level,\nthe TaskSetManager doesn't ever use that as the maximum\nallowed locality level (this optimization ensures that a\njob doesn't wait extra time in an attempt to satisfy\na scheduling locality level that is impossible).\n\n@mateiz and @lirui-intel this unit test illustrates an issue\nwith #892 (it fails with that patch).\n\nAuthor: Kay Ousterhout <kayousterhout@gmail.com>\n\nCloses #1024 from kayousterhout/scheduler_unit_test and squashes the following commits:\n\nde6a08f [Kay Ousterhout] Added a TaskSetManager unit test.","Kay Ousterhout",true],["e273447684779a18bd61d733bfe7958b78657ffd","9e5910599b22a2ff8c9ccd67d61bc3265a983719@gmail.com","[SPARK-1522] : YARN ClientBase throws a NPE if there is no YARN Application CP\n\nThe current implementation of ClientBase.getDefaultYarnApplicationClasspath inspects\nthe MRJobConfig class for the field DEFAULT_YARN_APPLICATION_CLASSPATH when it should\nbe really looking into YarnConfiguration. If the Application Configuration has no\nyarn.application.classpath defined a NPE exception will be thrown.\n\nAdditional Changes include:\n* Test Suite for ClientBase added\n\n[ticket: SPARK-1522] : https://issues.apache.org/jira/browse/SPARK-1522\n\nAuthor      : bernardo.gomezpalacio@gmail.com\nTesting     : SPARK_HADOOP_VERSION=2.3.0 SPARK_YARN=true ./sbt/sbt test\n\nAuthor: Bernardo Gomez Palacio <bernardo.gomezpalacio@gmail.com>\n\nCloses #433 from berngp/feature/SPARK-1522 and squashes the following commits:\n\n2c2e118 [Bernardo Gomez Palacio] [SPARK-1522]: YARN ClientBase throws a NPE if there is no YARN Application specific CP","Bernardo Gomez Palacio",true],["c6e041d171e3d9882ab15e2bd7a7217dc19647f6","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SQL] Simple framework for debugging query execution\n\nOnly records number of tuples and unique dataTypes output right now...\n\nExample:\n```scala\nscala> import org.apache.spark.sql.execution.debug._\nscala> hql(\"SELECT value FROM src WHERE key > 10\").debug(sparkContext)\n\nResults returned: 489\n== Project [value#1:0] ==\nTuples output: 489\n value StringType: {java.lang.String}\n== Filter (key#0:1 > 10) ==\nTuples output: 489\n value StringType: {java.lang.String}\n key IntegerType: {java.lang.Integer}\n== HiveTableScan [value#1,key#0], (MetastoreRelation default, src, None), None ==\nTuples output: 500\n value StringType: {java.lang.String}\n key IntegerType: {java.lang.Integer}\n```\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #1005 from marmbrus/debug and squashes the following commits:\n\ndcc3ca6 [Michael Armbrust] Add comments.\nc9dded2 [Michael Armbrust] Simple framework for debugging query execution","Michael Armbrust",true],["a9ec033c8cf489898cc47e2043bd9e86b7df1ff8","1ebd4c97b4950dc95b86f9c9aec05bcbebaf22af@gmail.com","[SPARK-1704][SQL] Fully support EXPLAIN commands as SchemaRDD.\n\nThis PR attempts to resolve [SPARK-1704](https://issues.apache.org/jira/browse/SPARK-1704) by introducing a physical plan for EXPLAIN commands, which just prints out the debug string (containing various SparkSQL's plans) of the corresponding QueryExecution for the actual query.\n\nAuthor: Zongheng Yang <zongheng.y@gmail.com>\n\nCloses #1003 from concretevitamin/explain-cmd and squashes the following commits:\n\n5b7911f [Zongheng Yang] Add a regression test.\n1bfa379 [Zongheng Yang] Modify output().\n719ada9 [Zongheng Yang] Override otherCopyArgs for ExplainCommandPhysical.\n4318fd7 [Zongheng Yang] Make all output one Row.\n439c6ab [Zongheng Yang] Minor cleanups.\n408f574 [Zongheng Yang] SPARK-1704: Add CommandStrategy and ExplainCommandPhysical.","Zongheng Yang",true],["6f2db8c2f51911f88a601ec5bf1509ea0e8173ed","24f70c800c13ae23cbad3ee3edbc742a6ecfb50e@dbtsai.com","Make sure that empty string is filtered out when we get the secondary jars from conf\n\nAuthor: DB Tsai <dbtsai@dbtsai.com>\n\nCloses #1027 from dbtsai/dbtsai-classloader and squashes the following commits:\n\n9ac6be3 [DB Tsai] Fixed line too long\nc9c7ad7 [DB Tsai] Make sure that empty string is filtered out when we get the secondary jars from conf.","DB Tsai",true],["f971d6cb60d642178d6544217a25fa16ece34889","25cbb71b4cb5d047244aacd790fc30e39dfd6a54@gmail.com","SPARK-1416: PySpark support for SequenceFile and Hadoop InputFormats\n\nSo I finally resurrected this PR. It seems the old one against the incubator mirror is no longer available, so I cannot reference it.\n\nThis adds initial support for reading Hadoop ```SequenceFile```s, as well as arbitrary Hadoop ```InputFormat```s, in PySpark.\n\n# Overview\nThe basics are as follows:\n1. ```PythonRDD``` object contains the relevant methods, that are in turn invoked by ```SparkContext``` in PySpark\n2. The SequenceFile or InputFormat is read on the Scala side and converted from ```Writable``` instances to the relevant Scala classes (in the case of primitives)\n3. Pyrolite is used to serialize Java objects. If this fails, the fallback is ```toString```\n4. ```PickleSerializer``` on the Python side deserializes.\n\nThis works \"out the box\" for simple ```Writable```s:\n* ```Text```\n* ```IntWritable```, ```DoubleWritable```, ```FloatWritable```\n* ```NullWritable```\n* ```BooleanWritable```\n* ```BytesWritable```\n* ```MapWritable```\n\nIt also works for simple, \"struct-like\" classes. Due to the way Pyrolite works, this requires that the classes satisfy the JavaBeans convenstions (i.e. with fields and a no-arg constructor and getters/setters). (Perhaps in future some sugar for case classes and reflection could be added).\n\nI've tested it out with ```ESInputFormat```  as an example and it works very nicely:\n```python\nconf = {\"es.resource\" : \"index/type\" }\nrdd = sc.newAPIHadoopRDD(\"org.elasticsearch.hadoop.mr.EsInputFormat\", \"org.apache.hadoop.io.NullWritable\", \"org.elasticsearch.hadoop.mr.LinkedMapWritable\", conf=conf)\nrdd.first()\n```\n\nI suspect for things like HBase/Cassandra it will be a bit trickier to get it to work out the box.\n\n# Some things still outstanding:\n1. ~~Requires ```msgpack-python``` and will fail without it. As originally discussed with Josh, add a ```as_strings``` argument that defaults to ```False```, that can be used if ```msgpack-python``` is not available~~\n2. ~~I see from https://github.com/apache/spark/pull/363 that Pyrolite is being used there for SerDe between Scala and Python. @ahirreddy @mateiz what is the plan behind this - is Pyrolite preferred? It seems from a cursory glance that adapting the ```msgpack```-based SerDe here to use Pyrolite wouldn't be too hard~~\n3. ~~Support the key and value \"wrapper\" that would allow a Scala/Java function to be plugged in that would transform whatever the key/value Writable class is into something that can be serialized (e.g. convert some custom Writable to a JavaBean or ```java.util.Map``` that can be easily serialized)~~\n4. Support ```saveAsSequenceFile``` and ```saveAsHadoopFile``` etc. This would require SerDe in the reverse direction, that can be handled by Pyrolite. Will work on this as a separate PR\n\nAuthor: Nick Pentreath <nick.pentreath@gmail.com>\n\nCloses #455 from MLnick/pyspark-inputformats and squashes the following commits:\n\n268df7e [Nick Pentreath] Documentation changes mer @pwendell comments\n761269b [Nick Pentreath] Address @pwendell comments, simplify default writable conversions and remove registry.\n4c972d8 [Nick Pentreath] Add license headers\nd150431 [Nick Pentreath] Merge remote-tracking branch 'upstream/master' into pyspark-inputformats\ncde6af9 [Nick Pentreath] Parameterize converter trait\n5ebacfa [Nick Pentreath] Update docs for PySpark input formats\na985492 [Nick Pentreath] Move Converter examples to own package\n365d0be [Nick Pentreath] Make classes private[python]. Add docs and @Experimental annotation to Converter interface.\neeb8205 [Nick Pentreath] Fix path relative to SPARK_HOME in tests\n1eaa08b [Nick Pentreath] HBase -> Cassandra app name oversight\n3f90c3e [Nick Pentreath] Merge remote-tracking branch 'upstream/master' into pyspark-inputformats\n2c18513 [Nick Pentreath] Add examples for reading HBase and Cassandra InputFormats from Python\nb65606f [Nick Pentreath] Add converter interface\n5757f6e [Nick Pentreath] Default key/value classes for sequenceFile asre None\n085b55f [Nick Pentreath] Move input format tests to tests.py and clean up docs\n43eb728 [Nick Pentreath] PySpark InputFormats docs into programming guide\n94beedc [Nick Pentreath] Clean up args in PythonRDD. Set key/value converter defaults to None for PySpark context.py methods\n1a4a1d6 [Nick Pentreath] Address @mateiz style comments\n01e0813 [Nick Pentreath] Merge remote-tracking branch 'upstream/master' into pyspark-inputformats\n15a7d07 [Nick Pentreath] Remove default args for key/value classes. Arg names to camelCase\n9fe6bd5 [Nick Pentreath] Merge remote-tracking branch 'upstream/master' into pyspark-inputformats\n84fe8e3 [Nick Pentreath] Python programming guide space formatting\nd0f52b6 [Nick Pentreath] Python programming guide\n7caa73a [Nick Pentreath] Merge remote-tracking branch 'upstream/master' into pyspark-inputformats\n93ef995 [Nick Pentreath] Add back context.py changes\n9ef1896 [Nick Pentreath] Recover earlier changes lost in previous merge for serializers.py\n077ecb2 [Nick Pentreath] Recover earlier changes lost in previous merge for context.py\n5af4770 [Nick Pentreath] Merge branch 'master' into pyspark-inputformats\n35b8e3a [Nick Pentreath] Another fix for test ordering\nbef3afb [Nick Pentreath] Merge remote-tracking branch 'upstream/master' into pyspark-inputformats\ne001b94 [Nick Pentreath] Fix test failures due to ordering\n78978d9 [Nick Pentreath] Add doc for SequenceFile and InputFormat support to Python programming guide\n64eb051 [Nick Pentreath] Scalastyle fix\ne7552fa [Nick Pentreath] Merge branch 'master' into pyspark-inputformats\n44f2857 [Nick Pentreath] Remove msgpack dependency and switch serialization to Pyrolite, plus some clean up and refactoring\nc0ebfb6 [Nick Pentreath] Change sequencefile test data generator to easily be called from PySpark tests\n1d7c17c [Nick Pentreath] Amend tests to auto-generate sequencefile data in temp dir\n17a656b [Nick Pentreath] remove binary sequencefile for tests\nf60959e [Nick Pentreath] Remove msgpack dependency and serializer from PySpark\n450e0a2 [Nick Pentreath] Merge branch 'master' into pyspark-inputformats\n31a2fff [Nick Pentreath] Scalastyle fixes\nfc5099e [Nick Pentreath] Add Apache license headers\n4e08983 [Nick Pentreath] Clean up docs for PySpark context methods\nb20ec7e [Nick Pentreath] Clean up merge duplicate dependencies\n951c117 [Nick Pentreath] Merge branch 'master' into pyspark-inputformats\nf6aac55 [Nick Pentreath] Bring back msgpack\n9d2256e [Nick Pentreath] Merge branch 'master' into pyspark-inputformats\n1bbbfb0 [Nick Pentreath] Clean up SparkBuild from merge\na67dfad [Nick Pentreath] Clean up Msgpack serialization and registering\n7237263 [Nick Pentreath] Add back msgpack serializer and hadoop file code lost during merging\n25da1ca [Nick Pentreath] Add generator for nulls, bools, bytes and maps\n65360d5 [Nick Pentreath] Adding test SequenceFiles\n0c612e5 [Nick Pentreath] Merge branch 'master' into pyspark-inputformats\nd72bf18 [Nick Pentreath] msgpack\ndd57922 [Nick Pentreath] Merge remote-tracking branch 'upstream/master' into pyspark-inputformats\ne67212a [Nick Pentreath] Add back msgpack dependency\nf2d76a0 [Nick Pentreath] Merge branch 'master' into pyspark-inputformats\n41856a5 [Nick Pentreath] Merge branch 'master' into pyspark-inputformats\n97ef708 [Nick Pentreath] Remove old writeToStream\n2beeedb [Nick Pentreath] Merge remote-tracking branch 'upstream/master' into pyspark-inputformats\n795a763 [Nick Pentreath] Change name to WriteInputFormatTestDataGenerator. Cleanup some var names. Use SPARK_HOME in path for writing test sequencefile data.\n174f520 [Nick Pentreath] Add back graphx settings\n703ee65 [Nick Pentreath] Add back msgpack\n619c0fa [Nick Pentreath] Merge remote-tracking branch 'upstream/master' into pyspark-inputformats\n1c8efbc [Nick Pentreath] Merge remote-tracking branch 'upstream/master' into pyspark-inputformats\neb40036 [Nick Pentreath] Remove unused comment lines\n4d7ef2e [Nick Pentreath] Fix indentation\nf1d73e3 [Nick Pentreath] mergeConfs returns a copy rather than mutating one of the input arguments\n0f5cd84 [Nick Pentreath] Remove unused pair UTF8 class. Add comments to msgpack deserializer\n4294cbb [Nick Pentreath] Add old Hadoop api methods. Clean up and expand comments. Clean up argument names\n818a1e6 [Nick Pentreath] Add seqencefile and Hadoop InputFormat support to PythonRDD\n4e7c9e3 [Nick Pentreath] Merge remote-tracking branch 'upstream/master' into pyspark-inputformats\nc304cc8 [Nick Pentreath] Adding supporting sequncefiles for tests. Cleaning up\n4b0a43f [Nick Pentreath] Refactoring utils into own objects. Cleaning up old commented-out code\nd86325f [Nick Pentreath] Initial WIP of PySpark support for SequenceFile and arbitrary Hadoop InputFormat","Nick Pentreath",true],["08ed9ad81397b71206c4dc903bfb94b6105691ed","1ebd4c97b4950dc95b86f9c9aec05bcbebaf22af@gmail.com","[SPARK-1508][SQL] Add SQLConf to SQLContext.\n\nThis PR (1) introduces a new class SQLConf that stores key-value properties for a SQLContext (2) clean up the semantics of various forms of SET commands.\n\nThe SQLConf class unlocks user-controllable optimization opportunities; for example, user can now override the number of partitions used during an Exchange. A SQLConf can be accessed and modified programmatically through its getters and setters. It can also be modified through SET commands executed by `sql()` or `hql()`. Note that users now have the ability to change a particular property for different queries inside the same Spark job, unlike settings configured in SparkConf.\n\nFor SET commands: \"SET\" will return all properties currently set in a SQLConf, \"SET key\" will return the key-value pair (if set) or an undefined message, and \"SET key=value\" will call the setter on SQLConf, and if a HiveContext is used, it will be executed in Hive as well.\n\nAuthor: Zongheng Yang <zongheng.y@gmail.com>\n\nCloses #956 from concretevitamin/sqlconf and squashes the following commits:\n\n4968c11 [Zongheng Yang] Very minor cleanup.\nd74dde5 [Zongheng Yang] Remove the redundant mkQueryExecution() method.\nc129b86 [Zongheng Yang] Merge remote-tracking branch 'upstream/master' into sqlconf\n26c40eb [Zongheng Yang] Make SQLConf a trait and have SQLContext mix it in.\ndd19666 [Zongheng Yang] Update a comment.\nbaa5d29 [Zongheng Yang] Remove default param for shuffle partitions accessor.\n5f7e6d8 [Zongheng Yang] Add default num partitions.\n22d9ed7 [Zongheng Yang] Fix output() of Set physical. Add SQLConf param accessor method.\ne9856c4 [Zongheng Yang] Use java.util.Collections.synchronizedMap on a Java HashMap.\n88dd0c8 [Zongheng Yang] Remove redundant SET Keyword.\n271f0b1 [Zongheng Yang] Minor change.\nf8983d1 [Zongheng Yang] Minor changes per review comments.\n1ce8a5e [Zongheng Yang] Invoke runSqlHive() in SQLConf#get for the HiveContext case.\nb766af9 [Zongheng Yang] Remove a test.\nd52e1bd [Zongheng Yang] De-hardcode number of shuffle partitions for BasicOperators (read from SQLConf).\n555599c [Zongheng Yang] Bullet-proof (relatively) parsing SET per review comment.\nc2067e8 [Zongheng Yang] Mark SQLContext transient and put it in a second param list.\n2ea8cdc [Zongheng Yang] Wrap long line.\n41d7f09 [Zongheng Yang] Fix imports.\n13279e6 [Zongheng Yang] Refactor the logic of eagerly processing SET commands.\nb14b83e [Zongheng Yang] In a HiveContext, make SQLConf a subset of HiveConf.\n6983180 [Zongheng Yang] Move a SET test to SQLQuerySuite and make it complete.\n5b67985 [Zongheng Yang] New line at EOF.\nc651797 [Zongheng Yang] Add commands.scala.\nefd82db [Zongheng Yang] Clean up semantics of several cases of SET.\nc1017c2 [Zongheng Yang] WIP in changing SetCommand to take two Options (for different semantics of SETs).\n0f00d86 [Zongheng Yang] Add a test for singleton set command in SQL.\n41acd75 [Zongheng Yang] Add a test for hql() in HiveQuerySuite.\n2276929 [Zongheng Yang] Fix default hive result for set commands in HiveComparisonTest.\n3b0c71b [Zongheng Yang] Remove Parser for set commands. A few other fixes.\nd0c4578 [Zongheng Yang] Tmux typo.\n0ecea46 [Zongheng Yang] Changes for HiveQl and HiveContext.\nce22d80 [Zongheng Yang] Fix parsing issues.\ncb722c1 [Zongheng Yang] Finish up SQLConf patch.\n4ebf362 [Zongheng Yang] First cut at SQLConf inside SQLContext.","Zongheng Yang",true],["a9a461c594fd20e46947e318095df60bddb67559","a50d2185142f97721f64187b05c8deeec99604b1@gmail.com","Moved hiveOperators.scala to the right package folder\n\nThe package is `org.apache.spark.sql.hive.execution`, while the file was placed under `sql/hive/src/main/scala/org/apache/spark/sql/hive/`.\n\nAuthor: Cheng Lian <lian.cs.zju@gmail.com>\n\nCloses #1029 from liancheng/moveHiveOperators and squashes the following commits:\n\nd632eb8 [Cheng Lian] Moved hiveOperators.scala to the right package folder","Cheng Lian",true]],"size":8,"ref":"refs/heads/master","head":"a9a461c594fd20e46947e318095df60bddb67559"}