GitHub PushEvent at 2014-08-02 17:05:54
Repository: spark
Actor: sarahgerweck
Payload: {"shas":[["f2eb84fe737e6b06f5625640b209cf02f80732cf","c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@apache.org","Wrap FWDIR in quotes.","Reynold Xin",true],["95cf203936c412bc689bd2345fec7f9ad3648c25","c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@apache.org","Wrap FWDIR in quotes in dev/check-license.","Reynold Xin",true],["0feb349ea07361f0363117404ffc9797c2c80dd1","c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@apache.org","More wrapping FWDIR in quotes.","Reynold Xin",true],["2248891a43d93cf2c05580211faf1e4f8dc7932d","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SQL] Fix compiling of catalyst docs.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #1653 from marmbrus/fixDocs and squashes the following commits:\n\n0aa1feb [Michael Armbrust] Fix compiling of catalyst docs.","Michael Armbrust",true],["437dc8c5b54f0dcf9564c1fb07e8dce9e771c8cd","c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@apache.org","dev/check-license wrap folders in quotes.","Reynold Xin",true],["94d1f46fc43c0cb85125f757fb40db9271caf1f4","b77a2672cd58da5852c1129f2fe5419176a54679@apache.org","[SPARK-2024] Add saveAsSequenceFile to PySpark\n\nJIRA issue: https://issues.apache.org/jira/browse/SPARK-2024\n\nThis PR is a followup to #455 and adds capabilities for saving PySpark RDDs using SequenceFile or any Hadoop OutputFormats.\n\n* Added RDD methods ```saveAsSequenceFile```, ```saveAsHadoopFile``` and ```saveAsHadoopDataset```, for both old and new MapReduce APIs.\n\n* Default converter for converting common data types to Writables. Users may specify custom converters to convert to desired data types.\n\n* No out-of-box support for reading/writing arrays, since ArrayWritable itself doesn't have a no-arg constructor for creating an empty instance upon reading. Users need to provide ArrayWritable subtypes. Custom converters for converting arrays to suitable ArrayWritable subtypes are also needed when writing. When reading, the default converter will convert any custom ArrayWritable subtypes to ```Object[]``` and they get pickled to Python tuples.\n\n* Added HBase and Cassandra output examples to show how custom output formats and converters can be used.\n\ncc MLnick mateiz ahirreddy pwendell\n\nAuthor: Kan Zhang <kzhang@apache.org>\n\nCloses #1338 from kanzhang/SPARK-2024 and squashes the following commits:\n\nc01e3ef [Kan Zhang] [SPARK-2024] code formatting\n6591e37 [Kan Zhang] [SPARK-2024] renaming pickled -> pickledRDD\nd998ad6 [Kan Zhang] [SPARK-2024] refectoring to get method params below 10\n57a7a5e [Kan Zhang] [SPARK-2024] correcting typo\n75ca5bd [Kan Zhang] [SPARK-2024] Better type checking for batch serialized RDD\n0bdec55 [Kan Zhang] [SPARK-2024] Refactoring newly added tests\n9f39ff4 [Kan Zhang] [SPARK-2024] Adding 2 saveAsHadoopDataset tests\n0c134f3 [Kan Zhang] [SPARK-2024] Test refactoring and adding couple unbatched cases\n7a176df [Kan Zhang] [SPARK-2024] Add saveAsSequenceFile to PySpark","Kan Zhang",true],["7c7ce54522015315c909e111d6c2cff83e9fb501","c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@apache.org","Wrap JAR_DL in dev/check-license.","Reynold Xin",true],["1097327538ec3870544f406775efcfe7722e48be","c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@apache.org","Set AMPLAB_JENKINS_BUILD_PROFILE.","Reynold Xin",true],["2f4b17056fdcba26fd3a7503b858364b883ab0b0","c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@apache.org","Properly pass SBT_MAVEN_PROFILES into sbt.","Reynold Xin",true],["6ab96a6fd0db7731c8c5d6478d9e28b619581687","d519fa33a62684a61ab04b24d26f83452d51b34c@gmail.com","SPARK-2749 [BUILD]. Spark SQL Java tests aren't compiling in Jenkins' Maven builds; missing junit:junit dep\n\nThe Maven-based builds in the build matrix have been failing for a few days:\n\nhttps://amplab.cs.berkeley.edu/jenkins/view/Spark/\n\nOn inspection, it looks like the Spark SQL Java tests don't compile:\n\nhttps://amplab.cs.berkeley.edu/jenkins/view/Spark/job/Spark-Master-Maven-pre-YARN/hadoop.version=1.0.4,label=centos/244/consoleFull\n\nI confirmed it by repeating the command vs master:\n\n`mvn -Dhadoop.version=1.0.4 -Dlabel=centos -DskipTests clean package`\n\nThe problem is that this module doesn't depend on JUnit. In fact, none of the modules do, but `com.novocode:junit-interface` (the SBT-JUnit bridge) pulls it in, in most places. However this module doesn't depend on `com.novocode:junit-interface`\n\nAdding the `junit:junit` dependency fixes the compile problem. In fact, the other modules with Java tests should probably depend on it explicitly instead of happening to get it via `com.novocode:junit-interface`, since that is a bit SBT/Scala-specific (and I am not even sure it's needed).\n\nAuthor: Sean Owen <srowen@gmail.com>\n\nCloses #1660 from srowen/SPARK-2749 and squashes the following commits:\n\n858ff7c [Sean Owen] Add explicit junit dep to other modules with Java tests for robustness\n9636794 [Sean Owen] Add junit dep so that Spark SQL Java tests compile","Sean Owen",true],["2ac37db7ac8f7ec5c99f3bfe459f8e2ac240961f","a34d47d11bef4e101aa6199cf903b81b91e09c75@apache.org","SPARK-2741 - Publish version of spark assembly which does not contain Hive\n\nProvide a version of the Spark tarball which does not package Hive. This is meant for HIve + Spark users.\n\nAuthor: Brock Noland <brock@apache.org>\n\nCloses #1667 from brockn/master and squashes the following commits:\n\n5beafb2 [Brock Noland] SPARK-2741 - Publish version of spark assembly which does not contain Hive","Brock Noland",true],["88a519db90d66ee5a1455ef4fcc1ad2a687e3d0b","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SPARK-2734][SQL] Remove tables from cache when DROP TABLE is run.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #1650 from marmbrus/dropCached and squashes the following commits:\n\ne6ab80b [Michael Armbrust] Support if exists.\n83426c6 [Michael Armbrust] Remove tables from cache when DROP TABLE is run.","Michael Armbrust",true],["e9b275b7697e7ad3b52b157d3274acc17ca8d828","d519fa33a62684a61ab04b24d26f83452d51b34c@gmail.com","SPARK-2341 [MLLIB] loadLibSVMFile doesn't handle regression datasets\n\nPer discussion at https://issues.apache.org/jira/browse/SPARK-2341 , this is a look at deprecating the multiclass parameter. Thoughts welcome of course.\n\nAuthor: Sean Owen <srowen@gmail.com>\n\nCloses #1663 from srowen/SPARK-2341 and squashes the following commits:\n\n8a3abd7 [Sean Owen] Suppress MIMA error for removed package private classes\n18a8c8e [Sean Owen] Updates from review\n83d0092 [Sean Owen] Deprecated methods with multiclass, and instead always parse target as a double (ie. multiclass = true)","Sean Owen",true],["da501766834453c9ac7095c7e8c930151f87cf11","5a2ea6e7e0ce7bbf598ce6694187311975d72ac5@gmail.com","Update DecisionTreeRunner.scala\n\nAuthor: strat0sphere <stratos.dimopoulos@gmail.com>\n\nCloses #1676 from strat0sphere/patch-1 and squashes the following commits:\n\n044d2fa [strat0sphere] Update DecisionTreeRunner.scala","strat0sphere",true],["e966284409f9355e1169960e73a2215617c8cb22","c481c5e387de8107aa9fe99c851e11e1ff44d0c3@databricks.com","SPARK-2045 Sort-based shuffle\n\nThis adds a new ShuffleManager based on sorting, as described in https://issues.apache.org/jira/browse/SPARK-2045. The bulk of the code is in an ExternalSorter class that is similar to ExternalAppendOnlyMap, but sorts key-value pairs by partition ID and can be used to create a single sorted file with a map task's output. (Longer-term I think this can take on the remaining functionality in ExternalAppendOnlyMap and replace it so we don't have code duplication.)\n\nThe main TODOs still left are:\n- [x] enabling ExternalSorter to merge across spilled files\n  - [x] with an Ordering\n  - [x] without an Ordering, using the keys' hash codes\n- [x] adding more tests (e.g. a version of our shuffle suite that runs on this)\n- [x] rebasing on top of the size-tracking refactoring in #1165 when that is merged\n- [x] disabling spilling if spark.shuffle.spill is set to false\n\nDespite this though, this seems to work pretty well (running successfully in cases where the hash shuffle would OOM, such as 1000 reduce tasks on executors with only 1G memory), and it seems to be comparable in speed or faster than hash-based shuffle (it will create much fewer files for the OS to keep track of). So I'm posting it to get some early feedback.\n\nAfter these TODOs are done, I'd also like to enable ExternalSorter to sort data within each partition by a key as well, which will allow us to use it to implement external spilling in reduce tasks in `sortByKey`.\n\nAuthor: Matei Zaharia <matei@databricks.com>\n\nCloses #1499 from mateiz/sort-based-shuffle and squashes the following commits:\n\nbd841f9 [Matei Zaharia] Various review comments\nd1c137fd [Matei Zaharia] Various review comments\na611159 [Matei Zaharia] Compile fixes due to rebase\n62c56c8 [Matei Zaharia] Fix ShuffledRDD sometimes not returning Tuple2s.\nf617432 [Matei Zaharia] Fix a failing test (seems to be due to change in SizeTracker logic)\n9464d5f [Matei Zaharia] Simplify code and fix conflicts after latest rebase\n0174149 [Matei Zaharia] Add cleanup behavior and cleanup tests for sort-based shuffle\neb4ee0d [Matei Zaharia] Remove customizable element type in ShuffledRDD\nfa2e8db [Matei Zaharia] Allow nextBatchStream to be called after we're done looking at all streams\na34b352 [Matei Zaharia] Fix tracking of indices within a partition in SpillReader, and add test\n03e1006 [Matei Zaharia] Add a SortShuffleSuite that runs ShuffleSuite with sort-based shuffle\n3c7ff1f [Matei Zaharia] Obey the spark.shuffle.spill setting in ExternalSorter\nad65fbd [Matei Zaharia] Rebase on top of Aaron's Sorter change, and use Sorter in our buffer\n44d2a93 [Matei Zaharia] Use estimateSize instead of atGrowThreshold to test collection sizes\n5686f71 [Matei Zaharia] Optimize merging phase for in-memory only data:\n5461cbb [Matei Zaharia] Review comments and more tests (e.g. tests with 1 element per partition)\ne9ad356 [Matei Zaharia] Update ContextCleanerSuite to make sure shuffle cleanup tests use hash shuffle (since they were written for it)\nc72362a [Matei Zaharia] Added bug fix and test for when iterators are empty\nde1fb40 [Matei Zaharia] Make trait SizeTrackingCollection private[spark]\n4988d16 [Matei Zaharia] tweak\nc1b7572 [Matei Zaharia] Small optimization\nba7db7f [Matei Zaharia] Handle null keys in hash-based comparator, and add tests for collisions\nef4e397 [Matei Zaharia] Support for partial aggregation even without an Ordering\n4b7a5ce [Matei Zaharia] More tests, and ability to sort data if a total ordering is given\ne1f84be [Matei Zaharia] Fix disk block manager test\n5a40a1c [Matei Zaharia] More tests\n614f1b4 [Matei Zaharia] Add spill metrics to map tasks\ncc52caf [Matei Zaharia] Add more error handling and tests for error cases\nbbf359d [Matei Zaharia] More work\n3a56341 [Matei Zaharia] More partial work towards sort-based shuffle\n7a0895d [Matei Zaharia] Some more partial work towards sort-based shuffle\nb615476 [Matei Zaharia] Scaffolding for sort-based shuffle","Matei Zaharia",true],["894d48ffb8c91e347ab60c58de983e1aaf181188","c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@apache.org","[SPARK-2758] UnionRDD's UnionPartition should not reference parent RDDs\n\nAuthor: Reynold Xin <rxin@apache.org>\n\nCloses #1675 from rxin/unionrdd and squashes the following commits:\n\n941d316 [Reynold Xin] Clear RDDs for checkpointing.\nc9f05f2 [Reynold Xin] [SPARK-2758] UnionRDD's UnionPartition should not reference parent RDDs","Reynold Xin",true],["118c1c422d3dfbfb2277995062678f0a808af6c3","5cc7173406045a3242d3595588aa610a46f4e47d@asiainfo-linkage.com","Required AM memory is \"amMem\", not \"args.amMemory\"\n\n\"ERROR yarn.Client: Required AM memory (1024) is above the max threshold (1048) of this cluster\" appears if this code is not changed. obviously, 1024 is less than 1048, so change this\n\nAuthor: derek ma <maji3@asiainfo-linkage.com>\n\nCloses #1494 from maji2014/master and squashes the following commits:\n\nb0f6640 [derek ma] Required AM memory is \"amMem\", not \"args.amMemory\"","derek ma",true],["a7c305b86b3b83645ae5ff5d3dfeafc20c443204","a5c478d606458f6c868a5126471748acd4730a6f@gmail.com","[SPARK-2340] Resolve event logging and History Server paths properly\n\nWe resolve relative paths to the local `file:/` system for `--jars` and `--files` in spark submit (#853). We should do the same for the history server.\n\nAuthor: Andrew Or <andrewor14@gmail.com>\n\nCloses #1280 from andrewor14/hist-serv-fix and squashes the following commits:\n\n13ff406 [Andrew Or] Merge branch 'master' of github.com:apache/spark into hist-serv-fix\nb393e17 [Andrew Or] Strip trailing \"/\" from logging directory\n622a471 [Andrew Or] Fix test in EventLoggingListenerSuite\n0e20f71 [Andrew Or] Shift responsibility of resolving paths up one level\nb037c0c [Andrew Or] Use resolved paths for everything in history server\nc7e36ee [Andrew Or] Resolve paths for event logging too\n40e3933 [Andrew Or] Resolve history server file paths","Andrew Or",true],["4fb259353f616822c32537e3f031944a6d2a09a8","e5dd8a3899598b2e85930c5ff06b11c053ef654f@apache.org","[SPARK-2737] Add retag() method for changing RDDs' ClassTags.\n\nThe Java API's use of fake ClassTags doesn't seem to cause any problems for Java users, but it can lead to issues when passing JavaRDDs' underlying RDDs to Scala code (e.g. in the MLlib Java API wrapper code). If we call collect() on a Scala RDD with an incorrect ClassTag, this causes ClassCastExceptions when we try to allocate an array of the wrong type (for example, see SPARK-2197).\n\nThere are a few possible fixes here. An API-breaking fix would be to completely remove the fake ClassTags and require Java API users to pass java.lang.Class instances to all parallelize() calls and add returnClass fields to all Function implementations. This would be extremely verbose.\n\nInstead, this patch adds internal APIs to \"repair\" a Scala RDD with an incorrect ClassTag by wrapping it and overriding its ClassTag. This should be okay for cases where the Scala code that calls collect() knows what type of array should be allocated, which is the case in the MLlib wrappers.\n\nAuthor: Josh Rosen <joshrosen@apache.org>\n\nCloses #1639 from JoshRosen/SPARK-2737 and squashes the following commits:\n\n572b4c8 [Josh Rosen] Replace newRDD[T] with mapPartitions().\n469d941 [Josh Rosen] Preserve partitioner in retag().\naf78816 [Josh Rosen] Allow retag() to get classTag implicitly.\nd1d54e6 [Josh Rosen] [SPARK-2737] Add retag() method for changing RDDs' ClassTags.","Josh Rosen",true],["5a110da25f15694773d6f7c6ee63c5b08ada4eb0","38ba40da102c3b82607690471c403766355c3e07@imaginea.com","[SPARK-2497] Included checks for module symbols too.\n\nAuthor: Prashant Sharma <prashant.s@imaginea.com>\n\nCloses #1463 from ScrapCodes/SPARK-2497/mima-exclude-all and squashes the following commits:\n\n72077b1 [Prashant Sharma] Check separately for module symbols.\ncd96192 [Prashant Sharma] SPARK-2497 Produce \"member excludes\" irrespective of the fact that class itself is excluded or not.","Prashant Sharma",true]],"size":83,"ref":"refs/heads/master","head":"87738bfa4051771ddfb8c4a4c1eb142fd77e3a46"}