GitHub PushEvent at 2014-11-02 19:55:34
Repository: spark
Actor: CodingCat
Payload: {"shas":[["97cf19f64e924569892e0a0417de19329855f4af","05f4ff6cfd932210ddb781d81debbbf760fcab75@gmail.com","Fix for sampling error in NumPy v1.9 [SPARK-3995][PYSPARK]\n\nChange maximum value for default seed during RDD sampling so that it is strictly less than 2 ** 32. This prevents a bug in the most recent version of NumPy, which cannot accept random seeds above this bound.\n\nAdds an extra test that uses the default seed (instead of setting it manually, as in the docstrings).\n\nmengxr\n\nAuthor: freeman <the.freeman.lab@gmail.com>\n\nCloses #2889 from freeman-lab/pyspark-sampling and squashes the following commits:\n\ndc385ef [freeman] Change maximum value for default seed","freeman",false],["813effc701fc27121c6f23ab32882932016fdbe0","e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","[SPARK-3426] Fix sort-based shuffle error when spark.shuffle.compress and spark.shuffle.spill.compress settings are different\n\nThis PR fixes SPARK-3426, an issue where sort-based shuffle crashes if the\n`spark.shuffle.spill.compress` and `spark.shuffle.compress` settings have\ndifferent values.\n\nThe problem is that sort-based shuffle's read and write paths use different\nsettings for determining whether to apply compression.  ExternalSorter writes\nruns to files using `TempBlockId` ids, which causes\n`spark.shuffle.spill.compress` to be used for enabling compression, but these\nspilled files end up being shuffled over the network and read as shuffle files\nusing `ShuffleBlockId` by BlockStoreShuffleFetcher, which causes\n`spark.shuffle.compress` to be used for enabling decompression.  As a result,\nthis leads to errors when these settings disagree.\n\nBased on the discussions in #2247 and #2178, it sounds like we don't want to\nremove the `spark.shuffle.spill.compress` setting.  Therefore, I've tried to\ncome up with a fix where `spark.shuffle.spill.compress` is used to compress\ndata that's read and written locally and `spark.shuffle.compress` is used to\ncompress any data that will be fetched / read as shuffle blocks.\n\nTo do this, I split `TempBlockId` into two new id types, `TempLocalBlockId` and\n`TempShuffleBlockId`, which map to `spark.shuffle.spill.compress` and\n`spark.shuffle.compress`, respectively.  ExternalAppendOnlyMap also used temp\nblocks for spilling data.  It looks like ExternalSorter was designed to be\na generic sorter but its configuration already happens to be tied to sort-based\nshuffle, so I think it's fine if we use `spark.shuffle.compress` to compress\nits spills; we can move the compression configuration to the constructor in\na later commit if we find that ExternalSorter is being used in other contexts\nwhere we want different configuration options to control compression.  To\nsummarize:\n\n**Before:**\n\n|       | ExternalAppendOnlyMap        | ExternalSorter               |\n|-------|------------------------------|------------------------------|\n| Read  | spark.shuffle.spill.compress | spark.shuffle.compress       |\n| Write | spark.shuffle.spill.compress | spark.shuffle.spill.compress |\n\n**After:**\n\n|       | ExternalAppendOnlyMap        | ExternalSorter         |\n|-------|------------------------------|------------------------|\n| Read  | spark.shuffle.spill.compress | spark.shuffle.compress |\n| Write | spark.shuffle.spill.compress | spark.shuffle.compress |\n\nThanks to andrewor14 for debugging this with me!\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #2890 from JoshRosen/SPARK-3426 and squashes the following commits:\n\n1921cf6 [Josh Rosen] Minor edit for clarity.\nc8dd8f2 [Josh Rosen] Add comment explaining use of createTempShuffleBlock().\n2c687b9 [Josh Rosen] Fix SPARK-3426.\n91e7e40 [Josh Rosen] Combine tests into single test of all combinations\n76ca65e [Josh Rosen] Add regression test for SPARK-3426.","Josh Rosen",false],["137d94235383cc49ccf8a7bb7f314f578aa1dede","5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","[SPARK-3877][YARN] Throw an exception when application is not successful so that the exit code wil be set to 1\n\nWhen an yarn application fails (yarn-cluster mode), the exit code of spark-submit is still 0. It's hard for people to write some automatic scripts to run spark jobs in yarn because the failure can not be detected in these scripts.\n\nThis PR added a status checking after `monitorApplication`. If an application is not successful, `run()` will throw an `SparkException`, so that Client.scala will exit with code 1. Therefore, people can use the exit code of `spark-submit` to write some automatic scripts.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #2732 from zsxwing/SPARK-3877 and squashes the following commits:\n\n1f89fa5 [zsxwing] Fix the unit test\na0498e1 [zsxwing] Update the docs and the error message\ne1cb9ef [zsxwing] Fix the hacky way of calling Client\nff16fec [zsxwing] Remove System.exit in Client.scala and add a test\n6a2c103 [zsxwing] [SPARK-3877] Throw an exception when application is not successful so that the exit code wil be set to 1","zsxwing",false],["c5882c663e054adcd3ecd9f11e91a1929dbc14a3","38ba40da102c3b82607690471c403766355c3e07@imaginea.com","[SPARK-3812] [BUILD] Adapt maven build to publish effective pom.\n\nI have tried maven help plugin first but that published all projects in top level pom. So I was left with no choice but to roll my own trivial plugin. This patch basically installs an effective pom after maven install is finished.\n\nThe problem it fixes is described as follows:\nIf you install using maven\n` mvn install -DskipTests -Dhadoop.version=2.2.0 -Phadoop-2.2 `\nThen without this patch the published pom(s) will have hadoop version as 1.0.4. This can be a problem at some point.\n\nAuthor: Prashant Sharma <prashant.s@imaginea.com>\n\nCloses #2673 from ScrapCodes/build-changes-effective-pom and squashes the following commits:\n\naa7b91d [Prashant Sharma] used an unused dep.\n0300dac [Prashant Sharma] improved comment messages..\n28f891e [Prashant Sharma] Added a useless dependency, so that we can shade it. And realized fake shading works for us.\n553d96b [Prashant Sharma] Shaded some unused class of an unused dep, to generate effective pom(s)","Prashant Sharma",false],["d6a302539213e8cdb51ca14b1769aeb33f9f435f","38ba40da102c3b82607690471c403766355c3e07@imaginea.com","[BUILD] Fixed resolver for scalastyle plugin and upgrade sbt version.\n\nAuthor: Prashant Sharma <prashant.s@imaginea.com>\n\nCloses #2877 from ScrapCodes/scalastyle-fix and squashes the following commits:\n\na17b9fe [Prashant Sharma] [BUILD] Fixed resolver for scalastyle plugin.","Prashant Sharma",false],["f799700eec4a5e33db9b2d6a4bee60a50fd5a099","7d45abe39fd242ee2f6575da28f13d66568418b3@oss.nttdata.co.jp","[SPARK-4055][MLlib] Inconsistent spelling 'MLlib' and 'MLLib'\n\nThare are some inconsistent spellings 'MLlib' and 'MLLib' in some documents and source codes.\n\nAuthor: Kousuke Saruta <sarutak@oss.nttdata.co.jp>\n\nCloses #2903 from sarutak/SPARK-4055 and squashes the following commits:\n\nb031640 [Kousuke Saruta] Fixed inconsistent spelling \"MLlib and MLLib\"","Kousuke Saruta",false],["6b485225271a3c616c4fa1231c20090a95c86f32","5d8655b7ac4ffddc00abf48d07490bd7766e006e@taboola.com","[SPARK-4006] In long running contexts, we encountered the situation of double registe...\n\n...r without a remove in between. The cause for that is unknown, and assumed a temp network issue.\n\nHowever, since the second register is with a BlockManagerId on a different port, blockManagerInfo.contains() returns false, while blockManagerIdByExecutor returns Some. This inconsistency is caught in a conditional statement that does System.exit(1), which is a huge robustness issue for us.\n\nThe fix - simply remove the old id from both maps during register when this happens. We are mimicking the behavior of expireDeadHosts(), by doing local cleanup of the maps before trying to add new ones.\n\nAlso - added some logging for register and unregister.\n\nThis is just like https://github.com/apache/spark/pull/2854 except it's on master\n\nAuthor: Tal Sliwowicz <tal.s@taboola.com>\n\nCloses #2886 from tsliwowicz/master-block-mgr-removal and squashes the following commits:\n\n094d508 [Tal Sliwowicz] some more white space change undone\n41a2217 [Tal Sliwowicz] some more whitspaces change undone\n7bcfc3d [Tal Sliwowicz] whitspaces fix\ndf9d98f [Tal Sliwowicz] Code review comments fixed\nf48bce9 [Tal Sliwowicz] In long running contexts, we encountered the situation of double register without a remove in between. The cause for that is unknown, and assumed a temp network issue.","Tal Sliwowicz",false],["293672c499911328eb27b48dbd7bdef4f4cc8adb","6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@pigscanfly.ca","specify unidocGenjavadocVersion of 0.8\n\nFixes an issue with being too strict generating javadoc causing errors.\n\nAuthor: Holden Karau <holden@pigscanfly.ca>\n\nCloses #2893 from holdenk/SPARK-3359-sbtunidoc-java8 and squashes the following commits:\n\n9379a70 [Holden Karau] specify unidocGenjavadocVersion of 0.8","Holden Karau",false],["222fa47f0dfd6c53aac513655a519521d9396e72","8bcd50e7b1eea79f965dde19d326001805380997@gmail.com","Revert \"[SPARK-3812] [BUILD] Adapt maven build to publish effective pom.\"\n\nThis reverts commit c5882c663e054adcd3ecd9f11e91a1929dbc14a3.\n\nI am reverting this becuase it appears to cause the maven tests\nto hang.","Patrick Wendell",false],["83b7a1c6503adce1826fc537b4db47e534da5cae","e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","[SPARK-4019] [SPARK-3740] Fix MapStatus compression bug that could lead to empty results or Snappy errors\n\nThis commit fixes a bug in MapStatus that could cause jobs to wrongly return\nempty results if those jobs contained stages with more than 2000 partitions\nwhere most of those partitions were empty.\n\nFor jobs with > 2000 partitions, MapStatus uses HighlyCompressedMapStatus,\nwhich only stores the average size of blocks.  If the average block size is\nzero, then this will cause all blocks to be reported as empty, causing\nBlockFetcherIterator to mistakenly skip them.\n\nFor example, this would return an empty result:\n\n    sc.makeRDD(0 until 10, 1000).repartition(2001).collect()\n\nThis can also lead to deserialization errors (e.g. Snappy decoding errors)\nfor jobs with > 2000 partitions where the average block size is non-zero but\nthere is at least one empty block.  In this case, the BlockFetcher attempts to\nfetch empty blocks and fails when trying to deserialize them.\n\nThe root problem here is that MapStatus has a (previously undocumented)\ncorrectness property that was violated by HighlyCompressedMapStatus:\n\n    If a block is non-empty, then getSizeForBlock must be non-zero.\n\nI fixed this by modifying HighlyCompressedMapStatus to store the average size\nof _non-empty_ blocks and to use a compressed bitmap to track which blocks are\nempty.\n\nI also removed a test which was broken as originally written: it attempted\nto check that HighlyCompressedMapStatus's size estimation error was < 10%,\nbut this was broken because HighlyCompressedMapStatus is only used for map\nstatuses with > 2000 partitions, but the test only created 50.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #2866 from JoshRosen/spark-4019 and squashes the following commits:\n\nfc8b490 [Josh Rosen] Roll back hashset change, which didn't improve performance.\n5faa0a4 [Josh Rosen] Incorporate review feedback\nc8b8cae [Josh Rosen] Two performance fixes:\n3b892dd [Josh Rosen] Address Reynold's review comments\nba2e71c [Josh Rosen] Add missing newline\n609407d [Josh Rosen] Use Roaring Bitmap to track non-empty blocks.\nc23897a [Josh Rosen] Use sets when comparing collect() results\n91276a3 [Josh Rosen] [SPARK-4019] Fix MapStatus compression bug that could lead to empty results.","Josh Rosen",false],["e595c8d08a20a122295af62d5e9cc4116f9727f6","399a71a219a2bbf15a9ad52c797d50eb921e7e28@gmail.com","[SPARK-3993] [PySpark] fix bug while reuse worker after take()\n\nAfter take(), maybe there are some garbage left in the socket, then next task assigned to this worker will hang because of corrupted data.\n\nWe should make sure the socket is clean before reuse it, write END_OF_STREAM at the end, and check it after read out all result from python.\n\nAuthor: Davies Liu <davies.liu@gmail.com>\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #2838 from davies/fix_reuse and squashes the following commits:\n\n8872914 [Davies Liu] fix tests\n660875b [Davies Liu] fix bug while reuse worker after take()","Davies Liu",false],["a29c9bd614e6d8088f759ae39218b945604d0b5b","e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","[SPARK-4000][BUILD] Sends archived unit tests logs to Jenkins master\n\nThis PR sends archived unit tests logs to the build history directory in Jenkins master, so that we can serve it via HTTP later to help debugging Jenkins build failures.\n\npwendell JoshRosen Please help review, thanks!\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #2845 from liancheng/log-archive and squashes the following commits:\n\nac8d9d4 [Cheng Lian] Includes build number in messages posted to GitHub\n68c7010 [Cheng Lian] Logs backup should be implemented in dev/run-tests-jenkins\n4b912f7 [Cheng Lian] Sends archived unit tests logs to Jenkins master","Cheng Lian",false],["0aea22895ce94f4f80faad5a2c41b76edac172b9","38ba40da102c3b82607690471c403766355c3e07@imaginea.com","SPARK-3812 Build changes to publish effective pom.\n\nAuthor: Prashant Sharma <prashant.s@imaginea.com>\n\nCloses #2921 from ScrapCodes/build-changes-effective-pom and squashes the following commits:\n\n8841491 [Prashant Sharma] Fixed broken maven build.\naa7b91d [Prashant Sharma] used an unused dep.\n0300dac [Prashant Sharma] improved comment messages..\n28f891e [Prashant Sharma] Added a useless dependency, so that we can shade it. And realized fake shading works for us.\n553d96b [Prashant Sharma] Shaded some unused class of an unused dep, to generate effective pom(s)","Prashant Sharma",false],["809c785bcc33e684a68ea14240a466def864199a","c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","[SPARK-2652] [PySpark] donot use KyroSerializer as default serializer\n\nKyroSerializer can not serialize customized class without registered explicitly, use it as default serializer in PySpark will introduce some regression in MLlib.\n\ncc mengxr\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #2916 from davies/revert and squashes the following commits:\n\n43eb6d3 [Davies Liu] donot use KyroSerializer as default serializer","Davies Liu",false],["d2987e8f7a2cb3bf971f381399d8efdccb51d3d2","7d45abe39fd242ee2f6575da28f13d66568418b3@oss.nttdata.co.jp","[SPARK-3900][YARN] ApplicationMaster's shutdown hook fails and IllegalStateException is thrown.\n\nApplicationMaster registers a shutdown hook and it calls ApplicationMaster#cleanupStagingDir.\n\ncleanupStagingDir invokes FileSystem.get(yarnConf) and it invokes FileSystem.getInternal. FileSystem.getInternal also registers shutdown hook.\nIn FileSystem of hadoop 0.23, the shutdown hook registration does not consider whether shutdown is in progress or not (In 2.2, it's considered).\n\n    // 0.23\n    if (map.isEmpty() ) {\n      ShutdownHookManager.get().addShutdownHook(clientFinalizer, SHUTDOWN_HOOK_PRIORITY);\n    }\n\n    // 2.2\n    if (map.isEmpty()\n                && !ShutdownHookManager.get().isShutdownInProgress()) {\n       ShutdownHookManager.get().addShutdownHook(clientFinalizer, SHUTDOWN_HOOK_PRIORITY);\n    }\n\nThus, in 0.23, another shutdown hook can be registered when ApplicationMaster's shutdown hook run.\n\nThis issue cause IllegalStateException as follows.\n\n    java.lang.IllegalStateException: Shutdown in progress, cannot add a shutdownHook\n            at org.apache.hadoop.util.ShutdownHookManager.addShutdownHook(ShutdownHookManager.java:152)\n            at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2306)\n            at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2278)\n            at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:316)\n            at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:162)\n            at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$cleanupStagingDir(ApplicationMaster.scala:307)\n            at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:118)\n            at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)\n\nAuthor: Kousuke Saruta <sarutak@oss.nttdata.co.jp>\n\nCloses #2924 from sarutak/SPARK-3900-2 and squashes the following commits:\n\n9112817 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-3900-2\n97018fa [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-3900\n2c2850e [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-3900\nee52db2 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-3900\na7d6c9b [Kousuke Saruta] Merge branch 'SPARK-3900' of github.com:sarutak/spark into SPARK-3900\n1cdf03c [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-3900\na5f6443 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-3900\n57b397d [Kousuke Saruta] Fixed IllegalStateException caused by shutdown hook registration in another shutdown hook","Kousuke Saruta",false],["d60a9d440b00beb107c1f1d7f42886c94f04a092","c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","[SPARK-4051] [SQL] [PySpark] Convert Row into dictionary\n\nAdded a method to Row to turn row into dict:\n\n```\n>>> row = Row(a=1)\n>>> row.asDict()\n{'a': 1}\n```\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #2896 from davies/dict and squashes the following commits:\n\n8d97366 [Davies Liu] convert Row into dict","Davies Liu",false],["0e886610eedd8ea24761cdcefa25ccedeca72dc8","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SPARK-4050][SQL] Fix caching of temporary tables with projections.\n\nPreviously cached data was found by `sameResult` plan matching on optimized plans.  This technique however fails to locate the cached data when a temporary table with a projection is queried with a further reduced projection.  The failure is due to the fact that optimization will collapse the projections, producing a plan that no longer produces the sameResult as the cached data (though the cached data still subsumes the desired data).  For example consider the following previously failing test case.\n\n```scala\nsql(\"CACHE TABLE tempTable AS SELECT key FROM testData\")\nassertCached(sql(\"SELECT COUNT(*) FROM tempTable\"))\n```\n\nIn this PR I change the matching to occur after analysis instead of optimization, so that in the case of temporary tables, the plans will always match.  I think this should work generally, however, this error does raise questions about the need to do more thorough subsumption checking when locating cached data.\n\nAnother question is what sort of semantics we want to provide when uncaching data from temporary tables.  For example consider the following sequence of commands:\n\n```scala\ntestData.select('key).registerTempTable(\"tempTable1\")\ntestData.select('key).registerTempTable(\"tempTable2\")\ncacheTable(\"tempTable1\")\n\n// This obviously works.\nassertCached(sql(\"SELECT COUNT(*) FROM tempTable1\"))\n\n// It seems good that this works ...\nassertCached(sql(\"SELECT COUNT(*) FROM tempTable2\"))\n\n// ... but is this valid?\nuncacheTable(\"tempTable2\")\n\n// Should this still be cached?\nassertCached(sql(\"SELECT COUNT(*) FROM tempTable1\"), 0)\n```\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #2912 from marmbrus/cachingBug and squashes the following commits:\n\n9c822d4 [Michael Armbrust] remove commented out code\n5c72fb7 [Michael Armbrust] Add a test case / question about uncaching semantics.\n63a23e4 [Michael Armbrust] Perform caching on analyzed instead of optimized plan.\n03f1cfe [Michael Armbrust] Clean-up / add tests to SameResult suite.","Michael Armbrust",false],["7c89a8f0c81ecf91dba34c1f44393f45845d438c","914a64bc573f63c3e7eba683b6cec5858e463b20@gmail.com","[SPARK-2706][SQL] Enable Spark to support Hive 0.13\n\nGiven that a lot of users are trying to use hive 0.13 in spark, and the incompatibility between hive-0.12 and hive-0.13 on the API level I want to propose following approach, which has no or minimum impact on existing hive-0.12 support, but be able to jumpstart the development of hive-0.13 and future version support.\n\nApproach: Introduce “hive-version” property,  and manipulate pom.xml files to support different hive version at compiling time through shim layer, e.g., hive-0.12.0 and hive-0.13.1. More specifically,\n\n1. For each different hive version, there is a very light layer of shim code to handle API differences, sitting in sql/hive/hive-version, e.g., sql/hive/v0.12.0 or sql/hive/v0.13.1\n\n2. Add a new profile hive-default active by default, which picks up all existing configuration and hive-0.12.0 shim (v0.12.0)  if no hive.version is specified.\n\n3. If user specifies different version (currently only 0.13.1 by -Dhive.version = 0.13.1), hive-versions profile will be activated, which pick up hive-version specific shim layer and configuration, mainly the hive jars and hive-version shim, e.g., v0.13.1.\n\n4. With this approach, nothing is changed with current hive-0.12 support.\n\nNo change by default: sbt/sbt -Phive\nFor example: sbt/sbt -Phive -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 assembly\n\nTo enable hive-0.13: sbt/sbt -Dhive.version=0.13.1\nFor example: sbt/sbt -Dhive.version=0.13.1 -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 assembly\n\nNote that in hive-0.13, hive-thriftserver is not enabled, which should be fixed by other Jira, and we don’t need -Phive with -Dhive.version in building (probably we should use -Phive -Dhive.version=xxx instead after thrift server is also supported in hive-0.13.1).\n\nAuthor: Zhan Zhang <zhazhan@gmail.com>\nAuthor: zhzhan <zhazhan@gmail.com>\nAuthor: Patrick Wendell <pwendell@gmail.com>\n\nCloses #2241 from zhzhan/spark-2706 and squashes the following commits:\n\n3ece905 [Zhan Zhang] minor fix\n410b668 [Zhan Zhang] solve review comments\ncbb4691 [Zhan Zhang] change run-test for new options\n0d4d2ed [Zhan Zhang] rebase\n497b0f4 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark\n8fad1cf [Zhan Zhang] change the pom file and make hive-0.13.1 as the default\nab028d1 [Zhan Zhang] rebase\n4a2e36d [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark\n4cb1b93 [zhzhan] Merge pull request #1 from pwendell/pr-2241\nb0478c0 [Patrick Wendell] Changes to simplify the build of SPARK-2706\n2b50502 [Zhan Zhang] rebase\na72c0d4 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark\ncb22863 [Zhan Zhang] correct the typo\n20f6cf7 [Zhan Zhang] solve compatability issue\nf7912a9 [Zhan Zhang] rebase and solve review feedback\n301eb4a [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark\n10c3565 [Zhan Zhang] address review comments\n6bc9204 [Zhan Zhang] rebase and remove temparory repo\nd3aa3f2 [Zhan Zhang] Merge branch 'master' into spark-2706\ncedcc6f [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark\n3ced0d7 [Zhan Zhang] rebase\nd9b981d [Zhan Zhang] rebase and fix error due to rollback\nadf4924 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark\n3dd50e8 [Zhan Zhang] solve conflicts and remove unnecessary implicts\nd10bf00 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark\ndc7bdb3 [Zhan Zhang] solve conflicts\n7e0cc36 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark\nd7c3e1e [Zhan Zhang] Merge branch 'master' into spark-2706\n68deb11 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark\nd48bd18 [Zhan Zhang] address review comments\n3ee3b2b [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark\n57ea52e [Zhan Zhang] Merge branch 'master' into spark-2706\n2b0d513 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark\n9412d24 [Zhan Zhang] address review comments\nf4af934 [Zhan Zhang] rebase\n1ccd7cc [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark\n128b60b [Zhan Zhang] ignore 0.12.0 test cases for the time being\naf9feb9 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark\n5f5619f [Zhan Zhang] restructure the directory and different hive version support\n05d3683 [Zhan Zhang] solve conflicts\ne4c1982 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark\n94b4fdc [Zhan Zhang] Spark-2706: hive-0.13.1 support on spark\n87ebf3b [Zhan Zhang] Merge branch 'master' into spark-2706\n921e914 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark\nf896b2a [Zhan Zhang] Merge branch 'master' into spark-2706\n789ea21 [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark\ncb53a2c [Zhan Zhang] Merge branch 'master' of https://github.com/apache/spark\nf6a8a40 [Zhan Zhang] revert\nba14f28 [Zhan Zhang] test\ndbedff3 [Zhan Zhang] Merge remote-tracking branch 'upstream/master'\n70964fe [Zhan Zhang] revert\nfe0f379 [Zhan Zhang] Merge branch 'master' of https://github.com/zhzhan/spark\n70ffd93 [Zhan Zhang] revert\n42585ec [Zhan Zhang] test\n7d5fce2 [Zhan Zhang] test","Zhan Zhang",false],["6a40a76848203d7266c134a26191579138c76903","e1f8d9790f67825b23faf6a373ab7c83bc8900f2@apache.org","[SPARK-4026][Streaming] Write ahead log management\n\nAs part of the effort to avoid data loss on Spark Streaming driver failure, we want to implement a write ahead log that can write received data to HDFS. This allows the received data to be persist across driver failures. So when the streaming driver is restarted, it can find and reprocess all the data that were received but not processed.\n\nThis was primarily implemented by @harishreedharan. This is still WIP, as he is going to improve the unitests by using HDFS mini cluster.\n\nAuthor: Hari Shreedharan <hshreedharan@apache.org>\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #2882 from tdas/driver-ha-wal and squashes the following commits:\n\ne4bee20 [Tathagata Das] Removed synchronized, Path.getFileSystem is threadsafe\n55514e2 [Tathagata Das] Minor changes based on PR comments.\nd29fddd [Tathagata Das] Merge pull request #20 from harishreedharan/driver-ha-wal\na317a4d [Hari Shreedharan] Directory deletion should not fail tests\n9514dc8 [Tathagata Das] Added unit tests to test reading of corrupted data and other minor edits\n3881706 [Tathagata Das] Merge pull request #19 from harishreedharan/driver-ha-wal\n4705fff [Hari Shreedharan] Sort listed files by name. Use local files for WAL tests.\neb356ca [Tathagata Das] Merge pull request #18 from harishreedharan/driver-ha-wal\n82ce56e [Hari Shreedharan] Fix file ordering issue in WALManager tests\n5ff90ee [Hari Shreedharan] Fix tests to not ignore ordering and also assert all data is present\nef8db09 [Tathagata Das] Merge pull request #17 from harishreedharan/driver-ha-wal\n7e40e56 [Hari Shreedharan] Restore old build directory after tests\n587b876 [Hari Shreedharan] Fix broken test. Call getFileSystem only from synchronized method.\nb4be0c1 [Hari Shreedharan] Remove unused method\nedcbee1 [Hari Shreedharan] Tests reading and writing data using writers now use Minicluster.\n5c70d1f [Hari Shreedharan] Remove underlying stream from the WALWriter.\n4ab602a [Tathagata Das] Refactored write ahead stuff from streaming.storage to streaming.util\nb06be2b [Tathagata Das] Adding missing license.\n5182ffb [Hari Shreedharan] Added documentation\n172358d [Tathagata Das] Pulled WriteAheadLog-related stuff from tdas/spark/tree/driver-ha-working","Hari Shreedharan",false],["7aacb7bfad4ec73fd8f18555c72ef6962c14358f","ae50d9f03dc526663c62caab7e3ac5b370caf973@intel.com","[SPARK-2713] Executors of same application in same host should only download files & jars once\n\nIf Spark lunched multiple executors in one host for one application, every executor would download it dependent files and jars (if not using local: url) independently. It maybe result in huge latency. In my case, it result in 20 seconds latency to download dependent jars(size about 17M) when I lunched 32 executors in every host(total 4 hosts).\n\nThis patch will cache downloaded files and jars for executors to reduce network throughput and download latency. In my case, the latency was reduced from 20 seconds to less than 1 second.\n\nAuthor: Li Zhihui <zhihui.li@intel.com>\nAuthor: li-zhihui <zhihui.li@intel.com>\n\nCloses #1616 from li-zhihui/cachefiles and squashes the following commits:\n\n36940df [Li Zhihui] Close cache for local mode\n935fed6 [Li Zhihui] Clean code.\nf9330d4 [Li Zhihui] Clean code again\n7050d46 [Li Zhihui] Clean code\n074a422 [Li Zhihui] Fix: deal with spark.files.overwrite\n03ed3a8 [li-zhihui] rename cache file name as XXXXXXXXX_cache\n2766055 [li-zhihui] Use url.hashCode + timestamp as cachedFileName\n76a7b66 [Li Zhihui] Clean code & use applcation work directory as cache directory\n3510eb0 [Li Zhihui] Keep fetchFile private\n2ffd742 [Li Zhihui] add comment for FileLock\ne0ebd48 [Li Zhihui] Try and finally lock.release\n7fb7c0b [Li Zhihui] Release lock before copy files\n6b997bf [Li Zhihui] Executors of same application in same host should only download files & jars once","Li Zhihui",false]],"size":93,"ref":"refs/heads/SPARK-3957","head":"6e4915dcfaf57a9b7363dce85135b675f2294981"}