GitHub PushEvent at 2014-10-24 23:00:39
Repository: apache-spark
Actor: ueshin
Payload: {"shas":[["386fc46bc039714fdfcf3db679f689287f8ef836","7d45abe39fd242ee2f6575da28f13d66568418b3@oss.nttdata.co.jp","[SPARK-4076] Parameter expansion in spark-config is wrong\n\nIn sbin/spark-config.sh, parameter expansion is used to extract source root as follows.\n\n    this=\"${BASH_SOURCE-$0}\"\n\nI think, the parameter expansion should be \":\" instead of \"\".\nIf we use \"-\" and BASH_SOURCE=\"\", (empty character is set, not unset),\n\"\" (empty character) is set to $this.\n\nAuthor: Kousuke Saruta <sarutak@oss.nttdata.co.jp>\n\nCloses #2930 from sarutak/SPARK-4076 and squashes the following commits:\n\n32a0370 [Kousuke Saruta] Fixed wrong parameter expansion\n\n(cherry picked from commit 30ea2868e7afbec20bfc83818249b6d2d7dc6aec)\nSigned-off-by: Andrew Or <andrew@databricks.com>\n\nConflicts:\n\tsbin/spark-config.sh","Kousuke Saruta",true],["80dde80a6d4f07d521dfeb471c425a67811504d9","7d45abe39fd242ee2f6575da28f13d66568418b3@oss.nttdata.co.jp","[SPARK-4075] [Deploy] Jar url validation is not enough for Jar file\n\nIn deploy.ClientArguments.isValidJarUrl, the url is checked as follows.\n\n    def isValidJarUrl(s: String): Boolean = s.matches(\"(.+):(.+)jar\")\n\nSo, it allows like 'hdfs:file.jar' (no authority).\n\nAuthor: Kousuke Saruta <sarutak@oss.nttdata.co.jp>\n\nCloses #2925 from sarutak/uri-syntax-check-improvement and squashes the following commits:\n\ncf06173 [Kousuke Saruta] Improved URI syntax checking\n\n(cherry picked from commit 098f83c7ccd7dad9f9228596da69fe5f55711a52)\nSigned-off-by: Andrew Or <andrew@databricks.com>","Kousuke Saruta",true],["59297e9510557edd4828a3961aa3559dbeae5f30","5d8655b7ac4ffddc00abf48d07490bd7766e006e@taboola.com","[SPARK-4006] In long running contexts, we encountered the situation of d...\n\n...ouble registe...\n\n...r without a remove in between. The cause for that is unknown, and assumed a temp network issue.\n\nHowever, since the second register is with a BlockManagerId on a different port, blockManagerInfo.contains() returns false, while blockManagerIdByExecutor returns Some. This inconsistency is caught in a conditional statement that does System.exit(1), which is a huge robustness issue for us.\n\nThe fix - simply remove the old id from both maps during register when this happens. We are mimicking the behavior of expireDeadHosts(), by doing local cleanup of the maps before trying to add new ones.\n\nAlso - added some logging for register and unregister.\n\nThis is just like https://github.com/apache/spark/pull/2886 except it's on branch-1.1\n\nAuthor: Tal Sliwowicz <tal.s@taboola.com>\n\nCloses #2915 from tsliwowicz/branch-1.1-block-mgr-removal and squashes the following commits:\n\nd122236 [Tal Sliwowicz] [SPARK-4006] In long running contexts, we encountered the situation of double registe...","Tal Sliwowicz",true],["6c10c2770c718287f9cc2af4109b701fa1057b70","e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","[SPARK-4080] Only throw IOException from [write|read][Object|External]\n\nIf classes implementing Serializable or Externalizable interfaces throw\nexceptions other than IOException or ClassNotFoundException from their\n(de)serialization methods, then this results in an unhelpful\n\"IOException: unexpected exception type\" rather than the actual exception that\nproduced the (de)serialization error.\n\nThis patch fixes this by adding a utility method that re-wraps any uncaught\nexceptions in IOException (unless they are already instances of IOException).\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #2932 from JoshRosen/SPARK-4080 and squashes the following commits:\n\ncd3a9be [Josh Rosen] [SPARK-4080] Only throw IOException from [write|read][Object|External].\n\n(cherry picked from commit 6c98c29ae0033556fd4424f41d1de005c509e511)\nSigned-off-by: Josh Rosen <joshrosen@databricks.com>\n\nConflicts:\n\tcore/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala\n\tcore/src/main/scala/org/apache/spark/scheduler/MapStatus.scala\n\tstreaming/src/main/scala/org/apache/spark/streaming/api/python/PythonDStream.scala","Josh Rosen",true]],"size":4,"ref":"refs/heads/branch-1.1","head":"6c10c2770c718287f9cc2af4109b701fa1057b70"}