[SPARK-4076] Parameter expansion in spark-config is wrong

In sbin/spark-config.sh, parameter expansion is used to extract source root as follows.

    this="${BASH_SOURCE-$0}"

I think, the parameter expansion should be ":" instead of "".
If we use "-" and BASH_SOURCE="", (empty character is set, not unset),
"" (empty character) is set to $this.

Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>

Closes #2930 from sarutak/SPARK-4076 and squashes the following commits:

32a0370 [Kousuke Saruta] Fixed wrong parameter expansion

(cherry picked from commit 30ea2868e7afbec20bfc83818249b6d2d7dc6aec)
Signed-off-by: Andrew Or <andrew@databricks.com>

Conflicts:
	sbin/spark-config.sh
[SPARK-4075] [Deploy] Jar url validation is not enough for Jar file

In deploy.ClientArguments.isValidJarUrl, the url is checked as follows.

    def isValidJarUrl(s: String): Boolean = s.matches("(.+):(.+)jar")

So, it allows like 'hdfs:file.jar' (no authority).

Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>

Closes #2925 from sarutak/uri-syntax-check-improvement and squashes the following commits:

cf06173 [Kousuke Saruta] Improved URI syntax checking

(cherry picked from commit 098f83c7ccd7dad9f9228596da69fe5f55711a52)
Signed-off-by: Andrew Or <andrew@databricks.com>
[SPARK-4006] In long running contexts, we encountered the situation of d...

...ouble registe...

...r without a remove in between. The cause for that is unknown, and assumed a temp network issue.

However, since the second register is with a BlockManagerId on a different port, blockManagerInfo.contains() returns false, while blockManagerIdByExecutor returns Some. This inconsistency is caught in a conditional statement that does System.exit(1), which is a huge robustness issue for us.

The fix - simply remove the old id from both maps during register when this happens. We are mimicking the behavior of expireDeadHosts(), by doing local cleanup of the maps before trying to add new ones.

Also - added some logging for register and unregister.

This is just like https://github.com/apache/spark/pull/2886 except it's on branch-1.1

Author: Tal Sliwowicz <tal.s@taboola.com>

Closes #2915 from tsliwowicz/branch-1.1-block-mgr-removal and squashes the following commits:

d122236 [Tal Sliwowicz] [SPARK-4006] In long running contexts, we encountered the situation of double registe...
[SPARK-4080] Only throw IOException from [write|read][Object|External]

If classes implementing Serializable or Externalizable interfaces throw
exceptions other than IOException or ClassNotFoundException from their
(de)serialization methods, then this results in an unhelpful
"IOException: unexpected exception type" rather than the actual exception that
produced the (de)serialization error.

This patch fixes this by adding a utility method that re-wraps any uncaught
exceptions in IOException (unless they are already instances of IOException).

Author: Josh Rosen <joshrosen@databricks.com>

Closes #2932 from JoshRosen/SPARK-4080 and squashes the following commits:

cd3a9be [Josh Rosen] [SPARK-4080] Only throw IOException from [write|read][Object|External].

(cherry picked from commit 6c98c29ae0033556fd4424f41d1de005c509e511)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>

Conflicts:
	core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala
	core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala
	streaming/src/main/scala/org/apache/spark/streaming/api/python/PythonDStream.scala
[SPARK-4056] Upgrade snappy-java to 1.1.1.5

This upgrades snappy-java to 1.1.1.5, which improves error messages when attempting to deserialize empty inputs using SnappyInputStream (see https://github.com/xerial/snappy-java/issues/89).

Author: Josh Rosen <rosenville@gmail.com>
Author: Josh Rosen <joshrosen@databricks.com>

Closes #2911 from JoshRosen/upgrade-snappy-java and squashes the following commits:

adec96c [Josh Rosen] Use snappy-java 1.1.1.5
cc953d6 [Josh Rosen] [SPARK-4056] Upgrade snappy-java to 1.1.1.4

(cherry picked from commit 898b22ab1fe90e8a3935b19566465046f2256fa6)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>

Conflicts:
	pom.xml