SPARK-1658: Correctly identify if maven is installed and working

The current test is checking the exit code of "tail" rather than "mvn".
This new check will make sure that mvn is installed and was able to
execute the "version command".

Author: Rahul Singhal <rahul.singhal@guavus.com>

Closes #580 from rahulsinghaliitd/SPARK-1658 and squashes the following commits:

83c0313 [Rahul Singhal] SPARK-1658: Correctly identify if maven is installed and working
bf821b9 [Rahul Singhal] SPARK-1658: Correctly identify if maven is installed and working
(cherry picked from commit e97a2e6717e75c70c5474f2d211682565909e557)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1663. (Addendum) Fix signature of one version of JavaPairRDDStream.reduceByKeyAndWindow()

It appears that one of these methods doesn't use `org.apache.spark.api.java.function.Function2` like all the others, but uses Scala's `Function2`.

Author: Sean Owen <sowen@cloudera.com>

Closes #633 from srowen/SPARK-1663.2 and squashes the following commits:

1e0232d [Sean Owen] Fix signature of one version of reduceByKeyAndWindow to use Java API Function2, as apparently intended
(cherry picked from commit 0088cede592540f35f9aec0f24dc1d9bd690d878)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1703 Warn users if Spark is run on JRE6 but compiled with JDK7.

This add some guards and good warning messages if users hit this issue. /cc @aarondav with whom I discussed parts of the design.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #627 from pwendell/jdk6 and squashes the following commits:

a38a958 [Patrick Wendell] Code review feedback
94e9f84 [Patrick Wendell] SPARK-1703 Warn users if Spark is run on JRE6 but compiled with JDK7.
(cherry picked from commit 0c98a8f6a761b941a9e0cf6fde6df8d0b8d24057)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1629. Addendum: Depend on commons lang3 (already used by tachyon) as it's used in ReplSuite, and return to use lang3 utility in Utils.scala

For consideration. This was proposed in related discussion: https://github.com/apache/spark/pull/569

Author: Sean Owen <sowen@cloudera.com>

Closes #635 from srowen/SPARK-1629.2 and squashes the following commits:

a442b98 [Sean Owen] Depend on commons lang3 (already used by tachyon) as it's used in ReplSuite, and return to use lang3 utility in Utils.scala
(cherry picked from commit f5041579ff573f988b673c2506fa4edc32f5ad84)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1693: Most of the tests throw a java.lang.SecurityException when s...

...park built for hadoop 2.3.0 , 2.4.0

Author: witgo <witgo@qq.com>

Closes #628 from witgo/SPARK-1693_new and squashes the following commits:

e3af968 [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1693_new
dc63905 [witgo] SPARK-1693: Most of the tests throw a java.lang.SecurityException when spark built for hadoop 2.3.0 , 2.4.0
(cherry picked from commit d940e4c16aaa7b60daf1229a99bc4d3455c0240d)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
EC2 script should exit with non-zero code on UsageError

This is specially import because some ssh errors are raised as UsageError, preventing an automated usage of the script from detecting the failure.

Author: Allan Douglas R. de Oliveira <allan@chaordicsystems.com>

Closes #638 from douglaz/ec2_exit_code_fix and squashes the following commits:

5915e6d [Allan Douglas R. de Oliveira] EC2 script should exit with non-zero code on UsageError
(cherry picked from commit bcb9b7fd4a656f9a6741220a6623441567ded0a4)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1710: spark-submit should print better errors than "InvocationTargetException"

Catching the InvocationTargetException, printing getTargetException.

Author: Sandeep <sandeep@techaddict.me>

Closes #630 from techaddict/SPARK-1710 and squashes the following commits:

834d79b [Sandeep] changes from srowen  suggestions
109d604 [Sandeep] SPARK-1710: spark-submit should print better errors than "InvocationTargetException"
(cherry picked from commit b48a55ae9ff2976c5fe6f5776a6d4659e828ee24)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Address SPARK-1717

I tested the change locally with Spark 0.9.1, but I can't test with 1.0.0 because there was no AMI for it at the time. It's a trivial fix, so it shouldn't cause any problems.

Author: msiddalingaiah <madhu@madhu.com>

Closes #641 from msiddalingaiah/master and squashes the following commits:

a4f7404 [msiddalingaiah] Address SPARK-1717
(cherry picked from commit bb2bb0cf6efcb1d18846e6c4904f3cd2da452f9f)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Updated doc for spark.closure.serializer to indicate only Java serializer work.

See discussion from http://apache-spark-developers-list.1001551.n3.nabble.com/bug-using-kryo-as-closure-serializer-td6473.html

Author: Reynold Xin <rxin@apache.org>

Closes #642 from rxin/docs-ser and squashes the following commits:

a507db5 [Reynold Xin] Use "Java" instead of default.
5eb8cdd [Reynold Xin] Updated doc for spark.closure.serializer to indicate only the default serializer work.

(cherry picked from commit f2eb070acc81e60096ee8d4ddf8da2b24a11da72)
Signed-off-by: Reynold Xin <rxin@apache.org>