Whitelist Hive Tests

This is ready when Jenkins is.

Author: Michael Armbrust <michael@databricks.com>

Closes #596 from marmbrus/moreTests and squashes the following commits:

85be703 [Michael Armbrust] Blacklist MR required tests.
35bc311 [Michael Armbrust] Add hive golden answers.
ede98fd [Michael Armbrust] More hive gitignore
da096ea [Michael Armbrust] update whitelist
(cherry picked from commit 92b2902ca06eddd250bb0f50f365ed69f127a842)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
The default version of yarn is equal to the hadoop version

This is a part of [PR 590](https://github.com/apache/spark/pull/590)

Author: witgo <witgo@qq.com>

Closes #626 from witgo/yarn_version and squashes the following commits:

c390631 [witgo] restore  the yarn dependency declarations
f8a4ad8 [witgo] revert remove the dependency of avro in yarn-alpha
2df6cf5 [witgo] review commit
a1d876a [witgo] review commit
20e7e3e [witgo] review commit
c76763b [witgo] The default value of yarn.version is equal to hadoop.version
(cherry picked from commit fb0543224bcedb8ae3aab4a7ddcc6111a03378fe)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1658: Correctly identify if maven is installed and working

The current test is checking the exit code of "tail" rather than "mvn".
This new check will make sure that mvn is installed and was able to
execute the "version command".

Author: Rahul Singhal <rahul.singhal@guavus.com>

Closes #580 from rahulsinghaliitd/SPARK-1658 and squashes the following commits:

83c0313 [Rahul Singhal] SPARK-1658: Correctly identify if maven is installed and working
bf821b9 [Rahul Singhal] SPARK-1658: Correctly identify if maven is installed and working
(cherry picked from commit e97a2e6717e75c70c5474f2d211682565909e557)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1663. (Addendum) Fix signature of one version of JavaPairRDDStream.reduceByKeyAndWindow()

It appears that one of these methods doesn't use `org.apache.spark.api.java.function.Function2` like all the others, but uses Scala's `Function2`.

Author: Sean Owen <sowen@cloudera.com>

Closes #633 from srowen/SPARK-1663.2 and squashes the following commits:

1e0232d [Sean Owen] Fix signature of one version of reduceByKeyAndWindow to use Java API Function2, as apparently intended
(cherry picked from commit 0088cede592540f35f9aec0f24dc1d9bd690d878)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1703 Warn users if Spark is run on JRE6 but compiled with JDK7.

This add some guards and good warning messages if users hit this issue. /cc @aarondav with whom I discussed parts of the design.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #627 from pwendell/jdk6 and squashes the following commits:

a38a958 [Patrick Wendell] Code review feedback
94e9f84 [Patrick Wendell] SPARK-1703 Warn users if Spark is run on JRE6 but compiled with JDK7.
(cherry picked from commit 0c98a8f6a761b941a9e0cf6fde6df8d0b8d24057)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1629. Addendum: Depend on commons lang3 (already used by tachyon) as it's used in ReplSuite, and return to use lang3 utility in Utils.scala

For consideration. This was proposed in related discussion: https://github.com/apache/spark/pull/569

Author: Sean Owen <sowen@cloudera.com>

Closes #635 from srowen/SPARK-1629.2 and squashes the following commits:

a442b98 [Sean Owen] Depend on commons lang3 (already used by tachyon) as it's used in ReplSuite, and return to use lang3 utility in Utils.scala
(cherry picked from commit f5041579ff573f988b673c2506fa4edc32f5ad84)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1693: Most of the tests throw a java.lang.SecurityException when s...

...park built for hadoop 2.3.0 , 2.4.0

Author: witgo <witgo@qq.com>

Closes #628 from witgo/SPARK-1693_new and squashes the following commits:

e3af968 [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1693_new
dc63905 [witgo] SPARK-1693: Most of the tests throw a java.lang.SecurityException when spark built for hadoop 2.3.0 , 2.4.0
(cherry picked from commit d940e4c16aaa7b60daf1229a99bc4d3455c0240d)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
EC2 script should exit with non-zero code on UsageError

This is specially import because some ssh errors are raised as UsageError, preventing an automated usage of the script from detecting the failure.

Author: Allan Douglas R. de Oliveira <allan@chaordicsystems.com>

Closes #638 from douglaz/ec2_exit_code_fix and squashes the following commits:

5915e6d [Allan Douglas R. de Oliveira] EC2 script should exit with non-zero code on UsageError
(cherry picked from commit bcb9b7fd4a656f9a6741220a6623441567ded0a4)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1710: spark-submit should print better errors than "InvocationTargetException"

Catching the InvocationTargetException, printing getTargetException.

Author: Sandeep <sandeep@techaddict.me>

Closes #630 from techaddict/SPARK-1710 and squashes the following commits:

834d79b [Sandeep] changes from srowen  suggestions
109d604 [Sandeep] SPARK-1710: spark-submit should print better errors than "InvocationTargetException"
(cherry picked from commit b48a55ae9ff2976c5fe6f5776a6d4659e828ee24)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Address SPARK-1717

I tested the change locally with Spark 0.9.1, but I can't test with 1.0.0 because there was no AMI for it at the time. It's a trivial fix, so it shouldn't cause any problems.

Author: msiddalingaiah <madhu@madhu.com>

Closes #641 from msiddalingaiah/master and squashes the following commits:

a4f7404 [msiddalingaiah] Address SPARK-1717
(cherry picked from commit bb2bb0cf6efcb1d18846e6c4904f3cd2da452f9f)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Updated doc for spark.closure.serializer to indicate only Java serializer work.

See discussion from http://apache-spark-developers-list.1001551.n3.nabble.com/bug-using-kryo-as-closure-serializer-td6473.html

Author: Reynold Xin <rxin@apache.org>

Closes #642 from rxin/docs-ser and squashes the following commits:

a507db5 [Reynold Xin] Use "Java" instead of default.
5eb8cdd [Reynold Xin] Updated doc for spark.closure.serializer to indicate only the default serializer work.

(cherry picked from commit f2eb070acc81e60096ee8d4ddf8da2b24a11da72)
Signed-off-by: Reynold Xin <rxin@apache.org>
SPARK-1556. jets3t dep doesn't update properly with newer Hadoop versions

See related discussion at https://github.com/apache/spark/pull/468

This PR may still overstep what you have in mind, but let me put it on the table to start. Besides fixing the issue, it has one substantive change, and that is to manage Hadoop-specific things only in Hadoop-related profiles. This does _not_ remove `yarn.version`.

- Moves the YARN and Hadoop profiles together in pom.xml. Sorry that this makes the diff a little hard to grok but the changes are only as follows.
- Removes `hadoop.major.version`
- Introduce `hadoop-2.2` and `hadoop-2.3` profiles to control Hadoop-specific changes:
  - like the protobuf version issue - this was only 'solved' now by enabling YARN for 2.2+, which is really an orthogonal issue
  - like the jets3t version issue now
- Hadoop profiles set an appropriate default `hadoop.version`, that can be overridden
- _(YARN profiles in the parent now only exist to add the sub-module)_
- Fixes the jets3t dependency issue
 - and makes it a runtime dependency
 - and centralizes config of this guy in the parent pom
- Updates build docs
- Updates SBT build too
  - and fixes a regex problem along the way

Author: Sean Owen <sowen@cloudera.com>

Closes #629 from srowen/SPARK-1556 and squashes the following commits:

c3fa967 [Sean Owen] Fix hadoop-2.4 profile typo in doc
a2105fd [Sean Owen] Add hadoop-2.4 profile and don't set hadoop.version in profiles
274f4f9 [Sean Owen] Make jets3t a runtime dependency, and bring its exclusion up into parent config
bbed826 [Sean Owen] Use jets3t 0.9.0 for Hadoop 2.3+ (and correct similar regex issue in SBT build)
f21f356 [Sean Owen] Build changes to set up for jets3t fix
(cherry picked from commit 73b0cbcc241cca3d318ff74340e80b02f884acbd)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1721: Reset the thread classLoader in the Mesos Executor

This is because Mesos calls it with a different environment or something, the result is that the Spark jar is missing and it can't load classes.

This fixes http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-ClassNotFoundException-spark-on-mesos-td3510.html

I have no idea whether this is the right fix, I can only confirm that it fixes the issue for us.

The `registered` method is called from mesos (https://github.com/apache/mesos/blob/765ff9bc2ac5a12d4362f8235b572a37d646390a/src/java/jni/org_apache_mesos_MesosExecutorDriver.cpp)

I am unsure which commit caused this regression

Author: Bouke van der Bijl <boukevanderbijl@gmail.com>

Closes #620 from bouk/mesos-classloader-fix and squashes the following commits:

c13eae0 [Bouke van der Bijl] Use getContextOrSparkClassLoader in SparkEnv and CompressionCodec
(cherry picked from commit 3292e2a71bfb5df5ba156cf7557747d164d12291)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[SPARK-1504], [SPARK-1505], [SPARK-1558] Updated Spark Streaming guide

- SPARK-1558: Updated custom receiver guide to match it with the new API
- SPARK-1504: Added deployment and monitoring subsection to streaming
- SPARK-1505: Added migration guide for migrating from 0.9.x and below to Spark 1.0
- Updated various Java streaming examples to use JavaReceiverInputDStream to highlight the API change.
- Removed the requirement for cleaner ttl from streaming guide

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes #652 from tdas/doc-fix and squashes the following commits:

cb4f4b7 [Tathagata Das] Possible fix for flaky graceful shutdown test.
ab71f7f [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into doc-fix
8d6ff9b [Tathagata Das] Addded migration guide to Spark Streaming.
7d171df [Tathagata Das] Added reference to JavaReceiverInputStream in examples and streaming guide.
49edd7c [Tathagata Das] Change java doc links to use Java docs.
11528d7 [Tathagata Das] Updated links on index page.
ff80970 [Tathagata Das] More updates to streaming guide.
4dc42e9 [Tathagata Das] Added monitoring and other documentation in the streaming guide.
14c6564 [Tathagata Das] Updated custom receiver guide.

(cherry picked from commit a975a19f21e71f448b3fdb2ed4461e28ef439900)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
[SPARK-1681] Include datanucleus jars in Spark Hive distribution

This copies the datanucleus jars over from `lib_managed` into `dist/lib`, if any. The `CLASSPATH` must also be updated to reflect this change.

Author: Andrew Or <andrewor14@gmail.com>

Closes #610 from andrewor14/hive-distribution and squashes the following commits:

a4bc96f [Andrew Or] Rename search path in jar error check
fa205e1 [Andrew Or] Merge branch 'master' of github.com:apache/spark into hive-distribution
7855f58 [Andrew Or] Have jar command respect JAVA_HOME + check for jar errors both cases
c16bbfd [Andrew Or] Merge branch 'master' of github.com:apache/spark into hive-distribution
32f6826 [Andrew Or] Leave the double colons
940a1bb [Andrew Or] Add back 2>/dev/null
58357cc [Andrew Or] Include datanucleus jars in Spark distribution built with Hive support
(cherry picked from commit cf0a8f0204bb8acdaf441b03c924c278fef08e28)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1728. JavaRDDLike.mapPartitionsWithIndex requires ClassTag

Author: Sandy Ryza <sandy@cloudera.com>

Closes #657 from sryza/sandy-spark-1728 and squashes the following commits:

4751443 [Sandy Ryza] SPARK-1728. JavaRDDLike.mapPartitionsWithIndex requires ClassTag
(cherry picked from commit 8e724dcbad411c533e4f0cd053aa878e8a60694d)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Expose SparkListeners and relevant classes as DeveloperApi

Hopefully this can go into 1.0, as a few people on the user list have asked for this.

Author: Andrew Or <andrewor14@gmail.com>

Closes #648 from andrewor14/expose-listeners and squashes the following commits:

e45e1ef [Andrew Or] Add missing colons (minor)
350d643 [Andrew Or] Expose SparkListeners and relevant classes as DeveloperApi
(cherry picked from commit ea10b3126167af3f50f7c2a70e1d942e839fcb66)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[SPARK-1594][MLLIB] Cleaning up MLlib APIs and guide

Final pass before the v1.0 release.

* Remove `VectorRDDs`
* Move `BinaryClassificationMetrics` from `evaluation.binary` to `evaluation`
* Change default value of `addIntercept` to false and allow to add intercept in Ridge and Lasso.
* Clean `DecisionTree` package doc and test suite.
* Mark model constructors `private[spark]`
* Rename `loadLibSVMData` to `loadLibSVMFile` and hide `LabelParser` from users.
* Add `saveAsLibSVMFile`.
* Add `appendBias` to `MLUtils`.

Author: Xiangrui Meng <meng@databricks.com>

Closes #524 from mengxr/mllib-cleaning and squashes the following commits:

295dc8b [Xiangrui Meng] update loadLibSVMFile doc
1977ac1 [Xiangrui Meng] fix doc of appendBias
649fcf0 [Xiangrui Meng] rename loadLibSVMData to loadLibSVMFile; hide LabelParser from user APIs
54b812c [Xiangrui Meng] add appendBias
a71e7d0 [Xiangrui Meng] add saveAsLibSVMFile
d976295 [Xiangrui Meng] Merge branch 'master' into mllib-cleaning
b7e5cec [Xiangrui Meng] remove some experimental annotations and make model constructors private[mllib]
9b02b93 [Xiangrui Meng] minor code style update
a593ddc [Xiangrui Meng] fix python tests
fc28c18 [Xiangrui Meng] mark more classes experimental
f6cbbff [Xiangrui Meng] fix Java tests
0af70b0 [Xiangrui Meng] minor
6e139ef [Xiangrui Meng] Merge branch 'master' into mllib-cleaning
94e6dce [Xiangrui Meng] move BinaryLabelCounter and BinaryConfusionMatrixImpl to evaluation.binary
df34907 [Xiangrui Meng] clean DecisionTreeSuite to use LocalSparkContext
c81807f [Xiangrui Meng] set the default value of AddIntercept to false
03389c0 [Xiangrui Meng] allow to add intercept in Ridge and Lasso
c66c56f [Xiangrui Meng] move tree md to package object doc
a2695df [Xiangrui Meng] update guide for BinaryClassificationMetrics
9194f4c [Xiangrui Meng] move BinaryClassificationMetrics one level up
1c1a0e3 [Xiangrui Meng] remove VectorRDDs because it only contains one function that is not necessary for us to maintain

(cherry picked from commit 98750a74daf7e2b873da85d2d5067f47e3bbdc4e)
Signed-off-by: Matei Zaharia <matei@databricks.com>
[SPARK-1678][SPARK-1679] In-memory compression bug fix and made compression configurable, disabled by default

In-memory compression is now configurable in `SparkConf` by the `spark.sql.inMemoryCompression.enabled` property, and is disabled by default.

To help code review, the bug fix is in [the first commit](https://github.com/liancheng/spark/commit/d537a367edf0bf24d0b925cc58b21d805ccbc11f), compression configuration is in [the second one](https://github.com/liancheng/spark/commit/4ce09aa8aa820bbbbbaa0f3f084a6cff1d4e6195).

Author: Cheng Lian <lian.cs.zju@gmail.com>

Closes #608 from liancheng/spark-1678 and squashes the following commits:

66c3a8d [Cheng Lian] Renamed in-memory compression configuration key
f8fb3a0 [Cheng Lian] Added assertion for testing .hasNext of various decoder
4ce09aa [Cheng Lian] Made in-memory compression configurable via SparkConf
d537a36 [Cheng Lian] Fixed SPARK-1678
(cherry picked from commit 6d721c5f7131f7c9fe56c524133d70cb37f1222d)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[SPARK-1735] Add the missing special profiles to make-distribution.sh

73b0cbcc241cca3d318ff74340e80b02f884acbd introduced a few special profiles that are not covered in the `make-distribution.sh`. This affects hadoop versions 2.2.x, 2.3.x, and 2.4.x. Without these special profiles, a java version error for protobufs is thrown at run time.

I took the opportunity to rewrite the way we construct the maven command. Previously, the only hadoop version that triggered the `yarn-alpha` profile was 0.23.x, which was inconsistent with the [docs](https://github.com/apache/spark/blob/master/docs/building-with-maven.md). This is now generalized to hadoop versions from 0.23.x to 2.1.x.

Author: Andrew Or <andrewor14@gmail.com>

Closes #660 from andrewor14/hadoop-distribution and squashes the following commits:

6740126 [Andrew Or] Generalize the yarn profile to hadoop versions 2.2+
88f192d [Andrew Or] Add the required special profiles to make-distribution.sh
(cherry picked from commit a2262cdb7aa30e9f45043f1440d4b02bc3340f9f)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>