SPARK-1878: Fix the incorrect initialization order

JIRA: https://issues.apache.org/jira/browse/SPARK-1878

Author: zsxwing <zsxwing@gmail.com>

Closes #822 from zsxwing/SPARK-1878 and squashes the following commits:

4a47e27 [zsxwing] SPARK-1878: Fix the incorrect initialization order

(cherry picked from commit 1811ba8ccb580979aa2e12019e6a82805f09ab53)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
SPARK-1879. Increase MaxPermSize since some of our builds have many classes

See https://issues.apache.org/jira/browse/SPARK-1879 -- builds with Hadoop2 and Hive ran out of PermGen space in spark-shell, when those things added up with the Scala compiler.

Note that users can still override it by setting their own Java options with this change. Their options will come later in the command string than the -XX:MaxPermSize=128m.

Author: Matei Zaharia <matei@databricks.com>

Closes #823 from mateiz/spark-1879 and squashes the following commits:

6bc0ee8 [Matei Zaharia] Increase MaxPermSize to 128m since some of our builds have lots of classes

(cherry picked from commit 5af99d7617ba3b9fbfdb345ef9571b7dd41f45a1)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
[SPARK-1875]NoClassDefFoundError: StringUtils when building with hadoop 1.x and hive

Author: witgo <witgo@qq.com>

Closes #824 from witgo/SPARK-1875_commons-lang-2.6 and squashes the following commits:

ef7231d [witgo] review commit
ead3c3b [witgo] SPARK-1875:NoClassDefFoundError: StringUtils when building against Hadoop 1

(cherry picked from commit 6a2c5c610c259f62cb12d8cfc18bf59cdb334bb2)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
SPARK-1689: Spark application should die when removed by Master

scheduler.error() will mask the error if there are active tasks. Being removed is a cataclysmic event for Spark applications, and should probably be treated as such.

Author: Aaron Davidson <aaron@databricks.com>

Closes #832 from aarondav/i-love-u and squashes the following commits:

9f1200f [Aaron Davidson] SPARK-1689: Spark application should die when removed by Master

(cherry picked from commit b0ce22e071da4cc62ec5e29abf7b1299b8e4a6b0)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
[SPARK-1874][MLLIB] Clean up MLlib sample data

1. Added synthetic datasets for `MovieLensALS`, `LinearRegression`, `BinaryClassification`.
2. Embedded instructions in the help message of those example apps.

Per discussion with Matei on the JIRA page, new example data is under `data/mllib`.

Author: Xiangrui Meng <meng@databricks.com>

Closes #833 from mengxr/mllib-sample-data and squashes the following commits:

59f0a18 [Xiangrui Meng] add sample binary classification data
3c2f92f [Xiangrui Meng] add linear regression data
050f1ca [Xiangrui Meng] add a sample dataset for MovieLensALS example

(cherry picked from commit bcb9dce6f444a977c714117811bce0c54b417650)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
[Spark 1877] ClassNotFoundException when loading RDD with serialized objects

Updated version of #821

Author: Tathagata Das <tathagata.das1565@gmail.com>
Author: Ghidireac <bogdang@u448a5b0a73d45358d94a.ant.amazon.com>

Closes #835 from tdas/SPARK-1877 and squashes the following commits:

f346f71 [Tathagata Das] Addressed Patrick's comments.
fee0c5d [Ghidireac] SPARK-1877: ClassNotFoundException when loading RDD with serialized objects

(cherry picked from commit 52eb54d02403a3c37d84b9da7cc1cdb261048cf8)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
Updated CHANGES.txt
Revert "[maven-release-plugin] prepare for next development iteration"

This reverts commit f8e611955096c5c1c7db5764b9d2851b1d295f0d.
Revert "[maven-release-plugin] prepare release v1.0.0-rc9"

This reverts commit 920f947eb5a22a679c0c3186cf69ee75f6041c75.
[Hotfix] Blacklisted flaky HiveCompatibility test

`lateral_view_outer` query sometimes returns a different set of 10 rows.

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes #838 from tdas/hive-test-fix2 and squashes the following commits:

9128a0d [Tathagata Das] Blacklisted flaky HiveCompatibility test.

(cherry picked from commit 7f0cfe47f4709843d70ceccc25dee7551206ce0d)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
[maven-release-plugin] prepare release v1.0.0-rc10
[maven-release-plugin] prepare for next development iteration
[SPARK-1250] Fixed misleading comments in bin/pyspark, bin/spark-class

Fixed a couple of misleading comments in bin/pyspark and bin/spark-class. The comments make it seem like the script is looking for the Scala installation when in fact it is looking for Spark.

Author: Sumedh Mungee <smungee@gmail.com>

Closes #843 from smungee/spark-1250-fix-comments and squashes the following commits:

26870f3 [Sumedh Mungee] [SPARK-1250] Fixed misleading comments in bin/pyspark and bin/spark-class

(cherry picked from commit 6e337380fc47071fc7fb28d744e8209c729fe1e9)
Signed-off-by: Reynold Xin <rxin@apache.org>
[Docs] Correct example of creating a new SparkConf

The example code on the configuration page currently does not compile.

Author: Andrew Or <andrewor14@gmail.com>

Closes #842 from andrewor14/conf-docs and squashes the following commits:

aabff57 [Andrew Or] Correct example of creating a new SparkConf

(cherry picked from commit 1014668f2727863fe46f9c75201ee459d093bf0c)
Signed-off-by: Reynold Xin <rxin@apache.org>
[Minor] Move JdbcRDDSuite to the correct package

It was in the wrong package

Author: Andrew Or <andrewor14@gmail.com>

Closes #839 from andrewor14/jdbc-suite and squashes the following commits:

f948c5a [Andrew Or] cache -> cache()
b215279 [Andrew Or] Move JdbcRDDSuite to the correct package

(cherry picked from commit 7c79ef7d43de258ad9a5de15c590132bd78ce8dd)
Signed-off-by: Reynold Xin <rxin@apache.org>
[Typo] Stoped -> Stopped

Author: Andrew Or <andrewor14@gmail.com>

Closes #847 from andrewor14/yarn-typo and squashes the following commits:

c1906af [Andrew Or] Stoped -> Stopped

(cherry picked from commit ba5d4a99425a2083fea2a9759050c5e770197e23)
Signed-off-by: Reynold Xin <rxin@apache.org>
[SPARK-1519] Support minPartitions param of wholeTextFiles() in PySpark

Author: Kan Zhang <kzhang@apache.org>

Closes #697 from kanzhang/SPARK-1519 and squashes the following commits:

4f8d1ed [Kan Zhang] [SPARK-1519] Support minPartitions param of wholeTextFiles() in PySpark

(cherry picked from commit f18fd05b513b136363c94adb3e5b841f8bf48134)
Signed-off-by: Reynold Xin <rxin@apache.org>
[SPARK-1889] [SQL] Apply splitConjunctivePredicates to join condition while finding join ke...

...ys.

When tables are equi-joined by multiple-keys `HashJoin` should be used, but `CartesianProduct` and then `Filter` are used.
The join keys are paired by `And` expression so we need to apply `splitConjunctivePredicates` to join condition while finding join keys.

Author: Takuya UESHIN <ueshin@happy-camper.st>

Closes #836 from ueshin/issues/SPARK-1889 and squashes the following commits:

fe1c387 [Takuya UESHIN] Apply splitConjunctivePredicates to join condition while finding join keys.

(cherry picked from commit bb88875ad52e8209c25e8350af1fe4b7159086ae)
Signed-off-by: Reynold Xin <rxin@apache.org>
Configuration documentation updates

1. Add < code > to configuration options
2. List env variables in tabular format to be consistent with other pages.
3. Moved Viewing Spark Properties section up.

This is against branch-1.0, but should be cherry picked into master as well.

Author: Reynold Xin <rxin@apache.org>

Closes #851 from rxin/doc-config and squashes the following commits:

28ac0d3 [Reynold Xin] Add <code> to configuration options, and list env variables in a table.
[SPARK-1870] Make spark-submit --jars work in yarn-cluster mode.

Sent secondary jars to distributed cache of all containers and add the cached jars to classpath before executors start. Tested on a YARN cluster (CDH-5.0).

`spark-submit --jars` also works in standalone server and `yarn-client`. Thanks for @andrewor14 for testing!

I removed "Doesn't work for drivers in standalone mode with "cluster" deploy mode." from `spark-submit`'s help message, though we haven't tested mesos yet.

CC: @dbtsai @sryza

Author: Xiangrui Meng <meng@databricks.com>

Closes #848 from mengxr/yarn-classpath and squashes the following commits:

23e7df4 [Xiangrui Meng] rename spark.jar to __spark__.jar and app.jar to __app__.jar to avoid confliction apped $CWD/ and $CWD/* to the classpath remove unused methods
a40f6ed [Xiangrui Meng] standalone -> cluster
65e04ad [Xiangrui Meng] update spark-submit help message and add a comment for yarn-client
11e5354 [Xiangrui Meng] minor changes
3e7e1c4 [Xiangrui Meng] use sparkConf instead of hadoop conf
dc3c825 [Xiangrui Meng] add secondary jars to classpath in yarn

(cherry picked from commit dba314029b4c9d72d7e48a2093b39edd01931f57)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>