HOTFIX: Removing out dated python path in testing tool.
[SPARK-1516]Throw exception in yarn client instead of run system.exit

    [SPARK-1516]Throw exception in yarn client instead of run system.exit directly.

    All the changes is in  the package of "org.apache.spark.deploy.yarn":
        1) Throw IllegalArgumentException in ClinetArguments  instead of exit directly.
        2) In Client's main method, if exception is caught, it will exit with code 1, otherwise exit with code 0.
        3) In YarnClientSchedulerBackend's start method, if IllegalArgumentException is caught, it will exit with code 1, otherwise throw that exception.
        4) Fix some message typo in the Client.scala
    After the fix, if user integrate the spark yarn client into their applications,
    when the argument is wrong or the running is finished, the application won't be terminated.

+CC dbtsai mengxr

Author: John Zhao <codeboyyong@gmail.com>

Closes #1099 from codeboyyong/branch-0.9 and squashes the following commits:

00144b5 [John Zhao] use e.printStackTrace() to replace "Console.err.println(e.getMessage)" so that client console can  get more useful information when somthing is wrong.
addcecb [John Zhao]   [SPARK-1516]Throw exception in yarn client instead of run system.exit directly.
[SPARK-2350] Don't NPE while launching drivers

Prior to this change, we could throw a NPE if we launch a driver while another one is waiting, because removing from an iterator while iterating over it is not safe.

Author: Aaron Davidson <aaron@databricks.com>

Closes #1289 from aarondav/master-fail and squashes the following commits:

1cf1cf4 [Aaron Davidson] SPARK-2350: Don't NPE while launching drivers
(cherry picked from commit 586feb5c9528042420f678f78bacb6c254a5eaf8)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-2282: Reuse PySpark Accumulator sockets to avoid crashing Spark

JIRA: https://issues.apache.org/jira/browse/SPARK-2282

This issue is caused by a buildup of sockets in the TIME_WAIT stage of TCP, which is a stage that lasts for some period of time after the communication closes.

This solution simply allows us to reuse sockets that are in TIME_WAIT, to avoid issues with the buildup of the rapid creation of these sockets.

Author: Aaron Davidson <aaron@databricks.com>

Closes #1220 from aarondav/SPARK-2282 and squashes the following commits:

2e5cab3 [Aaron Davidson] SPARK-2282: Reuse PySpark Accumulator sockets to avoid crashing Spark
(cherry picked from commit 97a0bfe1c0261384f09d53f9350de52fb6446d59)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[SPARK-2362] Fix for newFilesOnly logic in file DStream

The newFilesOnly logic should be inverted: the logic should be that if the flag newFilesOnly==true then only start reading files older than current time. As the code is now if newFilesOnly==true then it will start to read files that are older than 0L (that is: every file in the directory).

Author: Gabriele Nizzoli <mail@nizzoli.net>

Closes #1077 from gabrielenizzoli/master and squashes the following commits:

4f1d261 [Gabriele Nizzoli] Fix for newFilesOnly logic in file DStream

(cherry picked from commit e6f7bfcfbf6aff7a9f8cd8e0a2166d0bf62b0912)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
Merge branch 'branch-0.9' of github.com:apache/spark into master-csd
[SPARK-2433][MLLIB] fix NaiveBayesModel.predict

This is the same as https://github.com/apache/spark/pull/463 , which I forgot to merge into branch-0.9.

Author: Xiangrui Meng <meng@databricks.com>

Closes #1453 from mengxr/nb-transpose-0.9 and squashes the following commits:

bc53ce8 [Xiangrui Meng] fix NaiveBayes
[SPARK-1112, 2156] (0.9 edition) Use correct akka frame size and overhead amounts.

backport #1172 to branch-0.9.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #1455 from mengxr/akka-fix-0.9 and squashes the following commits:

a99f201 [Patrick Wendell] backport PR #1172 to branch-0.9
[branch-0.9] Fix github links in docs

We moved example code in v1.0. The links are no longer valid if still pointing to `tree/master`.

Author: Xiangrui Meng <meng@databricks.com>

Closes #1456 from mengxr/links-0.9 and squashes the following commits:

b7b9260 [Xiangrui Meng] change tree/master to tree/branch-0.9 in docs
[branch-0.9] bump versions for v0.9.2 release candidate

Manually update some version numbers.

Author: Xiangrui Meng <meng@databricks.com>

Closes #1458 from mengxr/v0.9.2-rc and squashes the following commits:

2c38419 [Xiangrui Meng] Merge remote-tracking branch 'apache/branch-0.9' into v0.9.2-rc
7d0fb76 [Xiangrui Meng] change tree/master to tree/branch-0.9 in docs
ea2b205 [Xiangrui Meng] update version in SparkBuild
162af66 [Xiangrui Meng] Merge remote-tracking branch 'apache/branch-0.9' into v0.9.2-rc
bc87035 [Xiangrui Meng] bump version numbers to 0.9.2
[branch-0.9] Update CHANGES.txt

Author: Xiangrui Meng <meng@databricks.com>

Closes #1459 from mengxr/v0.9.2-rc and squashes the following commits:

6fa5a65 [Xiangrui Meng] update CHANGES.txt
[maven-release-plugin] prepare release v0.9.2-rc1
Merge tag 'v0.9.2-rc1' of github.com:apache/spark into master-csd

[maven-release-plugin]  copy for tag v0.9.2-rc1

Conflicts:
	assembly/pom.xml
	bagel/pom.xml
	core/pom.xml
	examples/pom.xml
	external/flume/pom.xml
	external/kafka/pom.xml
	external/mqtt/pom.xml
	external/twitter/pom.xml
	external/zeromq/pom.xml
	extras/spark-ganglia-lgpl/pom.xml
	graphx/pom.xml
	mllib/pom.xml
	pom.xml
	project/SparkBuild.scala
	python/pyspark/shell.py
	repl/pom.xml
	streaming/pom.xml
	tools/pom.xml
	yarn/pom.xml
	yarn/stable/pom.xml
Changed default scheduling mode to FAIR
Merge branch 'master-csd' of github.com:clearstorydata/spark into master-csd
SPARK-2425 Don't kill a still-running Application because of some misbehaving Executors

Introduces a LOADING -> RUNNING ApplicationState transition and prevents Master from removing an Application with RUNNING Executors.

Two basic changes: 1) Instead of allowing MAX_NUM_RETRY abnormal Executor exits over the entire lifetime of the Application, allow that many since any Executor successfully began running the Application; 2) Don't remove the Application while Master still thinks that there are RUNNING Executors.

This should be fine as long as the ApplicationInfo doesn't believe any Executors are forever RUNNING when they are not.  I think that any non-RUNNING Executors will eventually no longer be RUNNING in Master's accounting, but another set of eyes should confirm that.  This PR also doesn't try to detect which nodes have gone rogue or to kill off bad Workers, so repeatedly failing Executors will continue to fail and fill up log files with failure reports as long as the Application keeps running.

Author: Mark Hamstra <markhamstra@gmail.com>

Closes #1360 from markhamstra/SPARK-2425 and squashes the following commits:

f099c0b [Mark Hamstra] Reuse appInfo
b2b7b25 [Mark Hamstra] Moved 'Application failed' logging
bdd0928 [Mark Hamstra] switched to string interpolation
1dd591b [Mark Hamstra] SPARK-2425 introduce LOADING -> RUNNING ApplicationState transition and prevent Master from removing Application with RUNNING Executors

Conflicts:
	core/src/main/scala/org/apache/spark/deploy/master/Master.scala
	core/src/main/scala/org/apache/spark/deploy/worker/ExecutorRunner.scala
	core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
Fixed mis-merge
Merge branch 'master-csd' of github.com:clearstorydata/spark into master-csd
Removed 'candidate' status
Merge branch 'master-csd' of github.com:markhamstra/spark into master-csd