[SPARK-1089] fix the regression problem on ADD_JARS in 0.9

https://spark-project.atlassian.net/browse/SPARK-1089

copied from JIRA, reported by @ash211

"Using the ADD_JARS environment variable with spark-shell used to add the jar to both the shell and the various workers. Now it only adds to the workers and importing a custom class in the shell is broken.
The workaround is to add custom jars to both ADD_JARS and SPARK_CLASSPATH.
We should fix ADD_JARS so it works properly again.
See various threads on the user list:
https://mail-archives.apache.org/mod_mbox/incubator-spark-user/201402.mbox/%3CCAJbo4neMLiTrnm1XbyqomWmp0m+EUcg4yE-txuRGSVKOb5KLeA@mail.gmail.com%3E
(another one that doesn't appear in the archives yet titled "ADD_JARS not working on 0.9")"

The reason of this bug is two-folds

in the current implementation of SparkILoop.scala, the settings.classpath is not set properly when the process() method is invoked

the weird behaviour of Scala 2.10, (I personally thought it is a bug)

if we simply set value of a PathSettings object (like settings.classpath), the isDefault is not set to true (this is a flag showing if the variable is modified), so it makes the PathResolver loads the default CLASSPATH environment variable value to calculated the path (see https://github.com/scala/scala/blob/2.10.x/src/compiler/scala/tools/util/PathResolver.scala#L215)

what we have to do is to manually make this flag set, (https://github.com/CodingCat/incubator-spark/blob/e3991d97ddc33e77645e4559b13bf78b9e68239a/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala#L884)

Author: CodingCat <zhunansjtu@gmail.com>

Closes #13 from CodingCat/SPARK-1089 and squashes the following commits:

8af81e7 [CodingCat] impose non-null settings
9aa2125 [CodingCat] code cleaning
ce36676 [CodingCat] code cleaning
e045582 [CodingCat] fix the regression problem on ADD_JARS in 0.9
(cherry picked from commit 345df5f4a9c16a6a87440afa2b09082fc3d224bd)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Removed reference to incubation in Spark user docs.

Author: Reynold Xin <rxin@apache.org>

Closes #2 from rxin/docs and squashes the following commits:

08bbd5f [Reynold Xin] Removed reference to incubation in Spark user docs.
(cherry picked from commit 40e080a68a8fd025435e9ff84fa9280b4aba4dcf)

Conflicts:

	docs/_config.yml
Add Jekyll tag to isolate "production-only" doc components. (0.9 version)

Author: Patrick Wendell <pwendell@gmail.com>

Closes #57 from pwendell/jekyll-prod-0.9 and squashes the following commits:

69a7614 [Patrick Wendell] Add Jekyll tag to isolate "production-only" doc components.
Merge branch 'branch-0.9' of github.com:apache/spark into csd-2.10
SPARK-1109 wrong API docs for pyspark map function

Author: Prashant Sharma <prashant.s@imaginea.com>

Closes #73 from ScrapCodes/SPARK-1109/wrong-API-docs and squashes the following commits:

1a55b58 [Prashant Sharma] SPARK-1109 wrong API docs for pyspark map function

(cherry picked from commit 02836657cfec50bc6cc357541e40f8d36c90b352)
Signed-off-by: Matei Zaharia <matei@databricks.com>
Merge branch 'branch-0.9' of github.com:apache/spark into csd-2.10
SPARK-1184: Update the distribution tar.gz to include spark-assembly jar

See JIRA for details.

Author: Mark Grover <mark@apache.org>

Closes #78 from markgrover/SPARK-1184 and squashes the following commits:

12b78e6 [Mark Grover] SPARK-1184: Update the distribution tar.gz to include spark-assembly jar
(cherry picked from commit cda381f88cc03340fdf7b2d681699babbae2a56e)

Conflicts:

	assembly/pom.xml
Merge branch 'branch-0.9' of github.com:apache/spark into csd-2.10