Removed reference to incubation in README.md.

Author: Reynold Xin <rxin@apache.org>

Closes #1 from rxin/readme and squashes the following commits:

b3a77cd [Reynold Xin] Removed reference to incubation in README.md.
(cherry picked from commit 84f7ca138165ca413897dada35c602676b0a614f)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[SPARK-1089] fix the regression problem on ADD_JARS in 0.9

https://spark-project.atlassian.net/browse/SPARK-1089

copied from JIRA, reported by @ash211

"Using the ADD_JARS environment variable with spark-shell used to add the jar to both the shell and the various workers. Now it only adds to the workers and importing a custom class in the shell is broken.
The workaround is to add custom jars to both ADD_JARS and SPARK_CLASSPATH.
We should fix ADD_JARS so it works properly again.
See various threads on the user list:
https://mail-archives.apache.org/mod_mbox/incubator-spark-user/201402.mbox/%3CCAJbo4neMLiTrnm1XbyqomWmp0m+EUcg4yE-txuRGSVKOb5KLeA@mail.gmail.com%3E
(another one that doesn't appear in the archives yet titled "ADD_JARS not working on 0.9")"

The reason of this bug is two-folds

in the current implementation of SparkILoop.scala, the settings.classpath is not set properly when the process() method is invoked

the weird behaviour of Scala 2.10, (I personally thought it is a bug)

if we simply set value of a PathSettings object (like settings.classpath), the isDefault is not set to true (this is a flag showing if the variable is modified), so it makes the PathResolver loads the default CLASSPATH environment variable value to calculated the path (see https://github.com/scala/scala/blob/2.10.x/src/compiler/scala/tools/util/PathResolver.scala#L215)

what we have to do is to manually make this flag set, (https://github.com/CodingCat/incubator-spark/blob/e3991d97ddc33e77645e4559b13bf78b9e68239a/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala#L884)

Author: CodingCat <zhunansjtu@gmail.com>

Closes #13 from CodingCat/SPARK-1089 and squashes the following commits:

8af81e7 [CodingCat] impose non-null settings
9aa2125 [CodingCat] code cleaning
ce36676 [CodingCat] code cleaning
e045582 [CodingCat] fix the regression problem on ADD_JARS in 0.9
(cherry picked from commit 345df5f4a9c16a6a87440afa2b09082fc3d224bd)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Removed reference to incubation in Spark user docs.

Author: Reynold Xin <rxin@apache.org>

Closes #2 from rxin/docs and squashes the following commits:

08bbd5f [Reynold Xin] Removed reference to incubation in Spark user docs.
(cherry picked from commit 40e080a68a8fd025435e9ff84fa9280b4aba4dcf)

Conflicts:

	docs/_config.yml
Add Jekyll tag to isolate "production-only" doc components. (0.9 version)

Author: Patrick Wendell <pwendell@gmail.com>

Closes #57 from pwendell/jekyll-prod-0.9 and squashes the following commits:

69a7614 [Patrick Wendell] Add Jekyll tag to isolate "production-only" doc components.
SPARK-1109 wrong API docs for pyspark map function

Author: Prashant Sharma <prashant.s@imaginea.com>

Closes #73 from ScrapCodes/SPARK-1109/wrong-API-docs and squashes the following commits:

1a55b58 [Prashant Sharma] SPARK-1109 wrong API docs for pyspark map function

(cherry picked from commit 02836657cfec50bc6cc357541e40f8d36c90b352)
Signed-off-by: Matei Zaharia <matei@databricks.com>
SPARK-1184: Update the distribution tar.gz to include spark-assembly jar

See JIRA for details.

Author: Mark Grover <mark@apache.org>

Closes #78 from markgrover/SPARK-1184 and squashes the following commits:

12b78e6 [Mark Grover] SPARK-1184: Update the distribution tar.gz to include spark-assembly jar
(cherry picked from commit cda381f88cc03340fdf7b2d681699babbae2a56e)

Conflicts:

	assembly/pom.xml
SPARK-1190: Do not initialize log4j if slf4j log4j backend is not being used

Author: Patrick Wendell <pwendell@gmail.com>

Closes #107 from pwendell/logging and squashes the following commits:

be21c11 [Patrick Wendell] Logging fix
(cherry picked from commit e59a3b6c415b95e8137f5a154716b12653a8aed0)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
For outputformats that are Configurable, call setConf before sending data to them.

[SPARK-1108] This allows us to use, e.g. HBase's TableOutputFormat with PairRDDFunctions.saveAsNewAPIHadoopFile, which otherwise would throw NullPointerException because the output table name hasn't been configured.

Note this bug also affects branch-0.9

Author: Bryn Keller <bryn.keller@intel.com>

Closes #638 from xoltar/SPARK-1108 and squashes the following commits:

7e94e7d [Bryn Keller] Import, comment, and format cleanup per code review
7cbcaa1 [Bryn Keller] For outputformats that are Configurable, call setConf before sending data to them. This allows us to use, e.g. HBase TableOutputFormat, which otherwise would throw NullPointerException because the output table name hasn't been configured
(cherry picked from commit 4d880304867b55a4f2138617b30600b7fa013b14)

Conflicts:

	core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala
	core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala
SPARK-1167: Remove metrics-ganglia from default build due to LGPL issues...

This patch removes Ganglia integration from the default build. It
allows users willing to link against LGPL code to use Ganglia
by adding build flags or linking against a new Spark artifact called
spark-ganglia-lgpl.

This brings Spark in line with the Apache policy on LGPL code
enumerated here:

https://www.apache.org/legal/3party.html#options-optional

Author: Patrick Wendell <pwendell@gmail.com>

Closes #108 from pwendell/ganglia and squashes the following commits:

326712a [Patrick Wendell] Responding to review feedback
5f28ee4 [Patrick Wendell] SPARK-1167: Remove metrics-ganglia from default build due to LGPL issues.
(cherry picked from commit 16788a654246067fd966033b5dc9bc0d4c759b70)

Conflicts:

	core/pom.xml
	dev/audit-release/sbt_app_core/src/main/scala/SparkApp.scala
	dev/create-release/create-release.sh
	pom.xml
	project/SparkBuild.scala
Log4j build fix on 0.9 branch

Spark should include log4j by default. Downstream packagers can exclude log4j
if they want to use another logging backend.
Version fix in pom file
SPARK-1162 Added top in python.

Author: Prashant Sharma <prashant.s@imaginea.com>

Closes #93 from ScrapCodes/SPARK-1162/pyspark-top-takeOrdered and squashes the following commits:

ece1fa4 [Prashant Sharma] Added top in python.

(cherry picked from commit b8afe3052086547879ebf28d6e36207e0d370710)
Signed-off-by: Matei Zaharia <matei@databricks.com>
Fix example bug: compile error

Author: jianghan <jianghan@xiaomi.com>

Closes #132 from pooorman/master and squashes the following commits:

54afbe0 [jianghan] Fix example bug: compile error
(cherry picked from commit 31a704004f9b4ad34f92ae5c95ae6e90d0ab62c7)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1019: pyspark RDD take() throws an NPE

Author: Patrick Wendell <pwendell@gmail.com>

Closes #112 from pwendell/pyspark-take and squashes the following commits:

daae80e [Patrick Wendell] SPARK-1019: pyspark RDD take() throws an NPE
(cherry picked from commit 4ea23db0efff2f39ac5b8f0bd1d9a6ffa3eceb0d)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Updated link for pyspark examples in docs

Author: Jyotiska NK <jyotiska123@gmail.com>

Closes #22 from jyotiska/pyspark_docs and squashes the following commits:

426136c [Jyotiska NK] Updated link for pyspark examples
SPARK-1168, Added foldByKey to pyspark.

Author: Prashant Sharma <prashant.s@imaginea.com>

Closes #115 from ScrapCodes/SPARK-1168/pyspark-foldByKey and squashes the following commits:

db6f67e [Prashant Sharma] SPARK-1168, Added foldByKey to pyspark.
Spark-1163, Added missing Python RDD functions

Author: prabinb <prabin.banka@imaginea.com>

Closes #92 from prabinb/python-api-rdd and squashes the following commits:

51129ca [prabinb] Added missing Python RDD functions Added __repr__ function to StorageLevel class. Added doctest for RDD.getStorageLevel().

Conflicts:

	python/pyspark/rdd.py
SPARK-977 Added Python RDD.zip function

was raised earlier as a part of  apache/incubator-spark#486

Author: Prabin Banka <prabin.banka@imaginea.com>

Closes #76 from prabinb/python-api-zip and squashes the following commits:

b1a31a0 [Prabin Banka] Added Python RDD.zip function

Conflicts:

	python/pyspark/rdd.py
SPARK-1240: handle the case of empty RDD when takeSample

https://spark-project.atlassian.net/browse/SPARK-1240

It seems that the current implementation does not handle the empty RDD case when run takeSample

In this patch, before calling sample() inside takeSample API, I add a checker for this case and returns an empty Array when it's a empty RDD; also in sample(), I add a checker for the invalid fraction value

In the test case, I also add several lines for this case

Author: CodingCat <zhunansjtu@gmail.com>

Closes #135 from CodingCat/SPARK-1240 and squashes the following commits:

fef57d4 [CodingCat] fix the same problem in PySpark
36db06b [CodingCat] create new test cases for takeSample from an empty red
810948d [CodingCat] further fix
a40e8fb [CodingCat] replace if with require
ad483fd [CodingCat] handle the case with empty RDD when take sample

Conflicts:
	core/src/main/scala/org/apache/spark/rdd/RDD.scala
	core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala
SPARK-1244: Throw exception if map output status exceeds frame size

This is a very small change on top of @andrewor14's patch in #147.

Author: Patrick Wendell <pwendell@gmail.com>
Author: Andrew Or <andrewor14@gmail.com>

Closes #152 from pwendell/akka-frame and squashes the following commits:

e5fb3ff [Patrick Wendell] Reversing test order
393af4c [Patrick Wendell] Small improvement suggested by Andrew Or
8045103 [Patrick Wendell] Breaking out into two tests
2b4e085 [Patrick Wendell] Consolidate Executor use of akka frame size
c9b6109 [Andrew Or] Simplify test + make access to akka frame size more modular
281d7c9 [Andrew Or] Throw exception on spark.akka.frameSize exceeded + Unit tests
(cherry picked from commit 796977acdb5c96ca5c08591657137fb3e44d2e94)

Conflicts:

	core/src/test/scala/org/apache/spark/AkkaUtilsSuite.scala
	core/src/test/scala/org/apache/spark/MapOutputTrackerSuite.scala