SPARK-1167: Remove metrics-ganglia from default build due to LGPL issues...

This patch removes Ganglia integration from the default build. It
allows users willing to link against LGPL code to use Ganglia
by adding build flags or linking against a new Spark artifact called
spark-ganglia-lgpl.

This brings Spark in line with the Apache policy on LGPL code
enumerated here:

https://www.apache.org/legal/3party.html#options-optional

Author: Patrick Wendell <pwendell@gmail.com>

Closes #108 from pwendell/ganglia and squashes the following commits:

326712a [Patrick Wendell] Responding to review feedback
5f28ee4 [Patrick Wendell] SPARK-1167: Remove metrics-ganglia from default build due to LGPL issues.
(cherry picked from commit 16788a654246067fd966033b5dc9bc0d4c759b70)

Conflicts:

	core/pom.xml
	dev/audit-release/sbt_app_core/src/main/scala/SparkApp.scala
	dev/create-release/create-release.sh
	pom.xml
	project/SparkBuild.scala
Log4j build fix on 0.9 branch

Spark should include log4j by default. Downstream packagers can exclude log4j
if they want to use another logging backend.
Version fix in pom file
SPARK-1162 Added top in python.

Author: Prashant Sharma <prashant.s@imaginea.com>

Closes #93 from ScrapCodes/SPARK-1162/pyspark-top-takeOrdered and squashes the following commits:

ece1fa4 [Prashant Sharma] Added top in python.

(cherry picked from commit b8afe3052086547879ebf28d6e36207e0d370710)
Signed-off-by: Matei Zaharia <matei@databricks.com>
Fix example bug: compile error

Author: jianghan <jianghan@xiaomi.com>

Closes #132 from pooorman/master and squashes the following commits:

54afbe0 [jianghan] Fix example bug: compile error
(cherry picked from commit 31a704004f9b4ad34f92ae5c95ae6e90d0ab62c7)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1019: pyspark RDD take() throws an NPE

Author: Patrick Wendell <pwendell@gmail.com>

Closes #112 from pwendell/pyspark-take and squashes the following commits:

daae80e [Patrick Wendell] SPARK-1019: pyspark RDD take() throws an NPE
(cherry picked from commit 4ea23db0efff2f39ac5b8f0bd1d9a6ffa3eceb0d)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Updated link for pyspark examples in docs

Author: Jyotiska NK <jyotiska123@gmail.com>

Closes #22 from jyotiska/pyspark_docs and squashes the following commits:

426136c [Jyotiska NK] Updated link for pyspark examples
SPARK-1168, Added foldByKey to pyspark.

Author: Prashant Sharma <prashant.s@imaginea.com>

Closes #115 from ScrapCodes/SPARK-1168/pyspark-foldByKey and squashes the following commits:

db6f67e [Prashant Sharma] SPARK-1168, Added foldByKey to pyspark.
Spark-1163, Added missing Python RDD functions

Author: prabinb <prabin.banka@imaginea.com>

Closes #92 from prabinb/python-api-rdd and squashes the following commits:

51129ca [prabinb] Added missing Python RDD functions Added __repr__ function to StorageLevel class. Added doctest for RDD.getStorageLevel().

Conflicts:

	python/pyspark/rdd.py
SPARK-977 Added Python RDD.zip function

was raised earlier as a part of  apache/incubator-spark#486

Author: Prabin Banka <prabin.banka@imaginea.com>

Closes #76 from prabinb/python-api-zip and squashes the following commits:

b1a31a0 [Prabin Banka] Added Python RDD.zip function

Conflicts:

	python/pyspark/rdd.py
SPARK-1240: handle the case of empty RDD when takeSample

https://spark-project.atlassian.net/browse/SPARK-1240

It seems that the current implementation does not handle the empty RDD case when run takeSample

In this patch, before calling sample() inside takeSample API, I add a checker for this case and returns an empty Array when it's a empty RDD; also in sample(), I add a checker for the invalid fraction value

In the test case, I also add several lines for this case

Author: CodingCat <zhunansjtu@gmail.com>

Closes #135 from CodingCat/SPARK-1240 and squashes the following commits:

fef57d4 [CodingCat] fix the same problem in PySpark
36db06b [CodingCat] create new test cases for takeSample from an empty red
810948d [CodingCat] further fix
a40e8fb [CodingCat] replace if with require
ad483fd [CodingCat] handle the case with empty RDD when take sample

Conflicts:
	core/src/main/scala/org/apache/spark/rdd/RDD.scala
	core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala
SPARK-1244: Throw exception if map output status exceeds frame size

This is a very small change on top of @andrewor14's patch in #147.

Author: Patrick Wendell <pwendell@gmail.com>
Author: Andrew Or <andrewor14@gmail.com>

Closes #152 from pwendell/akka-frame and squashes the following commits:

e5fb3ff [Patrick Wendell] Reversing test order
393af4c [Patrick Wendell] Small improvement suggested by Andrew Or
8045103 [Patrick Wendell] Breaking out into two tests
2b4e085 [Patrick Wendell] Consolidate Executor use of akka frame size
c9b6109 [Andrew Or] Simplify test + make access to akka frame size more modular
281d7c9 [Andrew Or] Throw exception on spark.akka.frameSize exceeded + Unit tests
(cherry picked from commit 796977acdb5c96ca5c08591657137fb3e44d2e94)

Conflicts:

	core/src/test/scala/org/apache/spark/AkkaUtilsSuite.scala
	core/src/test/scala/org/apache/spark/MapOutputTrackerSuite.scala
[Spark-1261] add instructions for running python examples to doc overview page

Author: Diana Carroll <dcarroll@cloudera.com>

Closes #162 from dianacarroll/SPARK-1261 and squashes the following commits:

14ac602 [Diana Carroll] typo in python example text
5121e3e [Diana Carroll] Add explanation of how to run Python examples to main doc overview page
Bundle tachyon:  SPARK-1269

This should all work as expected with the current version of the tachyon tarball (0.4.1)

Author: Nick Lanham <nick@afternight.org>

Closes #137 from nicklan/bundle-tachyon and squashes the following commits:

2eee15b [Nick Lanham] Put back in exec, start tachyon first
738ba23 [Nick Lanham] Move tachyon out of sbin
f2f9bc6 [Nick Lanham] More checks for tachyon script
111e8e1 [Nick Lanham] Only try tachyon operations if tachyon script exists
0561574 [Nick Lanham] Copy over web resources so web interface can run
4dc9809 [Nick Lanham] Update to tachyon 0.4.1
0a1a20c [Nick Lanham] Add scripts using tachyon tarball
(cherry picked from commit a18ea00f3af0fa4c6b2c59933e22b6c9f0f636c8)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[SPARK-1274] Add dev scripts to merge PRs and create releases from master to branch-0.9

All the files are one-to-one copied from master. Only the Spark version numbers were changed.

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes #176 from tdas/branch-0.9 and squashes the following commits:

fb1b913 [Tathagata Das] Changed version numbers for 0.9.1 release.
3303d5e [Tathagata Das] Copied Apache release scripts in master to branch-0.9
bugfix: Wrong "Duration" in "Active Stages" in stages page

If a stage which has completed once loss parts of data, it will be resubmitted. At this time, it appears that stage.completionTime > stage.submissionTime.

Author: shiyun.wxm <shiyun.wxm@taobao.com>

Closes #170 from BlackNiuza/duration_problem and squashes the following commits:

a86d261 [shiyun.wxm] tow space indent
c0d7b24 [shiyun.wxm] change the style
3b072e1 [shiyun.wxm] fix scala style
f20701e [shiyun.wxm] bugfix: "Duration" in "Active Stages" in stages page

(cherry picked from commit d55ec86de2e96f7dc9d1dd107daa35c3823791ec)
Signed-off-by: Reynold Xin <rxin@apache.org>
SPARK-1203 fix saving to hdfs from yarn

Author: Thomas Graves <tgraves@apache.org>

Closes #173 from tgravescs/SPARK-1203 and squashes the following commits:

4fd5ded [Thomas Graves] adding import
964e3f7 [Thomas Graves] SPARK-1203 fix saving to hdfs from yarn
Update the yarn alpha version to 0.9.1-incubating-SNAPSHOT
[SPARK-1275] Made dev/run-tests executable.

This was causing Jenkins tests to fail for PRs against branch 0.9.

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes #178 from tdas/branch-0.9-fix and squashes the following commits:

a633bce [Tathagata Das] Merge remote-tracking branch 'apache-github/branch-0.9' into branch-0.9-fix
9b043cc [Tathagata Das] Made dev/run-tests executable.
[SPARK-1273] MLlib bug fixes, improvements, and doc updates for v0.9.1

Cherry-picked a few MLlib commits that are bug fixes, optimization, or doc updates for the v0.9.1 release.

JIRA: https://spark-project.atlassian.net/browse/SPARK-1273

Author: Xiangrui Meng <meng@databricks.com>
Author: Sean Owen <sowen@cloudera.com>
Author: Andrew Tulloch <andrew@tullo.ch>
Author: Chen Chao <crazyjvm@gmail.com>

Closes #175 from mengxr/branch-0.9 and squashes the following commits:

d8928ea [Xiangrui Meng] add Apache header to LocalSparkContext
a66d386 [Xiangrui Meng] Merge remote-tracking branch 'apache/branch-0.9' into branch-0.9
a899894 [Xiangrui Meng] [SPARK-1237, 1238] Improve the computation of YtY for implicit ALS
46fe493 [Xiangrui Meng] [SPARK-1260]: faster construction of features with intercept
6340a18 [Sean Owen] MLLIB-22. Support negative implicit input in ALS
f27441a [Chen Chao] MLLIB-24:  url of "Collaborative Filtering for Implicit Feedback Datasets" in ALS is invalid now
a26ac90 [Sean Owen] Merge pull request #460 from srowen/RandomInitialALSVectors
0564985 [Andrew Tulloch] Fixed import order
2512e67 [Andrew Tulloch] LocalSparkContext for MLlib