Merge pull request #501 from JoshRosen/cartesian-rdd-fixes

Fix two bugs in PySpark cartesian(): SPARK-978 and SPARK-1034

This pull request fixes two bugs in PySpark's `cartesian()` method:

- [SPARK-978](https://spark-project.atlassian.net/browse/SPARK-978): PySpark's cartesian method throws ClassCastException exception
- [SPARK-1034](https://spark-project.atlassian.net/browse/SPARK-1034): Py4JException on PySpark Cartesian Result

The JIRAs have more details describing the fixes.
(cherry picked from commit cad3002fead89d3c9a8de4fa989e88f367bc0b05)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #502 from pwendell/clone-1

Remove Hadoop object cloning and warn users making Hadoop RDD's.

The code introduced in #359 used Hadoop's WritableUtils.clone() to
duplicate objects when reading from Hadoop files. Some users have
reported exceptions when cloning data in various file formats,
including Avro and another custom format.

This patch removes that functionality to ensure stability for the
0.9 release. Instead, it puts a clear warning in the documentation
that copying may be necessary for Hadoop data sets.
(cherry picked from commit c3196171f3dffde6c9e67e3d35c398a01fbba846)

Conflicts:

	core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala
Merge pull request #503 from pwendell/master

Fix bug on read-side of external sort when using Snappy.

This case wasn't handled correctly and this patch fixes it.
(cherry picked from commit 3d6e75419330d27435becfdf8cfb0b6d20d56cf8)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #505 from JoshRosen/SPARK-1026

Deprecate mapPartitionsWithSplit in PySpark (SPARK-1026)

This commit deprecates `mapPartitionsWithSplit` in PySpark (see [SPARK-1026](https://spark-project.atlassian.net/browse/SPARK-1026) and removes the remaining references to it from the docs.
(cherry picked from commit 05be7047744c88e64e7e6bd973f9bcfacd00da5f)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Revert "[maven-release-plugin] prepare for next development iteration"

This reverts commit e1dc5bedb48d2ac9b9e1c9b3b1a15c41b7d90ad8.
Revert "[maven-release-plugin] prepare release v0.9.0-incubating"

This reverts commit 0771df675363c69622404cb514bd751bc90526af.
Updating changes file
[maven-release-plugin] prepare release v0.9.0-incubating
[maven-release-plugin] prepare for next development iteration
Merge pull request #523 from JoshRosen/SPARK-1043

Switch from MUTF8 to UTF8 in PySpark serializers.

This fixes SPARK-1043, a bug introduced in 0.9.0 where PySpark couldn't serialize strings > 64kB.

This fix was written by @tyro89 and @bouk in #512. This commit squashes and rebases their pull request in order to fix some merge conflicts.
(cherry picked from commit f8c742ce274fbae2a9e616d4c97469b6a22069bb)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #497 from tdas/docs-update

Updated Spark Streaming Programming Guide

Here is the updated version of the Spark Streaming Programming Guide. This is still a work in progress, but the major changes are in place. So feedback is most welcome.

In general, I have tried to make the guide to easier to understand even if the reader does not know much about Spark. The updated website is hosted here -

http://www.eecs.berkeley.edu/~tdas/spark_docs/streaming-programming-guide.html

The major changes are:
- Overview illustrates the usecases of Spark Streaming - various input sources and various output sources
- An example right after overview to quickly give an idea of what Spark Streaming program looks like
- Made Java API and examples a first class citizen like Scala by using tabs to show both Scala and Java examples (similar to AMPCamp tutorial's code tabs)
- Highlighted the DStream operations updateStateByKey and transform because of their powerful nature
- Updated driver node failure recovery text to highlight automatic recovery in Spark standalone mode
- Added information about linking and using the external input sources like Kafka and Flume
- In general, reorganized the sections to better show the Basic section and the more advanced sections like Tuning and Recovery.

Todos:
- Links to the docs of external Kafka, Flume, etc
- Illustrate window operation with figure as well as example.

Author: Tathagata Das <tathagata.das1565@gmail.com>

== Merge branch commits ==

commit 18ff10556570b39d672beeb0a32075215cfcc944
Author: Tathagata Das <tathagata.das1565@gmail.com>
Date:   Tue Jan 28 21:49:30 2014 -0800

    Fixed a lot of broken links.

commit 34a5a6008dac2e107624c7ff0db0824ee5bae45f
Author: Tathagata Das <tathagata.das1565@gmail.com>
Date:   Tue Jan 28 18:02:28 2014 -0800

    Updated github url to use SPARK_GITHUB_URL variable.

commit f338a60ae8069e0a382d2cb170227e5757cc0b7a
Author: Tathagata Das <tathagata.das1565@gmail.com>
Date:   Mon Jan 27 22:42:42 2014 -0800

    More updates based on Patrick and Harvey's comments.

commit 89a81ff25726bf6d26163e0dd938290a79582c0f
Author: Tathagata Das <tathagata.das1565@gmail.com>
Date:   Mon Jan 27 13:08:34 2014 -0800

    Updated docs based on Patricks PR comments.

commit d5b6196b532b5746e019b959a79ea0cc013a8fc3
Author: Tathagata Das <tathagata.das1565@gmail.com>
Date:   Sun Jan 26 20:15:58 2014 -0800

    Added spark.streaming.unpersist config and info on StreamingListener interface.

commit e3dcb46ab83d7071f611d9b5008ba6bc16c9f951
Author: Tathagata Das <tathagata.das1565@gmail.com>
Date:   Sun Jan 26 18:41:12 2014 -0800

    Fixed docs on StreamingContext.getOrCreate.

commit 6c29524639463f11eec721e4d17a9d7159f2944b
Author: Tathagata Das <tathagata.das1565@gmail.com>
Date:   Thu Jan 23 18:49:39 2014 -0800

    Added example and figure for window operations, and links to Kafka and Flume API docs.

commit f06b964a51bb3b21cde2ff8bdea7d9785f6ce3a9
Author: Tathagata Das <tathagata.das1565@gmail.com>
Date:   Wed Jan 22 22:49:12 2014 -0800

    Fixed missing endhighlight tag in the MLlib guide.

commit 036a7d46187ea3f2a0fb8349ef78f10d6c0b43a9
Merge: eab351d a1cd185
Author: Tathagata Das <tathagata.das1565@gmail.com>
Date:   Wed Jan 22 22:17:42 2014 -0800

    Merge remote-tracking branch 'apache/master' into docs-update

commit eab351d05c0baef1d4b549e1581310087158d78d
Author: Tathagata Das <tathagata.das1565@gmail.com>
Date:   Wed Jan 22 22:17:15 2014 -0800

    Update Spark Streaming Programming Guide.
(cherry picked from commit 793020961489e16e924c4531da3a13884d2b9175)

Conflicts:

	docs/mllib-guide.md
Merge pull request #527 from ankurdave/graphx-assembly-pom

Add GraphX to assembly/pom.xml

Author: Ankur Dave <ankurdave@gmail.com>

== Merge branch commits ==

commit bb0b33ef9eb1b3d4a4fc283d9abb2ece4abcac23
Author: Ankur Dave <ankurdave@gmail.com>
Date:   Fri Jan 31 15:24:52 2014 -0800

    Add GraphX to assembly/pom.xml
(cherry picked from commit a8cf3ec157fc9a512421b319cfffc5e4f07cf1f3)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #524 from rxin/doc

Added spark.shuffle.file.buffer.kb to configuration doc.

Author: Reynold Xin <rxin@apache.org>

== Merge branch commits ==

commit 0eea1d761ff772ff89be234e1e28035d54e5a7de
Author: Reynold Xin <rxin@apache.org>
Date:   Wed Jan 29 14:40:48 2014 -0800

    Added spark.shuffle.file.buffer.kb to configuration doc.
(cherry picked from commit ac712e48af3068672e629cec7766caae3cd77c37)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #490 from hsaputra/modify_checkoption_with_isdefined

Replace the check for None Option with isDefined and isEmpty in Scala code

Propose to replace the Scala check for Option "!= None" with Option.isDefined and "=== None" with Option.isEmpty.

I think this, using method call if possible then operator function plus argument, will make the Scala code easier to read and understand.

Pass compile and tests.
(cherry picked from commit f16c21e22f706b268419fefce44e9905db3ee485)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #516 from sarutak/master

modified SparkPluginBuild.scala to use https protocol for accessing gith...

We cannot build Spark behind a proxy although we execute sbt with -Dhttp(s).proxyHost -Dhttp(s).proxyPort -Dhttp(s).proxyUser -Dhttp(s).proxyPassword options.
It's because of using git protocol to clone junit_xml_listener.git.
I could build after modifying SparkPluginBuild.scala.

I reported this issue to JIRA.
https://spark-project.atlassian.net/browse/SPARK-1046
(cherry picked from commit 3d5c03e2305777b8a32f2e196e3b73ab221b3e79)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #504 from JoshRosen/SPARK-1025

Fix PySpark hang when input files are deleted (SPARK-1025)

This pull request addresses [SPARK-1025](https://spark-project.atlassian.net/browse/SPARK-1025), an issue where PySpark could hang if its input files were deleted.
(cherry picked from commit c40619d4873f36ffb96a2e6292b32d5b64eab153)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #511 from JoshRosen/SPARK-1040

Fix ClassCastException in JavaPairRDD.collectAsMap() (SPARK-1040)

This fixes [SPARK-1040](https://spark-project.atlassian.net/browse/SPARK-1040), an issue where JavaPairRDD.collectAsMap() could sometimes fail with ClassCastException.  I applied the same fix to the Spark Streaming Java APIs.  The commit message describes the fix in more detail.

I also increased the verbosity of JUnit test output under SBT to make it easier to verify that the Java tests are actually running.
(cherry picked from commit c66a2ef1c2dc9c218069b3ce8c39a49e5b92fc16)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #493 from kayousterhout/double_add

Fixed bug where task set managers are added to queue twice

@mateiz can you verify that this is a bug and wasn't intentional? (https://github.com/apache/incubator-spark/commit/90a04dab8d9a2a9a372cea7cdf46cc0fd0f2f76c#diff-7fa4f84a961750c374f2120ca70e96edR551)

This bug leads to a small performance hit because task
set managers will get offered each rejected resource
offer twice, but doesn't lead to any incorrect functionality.

Thanks to @hdc1112 for pointing this out.
(cherry picked from commit 5bcfd798117e8617e604c1dd3b5c9b67e83100bb)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #489 from ash211/patch-6

Clarify spark.default.parallelism

It's the task count across the cluster, not per worker, per machine, per core, or anything else.
(cherry picked from commit 749f842827c7e7766a342b6b0a437803044a9f90)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #445 from kayousterhout/exec_lost

Fail rather than hanging if a task crashes the JVM.

Prior to this commit, if a task crashes the JVM, the task (and
all other tasks running on that executor) is marked at KILLED rather
than FAILED.  As a result, the TaskSetManager will retry the task
indefinitely rather than failing the job after maxFailures. Eventually,
this makes the job hang, because the Standalone Scheduler removes
the application after 10 works have failed, and then the app is left
in a state where it's disconnected from the master and waiting to reconnect.
This commit fixes that problem by marking tasks as FAILED rather than
killed when an executor is lost.

The downside of this commit is that if task A fails because another
task running on the same executor caused the VM to crash, the failure
will incorrectly be counted as a failure of task A. This should not
be an issue because we typically set maxFailures to 3, and it is
unlikely that a task will be co-located with a JVM-crashing task
multiple times.
(cherry picked from commit c06a307ca22901839df00d25fe623f6faa6af17e)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>