Merge pull request #90 from pwendell/master

SPARK-940: Do not directly pass Stage objects to SparkListener.

This patch updates the SparkListener interface to pass StageInfo objects rather than directly pass spark Stages. The reason for this patch is explained in detail in SPARK-940.
(cherry picked from commit c404adb9d2238644415fd93270871f5a8a9ce455)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #100 from JoshRosen/spark-902

Remove redundant Java Function call() definitions

This should fix [SPARK-902](https://spark-project.atlassian.net/browse/SPARK-902), an issue where some Java API Function classes could cause AbstractMethodErrors when user code is compiled using the Eclipse compiler.

Thanks to @MartinWeindel for diagnosing this problem.

(This PR subsumes #30).

(cherry picked from commit 9dfcf53a08645227b8722f07068466c7b7909721)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge branch 'branch-0.8' of https://git-wip-us.apache.org/repos/asf/incubator-spark into branch-0.8
Merge pull request #103 from JoshRosen/unpersist-fix

Add unpersist() to JavaDoubleRDD and JavaPairRDD.

This fixes a minor inconsistency where [unpersist() was only available on JavaRDD](https://mail-archives.apache.org/mod_mbox/incubator-spark-user/201310.mbox/%3CCE8D8748.68C0%25YannLuppo%40livenation.com%3E) and not JavaPairRDD / JavaDoubleRDD.   I also added support for the new optional `blocking` argument added in 0.8.

Please merge this into branch-0.8, too.

(cherry picked from commit a098438c4824891fb44e9119e038e6e796c69741)
Signed-off-by: Reynold Xin <rxin@apache.org>
Fixing broken links in programming guide

(cherry picked from commit 4e093b88f8f07885ca3411b4c332da94401cc29f)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #106 from pwendell/master

Add a `repartition` operator.

This patch adds an operator called repartition with more straightforward
semantics than the current `coalesce` operator. There are a few use cases
where this operator is useful:

1. If a user wants to increase the number of partitions in the RDD. This
is more common now with streaming. E.g. a user is ingesting data on one
node but they want to add more partitions to ensure parallelism of
subsequent operations across threads or the cluster.

Right now they have to call rdd.coalesce(numSplits, shuffle=true) - that's
super confusing.

2. If a user has input data where the number of partitions is not known. E.g.

> sc.textFile("some file").coalesce(50)....

This is both vague semantically (am I growing or shrinking this RDD) but also,
may not work correctly if the base RDD has fewer than 50 partitions.

The new operator forces shuffles every time, so it will always produce exactly
the number of new partitions. It also throws an exception rather than silently
not-working if a bad input is passed.

I am currently adding streaming tests (requires refactoring some of the test
suite to allow testing at partition granularity), so this is not ready for
merge yet. But feedback is welcome.

(cherry picked from commit 99ad4a613a859d1ea246829b3681b3f30fa92527)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #83 from ewencp/pyspark-accumulator-add-method

Add an add() method to pyspark accumulators.

Add a regular method for adding a term to accumulators in
pyspark. Currently if you have a non-global accumulator, adding to it
is awkward. The += operator can't be used for non-global accumulators
captured via closure because it's involves an assignment. The only way
to do it is using __iadd__ directly.

Adding this method lets you write code like this:

def main():
    sc = SparkContext()
    accum = sc.accumulator(0)

    rdd = sc.parallelize([1,2,3])
    def f(x):
        accum.add(x)
    rdd.foreach(f)
    print accum.value

where using accum += x instead would have caused UnboundLocalError
exceptions in workers. Currently it would have to be written as
accum.__iadd__(x).

(cherry picked from commit 747f53892546eb8edf1c75788c9ecf9c939d375c)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #93 from kayousterhout/ui_new_state

Show "GETTING_RESULTS" state in UI.

This commit adds a set of calls using the SparkListener interface
that indicate when a task is remotely fetching results, so that
we can display this (potentially time-consuming) phase of execution
to users through the UI.

(cherry picked from commit 1dc776b863663af713920d18cecaf57762c2fd77)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #97 from ewencp/pyspark-system-properties

Add classmethod to SparkContext to set system properties.

Add a new classmethod to SparkContext to set system properties like is
possible in Scala/Java. Unlike the Java/Scala implementations, there's
no access to System until the JVM bridge is created. Since
SparkContext handles that, move the initialization of the JVM
connection to a separate classmethod that can safely be called
repeatedly as long as the same instance (or no instance) is provided.

(cherry picked from commit 452aa36d671d3fdf53058e80cbd86787c8870cd7)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #109 from pwendell/master

Adding Java/Java Streaming versions of `repartition` with associated tests

(cherry picked from commit 4f2c9438b43cfb63f118b6b000da317d518a8577)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #114 from soulmachine/master

A little revise for the document

(cherry picked from commit 662ee9f3216e670e9582861fda92648d5abc51e1)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #111 from kayousterhout/ui_name

Properly display the name of a stage in the UI.

This fixes a bug introduced by the fix for SPARK-940, which
changed the UI to display the RDD name rather than the stage
name. As a result, no name for the stage was shown when
using the Spark shell, which meant that there was no way to
click on the stage to see more details (e.g., the running
tasks). This commit changes the UI back to using the
stage name.

@pwendell -- let me know if this change was intentional

(cherry picked from commit 85e2cab6f68cbcb66bb348298463e2e86591d92d)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #113 from pwendell/master

Improve error message when multiple assembly jars are present.

This can happen easily if building different hadoop versions. Right now it gives a class not found exception.
(cherry picked from commit e018f2d0ae826b682dda30c1ecfb0c51ad7edf58)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #108 from alig/master

Changes to enable executing by using HDFS as a synchronization point between driver and executors, as well as ensuring executors exit properly.

(cherry picked from commit bab496c1201305f8ee4b0cfdd9b9e8041ea08d38)
Signed-off-by: Reynold Xin <rxin@apache.org>

Conflicts:
	core/src/main/scala/org/apache/spark/SparkContext.scala
Merge branch 'branch-0.8' of https://git-wip-us.apache.org/repos/asf/incubator-spark into branch-0.8
Merge pull request #115 from aarondav/shuffle-fix

Eliminate extra memory usage when shuffle file consolidation is disabled

Otherwise, we see SPARK-946 even when shuffle file consolidation is disabled.
Fixing SPARK-946 is still forthcoming.

(cherry picked from commit d4df4749a8738628aa79802c4fb2805d02f47bb8)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #119 from soulmachine/master

A little revise for the document

(cherry picked from commit f0e23a023ce1356bc0f04248605c48d4d08c2d05)
Signed-off-by: Reynold Xin <rxin@apache.org>
add javadoc to JobLogger, and some small format fix
Merge pull request #132 from Mistobaan/doc_fix

fix persistent-hdfs

(cherry picked from commit e7c7b804b529342d6de1c86ee02bc78da81279bc)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #134 from rxin/readme

Fixed a typo in Hadoop version in README.

(cherry picked from commit d407c0732aa7855a3b150975931425fe9ba30519)
Signed-off-by: Reynold Xin <rxin@apache.org>