Merge pull request #103 from JoshRosen/unpersist-fix

Add unpersist() to JavaDoubleRDD and JavaPairRDD.

This fixes a minor inconsistency where [unpersist() was only available on JavaRDD](https://mail-archives.apache.org/mod_mbox/incubator-spark-user/201310.mbox/%3CCE8D8748.68C0%25YannLuppo%40livenation.com%3E) and not JavaPairRDD / JavaDoubleRDD.   I also added support for the new optional `blocking` argument added in 0.8.

Please merge this into branch-0.8, too.

(cherry picked from commit a098438c4824891fb44e9119e038e6e796c69741)
Signed-off-by: Reynold Xin <rxin@apache.org>
Fixing broken links in programming guide

(cherry picked from commit 4e093b88f8f07885ca3411b4c332da94401cc29f)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #106 from pwendell/master

Add a `repartition` operator.

This patch adds an operator called repartition with more straightforward
semantics than the current `coalesce` operator. There are a few use cases
where this operator is useful:

1. If a user wants to increase the number of partitions in the RDD. This
is more common now with streaming. E.g. a user is ingesting data on one
node but they want to add more partitions to ensure parallelism of
subsequent operations across threads or the cluster.

Right now they have to call rdd.coalesce(numSplits, shuffle=true) - that's
super confusing.

2. If a user has input data where the number of partitions is not known. E.g.

> sc.textFile("some file").coalesce(50)....

This is both vague semantically (am I growing or shrinking this RDD) but also,
may not work correctly if the base RDD has fewer than 50 partitions.

The new operator forces shuffles every time, so it will always produce exactly
the number of new partitions. It also throws an exception rather than silently
not-working if a bad input is passed.

I am currently adding streaming tests (requires refactoring some of the test
suite to allow testing at partition granularity), so this is not ready for
merge yet. But feedback is welcome.

(cherry picked from commit 99ad4a613a859d1ea246829b3681b3f30fa92527)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #83 from ewencp/pyspark-accumulator-add-method

Add an add() method to pyspark accumulators.

Add a regular method for adding a term to accumulators in
pyspark. Currently if you have a non-global accumulator, adding to it
is awkward. The += operator can't be used for non-global accumulators
captured via closure because it's involves an assignment. The only way
to do it is using __iadd__ directly.

Adding this method lets you write code like this:

def main():
    sc = SparkContext()
    accum = sc.accumulator(0)

    rdd = sc.parallelize([1,2,3])
    def f(x):
        accum.add(x)
    rdd.foreach(f)
    print accum.value

where using accum += x instead would have caused UnboundLocalError
exceptions in workers. Currently it would have to be written as
accum.__iadd__(x).

(cherry picked from commit 747f53892546eb8edf1c75788c9ecf9c939d375c)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #93 from kayousterhout/ui_new_state

Show "GETTING_RESULTS" state in UI.

This commit adds a set of calls using the SparkListener interface
that indicate when a task is remotely fetching results, so that
we can display this (potentially time-consuming) phase of execution
to users through the UI.

(cherry picked from commit 1dc776b863663af713920d18cecaf57762c2fd77)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #97 from ewencp/pyspark-system-properties

Add classmethod to SparkContext to set system properties.

Add a new classmethod to SparkContext to set system properties like is
possible in Scala/Java. Unlike the Java/Scala implementations, there's
no access to System until the JVM bridge is created. Since
SparkContext handles that, move the initialization of the JVM
connection to a separate classmethod that can safely be called
repeatedly as long as the same instance (or no instance) is provided.

(cherry picked from commit 452aa36d671d3fdf53058e80cbd86787c8870cd7)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #109 from pwendell/master

Adding Java/Java Streaming versions of `repartition` with associated tests

(cherry picked from commit 4f2c9438b43cfb63f118b6b000da317d518a8577)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge branch 'branch-0.8' of https://github.com/apache/incubator-spark into master-csd