Merge pull request #85 from rxin/clean

Moved the top level spark package object from spark to org.apache.spark

This is a pretty annoying documentation bug ...

(cherry picked from commit e4abb75d70aa08377829f635fe6135d94e28f434)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #84 from rxin/kill1

Added documentation for setJobGroup. Also some minor cleanup in SparkContext.

(cherry picked from commit 2a7ae1736a9ea4c5dded91ebc34f7a33e93835e6)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #89 from rxin/executor

Don't setup the uncaught exception handler in local mode.

This avoids unit test failures for Spark streaming.

    java.util.concurrent.RejectedExecutionException: Task org.apache.spark.streaming.JobManager$JobHandler@38cf728d rejected from java.util.concurrent.ThreadPoolExecutor@3b69a41e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 14]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
	at org.apache.spark.streaming.JobManager.runJob(JobManager.scala:54)
	at org.apache.spark.streaming.Scheduler$$anonfun$generateJobs$2.apply(Scheduler.scala:108)
	at org.apache.spark.streaming.Scheduler$$anonfun$generateJobs$2.apply(Scheduler.scala:108)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at org.apache.spark.streaming.Scheduler.generateJobs(Scheduler.scala:108)
	at org.apache.spark.streaming.Scheduler$$anonfun$1.apply$mcVJ$sp(Scheduler.scala:41)
	at org.apache.spark.streaming.util.RecurringTimer.org$apache$spark$streaming$util$RecurringTimer$$loop(RecurringTimer.scala:66)
	at org.apache.spark.streaming.util.RecurringTimer$$anon$1.run(RecurringTimer.scala:34)

(cherry picked from commit 5b9380e0173b3d3d13235ae912e9ccc2a974b98b)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #41 from pwendell/shuffle-benchmark

Provide Instrumentation for Shuffle Write Performance

Shuffle write performance can have a major impact on the performance of jobs. This patch adds a few pieces of instrumentation related to shuffle writes. They are:

1. A listing of the time spent performing blocking writes for each task. This is implemented by keeping track of the aggregate delay seen by many individual writes.
2. An undocumented option `spark.shuffle.sync` which forces shuffle data to sync to disk. This is necessary for measuring shuffle performance in the absence of the OS buffer cache.
3. An internal utility which micro-benchmarks write throughput for simulated shuffle outputs.

I'm going to do some performance testing on this to see whether these small timing calls add overhead. From a feature perspective, however, I consider this complete. Any feedback is appreciated.
(cherry picked from commit 35886f347466b25625d5391c97c2deb8293ebc66)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #88 from rxin/clean

Made the following traits/interfaces/classes non-public:

Made the following traits/interfaces/classes non-public:
SparkHadoopWriter
SparkHadoopMapRedUtil
SparkHadoopMapReduceUtil
SparkHadoopUtil
PythonAccumulatorParam
BlockManagerSlaveActor
(cherry picked from commit aa61bfd39962a3e3c8ba8e4bcf25f48212933cce)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #94 from aarondav/mesos-fix

Fix mesos urls

This was a bug I introduced in https://github.com/apache/incubator-spark/pull/71.
Previously, we explicitly removed the mesos:// part; with #71, this no longer occurs.

(cherry picked from commit 39d2e9b293768b6eeb291313ee94f3e02a0ad522)
Signed-off-by: Reynold Xin <rxin@apache.org>
Revert "Merge pull request #94 from aarondav/mesos-fix"

This reverts commit fe974ba1484dbe6fffdae8680b3b37b26de83145.

The commit only applies to 0.9, not 0.8.x.
Merge pull request #87 from aarondav/shuffle-base

Basic shuffle file consolidation

The Spark shuffle phase can produce a large number of files, as one file is created
per mapper per reducer. For large or repeated jobs, this often produces millions of
shuffle files, which sees extremely degredaded performance from the OS file system.
This patch seeks to reduce that burden by combining multipe shuffle files into one.

This PR draws upon the work of @jason-dai in https://github.com/mesos/spark/pull/669.
However, it simplifies the design in order to get the majority of the gain with less
overall intellectual and code burden. The vast majority of code in this pull request
is a refactor to allow the insertion of a clean layer of indirection between logical
block ids and physical files. This, I feel, provides some design clarity in addition
to enabling shuffle file consolidation.

The main goal is to produce one shuffle file per reducer per active mapper thread.
This allows us to isolate the mappers (simplifying the failure modes), while still
allowing us to reduce the number of mappers tremendously for large tasks. In order
to accomplish this, we simply create a new set of shuffle files for every parallel
task, and return the files to a pool which will be given out to the next run task.

I have run some ad hoc query testing on 5 m1.xlarge EC2 nodes with 2g of executor memory and the following microbenchmark:

    scala> val nums = sc.parallelize(1 to 1000, 1000).flatMap(x => (1 to 1e6.toInt))
    scala> def time(x: => Unit) = { val now = System.currentTimeMillis; x; System.currentTimeMillis - now }
    scala> (1 to 8).map(_ => time(nums.map(x => (x % 100000, 2000, x)).reduceByKey(_ + _).count) / 1000.0)

For this particular workload, with 1000 mappers and 2000 reducers, I saw the old method running at around 15 minutes, with the consolidated shuffle files running at around 4 minutes. There was a very sharp increase in running time for the non-consolidated version after around 1 million total shuffle files. Below this threshold, however, there wasn't a significant difference between the two.

Better performance measurement of this patch is warranted, and I plan on doing so in the near future as part of a general investigation of our shuffle file bottlenecks and performance.

(cherry picked from commit 48952d67e6dde25faaba241b9deba737b83a5372)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #92 from tgravescs/sparkYarnFixClasspath

Fix the Worker to use CoarseGrainedExecutorBackend and modify classpath ...

...to be explicit about inclusion of spark.jar and app.jar.  Be explicit so if there are any conflicts in packaging between spark.jar and app.jar we don't get random results due to the classpath having /*, which can including things in different order.

(cherry picked from commit b84193c5b86d60f36624ae29169cf62e489005af)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #90 from pwendell/master

SPARK-940: Do not directly pass Stage objects to SparkListener.

This patch updates the SparkListener interface to pass StageInfo objects rather than directly pass spark Stages. The reason for this patch is explained in detail in SPARK-940.
(cherry picked from commit c404adb9d2238644415fd93270871f5a8a9ce455)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #100 from JoshRosen/spark-902

Remove redundant Java Function call() definitions

This should fix [SPARK-902](https://spark-project.atlassian.net/browse/SPARK-902), an issue where some Java API Function classes could cause AbstractMethodErrors when user code is compiled using the Eclipse compiler.

Thanks to @MartinWeindel for diagnosing this problem.

(This PR subsumes #30).

(cherry picked from commit 9dfcf53a08645227b8722f07068466c7b7909721)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge branch 'branch-0.8' of https://git-wip-us.apache.org/repos/asf/incubator-spark into branch-0.8
Merge pull request #103 from JoshRosen/unpersist-fix

Add unpersist() to JavaDoubleRDD and JavaPairRDD.

This fixes a minor inconsistency where [unpersist() was only available on JavaRDD](https://mail-archives.apache.org/mod_mbox/incubator-spark-user/201310.mbox/%3CCE8D8748.68C0%25YannLuppo%40livenation.com%3E) and not JavaPairRDD / JavaDoubleRDD.   I also added support for the new optional `blocking` argument added in 0.8.

Please merge this into branch-0.8, too.

(cherry picked from commit a098438c4824891fb44e9119e038e6e796c69741)
Signed-off-by: Reynold Xin <rxin@apache.org>
Fixing broken links in programming guide

(cherry picked from commit 4e093b88f8f07885ca3411b4c332da94401cc29f)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #106 from pwendell/master

Add a `repartition` operator.

This patch adds an operator called repartition with more straightforward
semantics than the current `coalesce` operator. There are a few use cases
where this operator is useful:

1. If a user wants to increase the number of partitions in the RDD. This
is more common now with streaming. E.g. a user is ingesting data on one
node but they want to add more partitions to ensure parallelism of
subsequent operations across threads or the cluster.

Right now they have to call rdd.coalesce(numSplits, shuffle=true) - that's
super confusing.

2. If a user has input data where the number of partitions is not known. E.g.

> sc.textFile("some file").coalesce(50)....

This is both vague semantically (am I growing or shrinking this RDD) but also,
may not work correctly if the base RDD has fewer than 50 partitions.

The new operator forces shuffles every time, so it will always produce exactly
the number of new partitions. It also throws an exception rather than silently
not-working if a bad input is passed.

I am currently adding streaming tests (requires refactoring some of the test
suite to allow testing at partition granularity), so this is not ready for
merge yet. But feedback is welcome.

(cherry picked from commit 99ad4a613a859d1ea246829b3681b3f30fa92527)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #83 from ewencp/pyspark-accumulator-add-method

Add an add() method to pyspark accumulators.

Add a regular method for adding a term to accumulators in
pyspark. Currently if you have a non-global accumulator, adding to it
is awkward. The += operator can't be used for non-global accumulators
captured via closure because it's involves an assignment. The only way
to do it is using __iadd__ directly.

Adding this method lets you write code like this:

def main():
    sc = SparkContext()
    accum = sc.accumulator(0)

    rdd = sc.parallelize([1,2,3])
    def f(x):
        accum.add(x)
    rdd.foreach(f)
    print accum.value

where using accum += x instead would have caused UnboundLocalError
exceptions in workers. Currently it would have to be written as
accum.__iadd__(x).

(cherry picked from commit 747f53892546eb8edf1c75788c9ecf9c939d375c)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #93 from kayousterhout/ui_new_state

Show "GETTING_RESULTS" state in UI.

This commit adds a set of calls using the SparkListener interface
that indicate when a task is remotely fetching results, so that
we can display this (potentially time-consuming) phase of execution
to users through the UI.

(cherry picked from commit 1dc776b863663af713920d18cecaf57762c2fd77)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #97 from ewencp/pyspark-system-properties

Add classmethod to SparkContext to set system properties.

Add a new classmethod to SparkContext to set system properties like is
possible in Scala/Java. Unlike the Java/Scala implementations, there's
no access to System until the JVM bridge is created. Since
SparkContext handles that, move the initialization of the JVM
connection to a separate classmethod that can safely be called
repeatedly as long as the same instance (or no instance) is provided.

(cherry picked from commit 452aa36d671d3fdf53058e80cbd86787c8870cd7)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #109 from pwendell/master

Adding Java/Java Streaming versions of `repartition` with associated tests

(cherry picked from commit 4f2c9438b43cfb63f118b6b000da317d518a8577)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #114 from soulmachine/master

A little revise for the document

(cherry picked from commit 662ee9f3216e670e9582861fda92648d5abc51e1)
Signed-off-by: Reynold Xin <rxin@apache.org>