Merge pull request #34 from kayousterhout/rename

Renamed StandaloneX to CoarseGrainedX.

(as suggested by @rxin here https://github.com/apache/incubator-spark/pull/14)

The previous names were confusing because the components weren't just
used in Standalone mode.  The scheduler used for Standalone
mode is called SparkDeploySchedulerBackend, so referring to the base class
as StandaloneSchedulerBackend was misleading.

(cherry picked from commit 6dbd2208ffe1df0b2a871f7f18b08bd825f61001)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #65 from tgravescs/fixYarn

Fix yarn build

Fix the yarn build after renaming StandAloneX to CoarseGrainedX from pull request 34.

(cherry picked from commit f9973cae3aff39c29a2cdad5b54b7674d1126132)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #59 from rxin/warning

Bump up logging level to warning for failed tasks.

(cherry picked from commit 3249e0e90dd9a7b422f561c42407b6a2b3feab17)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #61 from kayousterhout/daemon_thread

Unified daemon thread pools

As requested by @mateiz in an earlier pull request, this refactors various daemon thread pools to use a set of methods in utils.scala, and also changes the thread-pool-creation methods in utils.scala to use named thread pools for improved debugging.

(cherry picked from commit 983b83f24da9f7f0251833b409cdcd37acae3d39)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #62 from harveyfeng/master

Make TaskContext's stageId publicly accessible.

(cherry picked from commit 4e46fde818a9b94960ab83fa6390952e2d5dd3e6)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #67 from kayousterhout/remove_tsl

Removed TaskSchedulerListener interface.

The interface was used only by the DAG scheduler (so it wasn't necessary
to define the additional interface), and the naming makes it very
confusing when reading the code (because "listener" was used
to describe the DAG scheduler, rather than SparkListeners, which
implement a nearly-identical interface but serve a different
function).

@mateiz - is there a reason for this interface that I'm missing?

(cherry picked from commit cf64f63f8a3b54dec37e991856260ac63f7e222e)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #69 from KarthikTunga/master

Fix for issue SPARK-627. Implementing --config argument in the scripts.

This code fix is for issue SPARK-627. I added code to consider --config arguments in the scripts. In case the  <conf-dir> is not a directory the scripts exit. I removed the --hosts argument. It can be achieved by giving a different config directory. Let me know if an explicit --hosts argument is required.

(cherry picked from commit fc26e5b8320556b9edb93741391b759813b4079b)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #74 from rxin/kill

Job cancellation via job group id.

This PR adds a simple API to group together a set of jobs belonging to a thread and threads spawned from it. It also allows the cancellation of all jobs in this group.

An example:

    sc.setJobDescription("this_is_the_group_id", "some job description")
    sc.parallelize(1 to 10000, 2).map { i => Thread.sleep(10); i }.count()

In a separate thread:

    sc.cancelJobGroup("this_is_the_group_id")

(cherry picked from commit 599dcb0ddf740e028cc8faac163303be8f9400a6)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #68 from mosharaf/master

Faster and stable/reliable broadcast

HttpBroadcast is noticeably slow, but the alternatives (TreeBroadcast or BitTorrentBroadcast) are notoriously unreliable. The main problem with them is they try to manage the memory for the pieces of a broadcast themselves. Right now, the BroadcastManager does not know which machines the tasks reading from a broadcast variable is running and when they have finished. Consequently, we try to guess and often guess wrong, which blows up the memory usage and kills/hangs jobs.

This very simple implementation solves the problem by not trying to manage the intermediate pieces; instead, it offloads that duty to the BlockManager which is quite good at juggling blocks. Otherwise, it is very similar to the BitTorrentBroadcast implementation (without fancy optimizations). And it runs much faster than HttpBroadcast we have right now.

I've been using this for another project for last couple of weeks, and just today did some benchmarking against the Http one. The following shows the improvements for increasing broadcast size for cold runs. Each line represent the number of receivers.
![fix-bc-first](https://f.cloud.github.com/assets/232966/1349342/ffa149e4-36e7-11e3-9fa6-c74555829356.png)

After the first broadcast is over, i.e., after JVM is wormed up and for HttpBroadcast the server is already running (I think), the following are the improvements for warm runs.
![fix-bc-succ](https://f.cloud.github.com/assets/232966/1349352/5a948bae-36e8-11e3-98ce-34f19ebd33e0.jpg)
The curves are not as nice as the cold runs, but the improvements are obvious, specially for larger broadcasts and more receivers.

Depending on how it goes, we should deprecate and/or remove old TreeBroadcast and BitTorrentBroadcast implementations, and hopefully, SPARK-889 will not be necessary any more.

(cherry picked from commit e5316d0685c41a40e54a064cf271f3d62df6c8e8)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #76 from pwendell/master

Clarify compression property.

Clarifies that this governs compression of internal data, not input
data or output data.

(cherry picked from commit f628804c023a44bf7506bdca017d9a895dd7eb32)
Signed-off-by: Reynold Xin <rxin@apache.org>
Added deprecation warning to BitTorrentBroadcast and TreeBroadcast implementations. TorrentBroadcast is replacing them.
Add deprecation warning to unrecognized MASTER urls

Unrecognized MASTER urls will throw an exception rather than be parsed as Mesos URLs
in 0.9. See https://github.com/apache/incubator-spark/pull/71
Merge pull request #79 from aarondav/scdefaults0.8

Add deprecation warning to unrecognized MASTER urls

Unrecognized MASTER urls will throw an exception rather than be parsed as Mesos URLs
in 0.9. See 74737264c4a9b2.
Merge branch 'branch-0.8' of github.com:mosharaf/incubator-spark into branch-0.8

Added deprecation warning to BitTorrentBroadcast and TreeBroadcast

Users should gradually move toward TorrentBroadcast.
Merge pull request #85 from rxin/clean

Moved the top level spark package object from spark to org.apache.spark

This is a pretty annoying documentation bug ...

(cherry picked from commit e4abb75d70aa08377829f635fe6135d94e28f434)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #84 from rxin/kill1

Added documentation for setJobGroup. Also some minor cleanup in SparkContext.

(cherry picked from commit 2a7ae1736a9ea4c5dded91ebc34f7a33e93835e6)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #89 from rxin/executor

Don't setup the uncaught exception handler in local mode.

This avoids unit test failures for Spark streaming.

    java.util.concurrent.RejectedExecutionException: Task org.apache.spark.streaming.JobManager$JobHandler@38cf728d rejected from java.util.concurrent.ThreadPoolExecutor@3b69a41e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 14]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
	at org.apache.spark.streaming.JobManager.runJob(JobManager.scala:54)
	at org.apache.spark.streaming.Scheduler$$anonfun$generateJobs$2.apply(Scheduler.scala:108)
	at org.apache.spark.streaming.Scheduler$$anonfun$generateJobs$2.apply(Scheduler.scala:108)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at org.apache.spark.streaming.Scheduler.generateJobs(Scheduler.scala:108)
	at org.apache.spark.streaming.Scheduler$$anonfun$1.apply$mcVJ$sp(Scheduler.scala:41)
	at org.apache.spark.streaming.util.RecurringTimer.org$apache$spark$streaming$util$RecurringTimer$$loop(RecurringTimer.scala:66)
	at org.apache.spark.streaming.util.RecurringTimer$$anon$1.run(RecurringTimer.scala:34)

(cherry picked from commit 5b9380e0173b3d3d13235ae912e9ccc2a974b98b)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #41 from pwendell/shuffle-benchmark

Provide Instrumentation for Shuffle Write Performance

Shuffle write performance can have a major impact on the performance of jobs. This patch adds a few pieces of instrumentation related to shuffle writes. They are:

1. A listing of the time spent performing blocking writes for each task. This is implemented by keeping track of the aggregate delay seen by many individual writes.
2. An undocumented option `spark.shuffle.sync` which forces shuffle data to sync to disk. This is necessary for measuring shuffle performance in the absence of the OS buffer cache.
3. An internal utility which micro-benchmarks write throughput for simulated shuffle outputs.

I'm going to do some performance testing on this to see whether these small timing calls add overhead. From a feature perspective, however, I consider this complete. Any feedback is appreciated.
(cherry picked from commit 35886f347466b25625d5391c97c2deb8293ebc66)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #88 from rxin/clean

Made the following traits/interfaces/classes non-public:

Made the following traits/interfaces/classes non-public:
SparkHadoopWriter
SparkHadoopMapRedUtil
SparkHadoopMapReduceUtil
SparkHadoopUtil
PythonAccumulatorParam
BlockManagerSlaveActor
(cherry picked from commit aa61bfd39962a3e3c8ba8e4bcf25f48212933cce)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #94 from aarondav/mesos-fix

Fix mesos urls

This was a bug I introduced in https://github.com/apache/incubator-spark/pull/71.
Previously, we explicitly removed the mesos:// part; with #71, this no longer occurs.

(cherry picked from commit 39d2e9b293768b6eeb291313ee94f3e02a0ad522)
Signed-off-by: Reynold Xin <rxin@apache.org>