mesos/spark — pull request #509 by rold50 (2013-03-04 22:10:26+00:00)
Title: Fix SPARK-663: Implement Fair scheduler within ClusterScheduler

The SparkContext now has a DynamicVariable Properties object that threads can set using addLocalProperties(). initLocalProperties() is used when you are sharing threads (e.g., processing requests from a thread pool).
SparkContext also can be configured to choose the type of scheduler. It uses the spark.cluster.taskscheduler system property. By default it will use the ClusterScheduler. To use the fair scheduler we can set it to spark.scheduler.cluster.fair.FairClusterScheduler.

Changes to DagScheduler, DagSchedulerEvent, TaskSet, Stage are for propagating meta information down to the TaskScheduler.

FairClusterScheduler and FairTaskSetManager contain the implementation of the fair scheduler. It is similar to the Hadoop fair scheduler but simplified. Within a pool, we just use FIFO. Pools are statically defined through an allocation XML, which we can set the path to through the system property: spark.fairscheduler.allocation.file. If that is not specified, it will just create 1 pool named default. The current implementation will always create a default pool. Also, the current implementation only supports "weight". Currently, it does not support minShares, etc.

The following is a sample allocation XML:
`

25

25

25

25

`

To assign a job to a specific pool, in the spark program, the user can set the LocalProperty:
sc.addLocalProperties("spark.scheduler.cluster.fair.pool","some pool name")
If the pool name does not exist (in the Scheduler) or the user did not specify any pool for their job, the scheduler will just assign the job to the default pool.