mesos/spark — pull request #529 by xiajunluan (2013-03-15 13:59:23+00:00)
Title: [SPARK-663]Implement Fair Scheduler in Spark Cluster Scheduler 

1. What features I have implemented?
Based on Harold's code, I mainly finished below functions
a) support FIFO (between TaskSetManagers)or fair scheduler(between pools) in cluster scheduler,default is FIFO
b) support FIFO or fair scheduler between TaskSetManagers within pool, default is FIFO
c) support priority and stageId comparator for FIFO
d) support miniShares, weight , numRunningTask comparator for fair scheduler
e) support to set which pool job will belong to by user when using Fair Scheduler Algorithm, default pool name is "default" if user does not set
f) support XML configuration for pool creation

2. How will user write spark application using fair scheduler
spark application will looks like:
//===============================
/* Set FIFO or FS class (spark.scheduler.cluster.FairTaskSetQueuesManager or spark.scheduler.cluster.FIFOTaskSetQueuesManager)
*/
System.setProperty("spark.cluster.taskscheduler", cluster/scheduler/class)

/*set XML configuration file path*/
System.setProperty("spark.fairscheduler.allocation.file", path/to/XML configuration file)

spark = new SparkContext("...")

/*Set pool on each spark thread*/
spark.addLocalProperties("spark.scheduler.cluster.fair.pool", poolName)
..........
//===============================

3. Format of XML configuration
\
\
\2\
\1\
\FIFO\
\
\
\10\
\2\
\FAIR\
\
\

4. Class relationship diagram
![Fair Scheduler another version](https://f.cloud.github.com/assets/1168928/263707/6bd2669e-8d78-11e2-807d-09ff3d3135e1.jpg)