apache/spark — pull request #3810 by YanTangZhai (2014-12-26 11:36:01+00:00)
Title: [SPARK-4962] [CORE] Put TaskScheduler.start back in SparkContext to shorten cluster resources occupation period

When SparkContext object is instantiated, TaskScheduler is started and some resources are allocated from cluster. However, these
resources may be not used for the moment. For example, DAGScheduler.JobSubmitted is processing and so on. These resources are wasted in
this period. Thus, we want to put TaskScheduler.start back to shorten cluster resources occupation period specially for busy cluster.
TaskScheduler could be started just before running stages.
We could analyse and compare the resources occupation period before and after optimization.
TaskScheduler.start execution time: [time1__]
DAGScheduler.JobSubmitted (excluding HadoopRDD.getPartitions or TaskScheduler.start) execution time: [time2_]
HadoopRDD.getPartitions execution time: [time3___]
Stages execution time: [time4_____]
The cluster resources occupation period before optimization is [time2_][time3___][time4_____].
The cluster resources occupation period after optimization is....[time3___][time4_____].
In summary, the cluster resources
occupation period after optimization is less than before.
If HadoopRDD.getPartitions could be put forward (SPARK-4961), the period may be shorten more which is [time4_____].
The resources saving is important for busy cluster.

The main purpose of this PR is to decrease resources waste for busy cluster.
For example, a process initializes a SparkContext instance, reads a few files from HDFS or many records from PostgreSQL, and then calls RDD's collect operation to submit a job.
When SparkContext is initialized, an app is submitted to cluster and some resources are hold by this app.
These resources are not used really until the job is submitted by RDD's action.
The resources in the period from initialization to actual use could be considered wasteful.
If app is submitted when SparkContext is initialized, all of resources needed by the app may be granted before running job.
Then the job could runs efficiently without resource constraint.
On the contrary, if app is submitted when job is submitted, resources needed by the app may be granted at different times. Then the job may run not so efficiently since some resources are applying.
Thus I use a configuration parameter spark.scheduler.app.slowstart (default false) to let user make tradeoffs between economy and efficiency.
There are 9 kinds of master URL and 6 kinds of SchedulerBackend.
LocalBackend and SimrSchedulerBackend don't need to put starting back since there is no difference.
SparkClusterSchedulerBackend (yarn-standalone or yarn-cluster) does not put starting back since the app should be submitted in advance by SparkSubmit.
CoarseMesosSchedulerBackend and MesosSchedulerBackend could put starting back.
YarnClientSchedulerBackend (yarn-client) could put starting back.
This PR puts TaskScheduler.start back only for yarn-client mode in the early.