apache/spark — pull request #2798 by PraveenSeluka (2014-10-14 11:22:13+00:00)
Title: [Spark-3822] Ability to add/delete executors from Spark Context. Works in both yarn-client and yarn-cluster mode

sc.addExecutors(count : Int)
sc.deleteExecutors(List[String])
![image](https://cloud.githubusercontent.com/assets/5339198/4628165/8f281bd8-5393-11e4-8f28-f5cc3d7b0795.png)
Diagram : add/deleteExecutors in Yarn-Client mode

In Yarn-Cluster mode, driver program and Yarn allocator threads are in the same JVM. But, in client mode, they live in different JVM’s. To make this work in both client and cluster mode, we need a way for the driver and Yarn Allocator to communicate.

At high level, this works as follows,
AutoscaleClient is created by SparkContext and AutoscaleServer is created by Yarn Application Master code flow.
Once AutoscaleServerActor comes up, it registers itself to the AutoscaleClientActor.
SparkContext forwards the add/deleteExecutors call through AutoScaleClient -> AutoscaleServerActor.
AutoscaleServer has a reference of YarnAllocator. On receiving addExecutors, it just increments the maxExecutor count in YarnAllocator. And, reporter thread will allocate the respective containers.

Difference when compared to what is proposed
This creates separate AutoscaleClient whereas the current design proposed(https://issues.apache.org/jira/secure/attachment/12673181/dynamic-scaling-executors-10-6-14.pdf) was to add these new messages in CoarseGrainedSchedulerBackend itself. One minor advantage of doing this way is, the dynamic scaling itself lives as a separate module in Spark and none of the other Spark Core code is affected much.