apache/spark — pull request #2746 by andrewor14 (2014-10-10 02:41:23+00:00)
Title: [WIP][SPARK-3795] Heuristics for dynamically scaling executors

This is part of a bigger effort to provide elastic scaling of executors within a Spark application ([SPARK-3174](https://issues.apache.org/jira/browse/SPARK-3174)). This PR does not provide any functionality by itself; it is a skeleton that is missing a mechanism to be added later in [SPARK-3822](https://issues.apache.org/jira/browse/SPARK-3822).

The design doc can be found at [SPARK-3174](https://issues.apache.org/jira/browse/SPARK-3174) (under "Heuristics for Scaling Executors". This deviates from the design in two ways: (1) the remove policy is significantly simplified by cutting out the exponential increase logic that mirrors the add policy, and (2) the configuration of the add policy is split up into `addExecutorThreshold` and `addExecutorInterval`. The main reason for this is that the user may want to configure how quickly executors are added, but want to keep the trigger condition the same.

This is work-in-progress because it is missing two things: (1) logic for retrying a request to add/remove executors if the request is not fulfilled, and (2) unit tests.

Comments and feedback are most welcome, but please keep in mind that this is still WIP.