apache/spark — pull request #2840 by andrewor14 (2014-10-18 02:44:17+00:00)
Title: [WIP][SPARK-3822] Executor scaling mechanism for Yarn

This is part of a broader effort to enable dynamic scaling of executors ([SPARK-3174](https://issues.apache.org/jira/browse/SPARK-3174)). This is intended to work alongside with SPARK-3795 (#2746), SPARK-3796 and SPARK-3797.

The logic is built on top of @PraveenSeluka at #2798. This is different from the changes there in that the mechanism is implemented within the existing scheduler backend framework rather than in new `Actor` classes. This also introduces a parent abstract class `YarnSchedulerBackend` to encapsulate common logic to communicate with the Yarn `ApplicationMaster`.

I have tested this on a stable Yarn cluster. This is still WIP because when an executor is removed, `SparkContext` and its components react as if it has failed, resulting in many scary error messages and eventual timeouts. While it's not strictly necessary to fix this as of the first-cut implementation of this mechanism, it would be good to add logic to distinguish this case if it doesn't require too much more work.