apache/spark — pull request #2786 by chesterxgchen (2014-10-13 18:56:13+00:00)
Title: [SPARK-3913] Spark Yarn Client API change to expose Yarn Resource Capacity, Yarn Application Listener and KillApplication APIs

Spark Yarn Client API change to expose Yarn Resource Capacity, Yarn Application Listener and KillApplication APIs

When working with Spark with Yarn cluster mode, we have following issues:
1) We don't know how much yarn max capacity ( memory and cores) before we specify the number of executor and memories for spark drivers and executors. We we set a big number, the job can potentially exceeds the limit and got killed.
It would be better we let the application know that the yarn resource capacity a head of time and the spark config can adjusted dynamically.
2) Once job started, we would like some feedback from yarn application. Currently, the spark client basically block the call and returns when the job is finished or failed or killed.
If the job runs for few hours, we have no idea how far it has gone, the progress and resource usage, tracking URL etc. This Pull Request will not complete solve the issue #2, but it will allow expose Yarn Application status: such as when the job is started, killed, finished, the tracking URL etc, some limited progress reporting ( for CDH5 we found the progress only reports 0, 10 and 100%)

I will have another Pull Request to address the Yarn Application and Spark Job communication issue, that's not covered here.

3) If we decide to stop the spark job, the Spark Yarn Client expose a stop method. But the stop method, in many cases, does not stop the yarn application.

So we need to expose the yarn client's killApplication() API to spark client.

The proposed change is to change Client Constructor, change the first argument from ClientArguments to
YarnResourceCapacity => ClientArguments

Were YarnResourceCapacity contains yarn's max memory and virtual cores as well as overheads.

This allows application to adjust the memory and core settings accordingly.

For existing application that ignore the YarnResourceCapacity the

def toArgs (capacity: YarnResourceCapacity) = new ClientArguments(...)

We also defined the YarnApplicationListener interface that expose some of the information about YarnApplicationReport.

Client.addYarnApplicaitonListener(listerner)
will allow them to get call back at different state of the application, so they can react accordingly.

For example, onApplicationInit() the callback will invoked when the AppId is available but application is not yet started. Once can use this AppId to kill the application if the run is not longer desired.