SPARK-3913
Spark Yarn Client API change to expose Yarn Resource Capacity and Yarn Application Listener

When working with Spark with Yarn deployment mode, we have two issues:
1) We don't know how much yarn max capacity ( memory and cores) before we specify the number of executor and memories for spark drivers and executors. We we set a big number, the job can potentially exceeds the limit and got killed.
It would be better we let the application know that the yarn resource capacity a head of time and the spark config can adjusted dynamically.
2) Once job started, we would like some feedback from yarn application. Currently, the spark client basically block the call and returns when the job is finished or failed or killed.
If the job runs for few hours, we have no idea how far it has gone, the progress and resource usage, tracking URL etc.

This commit addresses the first issue.

The proposed change is to change Client Constructor, change the first argument from ClientArguments to
 YarnResourceCapacity => ClientArguments

 Were YarnResourceCapacity contains yarn's max memory and virtual cores as well as overheads.

This allows application to adjust the memory and core settings accordingly.

For existing application that ignore the YarnResourceCapacity the

def toArgs (capacity: YarnResourceCapacity) = new ClientArguments(...)