[SPARK-3456] YarnAllocator on alpha can lose container requests to RM

I understand the concern over the interfaces, but in the end hopefully this doesn't matter as hopefully we will be removing alpha support in another release.  Also its an internal private interface.


To me this is safer to have one unused variable vs exposing a private variable.  For instance, you will notice that if I removed all changes to YarnAllocator (except for making numPendingAllocate protected) and just grabbed it from the alpha/YarnAllocateHandler, it would be wrong -  it would ask for 2x the number because we increment the numberPendig before calling allocateContainers.  To me exposing that variable could more easily lead to issues like that and perhaps others accessing it and leading to more complicated usage numPending in the future. It also seems more brittle if someone comes in here and refactors things again.

I could change YarnAllocator to rearrange it to call allocateContainers before incrementing numPending for now and put comment there saying the order matters.

I would prefer to leave it, but If you would really like me to I will.