We were getting RemoteClientDisconnected on the master, even though the client(s) were alive.
The problem ended up being that the amount of data slave was sending to master was > 1 Mb (or so) : which caused a netty exception to be thrown, which akka notifies spark via RemoteClientDisconnected.

Ofcourse, getting to this took quite a while since the actual reason was suppressed and not indicated even with TRACE enabled (after fixing log4j handling, which other commits in this set do).


The log-remote-lifecycle-events = on allowed us to finally get to the reason why the RemoteClientDisconnected was getting thrown.


Note that we still have some work to do in this regard : which I will get to later.
When RemoteClientDisconnected is raised, the master does not close the connection : and so slaves continue to run, and no new slave is spawned to replace the one lost - which results in all slaves being lost in a cascade (master thinks they are gone, but yarn knows the slaves are working - so does not reallocate).