On Sat, Jan 19, 2013 at 4:52 AM, Mridul Muralidharan <
notifications@github.com> wrote:

> In core/src/main/scala/spark/util/AkkaUtils.scala:
>
> > @@ -35,6 +35,10 @@ private[spark] object AkkaUtils {
> >        akka.remote.netty.connection-timeout = 1s
> >        akka.remote.netty.execution-pool-size = %d
> >        akka.actor.default-dispatcher.throughput = %d
> > +
> > +      akka.remote.netty.message-
> frame-size = 100 MiB
> > +      akka.remote.log-remote-lifecycle-events = on
>
> We were getting RemoteClientDisconnected on the master, even though the
> client(s) were alive.
> The problem ended up being that the amount of data slave was sending to
> master was > 1 Mb (or so) : which caused a netty exception to be thrown,
> which akka notifies spark via RemoteClientDisconnected.
>
> Ofcourse, getting to this took quite a while since the actual reason was
> suppressed and not indicated even with TRACE enabled (after fixing log4j
> handling, which other commits in this set do).
>
> The log-remote-lifecycle-events = on allowed us to finally get to the
> reason why the RemoteClientDisconnected was getting thrown.
>
> Note that we still have some work to do in this regard : which I will get
> to later.
> When RemoteClientDisconnected is raised, the master does not close the
> connection : and so slaves continue to run, and no new slave is spawned to
> replace the one lost - which results in all slaves being lost in a cascade
> (master thinks they are gone, but yarn knows the slaves are working - so
> does not reallocate).
>

Should we also port to the 0.6 & master branches?  We use the standalaone
spark cluster, but it seems like this would help all cases.  Are there
other parts of this commit that are applicable cross-environment?

We've also observed random RemoteClientDisconnecteds, it would be great to
get more information about why that happens.

And, sometimes with the standalone cluster, the StandaloneExecutorBackend
does not get killed when a job dies.  It doesn't happen often, and we
haven't been able to pinpoint the circumstances under which this happens.
It sounds like you may be onto a reason why, which would really help us out.

thanks,
imran