mesos/spark — pull request #299 by squito (2012-11-06 19:38:27+00:00)
Title: Retry tasks when mesos slaves die

This addresses SPARK-604
https://spark-project.atlassian.net/browse/SPARK-604

This addresses several issues:
1) the slaveId from mesos in a slaveLost() msg was not getting parsed properly.
2) if a mesos slave died, spark simply hung (neither died nor retried)

Now, when a slave is lost, its correctly removed, and then we add a an event to the queue to resubmit the tasks. That request waits a bit though, to see if the slave will reconnect (in this patch its hard-coded to 10s).

This could certainly use some review -- its was mostly just reverse-engineered, I wasn't sure about the exact contract between Spark & Mesos. I have tested this on a cluster w/ several machines, where I manually killed one of the slaves. Spark successfully recovers in that case -- though admittedly it seems like it needs to resubmit the current stage more than once before its happy.

Any suggestions on how to add unit tests for interactions w/ mesos?