apache/spark — pull request #3548 by JoshRosen (2014-12-02 05:00:39+00:00)
Title: [SPARK-4498] [WIP] Add driver -> master heartbeat to detect exited applications and fix executor failure detection logic

This is a WIP fix for SPARK-4498; this isn't the final fix that I want to merge in, but I'm submitting this now to get early feedback from Jenkins and reviewers. The main idea here is to add a periodic driver -> master heartbeat that both signals driver liveness and carries information on whether it the driver has received executors, which allows us to implement proper "don't kill an application due to failed executors as long as it has some running executors" logic in the master.

See discussion at https://issues.apache.org/jira/browse/SPARK-4498 for context.

Before merging, this needs more comments and tests. Specifically, I need tests to check that the heartbeat's information actually corresponds to the right notion of application progress / liveness. There's also open questions about heartbeat interval configuration and failure thresholds. I'll edit this description to accurately reflect the PR before I remove the `[WIP]` tag.

/cc @markhamstra @aarondav @andrewor14 @pwendell @airhorns