apache/spark — pull request #3447 by andrewor14 (2014-11-25 08:09:11+00:00)
Title: [SPARK-4592] Avoid duplicate worker registrations in standalone mode

**Symptom.** On failover, the Master may receive duplicate registrations from the same worker, causing the worker to exit.

**Cause.** This commit https://github.com/apache/spark/commit/4afe9a4852ebeb4cc77322a14225cd3dec165f3f adds logic for the worker to re-register with the master in case of failures. However, the following race condition may occur:

(1) Master A fails and Worker attempts to reconnect to all masters
(2) Master B takes over and notifies Worker
(3) Worker responds by registering with Master B
(4) Meanwhile, Worker's previous reconnection attempt reaches Master B, causing the same Worker to register with Master B twice

**Fix.** Instead of attempting to register with all known masters, the worker should re-register with only the one that it has been communicating with. Then, when it is finally notified of the change in master, the worker gives up on the old master and communicates with the new one.

**Caveat.** Even this fix is subject to more obscure race conditions. For instance, if Master B fails and Master A recovers immediately, then Master A may still observed duplicate worker registrations. However, this, and other potential race conditions summarized in [SPARK-4592](https://issues.apache.org/jira/browse/SPARK-4592), are much, much less likely than the one described above, which is deterministically reproducible.