apache/spark — pull request #2784 by aarondav (2014-10-13 18:21:02+00:00)
Title: [SPARK-3923] Decrease Akka heartbeat interval below heartbeat pause

Something about the 2.3.4 upgrade seems to have made the issue manifest where all the services disconnect from each other after exactly 1000 seconds (which is the heartbeat interval). [This post](https://groups.google.com/forum/#!topic/akka-user/X3xzpTCbEFs) suggests that heartbeat pause should be less than heartbeat interval, and decreasing the interval from 1000s to below the 600s of the heartbeat pause seems to have rectified the issue. My current cluster has now exceeded 1400s of uptime without failure!

I do not know why this fixed it, because the threshold we have set for the failure detector is the exponent of a timeout, and 300 is extremely large. Perhaps the default failure detector changed in 2.3.4 and now ignores threshold.