hector-client/hector — pull request #625 by jancona (2013-06-24 22:20:41+00:00)
Title: Modify retryDownedHosts() to only delete hosts not in the ring if autodiscovery is on

We discovered that Hector queries the cluster for ring status in retryDownedHosts() even if AutoDiscoverHosts is off. It then removes any hosts it thinks have left the cluster from the downedHostQueue. These hosts can then never be re-added to the host list because autodiscovery is off.

It seems to me that setting AutoDiscoverHosts to false should mean "I'm managing my host list myself." This patch makes retryDownedHosts() respect that setting.

*Background:* We encountered this in Hector 1.0-5 when upgrading our cluster from 1.0 to 1.1. The behavior we saw was that every time we restarted an upgraded node, we got a "Removing host foo - It does no longer exist in the ring" message in the log and the node disappeared from the KnownHosts list. Since we were doing a rolling restart, this rapidly progressed to the dreaded "All host pools marked down."

My guess is that there is an incompatibility between Hector 1.0 and Cassandra 1.1 that leads to the ring metadata being misinterpreted and the host being marked as having left. So our fix is to apply this patch to 1.0-5 before we upgrade our production cluster. I'd be happy to do a pull request against 1.0 if you think it's worth fixing there.