rantav/hector — Issue #394: Downed host is removed from HConnectionManager and is not added when host is up again.
Author: voitishchuk
Created: 2012-01-19 10:46:27+00:00
State: open

We have recently upgraded our hector version from 0.7.0-30 to 1.0.2 and cassandra 1.0.5.
Since this upgrade, we encounter reconnect problems.

We start our system and create Cluster via new ThriftCluster(). If we use HFactory.getOrCreateCluser the bahaviour remains the same.
We perform several operations with cassandra via this Cluster object to ensure that it works.

Then we put Cassandra down, and we try to use the Cluster object to perform operations. This produces "All host pools marked down", as expected.
After we start cassandra the Cluster object remains in a unworkable state.

After investigation we found that the host is removed from the hostpool in HConnectionManager if all connections with the host are down.
We found this problem with single node cassandra cluster.

I expect the same problem with multiple cassandra nodes.

We use CassandraHostConfigurator with default settings; retryDownedHosts == true .
The behaviour remains the same with nodeautodiscovery enabled or disabled.