rantav/hector — Issue #397: META_WRITE + timeout + ON_FAIL_TRY_ALL_AVAILABLE = SchemaDisagreementException
Author: cherron
Created: 2012-01-20 15:30:12+00:00
State: open

Until https://issues.apache.org/jira/browse/CASSANDRA-1391, there really isn't any reliable way to perform schema mutations.
Hector provides the blockUntilComplete flag on some operations, which is used in combination with me.prettyprint.cassandra.service.AbstractCluster#waitForSchemaAgreement .

As currently implemented, Hector OperationType.META_WRITE operations further increase the likelihood of a SchemaDisagreementException. For example:

1. Client code calls me.prettyprint.cassandra.service.AbstractCluster#dropColumnFamily
2. Thrift call cassandra.system_drop_column_family(columnFamily) is made, but times out.
3. (drop is underway).
4. Hector FailoverPolicy.ON_FAIL_TRY_ALL_AVAILABLE reacts to the timeout, re-issuing the call to another node.
5. SchemaDisagreementException occurs on the second or subsequent attempts.

An additional problem here is that waitForSchemaAgreement is always called after OperationType.META_WRITE operations, but for Cassandra versions < 1.1, a OperationType.META_WRITE operation should not be issued if schema versions are known not to agree.

As a stopgap until Cassandra 1.1 and CASSANDRA-1391, I propose that Hector use this behavior:

All OperationType.META_WRITE operations should:
1. Check waitForSchemaAgreement before and after they are called.
2. Use FailoverPolicy.FAIL_FAST

This is still not enough to guarantee safety, but it reduces risk instead of creating ideal circumstances for SchemaDisgreementExceptions.
Only bulletproof workaround is a global cluster lock around all OperationType.META_WRITE operations which is not practical for large deployments.