rantav/hector — Issue #354: Consistency levels aren't work as expected
Author: Alexander-Kiselyov
Created: 2011-11-27 18:45:39+00:00
State: open

This issue is partially covered in my post at hector-users group.
I'm compelled to repeat that I am completely new to Cassandra/Hector and it might be just my paranoia or misunderstanding :).

Preamble.
Hector’s "Defining Consistency Levels" help page suggests to set consistency level “on per-column family basis”. No restrictions or peculiarities noted.

```java
// Create a customized Consistency Level
ConfigurableConsistencyLevel configurableConsistencyLevel = new ConfigurableConsistencyLevel();
Map clmap = new HashMap();

// Define CL.ONE for ColumnFamily "MyColumnFamily"
clmap.put("MyColumnFamily", HConsistencyLevel.ONE);

// In this we use CL.ONE for read and writes. But you can use different CLs if needed.
configurableConsistencyLevel.setReadCfConsistencyLevels(clmap);
configurableConsistencyLevel.setWriteCfConsistencyLevels(clmap);

// Then let the keyspace know
HFactory.createKeyspace("MyKeyspace", myCluster, configurableConsistencyLevel);
```

But methods described on "Getting started" help page (using templates to access data) in combination with setting of consistency levels (as described in aforementioned article) do not work as expected. Instead of honor per-CF values, they use defaultReadConsistencyLevel/defaultWriteConsistencyLevel (members of ConfigurableConsistencyLevel).

Example.
There are 2 Cassandra nodes. Code attempts to create a keyspace (with replication factor = 2) and a column family, then write (or read) some value to it. Сonsistency level, which is passed to keyspace creation method before performing of write/read contains consistency level ONE for particular column family. Before writing/reading a value one of the nodes goes down.
Operation won't complete, instead an exception will be thrown:

me.prettyprint.hector.api.exceptions.HUnavailableException: : May not be enough replicas present to handle consistency level.
at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:66)
at me.prettyprint.cassandra.service.KeyspaceServiceImpl$1.execute(KeyspaceServiceImpl.java:97)
at me.prettyprint.cassandra.service.KeyspaceServiceImpl$1.execute(KeyspaceServiceImpl.java:90)
at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101)
at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:233)
at me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:131)
at me.prettyprint.cassandra.service.KeyspaceServiceImpl.batchMutate(KeyspaceServiceImpl.java:102)
at me.prettyprint.cassandra.service.KeyspaceServiceImpl.batchMutate(KeyspaceServiceImpl.java:108)
at me.prettyprint.cassandra.model.MutatorImpl$3.doInKeyspace(MutatorImpl.java:248)
at me.prettyprint.cassandra.model.MutatorImpl$3.doInKeyspace(MutatorImpl.java:245)
at me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:85)
at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:245)
at me.prettyprint.cassandra.service.template.AbstractColumnFamilyTemplate.executeBatch(AbstractColumnFamilyTemplate.java:113)
at me.prettyprint.cassandra.service.template.AbstractColumnFamilyTemplate.executeIfNotBatched(AbstractColumnFamilyTemplate.java:147)
at me.prettyprint.cassandra.service.template.ColumnFamilyTemplate.update(ColumnFamilyTemplate.java:80)
at jar.App.main(App.java:111)
Caused by: UnavailableException()
at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:19487)
at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:1035)
at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:1009)
at me.prettyprint.cassandra.service.KeyspaceServiceImpl$1.execute(KeyspaceServiceImpl.java:95)
... 15 more

As log on Cassandra node shows - operation has a QUORUM consistency level. Read operation behaves in similar way.

Detailed description and fix suggestions.
Brief investigation of this issue shows that per-CF consistency levels aren't taken into account. Here's the stacktrace of read operation:

ThriftColumnFamilyTemplate$1.execute(Client) line: 97
ThriftColumnFamilyTemplate$1.execute(Cassandra$Client) line: 1
ThriftColumnFamilyTemplate$1(Operation).executeAndSetResult(Cassandra$Client, CassandraHost) line: 99
HConnectionManager.operateWithFailover(Operation) line: 243
ExecutingKeyspace.doExecuteOperation(Operation) line: 97
ThriftColumnFamilyTemplate.sliceInternal(K, HSlicePredicate) line: 88
ThriftColumnFamilyTemplate.doExecuteSlice(K, HSlicePredicate) line: 47
ThriftColumnFamilyTemplate(ColumnFamilyTemplate).queryColumns(K) line: 109
App.main(String[]) line: 177

me.prettyprint.cassandra.service.template.ThriftColumnFamilyTemplate.sliceInternal(K, HSlicePredicate) method executes get_slice method with consistency level got by operation type only:

```Java
cosc.put(sKey, cassandra.get_slice(sKey, columnParent,
workingSlicePredicate.toThrift(),
ThriftConverter.consistencyLevel(consistencyLevelPolicy.get(operationType))));
```

In this case I suggest to use this call instead:

```java
ThriftConverter.consistencyLevel(consistencyLevelPolicy.get(operationType, columnFamily));
```
cause particular CF name is already stored in columnFamily member and me.prettyprint.hector.api.ConsistencyLevelPolicy.get(OperationType, String) already has safety code in case of consistency level per CF is not set (default level by op. type will be returned instead).

The situation is more involved with write operations. Write op. stacktrace:

KeyspaceServiceImpl$1.execute(Cassandra$Client) line: 95
KeyspaceServiceImpl$1.execute(Cassandra$Client) line: 1
KeyspaceServiceImpl$1(Operation).executeAndSetResult(Cassandra$Client, CassandraHost) line: 99
HConnectionManager.operateWithFailover(Operation) line: 243
KeyspaceServiceImpl.operateWithFailover(Operation) line: 131
KeyspaceServiceImpl.batchMutate(Map>>) line: 102
KeyspaceServiceImpl.batchMutate(BatchMutation) line: 108
MutatorImpl$3.doInKeyspace(KeyspaceService) line: 248
MutatorImpl$3.doInKeyspace(KeyspaceService) line: 1
MutatorImpl$3(KeyspaceOperationCallback).doInKeyspaceAndMeasure(KeyspaceService) line: 20
ExecutingKeyspace.doExecute(KeyspaceOperationCallback) line: 85
MutatorImpl.execute() line: 245
ThriftColumnFamilyTemplate(AbstractColumnFamilyTemplate).executeBatch(Mutator) line: 113
ThriftColumnFamilyTemplate(AbstractColumnFamilyTemplate).executeIfNotBatched(AbstractTemplateUpdater) line: 147
ThriftColumnFamilyTemplate(ColumnFamilyTemplate).update(ColumnFamilyUpdater) line: 65
App.main(String[]) line: 170

me.prettyprint.cassandra.service.KeyspaceServiceImpl.batchMutate(Map>>) method executes Cassandra's batch_mutate method which takes multiple mutations (possibly to multiple column families) but only one consistency level:

```java
cassandra.batch_mutate(mutationMap, getThriftCl(OperationType.WRITE));
```
So, in this case personally I can imagine several possible solutions:

* introduce consistency level per keyspace. It's not the "miracle cure" but would help to define more "fine-grained" consistency levels;
* try to behave "more intellectual"; for example, inspect mutationMap and, if it contains mutations for several CFs - use default level, if only for one column family - use something like:

```java
private ConsistencyLevel getThriftCl(OperationType operationType, String cfName) {
return ThriftConverter.consistencyLevel(consistency.get(operationType, cfName));
}
```

Anyway, it would be good to at least describe discovered peculiarities in documentation.