How to Fix Cassandra Consistency Issues using Read Repair | by Andriy Zabavskyy | Towards Data Science

As many of you probably know, Cassandra is an AP big data storage. In other words, when a network partition happens, Cassandra remains available and relaxes the Consistency property. It is always said that it is eventually consistent or, in other words, it will be consistent at some point in time in future.
The important things to know which is not really obvious are:
The cluster does become inconsistent pretty often. Sure, there are many things influencing the stability of the cluster, such as proper configuration, dedicated resources, production load, professionalism of the ops guys etc, but the fact is the probability the nodes are going down from time to time and therefore the data become inconsistent are really high.
The cluster does NOT become consistent again automatically. This is something which goes against the god feeling towards the modern and mature distributed systems. Unless you have the enterprise version of Datastax and enable one the latest feature of DSE v6 you have to fix the inconsistency issues manually.
Fortunately, there are ways to fix the inconsistency issues. There a couple of options here:
nodetool repair tool. This is probably the main and default method to use. Running the command on a node which was down for all the tables or specific ones. One caveat though is: all the nodes should be UP while you are running the command.
read repair Cassandra feature. This is an important feature meaning, during the read requests the cluster organism is repairing itself, to be more precise it repairs the proper data replicas. If the replicas involved in a read requests are not consistent they are being aligned again
As it was stated above the default and main way to fix inconsistency is nodetool repair tool, so the natural question is when and why to use read repair method. Let me answer that using the experience from one of my project.
At some point we entered a period when our Cassandra cluster started to become very unstable and it took significant amount of time until all the nodes returned to the UP state again. That lead to the 2 main consequences:
data became inconsistent
nodetool repair tool was not possible to use to fix inconsistencies during that period
Having these circumstances the best we could do was using the read repair feature to make sure at least at the majority of replicas, at the Quorum level, the data is consistent so that for all the reads which are using the Quorum consistency level the data is consistent and up to date
Read repair feature fixes the consistency only on the records which are involved in the reads, so how to repair the whole table?
It is possible to use the following approach:
select one of narrowest column to read
read the whole table using the copy command to export the data to the file
So let’s consider we have a Cassandra table “event” in a keyspace “test” with one of the narrowest column called “id”; the copy command would look like this:
cqlsh -e "consistency QUORUM; copy test.event(fid) to '/tmp/tid'"
Alternatively, you could read whole records and send them to ‘/dev/null’:
cqlsh -e "consistency QUORUM; copy test.event to '/dev/null'"
And sure, when all the nodes are UP you could use consistency ALL, but in this case it’s better to use the nodetool repair tool like this:
nodetool repair test event
Cassandra is great big data storage but in order to leverage it to the fullest, it requires a good understanding of the main principles, how it works and, as any beautiful thing, it requires some care :)