Bug: Cassandra ||system logs|| Unable to compute ceiling for max when histogram overflowed 
Status: Resolved
Priority: Normal
Resolution: Not A Problem
Created: 02/Jun/22 15:50
Resolved: 02/Jun/22 15:59

Hi Team,

 

Please help to suggest how to fix the issue as our Cassandra DB is misbehaving.

 

There is a total of 8 Node clusters of Cassandra out of which 2 nodes are misbehaving and getting down over and again. while checking ./nodetool status two nodes always get down within 1 or 2 days.

And we have noticed that Cassandra's services also not running on these two faulty nodes.

Logs:

ERROR [CompactionExecutor:9433] 2022-06-02 16:32:29,997 CassandraDaemon.java:228 - Exception in thread Thread[CompactionExecutor:9433,1,main]
java.lang.IllegalStateException: Unable to compute ceiling for max when histogram overflowed
    at org.apache.cassandra.utils.EstimatedHistogram.rawMean(EstimatedHistogram.java:231) ~[apache-cassandra-3.11.3.jar:3.11.3]
    at org.apache.cassandra.utils.EstimatedHistogram.mean(EstimatedHistogram.java:220) ~[apache-cassandra-3.11.3.jar:3.11.3]
    at org.apache.cassandra.io.sstable.metadata.StatsMetadata.getEstimatedDroppableTombstoneRatio(StatsMetadata.java:115) ~[apache-cassandra-3.11.3.jar:3.11.3]
    at org.apache.cassandra.io.sstable.format.SSTableReader.getEstimatedDroppableTombstoneRatio(SSTableReader.java:1926) ~[apache-cassandra-3.11.3.jar:3.11.3]
    at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.worthDroppingTombstones(AbstractCompactionStrategy.java:423) ~[apache-cassandra-3.11.3.jar:3.11.3]
    at org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.getNextBackgroundSSTables(SizeTieredCompactionStrategy.java:99) ~[apache-cassandra-3.11.3.jar:3.11.3]
    at org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.getNextBackgroundTask(SizeTieredCompactionStrategy.java:183) ~[apache-cassandra-3.11.3.jar:3.11.3]
    at org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:153) ~[apache-cassandra-3.11.3.jar:3.11.3]
    at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:268) ~[apache-cassandra-3.11.3.jar:3.11.3]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_191]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_191]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_191]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_191]
    at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.3.jar:3.11.3]
    at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_191]
ERROR [CompactionExecutor:9435] 2022-06-02 16:32:29,997 CassandraDaemon.java:228 - Exception in thread Thread[CompactionExecutor:9435,1,main]
java.lang.IllegalStateException: Unable to compute ceiling for max when histogram overflowed
    at org.apache.cassandra.utils.EstimatedHistogram.rawMean(EstimatedHistogram.java:231) ~[apache-cassandra-3.11.3.jar:3.11.3]
    at org.apache.cassandra.utils.EstimatedHistogram.mean(EstimatedHistogram.java:220) ~[apache-cassandra-3.11.3.jar:3.11.3]
    at org.apache.cassandra.io.sstable.metadata.StatsMetadata.getEstimatedDroppableTombstoneRatio(StatsMetadata.java:115) ~[apache-cassandra-3.11.3.jar:3.11.3]
    at org.apache.cassandra.io.sstable.format.SSTableReader.getEstimatedDroppableTombstoneRatio(SSTableReader.java:1926) ~[apache-cassandra-3.11.3.jar:3.11.3]
    at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.worthDroppingTombstones(AbstractCompactionStrategy.java:423) ~[apache-cassandra-3.11.3.jar:3.11.3]
    at org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.getNextBackgroundSSTables(SizeTieredCompactionStrategy.java:99) ~[apache-cassandra-3.11.3.jar:3.11.3]