Bug: Leak detected issue in cassandra 3.11.1
Status: Resolved
Priority: Normal
Resolution: Invalid
Created: 30/May/23 05:27
Resolved: 30/May/23 14:46

We are running 36 nodes Cassandra cluster. All nodes have the some config, Cassandra version is 3.11.1. Suddenly, we are seeing fluctuation in storage available graphs for the 2 nodes. Attaching the last 7 days used storage graph for the Cassandra nodes. 

Further investigation, we saw errors in the logs of these 2 Cassandra node logs. 

{code:java}
ERROR [CompactionExecutor:164360] 2023-05-04 00:30:58,094 CassandraDaemon.java:228 - Exception in thread Thread[CompactionExecutor:164360,1,main]
java.lang.IllegalArgumentException: null
    at java.nio.Buffer.position(Buffer.java:244) ~[na:1.8.0_131]
    at org.apache.cassandra.io.util.SafeMemoryWriter.reallocate(SafeMemoryWriter.java:59) ~[apache-cassandra-3.11.1.jar:3.11.1]
    at org.apache.cassandra.io.util.SafeMemoryWriter.setCapacity(SafeMemoryWriter.java:68) ~[apache-cassandra-3.11.1.jar:3.11.1]
    at org.apache.cassandra.io.sstable.IndexSummaryBuilder.prepareToCommit(IndexSummaryBuilder.java:250) ~[apache-cassandra-3.11.1.jar:3.11.1]
    at org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.doPrepare(BigTableWriter.java:524) ~[apache-cassandra-3.11.1.jar:3.11.1]
    at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) ~[apache-cassandra-3.11.1.jar:3.11.1]
    at org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:364) ~[apache-cassandra-3.11.1.jar:3.11.1]
    at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) ~[apache-cassandra-3.11.1.jar:3.11.1]
    at org.apache.cassandra.io.sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.java:281) ~[apache-cassandra-3.11.1.jar:3.11.1]
    at org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:379) ~[apache-cassandra-3.11.1.jar:3.11.1]
    at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) ~[apache-cassandra-3.11.1.jar:3.11.1]
    at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:111) ~[apache-cassandra-3.11.1.jar:3.11.1]
    at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) ~[apache-cassandra-3.11.1.jar:3.11.1]
    at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184) ~[apache-cassandra-3.11.1.jar:3.11.1]
    at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:121) ~[apache-cassandra-3.11.1.jar:3.11.1]
    at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:220) ~[apache-cassandra-3.11.1.jar:3.11.1]
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.11.1.jar:3.11.1]
    at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85) ~[apache-cassandra-3.11.1.jar:3.11.1]
    at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) ~[apache-cassandra-3.11.1.jar:3.11.1]
    at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:268) ~[apache-cassandra-3.11.1.jar:3.11.1]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_131]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_131]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_131]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
    at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.1.jar:3.11.1]
    at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
WARN  [CompactionExecutor:164909] 2023-05-04 00:31:54,573 IndexSummaryBuilder.java:115 - min_index_interval of 128 is too low for 4298508892 expected keys of avg size 64; using interval of 145 instead
ERROR [Reference-Reaper:1] 2023-05-04 00:32:02,326 Ref.java:224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@4f374614) to class org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1106289026:Memory@[7fa6ab9ec010..7fa72ad798d0) was not released before the reference was garbage collected {code}
Compactions also running continually run on these nodes.

I am unable to understand the root cause of it. Any help is appreciated.