Bug: Cassandra crashes with OutOfMemory Exception
Status: Resolved
Priority: Normal
Resolution: Invalid
Created: 19/Aug/20 13:35
Resolved: 18/Feb/21 17:37

Out Cassandra instance has run perfectly for almost 5 years, but aout a month ago, the performance has dropped significantly. Reads are incredibly slow or time out, making the cluster almost useless. The load on the nodes has skyrocketed to almost 100% CPU usage. On top of that, the nodes crash with OutOfMemoryError. 
I also have [heap dumps|https://arcelormittal-my.sharepoint.com/:f:/g/personal/sidtdke_sidmar_be/Es7PaQq15jdKgIvDfMFfBRoBR6sr60t156AxbA0dFZvzJg?e=rohgHn]. 
Cassandra version: 3.11.1
Java version: OpenJDK 1.8.0_201

We have upgraded our nodes to version 3.11.7 and applied the recommended OS/system setting, but this has not improved performance.

{code:java}
ERROR [ReadStage-9] 2020-08-10 14:56:01,399 JVMStabilityInspector.java:142 - JVM state determined to be unstable.  Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
	at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[na:1.8.0_201]
	at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_201]
	at org.apache.cassandra.io.util.DataOutputBuffer.reallocate(DataOutputBuffer.java:159) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:413) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.db.rows.Cell$Serializer.serialize(Cell.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.db.rows.UnfilteredSerializer.lambda$serializeRowBody$0(UnfilteredSerializer.java:248) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.db.rows.UnfilteredSerializer$$Lambda$120/205506350.accept(Unknown Source) ~[na:na]
	at org.apache.cassandra.utils.btree.BTree.applyForwards(BTree.java:1242) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.utils.btree.BTree.apply(BTree.java:1197) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.db.rows.BTreeRow.apply(BTreeRow.java:172) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.db.rows.UnfilteredSerializer.serializeRowBody(UnfilteredSerializer.java:236) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:205) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:137) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:125) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:137) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:92) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:308) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:167) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:160) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:156) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:346) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:50) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_201]
	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) ~[apache-cassandra-3.11.1.jar:3.11.1]
	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134) ~[apache-cassandra-3.11.1.jar:3.11.1]
{code}