Bug: LEAK DETECTED: was not released before the reference was garbage collected
Status: Open
Priority: Normal
Created: 02/Nov/20 17:04

Hello,

I am getting the following errors and I am running 3.11.5
{code:java}
LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@356cb07f) to class org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1709177197:Memory@[7e7a3b64b910..7e7a3b96b910) was not released before the reference was garbage collected


ERROR [CompactionExecutor:3763] 2020-11-02 02:34:53,316  CassandraDaemon.java:228 - Exception in thread Thread[CompactionExecutor:3763,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
        at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:180) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:135) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:268) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_60]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_60]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_60]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) [apache-cassandra-3.11.5.jar:3.11.5]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_60]
Caused by: java.io.IOException: Map failed
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:939) ~[na:1.8.0_60]
        at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153) ~[apache-cassandra-3.11.5.jar:3.11.5]
        ... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
        at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_60]
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:936) ~[na:1.8.0_60]
        ... 24 common frames omitted
WARN  [GossipTasks:1] 2020-11-02 02:34:53,302  FailureDetector.java:278 - Not marking nodes down due to local pause of 7468254535 > 5000000000
ERROR [Reference-Reaper] 2020-11-02 02:34:53,208  Ref.java:229 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@33fef449) to class org.apache.cassandra.io.util.FileHandle$Cleanup@1098210945:/data/cassandra/data/elvisevallogks/decision_detail-7351f6205f3811ea922ce1efeeba3e49/md-3695-big-Index.db was not released before the reference was garbage collected
ERROR [CompactionExecutor:3762] 2020-11-02 02:34:53,208  CassandraDaemon.java:228 - Exception in thread Thread[CompactionExecutor:3762,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
        at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:181) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:180) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:135) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:268) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_60]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_60]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_60]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) [apache-cassandra-3.11.5.jar:3.11.5]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_60]
Caused by: java.io.IOException: Map failed
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:939) ~[na:1.8.0_60]
        at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153) ~[apache-cassandra-3.11.5.jar:3.11.5]
        ... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
        at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_60]
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:936) ~[na:1.8.0_60]
        ... 24 common frames omitted
INFO  [Service Thread] 2020-11-02 02:34:53,877  StatusLogger.java:51 - MemtablePostFlush                 0         0           3596         0                 0 {code}
 

These are the settings
{code:java}
# /etc/security/limits-cassandra.conf
# Cassandra-specfic parameters
# add
cassandra        soft    memlock         unlimited
cassandra        hard    memlock         unlimited
cassandra        soft    nproc           65535
#change
*                soft    nofile          655350
*                hard    nofile          655350 {code}
For cassandra process
{code:java}
cat /proc/25586/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    0                    bytes
Max resident set          unlimited            unlimited            bytes
Max processes             65535                692334               processes
Max open files            655350               655350               files
Max locked memory         unlimited            unlimited            bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       692334               692334               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us {code}
{{vm.max_map_count is set to 1048575}}
{code:java}
sysctl vm.max_map_count
vm.max_map_count = 1048575{code}