av9ash/GitBugs huggingface dataset:
Bug: nodes failed to start because of ConfigurationException during upgrade C* from 2.2 to 3.0
Status: Triage Needed
Priority: Normal
Created: 20/Nov/20 06:18

 

Error message:
{code:java}
ERROR 21:57:59 Fatal exception during initialization
org.apache.cassandra.exceptions.ConfigurationException: Found system keyspace files, but they couldn't be loaded!
        at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:953) ~[main/:na]
        at org.apache.cassandra.service.StartupChecks$9.execute(StartupChecks.java:407) ~[main/:na]
        at org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:120) ~[main/:na]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:199) [main/:na]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:607) [main/:na]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:774) [main/:na]
{code}
 

Reproduce steps: 
 # Start up a cluster of version 2.2 with 10 nodes and one seed
 # Use the stress test tool to generate some data. e.g., 

/cassandra/tools/bin/cassandra-stress write n=1000000 -node 252.11.1.2
 # Stop all nodes, and started the C* nodes running C* compiled from 3.0

5. After upgrading, some of the nodes failed to start because of the ConfigurationException.
Bug: nodetool removenode failing
Status: Triage Needed
Priority: Normal
Created: 19/Nov/20 16:51

Hi, i tied to remove node on Cassandra  4.0-beta3  and got a lot of errors and also nodetool command failed.
{code}
[root@foo ~]# nodetool removenode 2dfa2fde-15ee-40ad-8afc-0cb6db9aaad9
nodetool: Conflicting replica added (expected unique ranges): Full(/1.1.1.1:7000,(2012127519876180938,2107158795315043585]); existing: Full(/1.1.1.1:7000,(2012127519876180938,2107158795315043585])
See 'nodetool help' or 'nodetool help <command>'.
{code}
In log:
{code}
INFO  [Messaging-EventLoop-3-31] 2020-11-19 15:56:48,949 NoSpamLogger.java:92 - /1.1.1.1:7000->/2.2.2.2:7000-URGENT_MESSAGES-[no-channel] failed to connect
io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /2.2.2.2:7000
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
        at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124)
        at io.netty.channel.unix.Socket.finishConnect(Socket.java:243)
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:672)
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:649)
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:529)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:465)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:834)
WARN  [Messaging-EventLoop-3-31] 2020-11-19 16:01:18,973 NoSpamLogger.java:95 - /1.1.1.1:7000->/2.2.2.2:7000-URGENT_MESSAGES-[no-channel] dropping message of type GOSSIP_DIGEST_SYN whose timeout expired before reaching the network
{code}

How to reproduce:
Create 2 DC and remove one node. 
Maybe it possible to reproduce with one DC but in my case was 2 DCs.

I had to remove with "force"
Bug: Fix STCS documentation (the header is currently LCS)
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Nov/20 05:21
Resolved: 18/Nov/20 12:18

Currently, the header in the [documentation for STCS|https://cassandra.apache.org/doc/latest/operating/compaction/stcs.html] refers to LCS in the header, which also makes it hard to find the STCS documentation via search.
Bug: Update "Getting Started" document for Windows users
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 02/Oct/20 15:13
Resolved: 18/Nov/20 04:08

This is a documentation follow up to CASSANDRA-16171.

Since we are removing support for Windows, we should update ["Getting Started" guide|https://cassandra.apache.org/doc/latest/getting_started/index.html] to include how-to's for Windows users for setting up Cassandra for dev evnironment.
Bug: Adding a C* 1.2.0 node to a C* 1.1.0 cluster fails with connection failure
Status: Resolved
Priority: Normal
Resolution: Won't Fix
Created: 11/Nov/20 22:55
Resolved: 12/Nov/20 19:24

Steps to reproduce: 
 # start a 2-node C* 1.1.0 cluster
 # add a new C* 1.2.0 node

Error: 
{code:java}
ERROR [main] 2020-06-18 01:12:12,894 CassandraDaemon.java (line 414) Exception encountered during startup
java.lang.RuntimeException: No other nodes seen!  Unable to bootstrap.If you intended to start a single-node cluster, you should make sure your broadcast_address (or listen_address) is listed as a seed.  Otherwise, you need to determine why the seed being contacted has no knowledge of the rest of the cluster.  Usually, this can be solved by giving all nodes the same seed list.
  at org.apache.cassandra.dht.BootStrapper.getBootstrapSource(BootStrapper.java:154)
  at org.apache.cassandra.dht.BootStrapper.getBalancedToken(BootStrapper.java:135)
  at org.apache.cassandra.dht.BootStrapper.getBootstrapTokens(BootStrapper.java:115)
  at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:611)
  at org.apache.cassandra.service.StorageService.initServer(StorageService.java:499)
  at org.apache.cassandra.service.StorageService.initServer(StorageService.java:397)
  at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:309)
  at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:397)
  at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:440)
ERROR [StorageServiceShutdownHook] 2020-06-18 01:12:12,898 CassandraDaemon.java (line 133) Exception in thread Thread[StorageServiceShutdownHook,5,main]
java.lang.NullPointerException
  at org.apache.cassandra.service.StorageService.stopRPCServer(StorageService.java:307)
  at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:464)
  at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
  at java.lang.Thread.run(Thread.java:748)
{code}
Root cause: 

In 1.2.0, when a node wants to connect another node, it will expect to read a int from it. See https://github.com/apache/cassandra/blob/cassandra-1.2.0/src/java/org/apache/cassandra/net/OutboundTcpConnection.java#L276 .

There *seems* to exist a check (line 269 in the same file) about the target version before trying to read int. It is based on variable `targetVersion`, which is from `MessagingService.instance().getVersion(poolReference.endPoint())`. This function, however, returns the version of itself when the end point is unknown.

Also the target node will send this int if it is also running 1.2.0. See https://github.com/apache/cassandra/blob/cassandra-1.2.0/src/java/org/apache/cassandra/net/IncomingTcpConnection.java#L90 .

However, in 1.1.0, there is no such mechanism, the 1.1.0 node won't send this int. So when a 1.2.0 node tries to connect to a 1.1.0 node, it will stuck at "readInt".
Bug: During rolling upgrade from C* 2.0.0 to 2.1.0, upgraded node fails to gossip with seed
Status: Resolved
Priority: Normal
Resolution: Won't Fix
Created: 11/Nov/20 22:06
Resolved: 12/Nov/20 19:29

Steps to reproduce: 
 # Start a 2-node C* 2.0.0 cluster with one seed
 # upgrade the non-seed node to 2.1.0

Error stacktrace on upgraded node: 

 
{code:java}
ERROR [main] 2020-07-22 08:12:53,406 CassandraDaemon.java:474 - Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any seeds
  at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1200) ~[apache-cassandra-2.1.0.jar:2.1.0]
  at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:451) ~[apache-cassandra-2.1.0.jar:2.1.0]
  at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:667) ~[apache-cassandra-2.1.0.jar:2.1.0]
  at org.apache.cassandra.service.StorageService.initServer(StorageService.java:615) ~[apache-cassandra-2.1.0.jar:2.1.0]
  at org.apache.cassandra.service.StorageService.initServer(StorageService.java:509) ~[apache-cassandra-2.1.0.jar:2.1.0]
  at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:338) [apache-cassandra-2.1.0.jar:2.1.0]
  at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:457) [apache-cassandra-2.1.0.jar:2.1.0]
  at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:546) [apache-cassandra-2.1.0.jar:2.1.0]

{code}
Analysis:

When a 2.1.0 node tries to make a connection with a 2.0.0 node, the 2.0.0 node will send an integer representing its version. After receiving this integer, the 2.1.0 node will compare it to the guessed version, and will update the guessed version and shutdown the connection if it is older than the guessed version, then it will make a second trial to connect to the 2.0.0 node. 

However, the 2.1.0 node somehow got stuck at 

https://github.com/apache/cassandra/blob/c6a2c65a75adea9a62896269da98dd036c8e57f3/src/java/org/apache/cassandra/net/OutboundTcpConnection.java#L138 

and was never able to make the second connection trial. And thus the failure.
Bug: During rolling upgrade from C* 2.0.0 to 2.1.0, upgraded node fails with IllegalArgumentException
Status: Resolved
Priority: Normal
Resolution: Won't Fix
Created: 11/Nov/20 22:14
Resolved: 12/Nov/20 19:28

Steps to reproduce: 
 # start a 2-node C* 2.0.0 cluster with 1 seed. 
 # upgrade the non-seed node to 2.1.0
 # run stress testing using /tools/bin/stress on two nodes, wait or it to finish. 

Error stack trace:
{code:java}
java.lang.IllegalArgumentException: No enum constant org.apache.cassandra.config.CFMetaData.Caching.{"keys":"ALL", "rows_per_partition":"NONE"}
  at java.lang.Enum.valueOf(Enum.java:238)
  at org.apache.cassandra.config.CFMetaData$Caching.valueOf(CFMetaData.java:261)
  at org.apache.cassandra.config.CFMetaData.fromSchemaNoColumnsNoTriggers(CFMetaData.java:1567)
  at org.apache.cassandra.config.CFMetaData.fromSchema(CFMetaData.java:1642)
  at org.apache.cassandra.config.KSMetaData.deserializeColumnFamilies(KSMetaData.java:312)
  at org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:309)
  at org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:184)
  at org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:67)
  at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:46)
  at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:57)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
{code}
Root cause: Incompatible data 

In 2.0.0 `CFMetaData.Caching` is an enum of 4 possible values. It is serialized and deserialized using the Java builtin methods `valueOf` and `toString` for enum class. See [https://github.com/apache/cassandra/blob/03045ca22b11b0e5fc85c4fabd83ce6121b5709b/src/java/org/apache/cassandra/config/CFMetaData.java#L1567] (ser) and [https://github.com/apache/cassandra/blob/03045ca22b11b0e5fc85c4fabd83ce6121b5709b/src/java/org/apache/cassandra/config/CFMetaData.java#L1524] (de).

In 2.1.0, `CachingOptions` is a class consisting of 2 fields. It is serialized and deserialized using a textual form similar to JSON (e.g. the string \{"keys":"ALL", "rows_per_partition":"NONE"} in the above log). See [https://github.com/apache/cassandra/blob/c6a2c65a75adea9a62896269da98dd036c8e57f3/src/java/org/apache/cassandra/cache/CachingOptions.java#L95] (ser) and [https://github.com/apache/cassandra/blob/c6a2c65a75adea9a62896269da98dd036c8e57f3/src/java/org/apache/cassandra/cache/CachingOptions.java#L50] (de).

Note that in 2.1.0 it will first check whether the input string is one of the 4 possible values in the old version, and will return a predefined constant for these values, so it is compatible with the old versions. Unfortunately the old versions have no idea about how to deserialize the string from the new versions.
Bug: centos 7 
Status: Resolved
Priority: Normal
Resolution: Invalid
Created: 06/Nov/20 23:44
Resolved: 09/Nov/20 13:47

We are using centos 7. we have an 10 node Cassandra Cluster and currently we are facing issues with one of them with following error:

 

Fatal exception in thread Thread[CompactionExecutor:1,1,main]
 java.io.IOError: java.io.EOFException
  
 or 
  
 java.lang.AssertionError: Added column does not sort as the last colum I already tried to remove the data folder, start scrub and repair. Error happens after 24h again.
  
 Need root cause and workaround for fixed
Bug: Decomissioned nodes are picked for gossip when unreachable nodes are considered for gossiping 
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 01/Oct/20 14:27
Resolved: 22/Oct/20 07:53

After node is decommissioned, it is still considered for gossip via “unreachable” nodes, which results into following exceptions:
 
{code}
INFO  [node4_Messaging-EventLoop-3-3] node4 2020-09-29 16:37:37,527 NoSpamLogger.java:91 - /127.0.0.4:7012->/127.0.0.1:7012-URGENT_MESSAGES-[no-channel] failed to connect
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:7012
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
 {code}

Trace of the method that attempts to establish connection:

{code} 
org.apache.cassandra.net.MessagingService.getOutbound(MessagingService.java:492)
	at org.apache.cassandra.net.MessagingService.doSend(MessagingService.java:335)
	at org.apache.cassandra.net.OutboundSink$Filtered.accept(OutboundSink.java:55)
	at org.apache.cassandra.net.OutboundSink.accept(OutboundSink.java:70)
	at org.apache.cassandra.net.MessagingService.send(MessagingService.java:327)
	at org.apache.cassandra.net.MessagingService.send(MessagingService.java:314)
	at org.apache.cassandra.gms.Gossiper.sendGossip(Gossiper.java:813)
	at org.apache.cassandra.gms.Gossiper.maybeGossipToUnreachableMember(Gossiper.java:840)
	at org.apache.cassandra.gms.Gossiper.access$400(Gossiper.java:86)
 {code}

LEFT and other nodes that are considered dead should not be picked for gossip with unreachable nodes.
Bug: jvm_upgrade_dtests job issue in CircleCI MIDRES
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/Oct/20 20:40
Resolved: 20/Oct/20 21:16

jvm_upgrade_dtests work well in HIGHRES, but we see the following issue with MIDRES:

https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/349/workflows/04bccc52-4e3e-41e2-9c04-93501ea4ce77/jobs/2167/steps
Bug: Rename master branch to trunk in cassandra-harry
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/Oct/20 09:10
Resolved: 20/Oct/20 12:59

ref: https://the-asf.slack.com/archives/CK23JSY2K/p1603178258238100?thread_ts=1603177458.237800&cid=CK23JSY2K
Bug: Synchronize Keyspace instance store/clear
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 14/Oct/20 14:22
Resolved: 20/Oct/20 07:31

DTest failure: dtest-large.repair_tests.repair_test.TestRepairDataSystemTable.test_repair_table (vnodes) - one random failure was reported which pointed to a race condition to be spotted.
Bug: Upgrade the metrics version
Status: Resolved
Priority: Normal
Resolution: Won't Fix
Created: 07/Oct/20 15:29
Resolved: 08/Oct/20 08:31

The current metrics version used by Cassandra is 3.1.5 which was not compiled and targeted for the JDK 8 (https://metrics.dropwizard.io/4.1.2/about/release-notes.html). 
There are several bug fixes that would also be interesting to get.
Bug: PicklingError: Can't pickle <class 'cqlshlib.copyutil.video_encoding'>: attribute lookup video_encoding on cqlshlib.copyutil failed
Status: Triage Needed
Priority: Normal
Created: 09/Oct/20 16:25

This seems to be a different issue than Cassandra-14982 :

$ cqlsh --version
 cqlsh 6.8.0

Following Datastax Academy course DS220, in the Denormalization exercise. I received 
{code:java}
PicklingError: Can't pickle <class 'cqlshlib.copyutil.video_encoding'>: attribute lookup video_encoding on cqlshlib.copyutil failed{code}
on the COPY command into the following table:
{code:java}
CREATE TYPE IF NOT EXISTS video_encoding (
encoding TEXT,
height INT,
width INT,
bit_rates SET<TEXT>
);

CREATE TABLE videos_by_actor (
actor text,
added_date timestamp,
video_id timeuuid,
character_name text,
description text,
encoding frozen<video_encoding>,
tags set<text>,
title text,
user_id uuid,
PRIMARY KEY ( (actor), added_date, video_id )
) WITH CLUSTERING ORDER BY ( added_date desc, video_id asc);

COPY videos_by_actor(actor,added_date,video_id,character_name,description,encoding,tags,title,user_id) FROM 'videos_by_actor.csv' WITH HEADER = true{code}
Now, as it turns out my PRIMARY KEY was non-unique (noted when I failed to load as many records as were in the file), and when I changed to:
{code:java}
 PRIMARY KEY ((actor), added_date, video_id, character_name){code}
the command worked. BUT the following options also worked (though they both dropped records):
{code:java}
 WITH HEADER = true AND MINBATCHSIZE=1 AND MAXBATCHSIZE=1 AND PAGESIZE=10;{code}
and
{code:java}
 WITH HEADER = true AND NUMPROCESSES=1;{code}
So this seems to be a problem of multi-threading and user-defined TYPEs?

I'll note that I'm running inside a WSL2 Docker Container based on ubuntu:bionic:
 $ uname -a
 Linux node-1 4.19.104-microsoft-standard #1 SMP Wed Feb 19 06:37:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Bug: A replacement node, although completed bootstrap and joined ring according to itself, stuck in Joining state as per the peers
Status: Triage Needed
Priority: Normal
Created: 05/Oct/20 16:16

This issue occurred in a production 3.0.21 cluster.

Here is what happened
# We had, say, a three node Cassandra cluster with nodes A, B and C
# C got "terminated by cloud provider" due to health check failure and a replacement node C' got launched.
# C' started bootstrapping data from its neighbors
# Network flaw: Nodes A,B were still able to communicate with terminated node C and consequently still have C as alive.
# The replacement node C' learnt about C through gossip but was unable to communicate with C and marked C as DOWN.
# C' completed bootstrapping successfully and itself and its peers logged this statement "Node C' will complete replacement of C for tokens [-7686143363672898397]"
# C' logged the statement "Nodes C' and C have the same token -7686143363672898397. C' is the new owner"
# C' started listening for thrift and cql clients
# Peer nodes A and B logged "Node C' cannot complete replacement of alive node C "
# A few seconds later, A and B marked C as DOWN

C' continued to log below lines in an endless fashion

{code:java}
Node C is now part of the cluster
Nodes () and C' have the same token C.  Ignoring -7686143363672898397 (Needs a log statement fix)
FatClient C has been silent for 30000ms, removing from gossip
{code}


My reasoning of what happened: 
By the time replacement node (C') finished bootstrapping and announced it's state to Normal, A and B were still able to communicate with the replacing node C (while C' was not able to with C), and hence rejected C' replacing C. C' does not know this and does not attempt to recommunicate its "Normal" state to rest of the cluster. (Worth noting that A and B marked C as down soon after)
Gossip keeps telling C' to add C to its metadata, and C' keeps kicking C out eventually based on FailureDetector. 

Proposed fix:
When C' is notified through gossip about C, and given both own the same token and given C' has finished bootstrapping, C' can emit its Normal state again which should fix this in my opinion (so long as A and B have marked C as DOWN, which they did eventually)

I ended up manually fixing this by restarting Cassandra on C', which forced it to announce its "Normal" state via
StorageService.initServer --> joinTokenRing() --> finishJoiningRing() --> setTokens() --> setGossipTokens()
Alternately, I could have possibly achieved the same behavior if I disabled and enabled gossip via jmx/nodetool.
Bug: Cassandra 4b2 - JVM options from *.options not read/set
Status: Resolved
Priority: Normal
Resolution: Not A Problem
Created: 01/Oct/20 05:11
Resolved: 06/Oct/20 17:39

Trying out Cassandra 4 beta 2 with Java 8 (AdoptOpenJDK) in AWS.
{noformat}
NAME="Amazon Linux AMI"
VERSION="2018.03"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2018.03"
PRETTY_NAME="Amazon Linux AMI 2018.03"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2018.03:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"
{noformat}

It seems the Cassandra JVM results in using Parallel GC.
{noformat}
INFO  [Service Thread] 2020-10-01 00:00:56,233 GCInspector.java:299 - PS Scavenge GC in 541ms.  PS Old Gen: 5152844776 -> 5726724752;
WARN  [Service Thread] 2020-10-01 00:00:56,234 GCInspector.java:297 - PS MarkSweep GC in 1969ms.  PS Eden Space: 2111307776 -> 0; PS Old Gen: 5726724752 -> 2581334376; PS Survivor Space: 363850224 -> 0
{noformat}

Although {{jvm8-server.options}} is using CMS.
{noformat}
#################
#  GC SETTINGS  #
#################

### CMS Settings
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSWaitDuration=10000
-XX:+CMSParallelInitialMarkEnabled
-XX:+CMSEdenChunksRecordAlways
## some JVMs will fill up their heap when accessed via JMX, see CASSANDRA-6541
-XX:+CMSClassUnloadingEnabled
...
{noformat}

In Cassandra 3, default has been CMS.

So, possibly there is something wrong in reading/processing {{jvm8-server.options}}?
Bug: Add droppable tombstone metrics to nodetool tablestats
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 02/Dec/20 13:43
Resolved: 03/Jan/22 15:00

This is a useful metric to troubleshoot tombstone cleanup problems and is not currently exposed on table stats.

While we're at it we should add the JMX metric to the table metrics documentation because it's currently missing.