av9ash/GitBugs huggingface dataset:
Bug: Standardize tablestats formatting and data units
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Nov/23 17:19
Resolved: 05/Jan/24 13:36

Tablestats reports output in plaintext, JSON or YAML. The human readable output currently has a mix of KiB, bytes with inconsistent spacing

Simplify and default output to 'human readable'. Machine readable output is available as an option and the current mixed output formatting is neither friendly for human or machine reading and can be replaced.

!image-2023-11-27-13-49-14-247.png!

*Not a goal now (consider a follow up Jira):*

Fix inconsistencies with KiB/MiB/GiB and KB/MB/GB formatting:
 * gcstats - uses MB
 * getcompactionthroughput - uses MB/s
 * getstreamthroughput - uses MB/s
 * info - uses MiB/GiB
Bug: Test Failure: org.apache.cassandra.index.sai.cql.VectorSegmentationTest.testMultipleSegmentsForCompaction
Status: Open
Priority: Normal
Created: 25/Nov/23 11:25

Seen in j11_utests_compression in CASSANDRA-19034
https://app.circleci.com/pipelines/github/michaelsembwever/cassandra/259/workflows/f343d3e3-00cf-4e13-bb4d-bbfff1d3658c/jobs/21135/tests

{noformat}
junit.framework.AssertionFailedError: 
Expecting actual:
  0.9666666666666667
to be greater than or equal to:
  0.99

	at org.apache.cassandra.index.sai.cql.VectorSegmentationTest.testMultipleSegmentsForCompaction(VectorSegmentationTest.java:107)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
{noformat}
Bug: Fix ClusterMetadataUpgradeHarryTest
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 19/Dec/23 13:37
Resolved: 22/Jan/24 14:03

Unfortunately I do not have a stack trace handy, but the test was essentially timing out after CMS was falling into an infinite loop during initialisation. A problem was caused by the fact that if there is a paxos timeout during propose phase, CMS would get stuck in an infinite loop trying to catch up, and not finding anything to catch up from.
Bug: Enhanced usage of test method names in CQLTester for better test debugging
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 22/Nov/23 08:08
Resolved: 18/Jan/24 06:16

All views, tables, ks,... created in CQLTester are of the form {{table_00}}:
- While debugging tests and flakies this makes it really hard to match logs to test methods.
- Some async operations can spill log lines from a previous test method into the next's log lines which is incredibly confusing
- It's hard on the eyes and easy to mix up
- When comparing logs from 2 different branches, envs,... it's really hard to match logs

The proposed solution is for {{CQLTester}} to decorate these with the test name.
Bug: Test Failure: 	SSTableCorruptionDetectionTest.testSinglePartitionIterator
Status: Resolved
Priority: Normal
Resolution: Cannot Reproduce
Created: 04/Dec/23 23:58
Resolved: 08/Jan/24 14:32

[https://ci-cassandra.apache.org/job/Cassandra-trunk/1785/testReport/org.apache.cassandra.io.sstable/SSTableCorruptionDetectionTest/testSinglePartitionIterator_cassandra_testtag_IS_UNDEFINED/]
h3.  
{code:java}
Error Message
Timeout occurred. Please note the time in the report does not reflect the time until the timeout.

Stacktrace
junit.framework.AssertionFailedError: Timeout occurred. Please note the time in the report does not reflect the time until the timeout. at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.util.Vector.forEach(Vector.java:1365) at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.util.Vector.forEach(Vector.java:1365) at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at jdk.internal.reflect.GeneratedMethodAccessor49.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.util.Vector.forEach(Vector.java:1365) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at org.apache.cassandra.anttasks.TestHelper.execute(TestHelper.java:53) at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.util.Vector.forEach(Vector.java:1365) at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
{code}
This test has timedout many times on trunk these days (5 out of 7), I found the same failure only once on 5.0 in Jenkins
[https://ci-cassandra.apache.org/job/Cassandra-5.0/116/testReport/org.apache.cassandra.io.sstable/SSTableCorruptionDetectionTest/testSinglePartitionIterator_cassandra_testtag_IS_UNDEFINED/]
Bug: Expose CDC via CQL
Status: Triage Needed
Priority: Normal
Created: 03/Nov/23 20:59

It looks like there was originally a plan to expose change data externally via CQL for any client to consume, as noted under Design Options #4 on the original design document: [https://docs.google.com/document/d/1ZxCWYkeZTquxsvf5hdPc0fiUnUHna8POvgt6TIzML4Y/edit?usp=sharing.]

 

It appears that this was ultimately not implemented. Is there a plan to improve CDC in the future with this functionality? The current implementation is tedious to use - you have to write a local agent that lives on every single c* node and then implement the Apache-provided CommitLogReader in order to parse the files. On top of that, it makes deduplication and ordering basically impossible if there isn't an inherently ordered property.
Bug: [Analytics] Column type mapping error for timestamp type during bulk writes
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Dec/23 19:41
Resolved: 10/Jan/24 22:57

When doing bulk reads with the analytics library, a user can specify the last modified column as an option. Bulk reader will add a column with the last modified column to the data frame. If a user wants to use the bulk-read data frame to persist data, and using the {{WriterOptions.TIMESTAMP}} feature from the last modified column from the bulk-read data frame, the bulk write will fail with a data type mapping error.

{code:java}
Caused by: java.lang.RuntimeException: Unsupported conversion for LONG from java.sql.Timestamp
	at org.apache.cassandra.spark.bulkwriter.SqlToCqlTypeConverter$LongConverter.convertInternal(SqlToCqlTypeConverter.java:245)
	at org.apache.cassandra.spark.bulkwriter.SqlToCqlTypeConverter$LongConverter.convertInternal(SqlToCqlTypeConverter.java:231)
	at org.apache.cassandra.spark.bulkwriter.SqlToCqlTypeConverter$Converter.convert(SqlToCqlTypeConverter.java:203)
	at org.apache.cassandra.spark.bulkwriter.SqlToCqlTypeConverter$NullableConverter.convert(SqlToCqlTypeConverter.java:212)
	at org.apache.cassandra.spark.bulkwriter.TableSchema.normalize(TableSchema.java:91)
	at org.apache.spark.api.java.JavaPairRDD$.$anonfun$toScalaFunction$1(JavaPairRDD.scala:1070)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
{code}
Bug: Test Failure: org.apache.cassandra.transport.CQLUserAuditTest
Status: Open
Priority: Normal
Created: 05/Dec/23 00:38

There are a number of tests in this test class that were failing, there might be dependency between them as they failed in the same run. 
- loginSuccessfulTest - [https://ci-cassandra.apache.org/job/Cassandra-trunk/1802/testReport/org.apache.cassandra.transport/CQLUserAuditTest/loginSuccessfulTest_trie_jdk17_x86_64/]
- prepareStmt - [https://ci-cassandra.apache.org/job/Cassandra-trunk/1802/testReport/org.apache.cassandra.transport/CQLUserAuditTest/prepareStmt_trie_jdk17_x86_64/]
- queryBatch - [https://ci-cassandra.apache.org/job/Cassandra-trunk/1802/testReport/org.apache.cassandra.transport/CQLUserAuditTest/queryBatch_trie_jdk17_x86_64/]
- queryInsert - [https://ci-cassandra.apache.org/job/Cassandra-trunk/1802/testReport/org.apache.cassandra.transport/CQLUserAuditTest/queryInsert_trie_jdk17_x86_64/]

- querySimpleSelect - https://ci-cassandra.apache.org/job/Cassandra-trunk/1802/testReport/org.apache.cassandra.transport/CQLUserAuditTest/querySimpleSelect_trie_jdk17_x86_64/
Bug: Fix flaky test accord.utils.RandomSourceTest.testBiasedInts
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 28/Nov/23 23:26
Resolved: 05/Jan/24 17:28

This test fails from time to time as binary search finds the first match it finds and not the first in the array.  This class is also problematic as it hand rolls property test framework but does so in a way where seeds may be lost or attributed to the wrong test.
Bug: Upgrade owasp to 9.0.x
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 01/Dec/23 16:54
Resolved: 20/Dec/23 12:17

From https://github.com/jeremylong/DependencyCheck :

{quote}
Upgrading to 9.0.0 or later is mandatory; previous versions of dependency-check utilize the NVD data feeds which will be deprecated on Dec 15th, 2023. 
{quote}
Bug: Test failure: org.apache.cassandra.distributed.test.MigrationCoordinatorTest.explicitEndpointIgnore
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Oct/23 10:49
Resolved: 22/Dec/23 11:11

Repeated run from `cassandra-4.1` [https://app.circleci.com/pipelines/github/jacek-lewandowski/cassandra/941/workflows/46fc6cb7-135e-4862-b9d3-6996c0993de8]
Bug: Test failure: org.apache.cassandra.simulator.test.ShortPaxosSimulationTest.simulationTest
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 19/Oct/23 12:08
Resolved: 30/Nov/23 15:05

The simulator test {{org.apache.cassandra.simulator.test.ShortPaxosSimulationTest#simulationTest}} is flaky on 5.0 and trunk:
* https://app.circleci.com/pipelines/github/mike-tr-adamson/cassandra/332/workflows/946e28f4-2dec-4384-ac38-de011093f6c6/jobs/25735/tests
* https://app.circleci.com/pipelines/github/adelapena/cassandra/3253/workflows/e48b49e9-cf36-412a-a811-d813031e6f01/jobs/83735/tests
* https://app.circleci.com/pipelines/github/adelapena/cassandra/3254/workflows/69f451ef-fb39-48e4-b1d1-40ee4141b0c1/jobs/83739/tests

{code}
org.apache.cassandra.simulator.SimulationException: Failed on seed 0xb01206bb3b021127
Caused by: java.lang.ClassCastException: class org.apache.cassandra.net.NoPayload cannot be cast to class org.apache.cassandra.gms.GossipShutdown (org.apache.cassandra.net.NoPayload and org.apache.cassandra.gms.GossipShutdown are in unnamed module of loader org.apache.cassandra.distributed.shared.InstanceClassLoader @68801feb)
	at org.apache.cassandra.gms.GossipShutdown$Serializer.serialize(GossipShutdown.java:41)
	at org.apache.cassandra.net.Message$Serializer.serialize(Message.java:722)
	at org.apache.cassandra.distributed.impl.Instance.serializeMessage(Instance.java:427)
	at org.apache.cassandra.distributed.impl.Instance.lambda$registerOutboundFilter$5(Instance.java:370)
	at org.apache.cassandra.net.OutboundSink$Filtered.accept(OutboundSink.java:54)
	at org.apache.cassandra.net.OutboundSink.accept(OutboundSink.java:70)
	at org.apache.cassandra.net.MessagingService.send(MessagingService.java:449)
	at org.apache.cassandra.net.MessagingService.send(MessagingService.java:419)
	at org.apache.cassandra.gms.Gossiper.unsafeSendShutdown(Gossiper.java:2634)
	at org.apache.cassandra.simulator.cluster.OnInstanceSendShutdown.lambda$invokableSendShutdown$1(OnInstanceSendShutdown.java:48)
	at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
	at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
	at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
	at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
	at org.apache.cassandra.concurrent.SyncFutureTask$1.call(SyncFutureTask.java:46)
	at org.apache.cassandra.concurrent.SyncFutureTask.run(SyncFutureTask.java:68)
	at org.apache.cassandra.simulator.systems.InterceptingExecutor$AbstractSingleThreadedExecutorPlus.lambda$new$0(InterceptingExecutor.java:584)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)
{code}
The test failure can easily be reproduced on CircleCI with:
{code}
.circleci/generate.sh -sp \
  -e REPEATED_SIMULATOR_DTESTS=org.apache.cassandra.simulator.test.ShortPaxosSimulationTest \
  -e REPEATED_SIMULATOR_DTESTS_COUNT=100
{code}
Flakiness seems ~18%.

The test failure is not reported on Butler because simulator tests are not run by Jenkins.
Bug: Unified Compaction Strategy is missing documentation
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Nov/23 15:42
Resolved: 04/Jan/24 01:27

UCS is missing from [the CQL documentation for 5.0|https://cassandra.apache.org/doc/5.0/cassandra/developing/cql/ddl.html#cql-compaction-options] and [the compaction page|https://cassandra.apache.org/doc/5.0/cassandra/managing/operating/compaction/index.html#compaction-options].

We need to create a documentation page for UCS and link it from both.
Bug: Multiple on-disk index versions on a single SSTable are not handled correctly
Status: Triage Needed
Priority: Normal
Created: 22/Dec/23 12:48

The SAI versioning does not consider having multiple on-disk per-column index versions against a single SSTable. 

The IndexDescriptor handles the versioning of all the index components for an SSTable but does not recognise multiple on-disk per-column versions.

The IndexDescriptor needs to look at all on-disk components and maintain a mapping of component to version.

NOTE: This is a precursor ticket to any ticket involving an on-disk version change for 5.x
Bug: Inquiry and Clarification on Addition of java-cup-runtime-11b-20160615.jar in Cassandra 4.0.10 Upgrade
Status: Triage Needed
Priority: Normal
Created: 19/Dec/23 03:37

During the recent upgrade from Cassandra 3 to Cassandra 4.0.10, I noticed the introduction of a GPL-Compatible dependency, specifically "java-cup-runtime-11b-20160615.jar." Upon further investigation, the declaration of this dependency seems to add an extra layer of complexity. While I understand that introducing GPL-Compatible dependencies into Apache is generally acceptable, the necessity of this declaration and whether it is an additional step has raised concerns.

!image-2023-12-19-13-44-16-660.png|width=841,height=196!

!image-2023-12-19-11-43-19-071.png|width=611,height=373!
[This LINCENSE|https://www2.cs.tum.edu/projects/cup/licence.php] refer to "Historical Permission Notice and Disclaimer"
!image-2023-12-19-11-45-06-345.png|width=556,height=430!

This uncommon license falls under Category A.

!image-2023-12-19-11-45-18-681.png|width=490,height=478!
Additionally, I observed that some portions of the code are explicitly marked as GPL (①). However, in the README file on the project's homepage, there is a clear indication of the licenses for different distribution packages. I believe we should reference Protocol ② or ③, namely the Historical protocol.

!image-2023-12-19-11-45-30-576.png|width=523,height=308!

I would like to bring attention to these two issues to ensure that we adhere to the correct protocols when using and distributing the code. Please consider explicitly stating in the relevant documentation or instructions that we are using the Historical protocol.

Thank you for your attention to these issues. I believe addressing them will contribute to ensuring the legality of our code usage and distribution.
Bug: A multi-line comment containing semi colon as the end of line throws error
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 15/Dec/23 01:18
Resolved: 02/Jan/24 17:00

{{Tested this with C* 4.0.latest.}}

Let's say I've a comment section as follows, it fails today,

 
{quote}{{/*}}
{{ Licensed under the Apache License, Version 2.0 (the "License");}}
{{ you may not use this file except in compliance with the License.}}
{{ You may obtain a copy of the License at}}
{{ [http://www.apache.org/licenses/LICENSE-2.0]}}
{{ Unless required by applicable law or agreed to in writing, software}}
{{ distributed under the License is distributed on an "AS IS" BASIS,}}
{{ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.}}
{{ See the License for the specific language governing permissions and}}
{{ limitations under the License.}}
{{*/}}

{{SELECT * FROM origin.table_a;}}
{quote}
which throws an exception as follows when execute it as,

{quote}
cqlsh -f test.cql
{quote}
 
error,
{quote}/tmp/test.cql:3:SyntaxException: line 2:64 mismatched character '<EOF>' expecting '*'
/tmp/test.cql:14:SyntaxException: line 1:0 no viable alternative at input 'you' ([you]...)
{quote}
This is because of having a semi-colon at the end of the [1st] line. If that same line ended as follows,
{quote}Version 2.0 (the "License"); you
{quote}
This error will not occur.
Bug: ArrayIndexOutOfBoundsException seen in mixed mode cluster during upgrade
Status: Triage Needed
Priority: Normal
Created: 29/Dec/23 11:41

I am trying to upgrade Cassandra 3.11.13 cluster to 4.1.3, and during the upgrade observed following Exception on 3.11.x nodes.

 
{code:java}
ERROR [MessagingService-Incoming-/X.X.X.X] 2023-12-28 07:03:18,940 CassandraDaemon.java:244 - Exception in thread Thread[MessagingService-Incoming-/X.X.X.X,5,main] 
java.lang.ArrayIndexOutOfBoundsException: 29
     at 
org.apache.cassandra.gms.EndpointStateSerializer.deserialize(EndpointState.java:220)
     at 
org.apache.cassandra.gms.EndpointStateSerializer.deserialize(EndpointState.java:191)
     at 
org.apache.cassandra.gms.GossipDigestAck2Serializer.deserialize(GossipDigestAck2.java:73)
     at 
org.apache.cassandra.gms.GossipDigestAck2Serializer.deserialize(GossipDigestAck2.java:52)
     at 
org.apache.cassandra.net.MessageIn.read(MessageIn.java:123)
     at 
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:207)
     at 
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:195)
     at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:98)  {code}

There is a difference between the ApplicationStates Enum in between 3.11 & 4.1, with the former containing 25 states only. However, I think that padding state shouldn't be used by the 4.1 during gossip exchange. This only happens when a Gossip is initiated by a 4.1 node to a 3.11 node, and 3.11 fails to send a GOSSIP_DIGEST_ACK.


{code:java}
GossipDigestSynVerbHandler.java:110 - Sending a GossipDigestAckMessage to /X.X.X.X 
TRACE [****] [GossipStage:1] 2023-12-28 05:21:45,620 MessagingService.java:942 - Y.Y.Y.Y sending GOSSIP_DIGEST_ACK to 731645@/X.X.X.X 
TRACE [****] [MessagingService-Incoming-/X.X.X.X] 2023-12-28 05:21:45,645 IncomingTcpConnection.java:125 - Closing socket Socket[addr=/X.X.X.X,port=51518,localport=7001] - isclosed: false
ERROR [****] [MessagingService-Incoming-/X.X.X.X] 2023-12-28 05:21:45,645 CassandraDaemon.java:244 - Exception in thread Thread[MessagingService-Incoming-/X.X.X.X,5,main] 
java.lang.ArrayIndexOutOfBoundsException: 29 {code}
Bug: Remove pre-4.0 compatibility code for 5.0
Status: Triage Needed
Priority: Normal
Created: 28/Dec/23 15:40

This is an umbrella ticket to discuss removing pre-4.0 compatibility code from 5.0, similar to CASSANDRA-12716 for 4.x.

A few considerations:
- Discuss/ratify removal of pre-compatibility code on dev mailing list
- What compatibility features are being removed?
- What upgrade tests are being removed ? Are they still relevant and can be reused?
- Should upgrade from 3.x to 5.X fail on startup with an override flag?
- Can/should we make it easier to deprecate/remove compatibility code for future major releases?
Bug: Histogram overflow causes client timeouts and message drops
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 24/Nov/23 11:45
Resolved: 27/Dec/23 16:15

Hi,

We have recently noticed that sometimes this exception happens on our Cassandra cluster: 
{code:java}
ERROR [ScheduledTasks:1] 2023-11-24 06:24:12,680 CassandraDaemon.java:244 - Exception in thread Thread[ScheduledTasks:1,5,main]
java.lang.IllegalStateException: Unable to compute when histogram overflowed
        at org.apache.cassandra.metrics.DecayingEstimatedHistogramReservoir$EstimatedHistogramReservoirSnapshot.getMean(DecayingEstimatedHistogramReservoir.java:472)
        at org.apache.cassandra.net.MessagingService.getDroppedMessagesLogs(MessagingService.java:1272)
        at org.apache.cassandra.net.MessagingService.logDroppedMessages(MessagingService.java:1244)
        at org.apache.cassandra.net.MessagingService.access$200(MessagingService.java:84)
        at org.apache.cassandra.net.MessagingService$4.run(MessagingService.java:512)
        at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
        at java.lang.Thread.run(Thread.java:750)
 {code}
It happens on all 6 nodes at the same time. Also we see increased client timeouts and dropped READ and READ_RESPONSE messages. Our Cassandra is 3.11.16, 2 DC setup, 6 node in each DC. RF is 3. I have searched issues but could not find exactly same issue causing messages to be dropped. Any suggestion would be appreciated.