av9ash/GitBugs huggingface dataset:
Bug: Improve operator tooling for recovering the cluster metadata log
Status: Triage Needed
Priority: Normal
Created: 04/Dec/23 13:12

There are a handful of ways that the CMS can become permanently unavailable, in which case operators will need to intervene to recover it.
This may be due to misconfiguration, for example only running with a single instance or having all CMS members in the same DC and losing that DC.


In these cases, we need to be able to manually promote a non-CMS instance to the CMS, something like:


1. disable replication everywhere in the cluster to avoid interference from for example a flapping CMS instance
2. dump cluster metadata to a file, modify this cluster metadata to make one of the live instances in the cluster the CMS
3. stop the instance from #1, start with -Dcassandra.unsafe_boot_with_cluster_metadata=/tmp/cm.dump


Perhaps an offline tool to dump/export cluster metadata that could work by just reading from the local table without actually starting the node itself. Such a tool should probably take an epoch param, so we could add a mode/other tool to list the local log table in order to select that epoch and then export it so it can be used to force the cluster to a desired state. Something along the lines of:
> bin/show_metadata_log


{code}
0 PRE_INIT
1 INITIALIZE
2 UNSAFE_JOIN 127.0.0.1
3 UNSAFE_JOIN 127.0.0.2
...

> bin/reconstruct_metadata --epoch 2 --output /tmp/recovery.dump


> bin/cassandra -Dcassandra.boot_with_metadata=/tmp/recovery.dump


{code}
Bug: Test failure: org.apache.cassandra.distributed.test.log.InProgressSequenceCoordinationTest:rejectSubsequentInProgressSequence
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Nov/23 10:12
Resolved: 04/Dec/23 13:05

Circleci Failure: https://app.circleci.com/pipelines/github/michaelsembwever/cassandra/256/workflows/c4fda8f1-a8d6-4523-be83-5e30b9de39fe/jobs/20500/tests

Also fais on jdk11: https://app.circleci.com/pipelines/github/michaelsembwever/cassandra/256/workflows/c4fda8f1-a8d6-4523-be83-5e30b9de39fe/jobs/20464/tests
{code}
 junit.framework.AssertionFailedError: Timeout occurred. Please note the time in the report does not reflect the time until the timeout.
 at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.util.Vector.forEach(Vector.java:1365)
 at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.util.Vector.forEach(Vector.java:1365)
 at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at jdk.internal.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.util.Vector.forEach(Vector.java:1365)
 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at org.apache.cassandra.anttasks.TestHelper.execute(TestHelper.java:53)
 at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.util.Vector.forEach(Vector.java:1365)
 at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
{code}
Bug: User documentation for CEP-21
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 24/Nov/23 10:20
Resolved: 02/Dec/23 13:48

The initial implementation contains some minimal docs in-tree (TransactionalClusterMetadata.md & TCM_implementation.md).

The CEP doc can be found at [https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-21%3A+Transactional+Cluster+Metadata]
Bug: Fix SAI statement
Status: Resolved
Priority: Normal
Resolution: Won't Do
Created: 01/Dec/23 20:38
Resolved: 01/Dec/23 21:10

Hi docs. There appears to be an error in the UDT section on the page for SAI queries.
[!https://www.datastax.com/favicon.ico!Query with SAI :: Cassandra Query Language (CQL)|https://docs.datastax.com/en/cql/astra/docs/developing/indexing/sai/sai-query.html#user-defined-type] The doco states "{_}SAI can index either a single field of a user-defined type (UDT) or a list of UDTs. This example shows how to index a list of UDTs.{_}"
I think the doco is trying to say that you can index a UDT or a list of UDTs, but it reads as though I can index a field within the UDT.Could we improve the doco to read:
"{_}SAI can index a user-defined type (UDT) or a list of UDTs. This example shows how to index a list of UDTs.{_}"
Bug: Describe statement may include inconsistent schema and schema version for paging
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 11/Oct/23 06:42
Resolved: 31/Oct/23 10:24

When the {{DescribeStatement}} is executed, it initially gets the current schema snapshot and the schema version with two separate unsynchronized calls. When the schema is being modified around that time, the schema snapshot and schema version may diverge which will result in failing or inconsistent paging. This is not super important but it is easy to fix.
Bug: Remove unused suppressions
Status: Resolved
Priority: Normal
Resolution: Later
Created: 30/Nov/23 00:43
Resolved: 01/Dec/23 17:27

After CASSANDRA-18875 we have some unused suppressions that can be removed:

{quote}
[dependency-check] Suppression Rule had zero matches: SuppressionRule{packageUrl=PropertyType{value=^pkg:maven/org\.yaml/snakeyaml@.*$, regex=true, caseSensitive=false},cve={CVE-2023-2251,CVE-2022-25857,CVE-2022-38749,CVE-2022-38750,CVE-2022-38751,CVE-2022-38752,CVE-2022-41854,CVE-2022-1471,CVE-2022-3064,CVE-2021-4235,CVE-2017-18640,}}
[dependency-check] Suppression Rule had zero matches: SuppressionRule{packageUrl=PropertyType{value=^pkg:maven/com\.fasterxml\.jackson\.core/jackson\-databind@.*$, regex=true, caseSensitive=false},cve={CVE-2022-42003,CVE-2022-42004,CVE-2023-35116,}}
{quote}
Bug: Test failure: org.apache.cassandra.distributed.test.ClearSnapshotTest#clearSnapshotSlowTest-_jdk11
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Nov/23 10:07
Resolved: 01/Dec/23 15:34

CircleCI failure: https://app.circleci.com/pipelines/github/michaelsembwever/cassandra/256/workflows/c4fda8f1-a8d6-4523-be83-5e30b9de39fe/jobs/20464/tests

Also on jdk17: https://app.circleci.com/pipelines/github/michaelsembwever/cassandra/256/workflows/c4fda8f1-a8d6-4523-be83-5e30b9de39fe/jobs/20500/tests

```
java.lang.RuntimeException: java.util.concurrent.TimeoutException
	at org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:79)
	at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:537)
	at org.apache.cassandra.distributed.impl.AbstractCluster.close(AbstractCluster.java:1098)
	at org.apache.cassandra.distributed.test.ClearSnapshotTest.clearSnapshotSlowTest(ClearSnapshotTest.java:124)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Caused by: java.util.concurrent.TimeoutException
	at org.apache.cassandra.utils.concurrent.AbstractFuture.get(AbstractFuture.java:253)
	at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:529)
	Suppressed: java.util.concurrent.TimeoutException
```
Bug: Test failure: org.apache.cassandra.distributed.test.log.RegisterTest#serializationVersionDisagreementTest-_jdk17
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Nov/23 10:09
Resolved: 01/Dec/23 14:51

Circleci Failure: [https://app.circleci.com/pipelines/github/michaelsembwever/cassandra/256/workflows/c4fda8f1-a8d6-4523-be83-5e30b9de39fe/jobs/20502/tests]

{code}
org.apache.cassandra.distributed.test.log.RegisterTest.lambda$serializationVersionDisagreementTest$81c80a4a$2(RegisterTest.java:132)
at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.NoSuchFieldException: modifiers
at java.base/java.lang.Class.getDeclaredField(Class.java:2610)
at org.apache.cassandra.distributed.test.log.RegisterTest.lambda$serializationVersionDisagreementTest$81c80a4a$2(RegisterTest.java:120)
{code}
Bug: Test Failure: org.apache.cassandra.db.TopPartitionTrackerTest.testRanges
Status: Triage Needed
Priority: Normal
Created: 01/Dec/23 12:31

Seen in j17_utests_oa in CASSANDRA-19011
https://app.circleci.com/pipelines/github/mike-tr-adamson/cassandra/412/workflows/e5e1311f-f9c3-4796-a857-f789dd3c76d1/jobs/36955/tests

{noformat}
junit.framework.AssertionFailedError
	at org.apache.cassandra.db.TopPartitionTrackerTest.testRanges(TopPartitionTrackerTest.java:285)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
{noformat}
Bug: Test Failure: org.apache.cassandra.distributed.upgrade.MixedModeFrom3LoggedBatchTest.testSimpleStrategy-_jdk11
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Nov/23 09:57
Resolved: 01/Dec/23 12:04

Failed in Circle:

[https://app.circleci.com/pipelines/github/michaelsembwever/cassandra/256/workflows/c4fda8f1-a8d6-4523-be83-5e30b9de39fe/jobs/20534/tests]
Bug: Make roles setup a synchronous task
Status: Triage Needed
Priority: Normal
Created: 01/Dec/23 08:36

Right now, roles set up is (while asynchronous), is schedule / delay-based. In other words, there is a task that wakes up every so often and checks if the node is joined, and attempts to run auth setup. But with TCM, we can actually be certain about whether or not we are done with setup, and can deterministically run this task.This is a problem only for simulation (that said, does not cause any failures), but can still be a good general improvement.
Bug: Audit table properties passed through Analytics CqlUtils
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Nov/23 20:27
Resolved: 30/Nov/23 05:37

In mixed version clusters, table properties (`WITH` clauses in the table definition) have historically been moved or renamed without much thought to backward-compatibility of external tools. This has caused breaking changes to the Analytics library in the past. Therefore, the Analytics library removes most table properties from the table schema it uses in order to avoid these issues. However, it is currently too aggressive in this removal process.  For example, the `default_time_to_live` parameter is currently removed from the table options, but the Bulk Writer should use this to set the appropriate TTL when the user doesn’t specify it in the bulk job.

Similarly, compression options passed to the Bulk Writer should be honored so that the appropriate compression algorithm is used when generating sstables.

The allowlist of options needs to be audited and improved to add additional options where appropriate.
Bug: Add metrics and logging to repair retries
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/Oct/23 21:24
Resolved: 30/Oct/23 21:49

In CASSANDRA-18816 we added repair message retries but forgot to add logging and metrics to actually let operators know repairs happened and how often… we should add such visibility to help operators know how best to tune it for their environment.
Bug: Test Failure: org.apache.cassandra.distributed.upgrade.ClusterMetadataUpgradeTest.upgradeWithHintsTest-_jdk11
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Nov/23 09:54
Resolved: 30/Nov/23 12:13

Fails in circle with OOM

[https://app.circleci.com/pipelines/github/michaelsembwever/cassandra/256/workflows/c4fda8f1-a8d6-4523-be83-5e30b9de39fe/jobs/20534/tests]
Bug: Test failure: org.apache.cassandra.db.lifecycle.ViewTest
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 24/Nov/23 09:53
Resolved: 27/Nov/23 07:07

Fails on both j17_unit_tests and j17_utests_cdc, same error as CASSANDRA-19059
{code}
FSWriteError in build/test/cassandra/data/system
	at org.apache.cassandra.io.util.PathUtils.propagateUnchecked(PathUtils.java:862)
	at org.apache.cassandra.io.util.PathUtils.propagateUnchecked(PathUtils.java:845)
	at org.apache.cassandra.io.util.PathUtils.deleteRecursiveUsingNixCommand(PathUtils.java:384)
	at org.apache.cassandra.io.util.PathUtils.deleteRecursive(PathUtils.java:402)
	at org.apache.cassandra.io.util.File.deleteRecursive(File.java:225)
	at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:678)
	at org.apache.cassandra.schema.MockSchema.cleanup(MockSchema.java:377)
	at org.apache.cassandra.db.lifecycle.ViewTest.setUp(ViewTest.java:58)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Caused by: java.io.IOException: [rm, -rd, /tmp/cassandra/build/test/cassandra/data/system] returned non-zero exit code: 1
stdout:


stderr:
rm: cannot remove '/tmp/cassandra/build/test/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3': Directory not empty
	at org.apache.cassandra.io.util.PathUtils.deleteRecursiveUsingNixCommand(PathUtils.java:377)
{code}

https://app.circleci.com/pipelines/github/michaelsembwever/cassandra/256/workflows/c4fda8f1-a8d6-4523-be83-5e30b9de39fe/jobs/20505/tests
Bug: Test Failure: org.apache.cassandra.distributed.upgrade.ClusterMetadataUpgradeTest.simpleUpgradeTest-_jdk11
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 24/Nov/23 09:55
Resolved: 28/Nov/23 09:23

Failed in Circle with OOM

[https://app.circleci.com/pipelines/github/michaelsembwever/cassandra/256/workflows/c4fda8f1-a8d6-4523-be83-5e30b9de39fe/jobs/20534/tests]
Bug: repair_admin summarize-pending command is throwing OpenDataException
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 09/Nov/23 13:09
Resolved: 29/Nov/23 17:46

Execution of repair_admin summarize-pending is throwing the following exception:
{code:java}
error: Item names do not match CompositeType: names in items but not in CompositeType: []; names in CompositeType but not in items: [failed]
-- StackTrace --
javax.management.openmbean.OpenDataException: Item names do not match CompositeType: names in items but not in CompositeType: []; names in CompositeType but not in items: [failed]
        at javax.management.openmbean.CompositeDataSupport.<init>(CompositeDataSupport.java:220)
        at javax.management.openmbean.CompositeDataSupport.<init>(CompositeDataSupport.java:176)
        at org.apache.cassandra.repair.consistent.admin.PendingStats.toComposite(PendingStats.java:86)
        at org.apache.cassandra.service.ActiveRepairService.getPendingStats(ActiveRepairService.java:296)
        at org.apache.cassandra.tools.nodetool.RepairAdmin$SummarizePendingCmd.execute(RepairAdmin.java:112)
        at org.apache.cassandra.tools.NodeTool$NodeToolCmd.runInternal(NodeTool.java:358)
        at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:343)
        at org.apache.cassandra.tools.NodeTool.execute(NodeTool.java:246)
        at org.apache.cassandra.distributed.impl.Instance$DTestNodeTool.execute(Instance.java:996)
        at org.apache.cassandra.distributed.impl.Instance.lambda$nodetoolResult$40(Instance.java:906)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:750) {code}
The bug is caused by the fact that the list of fields at [https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/repair/consistent/admin/PendingStats.java#L78-L82] does not align with [https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/repair/consistent/admin/PendingStats.java#L35.]
Bug: Test Failure: cql_tracing_test.TestCqlTracing.test_tracing_simple
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 25/Nov/23 07:45
Resolved: 29/Nov/23 16:28

Seen in cqlsh_dtests in CASSANDRA-19034
https://app.circleci.com/pipelines/github/michaelsembwever/cassandra/257/workflows/ddcb5f4e-e2c5-430b-b922-f7064a244971/jobs/20648/tests

{noformat}failed on teardown with "Failed: Unexpected error found in node logs (see stdout for full details). Errors: [[node3] 'ERROR [main] 2023-11-24 19:02:04,201 FailureDetector.java:309 - Unknown endpoint: /127.0.0.1:7000
java.lang.IllegalArgumentException: Unknown endpoint: /127.0.0.1:7000
    at org.apache.cassandra.gms.FailureDetector.isAlive(FailureDetector.java:309)
    at org.apache.cassandra.tcm.RemoteProcessor$CandidateIterator.computeNext(RemoteProcessor.java:326)
    at org.apache.cassandra.tcm.RemoteProcessor$CandidateIterator.computeNext(RemoteProcessor.java:256)
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
    at org.apache.cassandra.tcm.RemoteProcessor$1Request.retry(RemoteProcessor.java:187)
    at org.apache.cassandra.tcm.RemoteProcessor.sendWithCallbackAsync(RemoteProcessor.java:223)
    at org.apache.cassandra.tcm.RemoteProcessor.sendWithCallback(RemoteProcessor.java:170)
    at org.apache.cassandra.tcm.RemoteProcessor.commit(RemoteProcessor.java:76)
    at org.apache.cassandra.tcm.ClusterMetadataService$SwitchableProcessor.commit(ClusterMetadataService.java:837)
    at org.apache.cassandra.tcm.Processor.commit(Processor.java:45)
    at org.apache.cassandra.tcm.ClusterMetadataService.commit(ClusterMetadataService.java:502)
    at org.apache.cassandra.tcm.ClusterMetadataService.commit(ClusterMetadataService.java:467)
    at org.apache.cassandra.tcm.transformations.Register.register(Register.java:112)
    at org.apache.cassandra.tcm.transformations.Register.register(Register.java:95)
    at org.apache.cassandra.tcm.transformations.Register.register(Register.java:132)
    at org.apache.cassandra.tcm.transformations.Register.maybeRegister(Register.java:89)
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:807)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:367)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:728)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:879)']"
{noformat}
Bug: Test failure: j11_dtests_large.replace_address_test.TestReplaceAddress
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Nov/23 10:03
Resolved: 29/Nov/23 16:25

CircleCI failure: https://app.circleci.com/pipelines/github/michaelsembwever/cassandra/256/workflows/c4fda8f1-a8d6-4523-be83-5e30b9de39fe/jobs/20466/tests


{code}
assert 0 == 1
 +  where 0 = len(set())
self = <replace_address_test.TestReplaceAddress object at 0x7f390285a8d0>

    @pytest.mark.resource_intensive
    def test_replace_first_boot(self):
>       self._test_replace_node(jvm_option='replace_address_first_boot')

replace_address_test.py:281: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
replace_address_test.py:300: in _test_replace_node
    previous_log_size = self._verify_tokens_migrated_successfully()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <replace_address_test.TestReplaceAddress object at 0x7f390285a8d0>
previous_log_size = None

    def _verify_tokens_migrated_successfully(self, previous_log_size=None):
        if not self.dtest_config.use_vnodes:
            num_tokens = 1
        else:
            # a little hacky but grep_log returns the whole line...
            num_tokens = int(self.replacement_node.get_conf_option('num_tokens'))
    
        logger.debug("Verifying {} tokens migrated successfully".format(num_tokens))
        replmnt_address = self.replacement_node.address_for_current_version_slashy()
        repled_address = self.replaced_node.address_for_current_version_slashy()
        token_ownership_log = r"Token (.*?) changing ownership from {} to {}".format(repled_address,
                                                                                     replmnt_address)
        logs = self.replacement_node.grep_log(token_ownership_log)
    
        if (previous_log_size is not None):
            assert len(logs) == previous_log_size
    
        moved_tokens = set([l[1].group(1) for l in logs])
        logger.debug("number of moved tokens: {}".format(len(moved_tokens)))
>       assert len(moved_tokens) == num_tokens
E       assert 0 == 1
E        +  where 0 = len(set())

replace_address_test.py:207: AssertionError
{code}
Bug: Test failure: org.apache.cassandra.distributed.test.ClearSnapshotTest.clearSnapshotSlowTest-_jdk17
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Nov/23 10:12
Resolved: 24/Nov/23 10:19

Circleci Failure: https://app.circleci.com/pipelines/github/michaelsembwever/cassandra/256/workflows/c4fda8f1-a8d6-4523-be83-5e30b9de39fe/jobs/20500/tests

Also on jdk11: CircleCI failure: https://app.circleci.com/pipelines/github/michaelsembwever/cassandra/256/workflows/c4fda8f1-a8d6-4523-be83-5e30b9de39fe/jobs/20464/tests

 ```
 java.lang.RuntimeException: java.util.concurrent.TimeoutException
  at org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:79)
  at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:537)
  at org.apache.cassandra.distributed.impl.AbstractCluster.close(AbstractCluster.java:1098)
  at org.apache.cassandra.distributed.test.ClearSnapshotTest.clearSnapshotSlowTest(ClearSnapshotTest.java:124)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  Caused by: java.util.concurrent.TimeoutException
  at org.apache.cassandra.utils.concurrent.AbstractFuture.get(AbstractFuture.java:253)
  at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:529)
  Suppressed: java.util.concurrent.TimeoutException
```