av9ash/GitBugs huggingface dataset:
Bug: Indexing a frozen collection that is the clustering key and reversed is rejected
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Sep/24 18:04
Resolved: 23/Sep/24 15:30

{code}
CREATE TABLE tbl (
  pk int,
  ck frozen<list<int>>,
  value int,
  PRIMARY KEY(pk, ck)
)
WITH CLUSTERING ORDER BY (ck DESC)
{code}

If you index “ck” it will fail

{code}
CREATE INDEX ON tbl(FULL(ck));
{code}

The failure error is

{code}
Caused by: org.apache.cassandra.exceptions.InvalidRequestException: full() indexes can only be created on frozen collections
	at org.apache.cassandra.cql3.statements.schema.AlterSchemaStatement.ire(AlterSchemaStatement.java:222)
	at org.apache.cassandra.cql3.statements.schema.CreateIndexStatement.validateIndexTarget(CreateIndexStatement.java:250)
	at org.apache.cassandra.cql3.statements.schema.CreateIndexStatement.lambda$apply$1(CreateIndexStatement.java:177)
{code}

The reason?  We have a ReverseType column!  We must “unwrap” the type before this check...

Basic test

{code}
schemaChange(format("CREATE TABLE %s.tbl (\n" +
                            "  pk int,\n" +
                            "  ck frozen<list<int>>,\n" +
                            "  value int,\n" +
                            "  PRIMARY KEY(pk, ck)\n" +
                            ")\n" +
                            "WITH CLUSTERING ORDER BY (ck DESC)", KEYSPACE));
schemaChange(format("CREATE INDEX ON %s.tbl(FULL(ck));", KEYSPACE));
{code}
Bug: CommitLogArchiver in CommitLog should be final (19448 follow-up)
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Sep/24 09:38
Resolved: 27/Sep/24 12:17

This is follow-up for CASSANDRA-19448, we made CommitLogArchiver non-final but not volatile. This might cause issues. We just need to return it back to final and remove the setter. The fact it is non-final now is because it was more comfortable to write tests but there is not any reason to keep it like that if tests are rewritten a little bit.
Bug: Correct out-of-date metrics and configuration documentation for SAI
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 05/Sep/24 17:42
Resolved: 19/Sep/24 19:57

1.) The docs discuss adjusting {{file_cache_size}}, which is a DSE holdover, and should be removed from the OSS docs. SAI uses {{MmapRebufferer}}, which doesn't touch the chunk cache.

2.) The docs for SAI's metrics include a handful of scopes that no longer exist and should be removed. (ex. {{OffsetFetch}}, {{TokenFetch}}, etc.)
Bug: Reverse cursor and iteration support for Trie based memtables
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 23/Sep/24 16:13
Resolved: 27/Sep/24 12:24

Cherry- pick [https://github.com/datastax/cassandra/commit/196b931c677829d681406f14cf1da814ff5a6624]

For Accord in particular this is useful to avoid flushing memtables that don't intersect with the range that is going to start having metadata GCed so we can flush less frequently/later.
Bug: SAI's IndexInputLeakDetector is not correctly tracking index inputs
Status: Triage Needed
Priority: Normal
Created: 27/Sep/24 15:13

{{TrackingIndexFileUtils}} has a mechanism to detect duplicated closures of index file inputs, [here|https://github.com/apache/cassandra/blob/trunk/test/unit/org/apache/cassandra/index/sai/disk/io/TrackingIndexFileUtils.java#L71]. 

However, this mechanism doesn’t seem to actuate, because no one calls {{TrackingIndexFileUtils#openInput}} nor {{TrackingIndexFileUtils.TrackedForwardingIndexInput#close}}, [here|https://github.com/apache/cassandra/blob/trunk/test/unit/org/apache/cassandra/index/sai/utils/IndexInputLeakDetector.java#L43].
Bug: SASI mixed with `IN` condition returns invalid result, Cassandra 5.0.0GA
Status: Triage Needed
Priority: Normal
Created: 16/Sep/24 06:11

Given this sample table
{code:java}
CREATE TABLE usersbyorganization
(
    organizationid UUID,
    userid         UUID,
    displayname    TEXT,
    rolecode       TEXT,
    PRIMARY KEY (organizationid, userid)
) WITH CLUSTERING ORDER BY (userid ASC);

CREATE CUSTOM INDEX usersbyorganization_displayname_idx ON usersbyorganization (displayname)
    USING 'org.apache.cassandra.index.sasi.SASIIndex'
    WITH OPTIONS = {'analyzed': 'true', 'analyzer_class': 'org.apache.cassandra.index.sasi.analyzer.NonTokenizingAnalyzer', 'case_sensitive': 'false', 'mode': 'CONTAINS'};
{code}
and sample data
{code:java}
INSERT INTO usersbyorganization (organizationid, userid, displayname, rolecode)
VALUES (0326bd0a-f977-408f-b5fa-52902eb1fd0f, 233569aa-1d3d-4bac-a73a-5f382a7bf144, 'soheil', 'admin');
INSERT INTO usersbyorganization (organizationid, userid, displayname, rolecode)
VALUES (0326bd0a-f977-408f-b5fa-52902eb1fd0f, 811804f2-6ede-487d-9b2c-c9f487ff77d9, 'soheil', 'staff');
INSERT INTO usersbyorganization (organizationid, userid, displayname, rolecode)
VALUES (0326bd0a-f977-408f-b5fa-52902eb1fd0f, 104f8cbc-e22e-4629-b836-2f832e59c4a3, 'soheil', 'student');
INSERT INTO usersbyorganization (organizationid, userid, displayname, rolecode)
VALUES (0326bd0a-f977-408f-b5fa-52902eb1fd0f, b4b405df-0da6-4a99-b46a-4419a505e96f, 'soheil', 'teacher');
{code}
If I run this query with only 1 value in the `IN clause`, it returns appropriate result:
{code:java}
SELECT *
FROM usersbyorganization
WHERE organizationid = 0326bd0a-f977-408f-b5fa-52902eb1fd0f
  AND displayname = 'soheil'
  AND roleCode IN ('student')
    ALLOW FILTERING;
{code}
|| organizationid || userid || displayname || rolecode ||
| 0326bd0a-f977-408f-b5fa-52902eb1fd0f | 104f8cbc-e22e-4629-b836-2f832e59c4a3 | soheil | student |

But if I run the same query with 2 values in the `IN clause`, it returns invalid values for column `roleCode`:

{code:java}
SELECT *
FROM usersbyorganization
WHERE organizationid = 0326bd0a-f977-408f-b5fa-52902eb1fd0f
  AND displayname = 'soheil'
  AND roleCode IN ('student', 'teacher')
    ALLOW FILTERING;
{code}

|| organizationid || userid || displayname || rolecode ||
| 0326bd0a-f977-408f-b5fa-52902eb1fd0f | 104f8cbc-e22e-4629-b836-2f832e59c4a3 | soheil | student |
| 0326bd0a-f977-408f-b5fa-52902eb1fd0f | 233569aa-1d3d-4bac-a73a-5f382a7bf144 | soheil | admin |
| 0326bd0a-f977-408f-b5fa-52902eb1fd0f | 811804f2-6ede-487d-9b2c-c9f487ff77d9 | soheil | staff |
| 0326bd0a-f977-408f-b5fa-52902eb1fd0f | b4b405df-0da6-4a99-b46a-4419a505e96f | soheil | teacher |

`admin` and `staff` should not be returned.
Bug: Add sstabledump option to display only tombstones
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/Sep/24 17:03
Resolved: 26/Sep/24 08:03

The utility sstabledump would be more useful if it had an option to display just tombsones, much like the -e option to display only partition keys. 

When tombstones exist, it should display the partition key, clustering columns and the range, row or cell tombstone.

When investigating tombstone issues, a common way is to run sstabledump.  However, the output can be very large, on the order of gigabytes and scanning the JSON output for tombstone entries is a hunt-and-peck experience or requires using a tool like 'jq'.  Being able to just view the tombstones would eliminate the issue of the output being too large and not require extraneous tools.
Bug: Memtable allocation type unslabbed_heap_buffers_logged will cause an assertion error for TrieMemtables and SegmentedTrieMemtables
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 14/Aug/24 17:47
Resolved: 14/Aug/24 21:54

Config used

{code}
	---
	partitioner: Murmur3Partitioner
	commitlog_sync: periodic
	commitlog_sync_period: 9000ms
	commitlog_disk_access_mode: legacy
	memtable_allocation_type: unslabbed_heap_buffers_logged
	sstable:
	  selected_format: big
	disk_access_mode: standard
{code}

Error

{code}
Caused by: java.lang.AssertionError: null
	at org.apache.cassandra.config.Config$MemtableAllocationType.toBufferType(Config.java:1206)
	at org.apache.cassandra.index.sai.disk.v1.segment.SegmentTrieBuffer.<init>(SegmentTrieBuffer.java:48)
	at org.apache.cassandra.index.sai.disk.v1.segment.SegmentBuilder$TrieSegmentBuilder.<init>(SegmentBuilder.java:83)
	at org.apache.cassandra.index.sai.disk.v1.SSTableIndexWriter.newSegmentBuilder(SSTableIndexWriter.java:311)
	at org.apache.cassandra.index.sai.disk.v1.SSTableIndexWriter.addTerm(SSTableIndexWriter.java:195)
	at org.apache.cassandra.index.sai.disk.v1.SSTableIndexWriter.addRow(SSTableIndexWriter.java:99)
	at org.apache.cassandra.index.sai.disk.StorageAttachedIndexWriter.addRow(StorageAttachedIndexWriter.java:257)
	at org.apache.cassandra.index.sai.disk.StorageAttachedIndexWriter.nextUnfilteredCluster(StorageAttachedIndexWriter.java:131)
{code}

This was found by CASSANDRA-19833
Bug: CEP-15 (C*): Read accord repair cfk keys from sstable index.
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 12/Sep/24 22:33
Resolved: 24/Sep/24 17:00

In clusters where accord repairs are dealing with a lot of keys, reading the keys via cql can take up to 90% of cpu time. This patch switches to reading cfk keys out of the sstable index, which effectively removes that step of accord repair from the profiler output. Given the structure of the CFK table and how C* reads compound partition keys with local partitioner, this is a little involved since we need to properly handle exclusive starts and inclusive ends without a full partition key.
Bug: CEP-15 (C*) - misc accord perf improvements
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/Sep/24 23:17
Resolved: 24/Sep/24 16:59

This adds a handful of relatively minor changes that makes high key density accord workloads more performant.

Increase accord repair range splitting - given a cluster with 6 million keys, this splits up accord repairs into 600 key chunks.

Streamline table metadata fetching - removes some unnecessary abstraction from the table metadata lookup path, which we hit a lot during key deserialization.

Remote unnecessary set building when building lists of overlapping keys - we don’t need to dedupe keys and the set building was taking a decent chunk of time.

Add separate recover delay for repair and increase default recover delay - this significantly reduces the number of preemptions.

The range splitting and recovery delay changes stabilize performance today, but are kind of a stop gap - we should be able to support larger repair ranges and shorter recovery delays.
Bug: Failed test: replica_side_filtering_test.TestSecondaryIndexes test_complementary_deletion_with_limit_on_clustering_key_column
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/Sep/24 16:56
Resolved: 24/Sep/24 03:35

I noticed these two started to always fail in trunk

{code}
replica_side_filtering_test.TestSecondaryIndexes test_complementary_deletion_with_limit_on_clustering_key_column
replica_side_filtering_test.TestSecondaryIndexes test_complementary_deletion_with_limit_on_partition_key_column_with_not_empty_partitions
{code}

Fails on

{code}
        >       assert list_res == [], "Expected nothing from {}, but got {}".format(query, list_res)
        E       AssertionError: Expected nothing from SELECT * FROM t WHERE c = 0 LIMIT 1, but got [[1, 0]]
{code}

and

{code}
        >       assert list_res == [], "Expected nothing from {}, but got {}".format(query, list_res)
        E       AssertionError: Expected nothing from SELECT * FROM t WHERE k1 = 0 LIMIT 1, but got [[0, 1, 10, 100]]

{code}
Bug: Allow nodes to change IP address while upgrading to TCM
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/Sep/24 12:07
Resolved: 23/Sep/24 06:52

We currently only allow node version to change via gossip while upgrading to TCM (in mixed mode or before {{nodetool cms initialize}} has been run). We should also allow ip addresses to change since that can happen while bouncing nodes in some environments.
Bug: Move network operations outside the lock in Gossiper$GossipTask
Status: Triage Needed
Priority: Normal
Created: 21/Sep/24 01:58

To execute the gossip protocol and exchange state info with other nodes, _[GossiperTask.run()|https://github.com/apache/cassandra/blob/7b1eb1f0b717beb33e611157766701cd71e4ad8c/src/java/org/apache/cassandra/gms/Gossiper.java#L321]_ invokes _[doGossipToLiveMember()|https://github.com/apache/cassandra/blob/7b1eb1f0b717beb33e611157766701cd71e4ad8c/src/java/org/apache/cassandra/gms/Gossiper.java#L955]_, _[maybeGossipToUnreachableMember()|https://github.com/apache/cassandra/blob/7b1eb1f0b717beb33e611157766701cd71e4ad8c/src/java/org/apache/cassandra/gms/Gossiper.java#L964]_ and _[maybeGossipToSeed()|https://github.com/apache/cassandra/blob/7b1eb1f0b717beb33e611157766701cd71e4ad8c/src/java/org/apache/cassandra/gms/Gossiper.java#L982]_, with all 3 methods invoking _[sendGossip()|https://github.com/apache/cassandra/blob/7b1eb1f0b717beb33e611157766701cd71e4ad8c/src/java/org/apache/cassandra/gms/Gossiper.java#L933]_ to send a gossip message to a randomly selected endpoint. The interaction between GossiperTask.run() and sendGossip() creates a potential synchronization bottleneck due to the lock ([_taskLock_|https://github.com/apache/cassandra/blob/7b1eb1f0b717beb33e611157766701cd71e4ad8c/src/java/org/apache/cassandra/gms/Gossiper.java#L328]) being held during network-bound operations. GossiperTask.run() directly calls [_waitUntilListening()_|https://github.com/apache/cassandra/blob/7b1eb1f0b717beb33e611157766701cd71e4ad8c/src/java/org/apache/cassandra/gms/Gossiper.java#L326] which will wait for the MessagingService to start listening, but there could be delays if the messaging service is slow to start or has issues. Also, if sendGossip() encounters network-related delays (i.e. network latency, timeouts, slow or unresponsive nodes) when there is a large number of nodes, the taskLock could be held for longer periods,  possibly increasing the risk of a backlog of waiting threads (if delays are frequent) and also affecting the scheduling of subsequent tasks. 

One potential optimization for this could be to move network operations outside the taskLock. This way the lock is released before performing time consuming network operations. But before doing so, I wonder if the analysis above is correct and whether it is worth optimizing.
Bug: Pooled BTree.FastBuilder keeps references to data after reset
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 22/Sep/24 12:52
Resolved: 22/Sep/24 13:25

Hi I am analyzing of argecolumn_test.TestLargeColumn test failure (CASSANDRA-19702) and have noticed that sometimes when I run the test locally it fails with OOM. The test sends an insert with a large column value.

In the created heap dump I have found that there is a kind of memory leaking for the sent column values though thread-local variable used for re-usable Pooled BTree.FastBuilder:

!image-2024-09-22-13-51-02-751.png!

In normal cases it is probably not critical because the objects are not that huge but it is still a leak.

reset() logic in BTree.FastBuilder does not clean references to such values because during other operations like build() "count" value can be reset to 0.
Bug: Possible memory leak in BTree.FastBuilder 
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 19/Jul/24 07:48
Resolved: 16/Sep/24 17:28

We are having a problem with the heap growing in size, This is a large cluster > 1,000 nodes across a large number of dc’s. This is running version 4.0.11.

 

Each node has a 32GB heap, and the amount used continues to grow until it reaches 30GB, it then struggles with multiple Full GC pauses, as can be seen here:

!image-2024-07-19-08-44-56-714.png!

We took 2 heap dumps on one node a few days after it was restarted, and the heap had grown by 2.7GB

 

9{^}th{^} July

!image-2024-07-19-08-45-17-289.png!

11{^}th{^} July

!image-2024-07-19-08-45-33-933.png!

This can be seen as mainly an increase of memory used by FastThreadLocalThread, increasing from 5.92GB to 8.53GB

!image-2024-07-19-08-45-50-383.png!

!image-2024-07-19-08-46-06-919.png!

Looking deeper into this it can be seen that the growing heap is contained within the threads for the MutationStage, Native-transport-Requests, ReadStage etc. We would expect the memory used within these threads to be short lived, and not grow as time goes on.  We recently increased the size of theses threadpools, and that has increased the size of the problem.

 

Top memory usage for FastThreadLocalThread

9{^}th{^} July

!image-2024-07-19-08-46-42-979.png!

11{^}th{^} July

!image-2024-07-19-08-46-56-594.png!
This has led us to investigate whether there could be a memory leak, and we have found the following issues within the retained references in BTree.FastBuilder objects. The issue appears to stem from the reset() method, which does not properly clear all buffers.  We are not really sure how the BTree.FastBuilder works, but this this is our analysis of where a leak might occur.

 

Specifically:

Leaf Buffer Not Being Cleared:
When leaf().count is 0, the statement Arrays.fill(leaf().buffer, 0, leaf().count, null); does not clear the buffer because the end index is 0. This leaves the buffer with references to potentially large objects, preventing garbage collection and increasing heap usage.

Branch inUse Property:
If the inUse property of the branch is set to false elsewhere in the code, the while loop while (branch != null && branch.inUse) does not execute, resulting in uncleared branch buffers and retained references.

 

This is based on the following observations:

    Heap Dumps: Analysis of heap dumps shows that leaf().count is often 0, and as a result, the buffer is not being cleared, leading to high heap utilization.

!image-2024-07-19-08-47-19-517.png!

    Remote Debugging: Debugging sessions indicate that the drain() method sets count to 0, and the inUse flag for the parent branch is set to false, preventing the while loop in reset() from clearing the branch buffers.

!image-2024-07-19-08-47-34-582.png!
Bug: Deprecate use of gossip state for paxos electorate verification
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 09/Sep/24 08:51
Resolved: 18/Sep/24 11:00

In C* 5.0 and earlier, paxos uses gossip state to verify the paxos electorate remains stable during a round. 
In 5.1, we mostly use the current epoch for this purpose, but some usage of gossip state has been left in place and should be removed.

An {{Electorate}} is now constructed from {{ClusterMetadata}} and no longer relies on anything in gossip, so updating gossip in {{PaxosPrepare::onResponse}} is no longer necessary. Instead, in the case where the respondent has signalled a mismatch the requester should perform a metadata catchup to the appropriate epoch.

Currently, this mismatch is communicated by the presence of a non-empty map of endpoint states, but this is no longer necessary as {{Paxos::verifyElectorate}} can provide the epoch of the placements used to construct its local electorate if it differs from the remote one. That epoch can then be included in the {{Permitted}} response instead of the gossip state. When handling the {{Permitted}} response, the paxos coordinator can use this epoch to determine whether it needs to call {{{}permittedOrTerminateIfElectorateMismatch{}}}.

For now, we will need to include both the electorate epoch and gossip state in the {{Permitted}} response to ensure continuity during upgrades but eventually the gossip state can be removed from these messages.
Bug: NPE in AffectedRangesImpl#intersects when other has different keyspaces than local
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Sep/24 21:27
Resolved: 20/Sep/24 16:24

This error was found on Accord branch.

{code}
 org.apache.cassandra.simulator.SimulationException: Failed on seed 0xfa3d51da237d56e5; Failure creating the simulation
Caused by: java.lang.AssertionError: Errors detected during simulation
	Suppressed: java.lang.NullPointerException
		at org.apache.cassandra.tcm.sequences.LockedRanges$AffectedRangesImpl.intersects(LockedRanges.java:337)
		at org.apache.cassandra.tcm.sequences.LockedRanges.intersects(LockedRanges.java:97)
		at org.apache.cassandra.tcm.transformations.PrepareLeave.execute(PrepareLeave.java:108)
		at org.apache.cassandra.tcm.AbstractLocalProcessor.executeStrictly(AbstractLocalProcessor.java:167)
		at org.apache.cassandra.tcm.AbstractLocalProcessor.commit(AbstractLocalProcessor.java:77)
		at org.apache.cassandra.distributed.test.log.TestProcessor.commit(TestProcessor.java:62)
		at org.apache.cassandra.tcm.Commit$Handler.doVerb(Commit.java:368)
		at org.apache.cassandra.net.InboundSink.lambda$new$0(InboundSink.java:102)
		at org.apache.cassandra.net.InboundSink$Filtered.accept(InboundSink.java:70)
		at org.apache.cassandra.net.InboundSink$Filtered.accept(InboundSink.java:56)
		at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:122)
{code}

In debugger I see the following

{code}
other = {LockedRanges$AffectedRangesImpl@92088} "AffectedRangesImpl{map={ReplicationParams{class=org.apache.cassandra.locator.MetaStrategy, dc0=2, dc2=2, dc1=2}=[(-9223372036854775808,-9223372036854775808]]}}"
 map = {HashMap@93801}  size = 1
  {ReplicationParams@92097} "ReplicationParams{class=org.apache.cassandra.locator.MetaStrategy, dc0=2, dc2=2, dc1=2}" -> {HashSet@92098}  size = 1
{code}

{code}
map = {HashMap@92092}  size = 3
 {ReplicationParams@93810} "ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=2}" -> {HashSet@93811}  size = 3
 {ReplicationParams@93812} "ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=3}" -> {HashSet@93813}  size = 4
 {ReplicationParams@93814} "ReplicationParams{class=org.apache.cassandra.locator.NetworkTopologyStrategy, dc0=3, dc2=3, dc1=3}" -> {HashSet@93815}  size = 7
{code}

Locally we are missing the meta keyspace so can’t find it from the map and NPE.

The reason for this is that the current node is starting up

{code}
StorageService.instance.operationMode == STARTING
{code}
Bug: gocql: Update testdata/pki certificates as they expire on September 16, 2024
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/Aug/24 00:52
Resolved: 19/Sep/24 16:57

While familiarizing myself with the integration tests for gocql I noticed that they were expiring on September 16th, as the certificates were generated with a 10 year validity nearly 10 years ago as part of [#251|https://github.com/apache/cassandra-gocql-driver/pull/251] which added TLS support to gocql:

{noformat}
openssl x509 -in testdata/pki/gocql.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=cassandra
        Validity
            Not Before: Sep 19 21:18:33 2014 GMT
            Not After : Sep 16 21:18:33 2024 GMT
{noformat}

I'd like to regenerate the certificates and will also include a shell script for regenerating them as needed.