av9ash/GitBugs huggingface dataset:
Bug: Fix topology replay during bootstrap and startup, decouple Accord from TCM
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 12/Dec/24 15:27
Resolved: 14/Jan/25 12:27

Includes multiple changes, primary ones:
  * Removed nodes now live in TCM, no need to discover historic epochs in order to find removed nodes
  * CommandStore <-> RangesForEpochs mappings required for startup are now stored in journal, and CS can be set up _without_ topology replay
  * Topology replay is fully done via journal (where we store topologies themselves), and topology metadata table (where we store redundant/closed information)
  * Fixed various bugs related to propagation and staleness
     * TCM was previously relied on for "fetching" epoch: we can not rely on it as there's no guarantee we will see a consecutive epoch when grabbing Metadata#current
     * Redundant / closed during replay was set with incorrect ranges in 1 of the code paths
     * TCM was contacted multiple times for historical epochs, which made startup much longer under some circumstances
Bug: Refactor the ast package to enable harry model based testing
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 19/Dec/24 17:37
Resolved: 13/Jan/25 18:13

The AST package was created for pure random fuzz testing and has not had the ability to interact with a model to validate the results, which means that there are sharp edge cases when trying to interact with a model. This ticket is to cleanup these edge cases, improve human error messages, and pave the way for harry model testing.
Bug: Enhance CQLSH to support machine-readable output formatting
Status: Open
Priority: Normal
Created: 07/Oct/24 21:12

Existing CQLSH output formatting provides tabular formatting using grid-like separators and text alignment which is designed for user readability. 

When CQL is run for non-interactive tasks as part of a pipeline this formatting complicates the machine processing of the output. As an example, [Sqlite|https://www.sqlite.org/cli.html] has a command line flag -mode to switch between tablular, csv, and several other supported formats.

This enhancement will provide a new mode argument which will initially offer tabular and csv output formats. Tabular will remain the default so there will be no change for existing users.  In the future, other modes such as json and insert could be added.
{noformat}
--mode={tabular, csv}
Specify an output display format. The default is tabular.
{noformat}
The existing Copy To can be used for exporting to CSV, but it doesn't allow for query criteria and thus isn't a general solution for this issue.

In cqlshmain.py, the EXPAND CQLSH option which uses print_formatted_result() for result output is an example of alternative formatting.
Bug: IndexOutOfBoundsException when accessing partition where the column was deleted
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 22/Nov/24 22:34
Resolved: 08/Jan/25 01:50

{code}
Caused by: java.lang.IndexOutOfBoundsException
	at java.base/java.nio.Buffer.checkIndex(Buffer.java:687)
	at java.base/java.nio.HeapByteBuffer.get(HeapByteBuffer.java:169)
	at org.apache.cassandra.db.marshal.ByteBufferAccessor.getByte(ByteBufferAccessor.java:184)
	at org.apache.cassandra.db.marshal.ByteBufferAccessor.getByte(ByteBufferAccessor.java:42)
	at org.apache.cassandra.db.marshal.ByteType.compareCustom(ByteType.java:51)
	at org.apache.cassandra.db.marshal.AbstractType.compare(AbstractType.java:216)
	at org.apache.cassandra.db.marshal.AbstractType.compare(AbstractType.java:211)
	at org.apache.cassandra.db.marshal.AbstractType.compareForCQL(AbstractType.java:269)
	at org.apache.cassandra.cql3.Operator$1.isSatisfiedBy(Operator.java:73)
	at org.apache.cassandra.db.filter.RowFilter$SimpleExpression.isSatisfiedBy(RowFilter.java:725)
	at org.apache.cassandra.db.filter.RowFilter$1.applyToPartition(RowFilter.java:227)
	at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:94)
	at org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:1045)
	at org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:629)
	at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:665)
	at org.apache.cassandra.cql3.statements.SelectStatement.executeLocally(SelectStatement.java:635)
	at org.apache.cassandra.cql3.statements.SelectStatement.executeLocally(SelectStatement.java:151)
{code}

Table

{code}
CREATE TABLE keyspace_test_00."3W56TBuMmC11vPVxalpse84eS" (
		    pk0 date,
		    pk1 double,
		    ck0 int,
		    ck1 inet,
		    s0 tinyint static,
		    v0 int,
		    v1 varint,
		    v2 varint,
		    v3 timestamp,
		    PRIMARY KEY ((pk0, pk1), ck0, ck1)
		) WITH CLUSTERING ORDER BY (ck0 DESC, ck1 ASC)
{code}

The query

{code}
SELECT *
FROM keyspace_test_00."3W56TBuMmC11vPVxalpse84eS"
WHERE s0 = ? —- value is "(byte) -113"
ALLOW FILTERING
{code}

The issue is that we see the delete, but don’t properly handle null data

{code}
ByteBuffer foundValue = getValue(metadata, partitionKey, row);
// value is java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]; aka null (empty)
{code}

History of operations on this partition

{code}
	History:
		1: UPDATE pd0
		2: Select Whole Partition pd0
		4: Select Whole Partition pd0
		6: Delete COLUMN [s0, v0, v1, v2, v3] pd0
		7: Select Whole Partition pd0
		10: Select Whole Partition pd0
		12: Select Row pd0
		17: Delete COLUMN [s0, v0, v1, v2, v3] pd0
		20: INSERT pd0
		27: UPDATE pd0
		38: INSERT pd0
		41: Select Row pd0
		56: Select Row pd0
		66: Delete COLUMN [s0, v0, v1, v2, v3] pd0
		67: Search on column s0
{code}

Here we see an insert was done so liveness info is generated, but we do delete on all columns leaving only the partition/clustering keys around...
Bug: DELETE partition IF static column is rejected
Status: Resolved
Priority: Normal
Resolution: Won't Fix
Created: 23/Dec/24 23:05
Resolved: 08/Jan/25 00:00

{code}
CREATE TABLE tbl (
 pk int,
 ck int,
 s0 int,
 v0 int,
 PRIMARY KEY (pk, ck)
);

DELETE
FROM tbl
WHERE pk = ?
IF s0 = ?
{code}

The above fails with the exception

{code}
DELETE statements must restrict all PRIMARY KEY columns with equality relations in order to delete non static columns
{code}
Bug: Query construction is broken for SAI indexes on reversed types with fixed-length encodings
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/Nov/24 22:55
Resolved: 09/Dec/24 21:13

SAI indexes values in byte-comparable form, both in the in-memory trie that sits alongside the Memtable, and in the on-disk SSTable-adjacent indexes. In most cases, this means literally using {{asComparableBytes()}} from the type of the indexed column. There are, however, a few types that use a custom byte-comparable form, namely {{inet}}, {{bigint}}, {{varint}}, and {{decimal}}, to make sure we're dealing with a fixed-length piece of data for the numeric (balanced tree) index.

If we index one of these types as a reversed clustering key, however, we don't write terms as reversed comparable bytes, and this breaks some assumptions during query construction and post-filtering, where we generally assume that {{asComparableBytes()}} will reverse terms before they are indexed. We can make a short-term fix here without changing anything about the on-disk format by making sure we interpret these special types as being non-reversed (i.e. through the lens of their base types).

In the longer term, it might make sense to standardize on indexing everything in a non-reversed fashion in the index itself, although this might push some complexity into post-filtering, where we are going to have to filter data coming out of the normal read path anyway.
Bug: Suggestions from masokol (from ecchronos experience)
Status: Triage Needed
Priority: Normal
Created: 21/Oct/24 16:12

Here are a few changes suggested by [~masokol] 
 # Repair: Write x-version repair test cases to ensure the repair does not break while upgrading
 # Major versions: During major version upgrades like 3 -> 4, we weren't supposed to run repairs. If Cassandra plans to keep this, then it would be nice for repairs to automatically pause during major version upgrades.
 # Repair priority per table, not per node.
Bug: Check if MetaStrategy tables need to be repaired
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 11/Dec/24 00:21
Resolved: 07/Jan/25 18:56

TCM has introduced a new locator named org.apache.cassandra.locator.MetaStrategy, e.g., 

_system_cluster_metadata_

We need to assess if those tables need to be repaired or not because this strategy has been uniquely designed for TCM.
Bug: Flatten AutoRepair table parameters
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 09/Dec/24 04:43
Resolved: 07/Jan/25 18:56

With the current AutoRepair design, it would create one new table property per repair type. To simplify, flatten multiple repair type's properties into one. More discussion in #[cassandra-repair-scheduling-cep37|https://the-asf.slack.com/archives/C06MR8K2Q0Z/p1733605745111419]
Bug: Timing info for each test split in Jenkinsfile
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/Dec/24 07:49
Resolved: 05/Jan/25 18:20

Tuning the number of splits, and generally analysing where timeouts are (and will be) happening, is currently more difficult than it needs to be.

Splits are created dynamically in the Jenkinsfile, with a fixed number of splits per test type.  In trunk there's >900 splits.  While the blue ocean ui can help highlight which splits have timed out (past the one hour per-split timeout) twice (each split will retry once), it remains difficult to scroll through the blue ocean screen, to identify splits that only timed out once, and to see which splits are heavier (taking longer) than their counterparts and the timing trends of splits over time.

A simple log line reporting time taken for each split attempt solves all this.
Bug: Add cqlsh autocompletion for the identity mapping feature
Status: Open
Priority: Normal
Created: 23/Oct/24 23:22

Add {{cqlsh}} autocompletion for the [identity|https://cassandra.apache.org/doc/latest/cassandra/developing/cql/cql_singlefile.html#databaseIdentity] mapping feature so we have completions such as:
{code:java}
cqlsh> AD
ADD

cqlsh> ADD IDEN
IDENTITY

cqlsh> ADD IDENTITY 
<identity>    IF

cqlsh> DROP I
IDENTITY   INDEX
{code}
Bug: jenv .java-version and pyenv .python-version defaults
Status: Open
Priority: Normal
Created: 07/Dec/24 10:00

Commit in-tree both the .java-version and .python-version files.

Each branch has a default java version and python version.  We can honour that, and make life one less command when switching branches for those using jenv and/or pyenv. 

For those that do not have either jenv or pyenv installed, no harm.


It will need run… (as .gitignore won't work here) …so devs can still customise both…
{code}
git update-index --assume-unchanged .java-version .python-version
{code}
Bug: Avoid memory allocation in NativeCell.valueSize() and NativeClustering.dataSize()
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 23/Dec/24 09:58
Resolved: 04/Jan/25 21:01

Currently for memtable_allocation_type: offheap_objects we allocate memory to calculate dataSize() during a write to a memtable because we use default implementation of dataSize() which retrieves an object using value() method.
To avoid such memory allocation we can calculate data size directly for NativeCell and NativeClustering

test schema details:
{code:java}
CREATE TABLE memory_tests.test_table_text_0 (part_key text,clust_key text,value_0 text,value_1 text,value_2 text,value_3 text,value_4 text,value_5 text,value_6 text,value_7 text,value_8 text,value_9 text,PRIMARY KEY(part_key,clust_key))
{code}
write query:
{code:java}
INSERT INTO memory_tests.test_table_text_0 (part_key,clust_key,value_0,value_1,value_2,value_3,value_4,value_5,value_6,value_7,value_8,value_9) VALUES (?,?,?,?,?,?,?,?,?,?,?,?)
{code}
changed settings:
{code:java}
memtable_allocation_type: offheap_objects

memtable:
  configurations:
    skiplist:
      class_name: SkipListMemtable
    trie:
      class_name: TrieMemtable
    default:
      inherits: trie // changed default to trie
{code}
JFR allocation report:

!jfr_allocation_profile.png|width=570!

!image-2024-12-23-09-53-27-226.png|width=570!

Memtable flushing logic also is affected:

!Cell_valueSize_usage_during_flushing.png|width=570!
Bug: nodetool docs are broken for 5.0
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 25/Oct/24 17:55
Resolved: 02/Jan/25 15:10

The docs for nodetool on 4.1 all link to pages that outline their usage:

[https://cassandra.apache.org/doc/4.1/cassandra/tools/nodetool/nodetool.html]

For example, clicking assassinate brings you to the assassinate page: [https://cassandra.apache.org/doc/4.1/cassandra/tools/nodetool/assassinate.html]

With the 5.0 docs, the links don't seem to go anywhere:

[https://cassandra.apache.org/doc/5.0/cassandra/managing/tools/nodetool/nodetool.html]

 

The assassinate link, for example, links to here, but there's no content:

https://cassandra.apache.org/doc/5.0/cassandra/managing/tools/nodetool/nodetool.html#modules/cassandra/pages/managing/tools/nodetool/assassinate.adoc
Bug: Avoid ByteBuffer allocation during decoding of prepared CQL write requests
Status: In Progress
Priority: Normal
Created: 26/Dec/24 17:37

A lot of ByteBuffer objects are allocated when we decode CQL queries, frequently the space spent for such objects is large than the actual amount of data received.
There was a similar optimization (use byte[] directly and wrap them into ArrayCell instead of BufferCell) done some time ago for the place where a Mutation object is deserializing during a Cassandra cross-node communication or reading from a disk: CASSANDRA-15393
While a complete replacement of ByteBuffer with byte[] during CQL decoding step looks like a very complex task (ByteBuffer is a part of too many entities involved into CQL parsing) we can optimize 20% of logic to get 80% of benefit by focusing only on batch and modification statements when prepared statements are used and cell values are provided as bind variables.

!image-2024-12-26-17-33-39-031.jpg|width=570!

In case of 10-symbol String values I used for a test the wrapping ByteBuffer objects are costlier than inner byte[] with data:

 

!image-2024-12-26-17-35-05-485.png|width=570!
Bug: Update storage-engine.adoc - minor spelling fix in documentation
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Dec/24 12:31
Resolved: 31/Dec/24 14:47

Update storage-engine.adoc - minor spelling fix in documentation

https://github.com/apache/cassandra/blob/cassandra-5.0/doc/modules/cassandra/pages/architecture/storage-engine.adoc

line 247 : wrong : * da (5.0): initial version of the BIT format
line 247 : fixed : * da (5.0): initial version of the BTI format
Bug: Remove `REPEATED_TESTS` env var from repeated tests script
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 11/Dec/24 06:26
Resolved: 22/Dec/24 20:36

Quick follow-on from CASSANDRA-18942

The use of REPEATED_TESTS doesn't work with the dockerised scripts (see this [comment|https://issues.apache.org/jira/browse/CASSANDRA-18942?focusedCommentId=17904409&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17904409]), and duplicates the `-t` functionality.

Easiest solution is to just remove it.
Bug: BETWEEN where token(Y) > token(Z) returns wrong answer
Status: Open
Priority: Normal
Created: 19/Dec/24 16:21

Noticed it’s not compliant with the SQL specification for BETWEEN and so doesn’t match what other databases do (tested on Postgres and MySQL).

Here is the [SQL 92 spec|https://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt] (later versions probably don't change something so fundamental).

In particular rule 6 doesn't match the observed behavior which is to swap Y and Z and then return what is in between them which is pretty surprising as a result.
{noformat}
6) "X BETWEEN Y AND Z" is equivalent to "X>=Y AND X<=Z".
{noformat}

{noformat}
         8.3  <between predicate>

         Function

         Specify a range comparison.

         Format

         <between predicate> ::=
              <row value constructor> [ NOT ] BETWEEN
                <row value constructor> AND <row value constructor>


         Syntax Rules

         1) The three <row value constructor>s shall be of the same degree.

         2) Let respective values be values with the same ordinal position
            in the two <row value constructor>s.

         3) The data types of the respective values of the three <row value
            constructor>s shall be comparable.

         4) Let X, Y, and Z be the first, second, and third <row value con-
            structor>s, respectively.

         5) "X NOT BETWEEN Y AND Z" is equivalent to "NOT ( X BETWEEN Y AND
            Z )".

         6) "X BETWEEN Y AND Z" is equivalent to "X>=Y AND X<=Z".
{noformat}
Bug: Use larger internal buffers and don't block on network on streaming slow path
Status: Triage Needed
Priority: Normal
Created: 04/Oct/24 16:42

CASSANDRA-15452 is introducing an internal buffer to compaction in order to increase throughput while reducing IOPS.  We do the same thing with our streaming slow path, although it's not optimal.  There's a common misconception that the overhead comes from serde overhead, but I've found on a lot of devices the overhead is due to our read patterns. This is most commonly found on non-NVMe drives, especially disaggregated storage such as EBS where the latency is higher and more variable.

Attached is a perf profile showing the cost of streaming is dominated by pread.  The team I was working with was seeing they could stream only 12MB per streaming session.  Reducing the number of read operations by using larger buffered reads should improve this by at least 3-5x on some systems, as well as reduce CPU overhead from reduced system calls.

I think we need to do a few things:
 * Use a larger internal buffer on disk reads. 
 * Buffer writes to the network.  Writing constant small values to the network has a very high latency cost, we'd be better off flushing larger values more often
 * Move the blocking network writing to a separate thread.  We don't need to wait on the network transfer in order to read more data off disk.  Once we improve the internal buffer on reads I think we'll see this as the next problem so let's tackle it now. ExecutorService.newSingleThreadExecutor() would work well for this.

 

 

 

 

 

 

 

 

!image-2024-10-04-12-40-26-727.png|width=912,height=592!
Bug: Sort output in nodetool status
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 22/Nov/24 11:28
Resolved: 18/Dec/24 10:39

Lets have this output:

{code}
nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load        Tokens  Owns (effective)  Host ID                               Rack 
UN  172.19.0.9  385.29 KiB  256     66.6%             6d194555-f6eb-41d0-c000-000000000003  rack3
UN  172.19.0.5  609.07 KiB  256     68.4%             6d194555-f6eb-41d0-c000-000000000001  rack1
UN  172.19.0.8  377.5 KiB   256     65.0%             6d194555-f6eb-41d0-c000-000000000002  rack2
{code}

This is a very simple case but it will serve well for the explanation.

What I would like to see is to have this output sorted. If I am looking for a particular node, e.g. by IP address, these entries are not sorted by IP in general, so when I have 20-nodes cluster and I want to check a particular node, it is like a needle in a haystack so I would welcome if they are ordered by IP.

This can be generalized to other columns - sort by Load, for example, to see what nodes are the most loaded ones, sort by host id would be cool too. TCM introduced kind of predictable node ids so sorting them should give you the order in which nodes have been added etc.

There might be a new switch added like "--sort=ip/id/load/rack". For loads, it would sort it from the most loaded one to least loaded one. For IPs and IDs, it would sort them from the "lowest" to the "highest" node in terms of IP address and ID. We might also group nodes by rack and by node id inside a rack.

This should be done _per DC_.
Bug: Ensure downgrade from 5.1 to 4.x is possible
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 16/Dec/24 15:03
Resolved: 17/Dec/24 17:38

Downgrading from current trunk to 5.0 should be unproblematic as (currently) there are no changes to on disk format or existing system tables.

Rolling back to 4.x is also possible while the upgraded nodes are configured with storage compatibility mode {{CASSANDRA_4}}. However, this also means peers are unable to use messaging {{VERSION_51}}, so some of the guarantees around consistency and visibility of metadata changes are weakened as these depend on encoding metadata epochs in internode messages. Those limitations aside, the cluster can be operated as normal following the upgrade with schema and topology changes permitted. The effects of these changes persist through a downgrade. The downgrade requires operators to manually remove some directories on disk, but can be done online without taking down the entire cluster.    

There's one problem with this at the moment; in trunk an extra identifier is included in gossip digest syn messages to help guard against split brain scenarios when intialising a new cluster. Running with {{CASSANDRA_4}} compatibility currently causes the checking of this identifier to error which prevents the CMS being initialised (via {{nodetool cms initialize}}) during the 5.1 upgrade.
Bug: Check presence of a snapshot in a case-insensitive manner on macOS platform to prevent hardlinking failures
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 16/Dec/24 17:19
Resolved: 17/Dec/24 09:17

It looks like snapshot names are perhaps not case-sensitive, and there is a collision between "snapshot" and "Snapshot". In the logs, I see this when I run {{testExoticSnapshotNames()}}:

{noformat}
WARN  16:45:44 Unable to create snapshot Snapshot for distributed_test_keyspace.tbl
java.lang.RuntimeException: Tried to create duplicate hard link to /private/var/folders/4d/zfjs7m7s6x5_l93k33r5k6680000gn/T/dtests17605712756058855626/node1/data2/distributed_test_keyspace/tbl-1b255f4def2540a60000000000000005/snapshots/Snapshot/oa-3-big-Data.db
	at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:174)
	at org.apache.cassandra.io.sstable.format.SSTableReader.createLinks(SSTableReader.java:1114)
	at org.apache.cassandra.io.sstable.format.SSTableReader.createLinks(SSTableReader.java:1096)
	at org.apache.cassandra.service.snapshot.TakeSnapshotTask.createSnapshot(TakeSnapshotTask.java:166)
	at org.apache.cassandra.service.snapshot.TakeSnapshotTask.call(TakeSnapshotTask.java:140)
	at org.apache.cassandra.service.snapshot.SnapshotManager.executeTask(SnapshotManager.java:526)
	at org.apache.cassandra.service.snapshot.SnapshotManager.takeSnapshot(SnapshotManager.java:419)
	at org.apache.cassandra.service.snapshot.SnapshotManager.takeSnapshot(SnapshotManager.java:435)
	at org.apache.cassandra.tools.NodeProbe.takeMultipleTableSnapshot(NodeProbe.java:933)
	at org.apache.cassandra.tools.nodetool.Snapshot.execute(Snapshot.java:97)
	at org.apache.cassandra.tools.NodeTool$NodeToolCmd.runInternal(NodeTool.java:393)
	at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:378)
	at org.apache.cassandra.tools.NodeTool.execute(NodeTool.java:277)
{noformat}
Bug: Fix Accord SAI tests, Accord double apply, and add missing interop range reads
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Nov/24 15:40
Resolved: 16/Dec/24 15:50

Fixes double apply in Accord, adds a test mode for Accord to always do interop reads, adds SAI tests using this interop mode, add missing interop read support for range reads.