av9ash/GitBugs huggingface dataset:
Bug: When resetting bootstrap progress, all the streamed data should be deleted
Status: Needs Committer
Priority: Normal
Created: 20/Nov/24 19:07

As implemented in https://issues.apache.org/jira/browse/CASSANDRA-17679

If we reset the bootstrap progress, we will clear the data in available ranges system table, the part that was not implemented is that "// TODO: Also remove the files on disk. See discussion in CASSANDRA-17679"
Bug: Grant permission on keyspaces system_views and system_virtual_schema not possible
Status: Review In Progress
Priority: Normal
Created: 27/Dec/24 22:44

Trying to grant select permission on keyspace system_views or system_virtual_schema fails with an error message stating that these resources wouldn't exist.

{code}
cassandra@cqlsh> CREATE ROLE test WITH PASSWORD = 'test' AND LOGIN = true AND SUPERUSER = false ;
cassandra@cqlsh> LIST USERS ;

 name      | super | datacenters
-----------+-------+-------------
 cassandra |  True |         ALL
      test | False |         ALL

(2 rows)
cassandra@cqlsh> GRANT SELECT PERMISSION ON KEYSPACE system TO test;
cassandra@cqlsh> GRANT SELECT PERMISSION ON KEYSPACE system_schema TO test;
cassandra@cqlsh> GRANT SELECT PERMISSION ON KEYSPACE system_views TO test;
InvalidRequest: Error from server: code=2200 [Invalid query] message="Resource <keyspace system_views> doesn't exist"
cassandra@cqlsh> GRANT SELECT PERMISSION ON KEYSPACE system_ TO test;
system_auth           system_distributed    system_schema         system_traces         system_views          system_virtual_schema
cassandra@cqlsh> GRANT SELECT PERMISSION ON KEYSPACE system_virtual_schema TO test;
InvalidRequest: Error from server: code=2200 [Invalid query] message="Resource <keyspace system_virtual_schema> doesn't exist"
{code}

However, the above permission can be added by updating role_permissions directly

{code}
cassandra@cqlsh> UPDATE system_auth.role_permissions SET permissions = permissions + {'SELECT'} WHERE role = 'test' AND resource = 'data/system_views';
cassandra@cqlsh> LIST ALL PERMISSIONS OF test;

 role | username | resource                 | permission
------+----------+--------------------------+------------
 test |     test |        <keyspace system> |     SELECT
 test |     test | <keyspace system_schema> |     SELECT
 test |     test |  <keyspace system_views> |     SELECT

(3 rows)
{code}

Reading the code unveiled, that {{system_schema.keyspaces}} doesn't list those two keyspace, which causes the grant permission command to fail.
Bug: Typos in native protocol doc
Status: Triage Needed
Priority: Normal
Created: 27/Nov/24 14:21

Some native protocol spec files have typos. Example [https://github.com/apache/cassandra/blob/trunk/doc/native_protocol_v3.spec#L719-L740]

 

Affects [CASSANDRA-19915|https://issues.apache.org/jira/projects/CASSANDRA/issues/CASSANDRA-19915?filter=allopenissues ]
Bug: replacing existing images in documentation with better look and resolution
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Dec/24 12:40
Resolved: 24/Feb/25 09:53

replacing existing images in documentation with better look and resolution

https://github.com/apache/cassandra/blob/cassandra-5.0/doc/modules/cassandra/pages/developing/data-modeling/intro.adoc

line 153 : image::Figure_1_data_model.jpg[image]
line 162 : image::Figure_2_data_model.jpg[image]

 

https://github.com/apache/cassandra/blob/cassandra-5.0/doc/modules/cassandra/pages/managing/operating/backups.adoc

line 55 : image::Figure_1_backups.jpg[Data directory structure for backups]

 

[https://github.com/apache/cassandra/blob/trunk/doc/modules/cassandra/pages/managing/operating/read_repair.adoc]

Figure_1_read_repair.jpg
Figure_2_read_repair.jpg
Figure_3_read_repair.jpg
Figure_4_read_repair.jpg
Figure_5_read_repair.jpg
Figure_6_read_repair.jpg

 

https://github.com/apache/cassandra/blob/cassandra-5.0/doc/modules/cassandra/pages/architecture/guarantees.adoc

line 33 : image::Figure_1_guarantees.jpg[image]
Bug: Prepared statements cache computation improvements
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 08/Oct/24 11:40
Resolved: 04/Nov/24 09:07

Currently prepared statements size are computed twice which is duplicate effort:
 - [https://github.com/apache/cassandra/blob/cassandra-4.0/src/java/org/apache/cassandra/cql3/QueryProcessor.java#L672]
 - [https://github.com/apache/cassandra/blob/cassandra-4.0/src/java/org/apache/cassandra/cql3/QueryProcessor.java#L802]

But we also get the side effect of that 'slow' computation locking the underlying ConcurrentHashMap. We've had nodes becoming unresponsively slow for hours with hundreds of threads in jstack all queued up behind {{computeIfAbsent}} operations on cache insertion/eviction collisions.

Computing size only once and pulling that value instead of recomputing it on cache insertion was the solution
Bug: SSTableScanner can be vastly simplified for compaction
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 19/Nov/24 10:48
Resolved: 04/Dec/24 09:06

One of the main bottlenecks for compaction performance is its use of the {{SSTableScanner}} class, whose main purpose is to implement partition range queries and as such supports filtering by row and column that is not helpful to compaction. To implement the latter it must rely on the sstable's index, adding a lot of complexity and inefficiency.

Implementing a simpler version of a scanner that reads off the data file directly for given spans of offsets would speed up compaction significantly.
Bug: [Analytics] Add support for vnodes
Status: Open
Priority: Normal
Created: 12/Nov/24 17:41

Analytics currently assumes 1 token per node, but most people in the community use vnodes by default. We should add support for vnodes, at the most basic level we can improve to issue a Spark task per vnode range. After that optimizations can be added to more intelligently merge contiguous ranges, or sort tasks to avoid the Spark JVM connecting to the entire cluster.
Bug: To improve accord interoperability test coverage, need to extend the harry model domain to handle more possible CQL states
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 19/Dec/24 20:14
Resolved: 12/Feb/25 02:24

Interop touches every read/write path and as such the testing demand needs to cover the full domain as well with proper validation.  We need to extend harry’s model logic to handle more possible states such as: by token, and filtering/index (native to the model); the larger CQL domain is planned and expected to be follow up work in their own JIRAs.
Bug: Hints ignored during Upgrade from C*4 to C*5
Status: Patch Available
Priority: Normal
Created: 27/Nov/24 15:23

I have discovered that some hints were not being processed after nodes come back up when a cluster in in a mixed mode with some cassandra 4 nodes and some cassdandra 5 nodes ( these with a storage compatibility mode CASSANDRA_4 )

 

When in this mode there is a schema mismatch after the first node has been upgraded, which continues until the last node has been upgraded.

It seems that the hints are blocked from being sent if there is a schema mismatch between the 2 nodes, that can be seen at this line. 

[cassandra/src/java/org/apache/cassandra/hints/HintsDispatchTrigger.java at cassandra-5.0 · apache/cassandra|https://github.com/apache/cassandra/blob/cassandra-5.0/src/java/org/apache/cassandra/hints/HintsDispatchTrigger.java#L65]

I have tested removing this line, and that then does allow the hint to be transferred normally. However I am not sure of the implications for doing that if the hint is for part of the schema where the actual mismatch occurs.

 

This creates the problem when a node is being upgraded and is currently down, hint files will be created for it on the new cassandra 5 nodes and the old cassandra 4 nodes, but the hint files on the old cassandra 4 nodes will not be processed, due to the schema mismatch. Leading to potential data loss.
Bug: support copyAndAddIntervals in IntervalTree for faster replaceFlushed View update
Status: Patch Available
Priority: Normal
Created: 24/Dec/24 02:01

More details in CASSANDRA-20159.

 

With CASSANDRA-20159 and this patch (marked as new), compared to cassandra-4.1 trunk (marked as trunk) we no longer see this slow finish time for replaceFlushed.

!image-2024-12-23-18-33-28-490.png|width=1209,height=392!

These spikes and slowness are no longer observed under the same stress test

!image-2024-12-23-18-37-11-939.png|width=1314,height=303!
Bug: IntervalTree should support copyAndReplace for checkpoint when ranges are unchanged
Status: Patch Available
Priority: Normal
Created: 20/Dec/24 10:46

We observed very slow compaction and sometimes stuck memtable flushing hence caused write latency spikes when the cluster has large number of SSTables (~20K), similar to what was observed in CASSANDRA-19596.

Looking deeper into when these interval tree is rebuilt - there is actually no need to do rebuild all the time for checkpoint() calls.
 
updateLiveSet(toUpdate, staged.update) this is updating the current version of SSTableReader within the View. However the update isn't always changing the ranges of the SSTableReader (low, high).
 
One Example:
 * IndexSummaryRedistribution.adjustSamplingLevels()
 ** SSTableReader replacement = sstable.cloneWithNewSummarySamplingLevel(cfs, entry.newSamplingLevel);
 ** This is changing the Metadata only and the ranges are unchanged
 

Considering this, rebuilding the entire IntervalTree will not be required, instead IntervalTree should support replacing these SSTableReader.
 
If we're rebuilding the tree, complexity is O(n(logn)^2) in current trunk as we're repeating the O(nlogn) sort on every node creation, after CASSANDRA-19596 this will be O(nlogn), but with update supported, some of the updateLiveSet calls can be optimized to O(m(logn)^2) where m is the number of SSTableReaders we attempt to replace, which we have m << n (number of SSTables) in most cases.
 
This is achieved by
 # finding the node containing the SSTable (logn)
 # binary search and replacing the SSTableReader from the node (logn)
 # To support CAS update, 1 and 2 need to done by copying the path and re-create the affected nodes on the path

 
The experiment I did was on a 2 rings setup, one on 4.1+CASSANDRA-19596 (marked as trunk), and the other on  4.1+CASSANDRA-19596+this patch (marked as new). ~15K SSTables (with LCS, sstable size was 50MB, single_uplevel enabled). stress-test with 1:1 rw ratio.

Result shows that ~15% of the checkpoint calls don't necessarily need to rebuild the tree.

!image-2024-12-20-02-41-06-544.png|width=803,height=133!

Compaction throughput (scanner read throughput) was increased from 130MB/s to 200MB/s

!image-2024-12-20-02-39-53-420.png|width=1378,height=136!

Checkpoint finish time reduced from mean ~1.5s to ~800ms

!image-2024-12-20-02-42-52-003.png|width=1100,height=186!
Bug: Tighten up permissions on system keyspaces
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Nov/24 18:30
Resolved: 05/Feb/25 10:26

* Restrict which permissions can be granted on system keyspaces
* Ensure that GRANT... ON ALL KEYSPACES excludes system keyspaces
* Add system_traces to the always readable set


CVE-2025-23015
Bug: Avoid new ByteBuffer allocation for each NativeCell/NativeClustering during flushing of offheap_objects memtable
Status: Patch Available
Priority: Normal
Created: 29/Dec/24 13:08

Currently for memtable_allocation_type: offheap_objects when we flush a memtable we allocate new ByteBuffer object for each NativeCell and NativeClustering to write it to disk. It is one of the main contributors (together with CASSANDRA-20162) to memory allocation for MemtableFlushWriter thread:

!image-2024-12-29-13-06-13-115.png|width=570!

!native_clustering_byte_buffer_alloc.png|width=570!

Instead of retrieving of the value() as ByteBuffer we can introduce NativeValueAccessor to not expose ByteBuffer + keep it inside NativeCell as a thread-local to re-use. Same idea is applicable for NativeClustering

Note: for Cassandra 4.x the situation is even worse because we clone very BTreeRow from off-heap to heap due to using the same iterator as for usual reads (where we need to protect against memtable lifecycle changes). In case of flushing such protection is not needed. For TrieMemtable we skip such cloning (CASSANDRA-17240, [https://github.com/apache/cassandra/blob/cassandra-5.0/src/java/org/apache/cassandra/db/memtable/TrieMemtable.java#L393]). It makes sense to consider such skipping for over types of memtable too..
Bug: Reduce memory allocations in miscellaneous places along write path
Status: Needs Committer
Priority: Normal
Created: 26/Dec/24 22:04

Avoid unnecessary allocations in different places along write path:
* avoid iterator allocations if possible
* handle typical cases (such as a single row, single table writes) more efficiently
* add fast paths for typical scenarios (like pending ranges is empty)
* memorize things which can be computed once
Bug: cqlsh autocompletion for role names/user names is broken
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 31/Dec/24 17:53
Resolved: 04/Feb/25 10:06

In CASSANDRA-6307 cqlsh switched from using cassandra-dbapi2 to python-driver. The autocomplete functionality for role names and user names relied on the {{is_valid_cql3_name}} function which was removed as part of CASSANDRA-6307, but the {{username_name_completer}} and {{rolename_completer}} functions were missed and were not updated to use the new {{maybe_escape_name}} function.
Bug: Improve Accord Work Queueing (and misc perf improvements)
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 08/Nov/24 15:38
Resolved: 04/Dec/24 09:33

Work can now be cancelled, and commands serving remote requests now cancel queued work if they time out. This prevents run-away work growth, as work does not outlive its useful lifespan.

CommandStores, threads and caches are now only loosely coupled, with it possible to independently tune:
 * the total number of threads
 * the number of work queue/cache units we distribute the threads amongst
 * the number of CommandStores per table/shard (which are distributed between queue/cache units, the threads of which will execute CommandStore work)

Mutual exclusivity is managed separately for the queue/cache unit and each CommandStore, and the locks are held only for as long as necessary so we can have multiple threads servicing the same CommandStore(s). Given this threading model, it is now possible for Accord threads to perform all of the loading/saving work, reducing the queueing delay and context switching costs - and this is also configurable. The default configuration is now to do this work on the Accord work pool (as already the case with Paxos for most IO). Accord state reads can be scheduled and completed from any thread, so that we do not incur multiple queue delays when preparing work for a command store. There are further improvements that can be made here to permit the event loop to answer in-cache queries, and under a future async-io model to directly submit read requests.

Misc perf improvements:
 * Write directly to Memtable for cache evictions, using putIfAbsent immediately
 * Use LCS on CommandsForKey table for faster reads
 * Flatten UUID fields into TableId to reduce indirection for comparisons
 * Introduce asymmetric comparisons to BTreeMap for faster schema lookups
 * Read TimestampsForKey directly to avoid parsing CQL
 * Save summary information in RedundantBefore to short-circuit executions
 * Send Stable message only as necessary on Execute, to reduce load on replicas
 * Ensure journal entries are immediately visible to replay without handing over to another thread, so on normal path can avoid context switch and listener overheads
 * Journal periodic mode should fsync only as necessary
 * Use OpOrder to guard Journal Segment read access (avoiding taking individual references, which can be costly)
 * AccordCache can “shrink” (serialise) entries instead of evicting, to increase effective capacity (evicting any already-shrunk entries that are encountered)
 * EphemeralRead cache entries are evicted only on timeout of the remote request, and are not otherwise persisted
 * Work is scheduled by first arrival time, not by read completion time - work that is slow to read jumps the queue once data is in memory to serve it, to reduce latency variability
 * Some operations may now partially execute without waiting for all state to be brought into memory
Bug: Add metric and tracing event for scanned purgeable tombstones
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 08/Dec/24 13:02
Resolved: 27/Jan/25 21:18

Currently, Cassandra can read data from SSTables with tombstones and later drop them silently if the tombstones are older than gc_grace_seconds (aka purgeable tombstones). Such tombstones are not visible via readTombstoneHistogram metric, not reported in Cassandra logs if tombstone threshold is crossed and not mentioned in tracing events. As a result if a partition has a lot of purgeable tombstones we may have a slow read query without any signs why it is it slow. Example: [^trace_sample.txt]

This suggested improvement adds:
1) a new metric which tracks number of such tombstones: PurgeableTombstoneScannedHistogram
2) a new tracing event: "Read {} purgeable tombstone cells" if number of such tombstones > 0

 

Implementation notes: the logic of new withMetricsRecording iterator is an adjusted version of existing withMetricsRecording iterator
Bug: Test failure: compaction_test.py::TestCompaction::test_compaction_throughput 
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 17/Dec/24 17:10
Resolved: 23/Jan/25 18:43

Seeing this fail consistently on trunk with:

{code}
>       assert float(threshold) + 0.5 >= avgthroughput_mb
E       AssertionError: assert (5.0 + 0.5) >= 5.642
E        +  where 5.0 = float('5')

compaction_test.py:302: AssertionError
{code}
Bug: Reduce memory allocation in Cassandra write path
Status: In Progress
Priority: Normal
Created: 26/Dec/24 17:25

The epic accumulates improvements about reducing of memory allocations in Cassandra write path. 
The drivers for this epic are:
* to reduce GC pressure -> reduce GC overhead and GC pauses frequency/duration. In case of low-latency GC reducing of memory allocation also helps to move forward the threshold when the system throughput is limited by the memory allocation throughput a GC is able to process without slowing down an app. 
* memory allocations also help to highlight the places to optimize from CPU usage point of view by just doing less work

Related link: https://blog.vanillajava.blog/2022/09/java-is-very-fast-if-you-dont-create.html

Test results: https://docs.google.com/spreadsheets/d/1ZoIGKl08OXvM9tiV5q2IdfxwvMK9GJDd_jTsNdfdqyY/edit?usp=sharing
Bug: Discard hints based on write time, not timeout time
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Oct/24 18:28
Resolved: 21/Jan/25 20:18

Hints are created after a write timeout are created with the timeout time as the hint creation time. In the case of slow hint delivery, this can create a window of time where a write is applied after gcgs would have elapsed for tombstones written after the original write, and the tombstone has been purged, causing data resurrection. We should use the time the client request thread started working on the request as the hint creation time.
Bug: Migrate route index from commands table to journal, and drop the commands table
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/Dec/24 21:21
Resolved: 18/Jan/25 00:06

Right now commands table only has range transactions and this is to avoid breaking the route index; the durability of transactions was migrated to journal.  We should cleanup this relationship and push the index into the journal so we can drop this special logic.
Bug: Integrate SAI's index state and query performance metrics w/ nodetool tablestats
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Oct/24 17:17
Resolved: 17/Jan/25 18:51

The output structure of {{nodetool tablestats}} might be amenable to adding a number of SAI state/performance metrics, which are predominantly table-scoped. (There are some that are scoped at the column level, of course, but the table-level items could be useful if we don't want to directly call JMX and want a general idea of how we're doing with things like post-filtering read latency.)

The decisions we have to make are along the lines of "What percentile to use for post-filtering read latency?", but I imagine we can follow whatever existing pattern exists.
Bug: Enable filtering on keyspace, table and snapshot name in nodetool listsnapshots
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Dec/24 20:03
Resolved: 16/Jan/25 13:57

There might in practice hundreds of snapshots a user has a hard time navigate through in the current output of nodetool listsnapshots.

The patch would consist of adding flags to listsnapshots so we can narrow down what snapshots to display.
Bug: Create snapshot manifest upon loading where it does not exist or enrich it
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Dec/24 13:26
Resolved: 16/Jan/25 13:57

    In 4.0 times, there was no snapshot manifest created. When such
    node is upgraded to 5.1, no snapshots would be found, because even
    they would be loaded, snapshot's presence is logically determined by the
    existence of at least one manifest.json file in any data dir. That
    would result in snapshots not being shown e.g. in the output of
    nodetool listsnapshots.
    
    The fix consists of creating a manifest in each snapshot directory when not present.
    
    When a manifest.json exists, its older format has not contained created_at field.
    When this is detected, we will proceed to enrich such manifest by overwriting it
    with a manifest of new format.
Bug: nodetool listsnapshots initiates two jmx calls unnecessarily
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Dec/24 10:52
Resolved: 16/Jan/25 13:57

listsnapshots calls jmx on two occasions

1) when getting trueSnapshotsSize
2) when getting snapshot details

What it was doing in practice before CASSANDRA-18111 was that it was essentially loading the snapshots twice hence almost double the time to list.

What it is doing after CASSANDRA-18111 is that it still resolves sizes but in a way more efficient way however it still calls jmx twice which is not necessary.

The fix consists of returning resolved sizes together with snapshot details and then on the client size we just display and sum it up in case of wanting to know the total. 

It is more optimal to just return results in one response and sum few numbers on the client from the results than to call jmx twice. 

It is, actually, safer too. Because for now, total true snapshots size is returned _before_ we list snapshots. So what can happen, albeit quite improbably, is that a snapshot can be removed _after_ we call total true snapshot size method but _before_ we list all snapshots. So that means that total true disk size would be bigger than the sum of true disk sizes for all snapshots after one of them was removed. 

This also means that a user will get true raw size (raw means not human friendly, human friendly is returned in the response already) of a snapshot in the result of snapshot details automatically. They do not need to call extra MBean method for that (which this patch opportunistically introduces too).