av9ash/GitBugs huggingface dataset:
Bug: Use direct IO for compaction reads
Status: Triage Needed
Priority: Normal
Created: 09/Oct/24 18:54

If we use direct io to read SSTables during compaction, we can avoid polluting the page cache with data we're about to delete.  As another side effect, we also evict pages to make room for whatever we're putting in.  This unnecessary churn leads to higher CPU overhead and can cause dips in client read latency, as we're going to be evicting pages that could be used to serve those reads.

This is most notable with STCS as the SSTables get larger, potentially evicting the entire hot dataset out of cache, but is affected by every compaction strategy.

This is a follow up to be done after CASSANDRA-15452 since we will have an internal buffer.
Bug: use direct io when streaming through the JVM
Status: Triage Needed
Priority: Normal
Created: 09/Oct/24 18:56

In the case of the non-ZCS streaming path, we read entire SSTables.  This causes a high rate of page cache churn and should be avoided.  Using direct IO we will avoid the churn and we should see read latency remain stable, rather than jittery as a result of decreased page cache hit rate.

This affects both our streaming slow path and when sending entire files.
Bug: Consider setting -Djdk.nio.MaxCachedBufferSize by default
Status: Triage Needed
Priority: Normal
Created: 02/Oct/24 21:54

Long time ago CASSANDRA-8670 added a fix of native memory leaking through JDK-level thread-local caches for direct buffers. 
From one side since that time there were a lot of changes in Cassandra IO and networking logic. From another side https://bugs.openjdk.org/browse/JDK-8147468 - a JDK-level fix for the issue (by adding of -Djdk.nio.MaxCachedBufferSize option) was implemented some time after CASSANDRA-8670 and backported to JDK8.

Since that time there were many places where the JDK option has been mentioned in a similar context as a possible way to protect against direct byte buffer memory leakage:
- CASSANDRA-10689
- CASSANDRA-13931
- CASSANDRA-15006
- CASSANDRA-15400
- https://support.datastax.com/s/article/JVM-OOM-direct-buffer-errors-affected-by-unlimited-javanio-cache (DSE currently sets it by default to 1Mb)
- https://github.com/k8ssandra/cass-operator/blob/19ed28196f2549a763ec7d3794328565f6d4b127/docs/user/jvm_server_configuration.md?plain=1#L207

I have created the ticket to consider if it makes sense to add -Djdk.nio.MaxCachedBufferSize by default to Cassandra jvm-server.options.
Bug: Allow TWCS to use UCS within a window
Status: Triage Needed
Priority: Normal
Created: 01/Oct/24 16:58

Since TWCS is using STCS within a window, it makes the sstables inelligible for zero copy streaming.  That is a bummer, because time series use cases typically have denser nodes.  This makes the penalty from using TWCS very impactful when adding new nodes. 

We should be able to use UCS for the window so we can get the benefits of ZCS when bootstrapping new nodes.
Bug: Virtual table to query partition keys in a table
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 21/Dec/24 16:11
Resolved: 05/Feb/25 11:51

Introduce a new virtual table, {{{}PrimaryIdTable{}}}, to allow users to query partition keys and related metadata for a specific table within a keyspace. This feature provides insights into SSTable-level details, such as token values, size estimates, and SSTable counts, without requiring expensive disk-based operations.
Bug: Fix TRACING when PAGING so it doesn't repeat the tracing multiple times
Status: Open
Priority: Low
Created: 20/Nov/24 22:22

When paging is used, the trace is repeated multiple times.  For example,
{quote}> PAGING 2

> TRACING ON

> select * from system_schema.columns LIMIT 10;
{quote}
Also, for consistency with the other ON/OFF flags used with Paging, the Capture message should be updated as below.  This could work well if it *used on_off_switch_with_value,* but that only has OFF & Value and would need to be tweaked to support a string value.
 * Capture is ON, writing to '/tmp/output'
 * Capture set to OFF
 * Capture is OFF
Bug: Support simple length functions for blob and string
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 21/Nov/24 15:36
Resolved: 22/Jan/25 11:12

A common question from users is how to query for the length of {{blob}} and {{text}} columns, and the answer is often for users to enable UDFs and then write their own ([1|https://stackoverflow.com/questions/49674114/cassandra-query-string-length], [2|https://stackoverflow.com/questions/58401750/how-can-i-get-size-of-blob-column-of-cassandra-using-user-defined-functions], ...). I think it would make sense for Cassandra to support a subset of the SQL99 [binary length functions|https://sql-99.readthedocs.io/en/latest/chapters/05.html#octet-length-expression] and a UTF8 codepoint aware {{length}} function similar to PostgreSQL's [{{length}}|https://www.postgresql.org/docs/9.1/functions-string.html] function.

Specifically I propose two new system functions:
# {{octet_length}} - Defined on all types and returns the number of underlying bytes in the bytebuffer representation of the column. Does _not_ account for storage overhead (e.g. does not account for metadata)
# {{length}} - Defined on UTF8 strings, and returns the number of UTF8 codepoints in the string, equivalent to Java's [{{String.length}}|https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/lang/String.html#length()] function.

Patch coming shortly. Particularly interested in feedback if we A) think these functions are reasonable to have inside Cassandra, and B) like the names of the functions to align with SQL or if we want to go nonstandard and do like {{blength}} instead of octet length. 

An example of the difference between octet length and string length

{code}
// A table with text key and blob value
cqlsh> select key, value from test.test;

 key   | value
-------+--------
 hello | 0x12af

cqlsh> select length(key), octet_length(value) from test.test;

 system.length(key) | system.octet_length(value)
--------------------+---------------------------
                  5 |                         2

cqlsh> INSERT INTO test.test (key, value) VALUES ('こんにちは世界', 0x12AF);
cqlsh> select length(key), octet_length(key), octet_length(value) from test.test where key = 'こんにちは世界';

 system.length(key) | system.octet_length(key) | system.octet_length(value)
--------------------+-------------------------+---------------------------
                  7 |                      21 |                         2
{code}
Bug: AbstractCommitLogService#lastSyncedAt initialized with currentTimeMillis() but later compared and updated with System.nanoTime() 
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 12/Nov/24 13:12
Resolved: 15/Nov/24 11:12

Within org.apache.cassandra.db.commitlog.AbstractCommitLogService

we use System.currentMillis()-based time to initialize lastSyncedAt field:
{code:java}
protected volatile long lastSyncedAt = currentTimeMillis();{code}
but later we use clock.now() = System.nanoTime() to compare and update the field value:
{code:java}
long pollStarted = clock.now();
boolean flushToDisk = lastSyncedAt + syncIntervalNanos <= pollStarted || state != NORMAL || syncRequested; {code}
As of now we are lucky that in the current JDK implementations System.nanoTime() is much bigger than currentTimeMillis() but it is not guaranteed by a specification of the API. If  nanoTime() < currentTimeMillis() then we can stuck without sync of commit log to disk.

So, we need to align the time sources and change the init logic to:
{code:java}
protected volatile long lastSyncedAt = preciseTime.now();{code}
Bug: Remove SparkSQL dependency from CassandraBridge so that it can be used independent from Spark
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 04/Oct/24 17:02
Resolved: 17/Oct/24 17:02

The Spark dependency can be decoupled from CassandraBridge to enable a standalone version-independent Cassandra API to interface with Cassandra independent from Spark. This permits the sstable reader and commitlog code to be used without Spark dependency by CDC and other tooling projects.
Bug: MemtablePostFlush deadlock leads to stuck nodes and crashes
Status: Patch Available
Priority: Urgent
Created: 16/Apr/24 20:59

I've run into an issue on a 4.1.4 cluster where an entire node has locked up due to what I believe is a deadlock in memtable flushing. Here's what I know so far.  I've stitched together what happened based on conversations, logs, and some flame graphs.

*Log reports memtable flushing*

The last successful flush happens at 12:19. 
{noformat}
INFO  [NativePoolCleaner] 2024-04-16 12:19:53,634 AbstractAllocatorMemtable.java:286 - Flushing largest CFS(Keyspace='ks', ColumnFamily='version') to free up room. Used total: 0.24/0.33, live: 0.16/0.20, flushing: 0.09/0.13, this: 0.13/0.15
INFO  [NativePoolCleaner] 2024-04-16 12:19:53,634 ColumnFamilyStore.java:1012 - Enqueuing flush of ks.version, Reason: MEMTABLE_LIMIT, Usage: 660.521MiB (13%) on-heap, 790.606MiB (15%) off-heap
{noformat}

*MemtablePostFlush appears to be blocked*

At this point, MemtablePostFlush completed tasks stops incrementing, active stays at 1 and pending starts to rise.
{noformat}
MemtablePostFlush   1    1   3446   0   0
{noformat}
 
The flame graph reveals that PostFlush.call is stuck.  I don't have the line number, but I know we're stuck in {{org.apache.cassandra.db.ColumnFamilyStore.PostFlush#call}} given the visual below:

*!image-2024-04-16-13-43-11-064.png!*


*Memtable flushing is now blocked.*

All MemtableFlushWriter threads are Parked waiting on {{{}OpOrder.Barrier.await{}}}. A wall clock profile of 30s reveals all time is spent here.  Presumably we're waiting on the single threaded Post Flush.

!image-2024-04-16-12-29-15-386.png!

*Memtable allocations start to block*

Eventually it looks like the NativeAllocator stops successfully allocating memory. I assume it's waiting on memory to be freed, but since memtable flushes are blocked, we wait indefinitely.

Looking at a wall clock flame graph, all writer threads have reached the allocation failure path of {{MemtableAllocator.allocate()}}.  I believe we're waiting on {{signal.awaitThrowUncheckedOnInterrupt()}}
{noformat}
 MutationStage    48    828425      980253369      0    0{noformat}
!image-2024-04-16-11-55-54-750.png!

 

*Compaction Stops*

Since we write to the compaction history table, and that requires memtables, compactions are now blocked as well.

 

!image-2024-04-16-13-53-24-455.png!

 

The node is now doing basically nothing and must be restarted.
Bug: Unbounded queues in native transport requests lead to node instability
Status: Resolved
Priority: Urgent
Resolution: Fixed
Created: 05/Apr/24 20:11
Resolved: 31/May/24 09:36

When a node is under pressure, hundreds of thousands of requests can show up in the native transport queue, and it looks like it can take way longer to timeout than is configured.  We should be shedding load much more aggressively and use a bounded queue for incoming work.  This is extremely evident when we combine a resource consuming workload with a smaller one:

Running 5.0 HEAD on a single node as of today:
{noformat}
# populate only
easy-cass-stress run RandomPartitionAccess -p 100  -r 1 --workload.rows=100000 --workload.select=partition --maxrlat 100 --populate 10m --rate 50k -n 1

# workload 1 - larger reads
easy-cass-stress run RandomPartitionAccess -p 100  -r 1 --workload.rows=100000 --workload.select=partition --rate 200 -d 1d

# second workload - small reads
easy-cass-stress run KeyValue -p 1m --rate 20k -r .5 -d 24h{noformat}
It appears our results don't time out at the requested server time either:

 
{noformat}
                 Writes                                  Reads                                  Deletes                       Errors
  Count  Latency (p99)  1min (req/s) |   Count  Latency (p99)  1min (req/s) |   Count  Latency (p99)  1min (req/s) |   Count  1min (errors/s)
 950286       70403.93        634.77 |  789524       70442.07        426.02 |       0              0             0 | 9580484         18980.45
 952304       70567.62         640.1 |  791072       70634.34        428.36 |       0              0             0 | 9636658         18969.54
 953146       70767.34         640.1 |  791400       70767.76        428.36 |       0              0             0 | 9695272         18969.54
 956833       71171.28        623.14 |  794009        71175.6        412.79 |       0              0             0 | 9749377         19002.44
 959627       71312.58        656.93 |  795703       71349.87        435.56 |       0              0             0 | 9804907         18943.11{noformat}
 

After stopping the load test altogether, it took nearly a minute before the requests were no longer queued.
Bug: Split the metrics collection into smaller groups
Status: Patch Available
Priority: Urgent
Created: 28/May/24 19:02

Historically, all metrics are stored in a single collection indexed by a unique metric name - {{{}ConcurrentMap<String, Metric>{}}}. This storage technique doesn't offer any advantages over other options we might have below, and causes some performance and usability issues when we need to traverse this huge collection while filtering the metrics.

The following possible solutions could be used: 
- Use https://www.javadoc.io/doc/io.dropwizard.metrics/metrics-core/4.0.2/com/codahale/metrics/SharedMetricRegistries.html , which provides a way to store metrics in their dedicated groups;
- Implement a hierarchical storage structure, based on the knowledge of how the metric key is formed. For example, https://the-asf.slack.com/archives/CK23JSY2K/p1716568216701789?thread_ts=1716324924.466269&cid=CK23JSY2K

Each of the above options assumes, that the metric groups are statically registered and known at the time a node starts. This may also be an option to consider for dynamically registering metric groups and in turn registering corresponding virtual tables and JmxMbean. In the latter case, we'll need to implement the notification mechanism for the drivers, as they are not currently notified about newly created virtual tables.
Bug: Data Corruption during Schema Alterations 
Status: Triage Needed
Priority: Urgent
Created: 24/May/24 18:06

h2. Description

 
*Overview:* The primary issue is data corruption occurring during schema alterations (ADD/DROP column) on large tables(300+ columns and 6TB size ) in the production cluster. This problem has been replicated on multiple clusters, running Apache Cassandra version 4.0.12 and Datastax Java Driver Version: 4.17

*Details:*

*Main Issue:*
 * *Data Corruption:* When dynamically adding a column to a table, the data intended for the new column is shifted, causing misalignment in the data.
 * *Symptoms:* The object implementing {{com.datastax.oss.driver.api.core.cql.Row}} returns values shifted against the column names returned by {{{}row.getColumnDefinitions(){}}}. The driver returns a corrupted row, leading to incorrect data insertion.

*Additional Issues:*

*Exceptions:*
 * {{java.nio.BufferUnderflowException}} during batch reads when ALTER TABLE ADD/DROP column statements are issued.
 * {{java.lang.ArrayIndexOutOfBoundsException}} in some cases.

 * -Buffer underflow exceptions with messages like "Invalid 32-bits integer value, expecting 4 bytes but got 292".- [This is no longer issue after upgrading to V5]

 * {-}OOM errors mostly occur during ADD column operations, while other exceptions occur during DELETE column operations.{-}[This is no longer issue after upgrading to V5]

 * *Method Specific:* Errors occur specifically with {{{}row.getList(columnName, Float.class){}}}, returning incorrect values.

*Reproducibility:*
 * The issue is reproducible on larger tables (300 columns, 6 TB size) but not on smaller tables.
 * SELECT * statements are used during reads

 * *Method Specific:* Errors occur specifically with {{{}row.getList(columnName, Float.class){}}}, returning incorrect values. However, the code registers a driver exception when calling the method {{{}row.getList(columnName, Float.class){}}}. We pass the exact column name obtained from {{{}row.getColumnDefinition{}}}, but it returns the wrong value for a column with this name. This suggests that the issue lies with the driver returning an object with incorrect properties, rather than with the SQL query itself.

*Debugging Efforts:*
 * *Metadata Refresh:* Enabling metadata refresh did not resolve the issue.
 * *Schema Agreement:* {{session.getCqlSession().checkSchemaAgreement()}} did not detect inconsistencies during test execution.

*Additional information about misalignment:*

1) *Result Set returned by the driver has mixed up values for the column type “text”.*

This is observed in {*}custom application ({*}plain java application using Datastax java driver 4.17)

 

Example:

In Cassandra

Column 1 text – Value 1

Column 2 text – Value 2

Column 3 text – Value 3

 

In the result set retrieved by the *custom application using Datastax Java driver 4.17 before any processing* 

Column 1 text – Value 2

Column 2 text – Value 1

Column 3 text – Value 3
Bug: Fix building debian packages
Status: Resolved
Priority: Urgent
Resolution: Fixed
Created: 01/May/24 18:47
Resolved: 02/May/24 13:50

Trying to run cassandra-deb-packaging.sh will result in the docker image looping:

{noformat}
Errors were encountered while processing:
 ed
 quilt
 cassandra-build-deps
E: Sub-process /usr/bin/dpkg returned an error code (1)
(Reading database ... 36721 files and directories currently installed.)
Removing cassandra-build-deps (5.0~beta2-20240501gitae9be29918) ...
mk-build-deps: Unable to install all build-dep packages
mk-build-deps failed… trying again after 10s… 
{noformat}
Bug: Honour previous parameter defaults between builds in Jenkinsfile
Status: Resolved
Priority: High
Resolution: Duplicate
Created: 08/Apr/24 07:15
Resolved: 14/Apr/24 11:27

When the Jenkinsfile is being used in a job that was created by dsl and intended to be triggered by scm polling, e.g. post-commit, that dsl will want to set alternative defaults for the Jenkinsfile parameters.

 

By default each build of the Jenkinsfile will restore the job's default parameters (i.e. changing the defaults for the next build).

 

This is described in more detail here
[https://www.linkedin.com/pulse/build-jenkins-job-default-parameters-using-bipin-kumar-chaurasia/] 

and summarised in a quick answer here
[https://stackoverflow.com/questions/57745451/how-can-i-override-a-jenkinsfiles-default-parameters]

 

Patch for this is here:
[https://github.com/apache/cassandra/compare/trunk...thelastpickle:cassandra:mck/jenkinsfile-persist-parameters]

 

I'm waiting on INFRA-25694 to see if there are any other problems/changes in the Jenkinsfile when it's running in ci-cassandra.a.o
Bug: IntervalTree build throughput is low enough to be a bottleneck
Status: Patch Available
Priority: Normal
Created: 29/Apr/24 16:35

With several terabytes of data and 8 compactors it’s possible for the compactors to spend a lot of time blocked waiting on IntervalTrees to be built.

There is also a lot of wasted CPU because it’s updated optimistically so most of them end up being thrown away.

This can end up being quite painful because it can block memtable flushing as well and then a single slow CFS can block unrelated CFS because the memtable post flush executor is single threaded and shared across all CFS.
Bug: some unit tests should generate files in the tmp directory
Status: Patch Available
Priority: Normal
Created: 31/May/24 13:15

I run "{*}_ant test_{*}" to fire the whole test suit cases in my local machine, and found some UTs had generated files in current directory, otherwise the tmp directory.

 
{code:java}
[root@vm-24-5-centos cassandra]# git status
# audit/
# compaction.log
# import_cql_test_keyspace_table_testcopyonlythoserowsthatmatchvectortyp_04.err
{code}
 

These problematic UTs are

 
{code:java}
ant testsome -Dtest.name=org.apache.cassandra.service.StorageServiceServerTest -Dtest.methods=testAuditLogEnableLoggerNotFound
ant testsome -Dtest.name=org.apache.cassandra.service.StorageServiceServerTest -Dtest.methods=testAuditLogEnableLoggerTransitions
ant testsome -Dtest.name=org.apache.cassandra.tools.CompactionStressTest -Dtest.methods=testWriteAndCompact
ant testsome -Dtest.name=org.apache.cassandra.tools.cqlsh.CqlshTest -Dtest.methods=testCopyOnlyThoseRowsThatMatchVectorTypeSize
{code}
 

The patch is aimed to generate files in the tmp directory to fix it.
Bug: Replaced node is stuck in a loop calculating ranges
Status: Patch Available
Priority: Normal
Created: 11/May/24 04:25

Hello,

 

I am running into an issue where in a node that is replacing a dead (non-seed) node is stuck in calculating ranges forever. It eventually succeeds, however the time taken for calculating the ranges is not constant. I do sometimes see that it takes 24 hours to calculate ranges for each keyspace. Attached the flume graph of the cassandra process during this time, which points to the below code. 
{code:java}
Multimap<InetAddressAndPort, Range<Token>> getRangeFetchMapForNonTrivialRanges()
{
//Get the graph with edges between ranges and their source endpoints
MutableCapacityGraph<Vertex, Integer> graph = getGraph();
//Add source and destination vertex and edges
addSourceAndDestination(graph, getDestinationLinkCapacity(graph));
int flow = 0;
MaximumFlowAlgorithmResult<Integer, CapacityEdge<Vertex, Integer>> result = null;
//We might not be working on all ranges
while (flow < getTotalRangeVertices(graph))
{
if (flow > 0)
{ //We could not find a path with previous graph. Bump the capacity b/w endpoint vertices and destination by 1 incrementCapacity(graph, 1); }
MaximumFlowAlgorithm fordFulkerson = FordFulkersonAlgorithm.getInstance(DFSPathFinder.getInstance());
result = fordFulkerson.calc(graph, sourceVertex, destinationVertex, IntegerNumberSystem.getInstance());
int newFlow = result.calcTotalFlow();
assert newFlow > flow; //We are not making progress which should not happen
flow = newFlow;
}
return getRangeFetchMapFromGraphResult(graph, result);
}
{code}
Digging through the logs, I see the below log line for a given keyspace `system_auth`
{code:java}
INFO [main] 2024-05-10 17:35:02,489 RangeStreamer.java:330 - Bootstrap: range Full(/10.135.56.214:7000,(5080189126057290696,5081324396311791613]) exists on Full(/10.135.56.157:7000,(5080189126057290696,5081324396311791613]) for keyspace system_auth{code}
corresponding code:
{code:java}
for (Map.Entry<Replica, Replica> entry : fetchMap.flattenEntries())
logger.info("{}: range {} exists on {} for keyspace {}", description, entry.getKey(), entry.getValue(), keyspaceName);{code}
BUT do not see the below line for the corresponding keyspace
{code:java}
RangeStreamer.java:606 - Output from RangeFetchMapCalculator for keyspace{code}
this means the code it's stuck in `getRangeFetchMap();`
{code:java}
Multimap<InetAddressAndPort, Range<Token>> rangeFetchMapMap = calculator.getRangeFetchMap();
logger.info("Output from RangeFetchMapCalculator for keyspace {}", keyspace);{code}
Here is the cluster topology:
 * Cassandra version: 4.0.12
 * # of nodes: 190
 * Tokens (vnodes): 128
 * Profile Attached

Initial hypothesis was that the graph calculation was taking longer due to the combination of nodes + tokens + tables but in the same cluster I see one of the node joined without any issues. 
wondering if I am hitting a bug causing it to  work sometimes but get into an infinite loop some times?
Please let me know if you need any other details and appreciate any pointers to debug this further.