av9ash/GitBugs huggingface dataset:
Bug: Unable to contact any seeds with node in hibernate status
Status: Open
Priority: Normal
Created: 22/Apr/24 10:36

We have customer running into the error 'Unable to contact any seeds!' . I have been able to reproduce this issue if I kill Cassandra as its joining which will put the node into hibernate status. Once a node is in hibernate it will no longer receive any SYN messages from other nodes during startup and as it sends only itself as digest in outbound SYN messages it never receives any states in any of the ACK replies. So once it gets to the check `seenAnySeed` in it fails as the endpointStateMap is empty.

 

A workaround is copying the system.peers table from other node but this is less than ideal. I tested modifying maybeGossipToSeed as follows:
{code:java}
    /* Possibly gossip to a seed for facilitating partition healing */
    private void maybeGossipToSeed(MessageOut<GossipDigestSyn> prod)
    {
        int size = seeds.size();
        if (size > 0)
        {
            if (size == 1 && seeds.contains(FBUtilities.getBroadcastAddress()))
            {
                return;
            }
            if (liveEndpoints.size() == 0)
            {
                List<GossipDigest> gDigests = prod.payload.gDigests;
                if (gDigests.size() == 1 && gDigests.get(0).endpoint.equals(FBUtilities.getBroadcastAddress()))
                {
                    gDigests = new ArrayList<GossipDigest>();
                    GossipDigestSyn digestSynMessage = new GossipDigestSyn(DatabaseDescriptor.getClusterName(),
                                                                           DatabaseDescriptor.getPartitionerName(),
                                                                           gDigests);
                    MessageOut<GossipDigestSyn> message = new MessageOut<GossipDigestSyn>(MessagingService.Verb.GOSSIP_DIGEST_SYN,
                                                                                          digestSynMessage,
                                                                                          GossipDigestSyn.serializer);
                    sendGossip(message, seeds);
                }
                else
                {
                    sendGossip(prod, seeds);
                }
            }
            else
            {
                /* Gossip with the seed with some probability. */
                double probability = seeds.size() / (double) (liveEndpoints.size() + unreachableEndpoints.size());
                double randDbl = random.nextDouble();
                if (randDbl <= probability)
                    sendGossip(prod, seeds);
            }
        }
    }
 {code}
Only problem is this is the same as SYN from shadow round. It does resolve the issue however as then receive an ACK with all the states.
Bug: Unicode Code Points incorrectly sized in protocol response
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 08/Apr/24 11:20
Resolved: 22/Feb/25 22:32

Within a query, we have sent in a character which is \U0010FFFF - the highest permissible unicode character point. This is encoded in UTF-8 using 4 bytes and sent. When the query issues a warning in the response (such as a tombstone warning which includes the query sent), the warning string in the protocol is specified as a short , followed by the string.

 

CBUtil.WriteString gets the length using the following code:
{code:java}
int length = TypeSizes.encodedUTF8Length(str);{code}
This in turn gets the length of the string based on a calculation:
{noformat}
public static int encodedUTF8Length(String st)
{
int strlen = st.length();
int utflen = 0;
for (int i = 0; i < strlen; i++)
{
int c = st.charAt(i);
if ((c >= 0x0001) && (c <= 0x007F))
utflen++;
else if (c > 0x07FF)
utflen += 3;
else
utflen += 2;
}
return utflen;
}{noformat}
The use of the st.length within this function causes problems - its considering the string as utf-16, so the 4 byte UTF-8 value is treated as a 2 character utf-16 value, both of which are high values and considered to be 3 bytes in length each, making a total length of 6 bytes.
 
Using some test code:
{noformat}
import java.nio.charset.StandardCharsets;
byte[] utf8Bytes = {(byte)244, (byte)143, (byte)191, (byte)191};
var st = new String(utf8Bytes, StandardCharsets.UTF_8);
System.out.println(st);
int strlen = st.length();
System.out.println(strlen);
int utflen = 0;
for (int i = 0; i < strlen; i++)
{
  int c = st.charAt(i);
  if ((c >= 0x0001) && (c <= 0x007F))
    utflen++;
  else if (c > 0x07FF) {
    utflen += 3;
  }
  else
    utflen += 2;
}
System.out.println(utflen);
byte[] utf8Bytes = st.getBytes(StandardCharsets.UTF_8);
for (byte b : utf8Bytes) {
  System.out.print(b & 0xFF);
  System.out.printf(" ");
}
{noformat}
The 4 byte UTF-8, is seen by st.length as 2, which then considered the value of each utf-16  as 56319 and 57343 respectively, and since this is above the 2047 (0x07FF),  adds 3 to the length each time.

The response message at a byte level does correctly return the UTF-8 character in as 244 143 191 191, but the incorrect length results in a buffer overread, which offsets the following reads, resulting in a few different possible errors, but all relating to misalignment of the buffer read vs expected value at that point in the buffer.

 

Issue specifically found in 4.1, but appears to have existed for a while - and is specifically due to operating outside of the UTF-16 BMP range but into the higher planes.
Bug: Newly inserted prepared statements got evicted too early from cache that leads to race condition
Status: Triage Needed
Priority: Normal
Created: 13/Jun/24 20:34

We're upgrading from Cassandra 4.0 to Cassandra 4.1.3 and system.prepared_statements table size start growing to GB size after upgrade. This slows down node startup significantly when it's doing preloadPreparedStatements

I can't share the exact log but it's a race condition like this:
 # [Thread 1] Receives a prepared request for S1. Attempts to get S1 in cache
 # [Thread 1] Cache miss, put this S1 into cache
 # [Thread 1] Attempts to write S1 into local table
 # [Thread 2] Receives a prepared request for S2. Attempts to get S2 in cache
 # [Thread 2] Cache miss, put this S2 into cache
 # [Thread 2] Cache is full, evicting S1 from cache
 # [Thread 2] Attempts to delete S1 from local table
 # [Thread 2] Tombstone inserted for S1, delete finished
 # [Thread 1] Record inserted for S1, write finished

Thread 2 inserted a tombstone for S1 earlier than Thread 1 was able to insert the record in the table. Hence the data will not be removed because the later insert has newer write time than the tombstone.

Whether this would happen or not depends on how the cache decides what’s the next entry to evict when it’s full. We noticed that in 4.1.3 Caffeine was upgraded to 2.9.2 CASSANDRA-15153

 

I did a small research in Caffeine commits. It seems this commit was causing the entry got evicted to early: Eagerly evict an entry if it too large to fit in the cache(Feb 2021), available after 2.9.0: [https://github.com/ben-manes/caffeine/commit/464bc1914368c47a0203517fda2151fbedaf568b]

And later fixed in: Improve eviction when overflow or the weight is oversized(Aug 2022), available after 3.1.2: [https://github.com/ben-manes/caffeine/commit/25b7d17b1a246a63e4991d4902a2ecf24e86d234]
{quote}Previously an attempt to centralize evictions into one code path led to a suboptimal approach ([{{464bc19}}|https://github.com/ben-manes/caffeine/commit/464bc1914368c47a0203517fda2151fbedaf568b]
). This tried to move those entries into the LRU position for early eviction, but was confusing and could too aggressively evict something that is desirable to keep.
{quote}
 

I upgrade the Caffeine to 3.1.8 (same as 5.0 trunk) and this issue is gone. But I think this version is not compatible with Java 8.

I'm not 100% sure if this is the root cause and what's the correct fix here. Would appreciate if anyone can have a look, thanks
Bug: Add startup flag to load a local snapshot from disk
Status: Triage Needed
Priority: Normal
Created: 02/Apr/24 13:32

Add startup flag "cassandra.load_snapshot_unsafe=snapshot_id" that loads a snapshot with the specified ID into the sstable tracker in the initial startup phase.

The flag has the {{_unsafe}} prefix because it may cause data consistency issues if this is used incorrectly. For example, if a given snapshot is loaded in a single replica of a replicated keyspace, it may cause replicas to go out of sync. For this reason, this flag should only be accepted if the "allow_load_snapshot_unsafe" guardrail is enabled (it is disabled by default).

When the flag is detected during startup, snapshots with the given tag will be located. If no snapshot with the given tag exists, the startup should fail.

The snapshot loading mechanism should create a hard link to existing sstables into a staging area to ensure the existing data is secured. After this, it should replace the existing sstables with the snapshot data into the sstable tracker before proceeding normally with the startup.
Bug: Add per type max size guardrails
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Jun/24 22:07
Resolved: 17/Feb/25 20:30

Right now, we have column_value_size_warn_threshold and column_value_size_fail_threshold guardrails, which are generic guardrails that are applied at write time to all columns with applicable datatypes.

This Jira is about adding specific per type size guard rails to allow even further flexibility to specify those limits. Actual limit would be the smaller between the general column_value_size_*_threshold and the specific column_type_value_size_*_threshold.

The list of guardrails to add is:
 * column_ascii_value_size_warn_threshold
 * column_ascii_value_size_fail_threshold
 * column_blob_value_size_warn_threshold
 * column_blob_value_size_fail_threshold
 * column_text_value_size_warn_threshold
 * column_text_value_size_fail_threshold
 * column_varchar_value_size_warn_threshold
 * column_varchar_value_size_fail_threshold


For collections, we have a similar guardrail called collection_size_warn_threshold and collection_size_fail_threshold, with some differences on how it is applied (look at the cassandra.yaml [documentation|[https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L2080]]). Similarly, we should add:
 * collection_map_size_warn_threshold
 * collection_map_size_fail_threshold
 * collection_set_size_warn_threshold
 * collection_set_size_fail_threshold
 * collection_list_size_warn_threshold
 * collection_list_size_fail_threshold
Bug: Add autocompletion in CQLSH for built-in functions
Status: Patch Available
Priority: Normal
Created: 10/May/24 10:15

Why do we not autocomplete native functions in CQLSH shell? I am pretty lost in what functions are there to choose from without consulting the documentation.

Additionally, the documentation on [Blob conversion functions|https://github.com/apache/cassandra/blob/trunk/doc/cql3/CQL.textile#blob-conversion-functions] could use an example, as done for the other builtins.
Bug: unhandled exception handling in jvm dtest may see errors from other clusters
Status: Ready to Commit
Priority: Normal
Created: 03/Apr/24 05:21

From time to time we see tests failing with unexpected errors during close, but this shouldn’t be possible because we ignore errors for instances that are shutting down…. In debugging it turns out that the errors are from different clusters, so we add to the unexpected error list even though it’s not from this cluster!
Bug: Accord Journal Determinism: PreAccept replay stability 
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/May/24 12:58
Resolved: 31/May/24 13:53

Currently, some messages, such as PreAccept can have some of their context initialized on replay. This patch adds a concept of Context to Journal that can be used for arbitrary information necessary for replaying them just the way they were executed the first time.
Bug: Allow operators to disable the execution of triggers
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/Apr/24 21:38
Resolved: 16/Apr/24 09:23

Currently, triggers are discouraged but there's no explicit way to disable them. Similar configuration already exists to disable other features, such as "conf.materialized_views_enabled". There should be a means for operators to gracefully disable the creation and execution of triggers.
Bug: nodetool Docs missing titles
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 05/Jun/24 20:00
Resolved: 02/Jan/25 08:47

Looking at the docs for nodetool rebuild: [https://cassandra.apache.org/doc/stable/cassandra/tools/nodetool/rebuild.html]

I noticed the title:
{noformat}
 <title>Untitled | Apache Cassandra Documentation</title>
{noformat}
I think for all the nodetool commands, this should be {{{}nodetool COMMAND{}}}, for example:
{noformat}
 <title>nodetool rebuild | Apache Cassandra Documentation</title>
{noformat}
This is also the case for the Operating page: [https://cassandra.apache.org/doc/stable/cassandra/operating/index.html]

As well as nodetool overview: [https://cassandra.apache.org/doc/stable/cassandra/tools/nodetool/nodetool.html]

There might be others, I'll update this issue if I find more.
Bug: Add support for BETWEEN operator
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 01/May/24 07:17
Resolved: 06/Jun/24 16:01

CQL support the {{>=}} and {{<=}} but does not support yet the {{BETWEEN}} operator. After CASSANDRA-19341 adding new operators should be much simpler and safer than it use to be.
For the scope of this ticket {{BETWEEN}} support should be added for {{WHERE}} clauses of {{SELECT}} and {{DELETE}} queries (for single column and multi-column restrictions). NOT BETWEEN should be added and should be supported everywhere BETWEEN is.

+Additional information for newcomers:+
Parts that will need to be modified:
 * {{Lexer.g}} and {{Parser.g}} to add support for the new keyword and syntax
 * The {{Operator.class}} to add the new {{BETWEEN}} operator
 * Unit tests in {{SelectSingleColumnRelationTest}} and {{SelectMultiColumnRelationTest}} classes for the different types of columns (partition key, clustering, static and regular).
 * CQLSH auto completion in {{cql3handling.py}} and test for it in {{test_cqlsh_completion.py}}
 * Update the documentation

Of course this is just an overview and some other parts might have to be changed as well.
Bug: Improve debug around paused and disabled compaction
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Jun/24 10:16
Resolved: 27/Jun/24 16:53

Compactions can be paused (from other operations) or disabled (nodetool), and when compactions are not running it can be unclear what the reason is.

Add debug so this is clearer to catch for the operator.
Bug: Custom SSLFactory for jmx
Status: Resolved
Priority: Normal
Resolution: Invalid
Created: 05/Jun/24 11:58
Resolved: 15/Nov/24 07:46

I got a request to use custom SSLFactory for jmx instead of the {{SslRMIClientSocketFactory}} and {{SslRMIServerSocketFactory}} that are used today. I think it would be a small change in {{JMXServerUtils}} to add two properties that could be set in cassandra-env.sh with the other jmx settings so you could specify your own class that would be used instead of the default ones.
Bug: Improve migration to incremental repair
Status: Resolved
Priority: Normal
Resolution: Not A Problem
Created: 24/Apr/24 19:54
Resolved: 14/Nov/24 17:11

In order to make it easier to migrate from subrange repair to incremental repair, I think we should support an option to repair data already marked as repaired.  

The process would require the user to do the following:

1. A new command to mark all data for a table as repaired, cluster wide.
2. Going forward, the user runs incremental repairs on a regular basis
3. To ensure the data that was written prior to step 1 is consistent, the user would now run subrange repair on *only* the data marked as repaired.  

This would allow the repaired data to remain marked as repaired while also ensuring it's consistent. It also makes it easy for the user to have a regular repair job on the already repaired data to ensure that things are consistent, or after replacing a node from a backup that may have a different set of repaired data from the rest of the cluster.
Bug: Test failure: largecolumn_test.TestLargeColumn
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/Jun/24 15:49
Resolved: 04/Oct/24 19:33

https://app.circleci.com/pipelines/github/driftx/cassandra/1671/workflows/980121dd-ac7f-4747-91fa-391d961c9c5b/jobs/93813/tests

{quote}
failed on teardown with "Unexpected error found in node logs (see stdout for full details). Errors: [[node1] 'ERROR [epollEventLoopGroup-5-8] 2024-06-13 15:06:46,808 NoSpamLogger.java:111 - Could not aquire capacity while processing native protocol message', [node2] 'ERROR [epollEventLoopGroup-5-4] 2024-06-13 15:06:48,851 NoSpamLogger.java:111 - Could not aquire capacity while processing native protocol message']"
Unexpected error found in node logs (see stdout for full details). Errors: [[node1] 'ERROR [epollEventLoopGroup-5-8] 2024-06-13 15:06:46,808 NoSpamLogger.java:111 - Could not aquire capacity while processing native protocol message', [node2] 'ERROR [epollEventLoopGroup-5-4] 2024-06-13 15:06:48,851 NoSpamLogger.java:111 - Could not aquire capacity while processing native protocol message']
{quote}
Bug: Metric for Cross Node DC Latency records external DCs as UNKNOWN_DC until restart
Status: Open
Priority: Normal
Created: 21/Jun/24 11:16

When creating a new DC for a different region (In this case, on AWS), I've noticed that the metric (JMX Exporter): `org.apache.cassandra.metrics:type=Messaging,name=*-Latency` shows `UNKNOWN_DC` for external datacenters until the node is restarted, at which point it seems to resolve to the actual datacenter name.

I'm using GossipingFileSnitch and nodetool status and nodetool gossipinfo always show the correct DC for the nodes, which makes me suspect there's some variable that's responsible for the metric that isn't updated
Bug: idealCLWriteLatency metric reports the worst response time instead of the time when ideal CL is satisfied
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/May/24 14:27
Resolved: 21/Aug/24 20:09

org.apache.cassandra.service.AbstractWriteResponseHandler:
{code:java}
private final void decrementResponseOrExpired()
{
    int decrementedValue = responsesAndExpirations.decrementAndGet();
    if (decrementedValue == 0)
    {
        // The condition being signaled is a valid proxy for the CL being achieved
        // Only mark it as failed if the requested CL was achieved.
        if (!condition.isSignalled() && requestedCLAchieved)
        {
            replicaPlan.keyspace().metric.writeFailedIdealCL.inc();
        }
        else
        {
            replicaPlan.keyspace().metric.idealCLWriteLatency.addNano(nanoTime() - queryStartNanoTime);
        }
    }
} {code}
Actual result: responsesAndExpirations is a total number of replicas across all DCs which does not depend on the ideal CL, so the metric value for replicaPlan.keyspace().metric.idealCLWriteLatency is updated when we get the latest response/timeout for all replicas.
Expected result: replicaPlan.keyspace().metric.idealCLWriteLatency is updated when we get enough responses from replicas according to the ideal CL.
Bug: Enable color nodetool status output
Status: Open
Priority: Normal
Created: 14/Jun/24 20:17

To improve human readability of {{nodetool status}}, the following patch adds an option that enables status lines to be printed augmented by different colors. Warning color - yellow by default - is used for a node joining, leaving or moving and alert color - red by default - is used for nodes which are not up.

The option {{\-c}} or {{\-\-color}} enables the feature. Alternatively, the environment variable {{NODETOOL_COLOR_OUTPUT}} set {{true}} will enable it by default. Further environment variables {{NODETOOL_WARNING_COLOR}} and {{NODETOOL_ALERT_COLOR}} allow to override the default colors by setting to values 30-37 of the standard ANSI codes for foreground color.
Bug: Reconfigure CMS after move/bootstrap/replacement
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 14/Jun/24 11:33
Resolved: 02/Jul/24 07:04

The CMS placement uses SimpleStrategy/NTS to decide where it is placed to make it easier to safely bounce a cluster using existing tools (with CMS placement {{dc1: 3, dc2: 3}} we will   use the placements for min_token in a NetworkTopologyStrategy with the same replication setting). 

We need to reconfigure this after move/bootstrap/replacement though, since the placements might have changed.
Bug: Integrate tracing with Accord transaction execution
Status: Triage Needed
Priority: Normal
Created: 25/Jun/24 21:52

Accord transactions contain potentially multiple read and write operations, as well as the evaluation of conditions. As of right now, they are not integrated holistically into the CQL query tracing mechanism, at least not completely.

There are two things (or categories of events) we might want here. The first is logging events during transaction execution that would be new and specific to Accord. The second is making sure that existing tracing information around local reads and writes are persisted as expected to tracing sessions.
Bug: Rate limiting per-node on failed log-in attempts
Status: In Progress
Priority: Normal
Created: 28/Jun/24 14:30

If there is a malicious attacker who is brute-forcing passwords / usernames, we should just ban such user for some time. On the other hand, we should enable logging in for genuine users who just happened to provide invalid passwords for multiple times, we do not want to ban these completely. 

A rate limit might be something like "5 times per a minute".

This should be based on IP address of a client to identify the attacker. If we based this on invalid passwords only, an attacker might just change the usernames to bypass that.
Bug: Make 0 work as 0+unit for all three config classes (DataStorageSpec, DurationSpec, DataRateSpec)
Status: Needs Committer
Priority: Normal
Created: 23/Apr/24 23:16

The inline docs say:
{noformat}
Setting this to 0 disables throttling.
{noformat}
However, on startup, we throw this error:
{noformat}
Caused by: java.lang.IllegalArgumentException: Invalid data rate: 0 Accepted units: MiB/s, KiB/s, B/s where case matters and only non-negative values a>
Apr 23 23:12:01 cassandra0 cassandra[3424]:         at org.apache.cassandra.config.DataRateSpec.<init>(DataRateSpec.java:52)
Apr 23 23:12:01 cassandra0 cassandra[3424]:         at org.apache.cassandra.config.DataRateSpec.<init>(DataRateSpec.java:61)
Apr 23 23:12:01 cassandra0 cassandra[3424]:         at org.apache.cassandra.config.DataRateSpec$LongBytesPerSecondBound.<init>(DataRateSpec.java:232)
Apr 23 23:12:01 cassandra0 cassandra[3424]:         ... 27 common frames omitted
{noformat}
We should allow 0 without a unit attached for data, duration, and data spec config parameters, as 0 is always 0 no matter the unit.