av9ash/GitBugs huggingface dataset:
Bug: Getting tons of msgs "Failed to get peer certificates for peer /x.x.x.x:45796" when require_client_auth is set to false
Status: Resolved
Priority: Urgent
Resolution: Fixed
Created: 31/Mar/24 22:34
Resolved: 04/Apr/24 16:03

We recently upgraded our production clusters from 3.11.15 to 4.1.4. We started seeing thousands of msgs "Failed to get peer certificates for peer /x.x.x.x:45796". SSL is enabled but require_client_auth is disabled.  This is causing a huge problem for us because cassandra log files are growing very fast as our connections are short live connections, we open more than 1K connections per second and they stay live for 1-2 seconds. 
{code:java}
DEBUG [Native-Transport-Requests-2] 2024-03-31 21:26:38,026 ServerConnection.java:140 - Failed to get peer certificates for peer /172.31.2.23:45796
javax.net.ssl.SSLPeerUnverifiedException: peer not verified
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine$DefaultOpenSslSession.getPeerCertificateChain(ReferenceCountedOpenSslEngine.java:2414)
        at io.netty.handler.ssl.ExtendedOpenSslSession.getPeerCertificateChain(ExtendedOpenSslSession.java:140)
        at org.apache.cassandra.transport.ServerConnection.certificates(ServerConnection.java:136)
        at org.apache.cassandra.transport.ServerConnection.getSaslNegotiator(ServerConnection.java:120)
        at org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:76)
        at org.apache.cassandra.transport.Message$Request.execute(Message.java:255)
        at org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:166)
        at org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:185)
        at org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:212)
        at org.apache.cassandra.transport.Dispatcher$RequestProcessor.run(Dispatcher.java:109)
        at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
        at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
        at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:142)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) {code}
*Our SSL config:*
{code:java}
client_encryption_options:
  enabled: true
  keystore: /path/to/keystore
  keystore_password: xxxxx
  optional: false
  require_client_auth: false {code}
 

We should stop throwing this msg when require_client_auth is set to false. Or at least it should be logged in TRACE not DEBUG. 

I'm working on preparing a PR.
Bug: CQL Management API
Status: In Progress
Priority: High
Created: 15/Mar/24 17:04

We want to run management commands via CQL.
The goals are:
* To provide a way to run predefined management commands via CQL;
* To provide a mechanism for retrieving command definitions and metadata via CQL;
* To provide information on all available management commands via virtual tables;
* To provide a registry that stores all C* commands and their metadata accordingly;
* To internal instrumentation and a reasonable plan for migrating cluster management from JMX to CQL, taking into account backward compatibility and adopted deprecation policies;

The discussion on the ML:
https://lists.apache.org/thread/pow83q92m666nqtwyw4m3b18nnkgj2y8

The design document:
https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-38%3A+CQL+Management+API
Bug: Accord: Command to Exclude Replicas from Durability Status Coordination
Status: Resolved
Priority: High
Resolution: Fixed
Created: 24/Jan/24 11:41
Resolved: 30/Aug/24 20:51

So that other replicas may continue to cleanup their state, we must have an operator command for marking replicas as stale so that the remaining replicas do not wait for them to coordinate their durability status.
Bug: Allow configurable v4/v6 address matching behavior for SAI indexes on CQL inet columns
Status: Open
Priority: Normal
Created: 26/Mar/24 16:26

The {{inet}} type in CQL allows ipv4 and ipv6 address formats to mix in the same column, and while there is a standard conversion between v4 and v6, equivalent addresses are not considered equal by Cassandra, when it comes to their usage in keys or in filtering queries. However, in SAI queries, equivalent v4 and v6 addresses ARE considered equal. Even if this behavior is useful, it might be a good idea to make the default behavior consistent with filtering, and allow the index configuration to control whether or not to consider them equal. (We have precedent for this w/ case-insensitive indexing of text columns, etc.)
Bug: SAI does not index empty bytes even for types that allow empty bytes as a valid input
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 07/Mar/24 17:22
Resolved: 22/Mar/24 17:00

This is easy to reproduce with a test that looks something like this:

{noformat}
@Test
public void testEmptyString()
{
    createTable("CREATE TABLE %s (k TEXT PRIMARY KEY, v text)");
    createIndex(String.format(CREATE_INDEX_TEMPLATE, 'v'));

    execute("INSERT INTO %s (k, v) VALUES ('0', '')");
    execute("INSERT INTO %s (k) VALUES ('1')");
    
    // flush(); <---- there is not always a memtable index involved, a fix will have to pay attention to this

    List<Row> rows = executeNet("SELECT * FROM %s WHERE v = ''").all();
    assertEquals(1, rows.size()); <— FAILS! No matches...
}
{noformat}
Bug: Remove lock contention generated by getCapacity function in SSTableReader
Status: Changes Suggested
Priority: Normal
Created: 23/Feb/24 22:22

Profiling Cassandra 4.1.3 on large AWS instances, a high number of lock acquires is measured in the `getCapacity` function from `org/apache/cassandra/cache/InstrumentingCache` (1.9M lock acquires per 60 seconds). Based on our tests on r8g.24xlarge instances (using Ubuntu 22.04), this limits the CPU utilization of the system to under 50% when testing at full load and therefore limits the achieved throughput.

Removing the lock contention from the SSTableReader.java file by replacing the call to `getCapacity` with `size` achieves up to 2.95x increase in throughput on r8g.24xlarge and 2x on r7i.24xlarge:
|Instance type|Cass 4.1.3|Cass 4.1.3 patched|
|r8g.24xlarge|168k ops|496k ops (2.95x)|
|r7i.24xlarge|153k ops|304k ops (1.98x)|

 

Instructions to reproduce:
{code:java}
## Requirements for Ubuntu 22.04
sudo apt install -y ant git openjdk-11-jdk

## Build and run
CASSANDRA_USE_JDK11=true ant realclean && CASSANDRA_USE_JDK11=true ant jar && CASSANDRA_USE_JDK11=true ant stress-build  && rm -rf data && bin/cassandra -f -R

# Run
bin/cqlsh -e 'drop table if exists keyspace1.standard1;' && \
bin/cqlsh -e 'drop keyspace if exists keyspace1;' && \
bin/nodetool clearsnapshot --all && tools/bin/cassandra-stress write n=10000000 cl=ONE -rate threads=384 -node 127.0.0.1 -log file=cload.log -graph file=cload.html && \
bin/nodetool compact keyspace1   && sleep 30s && \
tools/bin/cassandra-stress mixed ratio\(write=10,read=90\) duration=10m cl=ONE -rate threads=406 -node localhost -log file=result.log -graph file=graph.html
{code}
Bug: AND queries involving both static and regular columns can hide matches from Memtable-attached indexes
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Jan/24 17:54
Resolved: 29/Jan/24 15:56

As part of our ongoing work on CASSANDRA-18275, Harry started surfacing errors shortly after we added static columns to the schema of {{SingleNodeSAITest}}. After a bit of debugging, I was able to come up with the following concrete repro:

{noformat}
import org.junit.Test;

import org.apache.cassandra.cql3.UntypedResultSet;
import org.apache.cassandra.index.sai.utils.SAIRandomizedTester;

public class InMemoryIntersectionsTest extends SAIRandomizedTester
{
    @Test
    public void staticAndRegularIntersection()
    {
        createTable("CREATE TABLE %s (pk int, ck int, v1 int, s1 int static, PRIMARY KEY(pk, ck))");
        createIndex("CREATE INDEX ON %s(v1) USING 'sai'");
        createIndex("CREATE INDEX ON %s(s1) USING 'sai'");

        execute("INSERT INTO %s (pk, ck, v1) VALUES (?, ?, ?)", 0, 1, 0);
        execute("INSERT INTO %s (pk, ck, v1) VALUES (?, ?, ?)", 0, 2, 1);
        execute("INSERT INTO %s (pk, ck, v1) VALUES (?, ?, ?)", 0, 3, 2);
        execute("INSERT INTO %s (pk, ck, v1) VALUES (?, ?, ?)", 0, 4, 3);
        execute("INSERT INTO %s (pk, ck, v1) VALUES (?, ?, ?)", 0, 5, 4);
        execute("INSERT INTO %s (pk, ck, v1) VALUES (?, ?, ?)", 0, 6, 5);

        execute("INSERT INTO %s (pk, s1) VALUES (?, ?)", 0, 100);

	// Flushing here passes test

        UntypedResultSet result1 = execute("SELECT * FROM %s WHERE pk = ? AND v1 > ?", 0, 2);
        assertRowCount(result1, 3);
        UntypedResultSet result2 = execute("SELECT * FROM %s WHERE pk = ? AND v1 > ? AND s1 = ?", 0, 2, 100);        
        assertRowCount(result2, 3); // Only returns one result!
    }
}
{noformat}

Flushing memtables immediately before the queries passes the test, but it seems we have an issue with how the {{PrimaryKey}} iterators produced by static and regular column indexes are intersected when the iterators come from Memtable-attached indexes. Once the root cause of this is determined, it might make some sense to enhance {{RandomIntersectionTest}} to cover this interaction as well, just in case it turns up further problems.
Bug: ALTER ROLE WITH LOGIN=FALSE and REVOKE ROLE do not disconnect existing users
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 09/Feb/24 17:24
Resolved: 14/Jan/25 20:42

Currently, if users want to block a role from connecting to Cassandra, ALTER ROLE WITH LOGIN=FALSE and REVOKE ROLE seem like the sensible options. But these commands do not disconnect existing connections authenticated with the given role, and these connections will stay alive until they're disconnected for another reason. Subsequent attempts to connect with that role will fail.

There is currently no way to disconnect all connections for a given user either. nodetool disablebinary will disconnect all client connections for a given node, and client sessions can be shut down. But in the case of a credential leak or a misconfigured user, it can be desirable to prevent login for a given role and disconnect all existing connections for that role.
Bug: Ensure snitches always defer to ClusterMetadata
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 22/Mar/24 16:11
Resolved: 16/Dec/24 15:31

Internally, C* always uses {{ClusterMetadata}} as the source of topology information when calculating data placements, replica plans etc and as such the role of the snitch has been somewhat reduced. 

Sorting and comparison functions as provided by specialisations like {{DynamicEndpointSnitch}} are still used, but the snitch should only be responsible for providing the DC and rack for a new node when it first joins a cluster.

Aside from initial startup and registration, snitch implementations should always defer to {{{}ClusterMetadata{}}}, for DC and rack otherwise there is a risk that the snitch config drifts out of sync with TCM and output from tools like {{nodetool ring}} and {{gossipinfo}} becomes incorrect.

A complication is that topology is used when opening connections to peers as certain internode connection settings are variable at the DC level, so at the time of connecting we want to check the location of the remote peer. Usually, this is available from {{{}ClusterMetadata{}}}, but in the case of a brand new node joining the cluster nothing is known a priori. The current implementation assumes that the snitch will know the location of the new node ahead of time, but in practice this is often not the case (though with variants of {{PropertyFileSnitch}} it _should_ be), and the remote node is temporarily assigned a default DC. This is problematic as it can cause the internode connection settings which depend on DC to be incorrectly set. Internode connections are long lived and any established while the DC is unknown (potentially with incorrect config) will persist indefinitely. This particular issue is not directly related to TCM and is present in earlier versions.
Bug: Non-serial reads need to handle racing with Accord topology changes
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/Feb/24 19:26
Resolved: 29/Jan/25 03:35

A key or range read could end up being sent to Accord when it's not managed by Accord and we might not find out until the execution epoch is known. In reality I think this already throws an exception in Accord for a key we just need to propagate and handle the exception and retry with the new topology until we can complete the read.
Bug: invalid EstimatedHistogramReservoirSnapshot::getValue values due to race condition in DecayingEstimatedHistogramReservoir
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 05/Feb/24 15:19
Resolved: 02/Oct/24 13:46

`DecayingEstimatedHistogramReservoir` has a race condition between `update` and `rescaleIfNeeded`.
A sample which ends up (`update`) in an already scaled decayingBucket (`rescaleIfNeeded`) may still use a non-scaled weight because `decayLandmark` has not been updated yet at the moment of `update`.
 
The observed consequence was flooding of the cluster with speculative retries (we happened to hit low-percentile buckets with overweight samples, which drove p99 below true p50 for a long time).

Please note that despite the manifestation being similar to CASSANDRA-19330, these are two distinct bugs in their own right.

This bug affects versions 4.0+
On 3.11 there's locking in DEHR. I did not check earlier versions.
Bug: cqlsh does not autocomplete table names in MacOS
Status: Triage Needed
Priority: Normal
Created: 07/Mar/24 19:20

I am on MacBook with Sonoma 

{code}
$ uname -a
Darwin stefanm-mac-0 23.3.0 Darwin Kernel Version 23.3.0: Wed Dec 20 21:30:59 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T6030 arm64
{code}

What is interesting is that ./bin/cqlsh works but when I am tabbing like 

{code}
select * from system_auth.<TAB>
{code}

This does not work.

It does work when I install cqlsh from pip though.

I have pyreadline installed,

I have two pythons 3.10 and 3.9.6. I installed 3.10 from brew and 3.9.6 is from Sonoma itself.

Could somebody with Mac verify that tab completion works (or does not) when invoking ./bin/cqlsh from Cassandra?
Bug: Test failure: org.apache.cassandra.tcm.DiscoverySimulationTest
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Mar/24 16:52
Resolved: 18/Dec/24 13:02

As seen at https://app.circleci.com/pipelines/github/driftx/cassandra/1551/workflows/be2ca359-49d7-4fb3-959d-454656f48d79/jobs/80604/tests :

{noformat}
junit.framework.AssertionFailedError: expected:<[/127.0.100.1:7012, /127.0.100.2:7012, /127.0.100.3:7012, /127.0.100.4:7012, /127.0.100.5:7012, /127.0.100.6:7012, /127.0.100.7:7012, /127.0.100.8:7012, /127.0.100.9:7012, /127.0.100.10:7012]> but was:<[/127.0.100.1:7012, /127.0.100.2:7012, /127.0.100.3:7012, /127.0.100.5:7012, /127.0.100.6:7012, /127.0.100.7:7012, /127.0.100.8:7012, /127.0.100.9:7012, /127.0.100.10:7012]>
	at org.apache.cassandra.tcm.DiscoverySimulationTest.discoveryTest(DiscoverySimulationTest.java:105)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
{noformat}
Bug: "nodetool bootstrap resume" starts unnecessary streaming session on joining node
Status: Patch Available
Priority: Normal
Created: 04/Jan/24 21:30

Calling {{nodetool boostrap resume}} triggers a new bootstrap streaming session on a joining node, even if there's a bootstrap streaming session currently running.

Each time this command is called, a new bootstrap streaming session is started, causing the same data to be needlessly streamed from peers.

It should only be possible to call {{nodetool bootstrap resume}} if a previous bootstrap attempt has failed.

An example of multiple invocations of {{nodetool bootstrap resume}} in a joining node is shown below:
{noformat}
$ nodetool netstats
Mode: JOINING
Bootstrap a1cf3bf0-ab3a-11ee-9fcf-5746a7aee9ca
    /A.B.C.D
        Receiving 13 files, 14302312660 bytes total. Already received 2 files, 52389676 bytes total
            ks1/tbl1 80/80 bytes(100%) received from idx:0/A.B.C.D
            ks2/tbl2 471/471 bytes(100%) received from idx:0/A.B.C.D
    /E.F.G.H
    /I.J.K.L
Bootstrap 7f1e7000-ab3d-11ee-9fcf-5746a7aee9ca
    /A.B.C.D
        Receiving 13 files, 14302312660 bytes total. Already received 0 files, 0 bytes total
    /E.F.G.H
    /I.J.K.L
Bootstrap 9ca42500-ab3a-11ee-9fcf-5746a7aee9ca
    /A.B.C.D
    /E.F.G.H
        Receiving 13 files, 14302312660 bytes total. Already received 2 files, 104838752 bytes total
            ks1/tbl1 80/80 bytes(100%) received from idx:0/E.F.G.H
            ks2/tbl2 471/471 bytes(100%) received from idx:0/E.F.G.H
    /I.J.K.L {noformat}
Bug: Relation and Restriction hierarchies are too complex and error prone
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 30/Jan/24 10:59
Resolved: 06/May/24 17:53

The {{Relation}} and {{Restriction}} hierarchy have been designed when C* was only supporting a limited amount of operators and columns expressions (single column, multi-column and token expressions). Over time they have grown in complexity making the code harder to understand and modify and error prone. Their design is also resulting in unnecessary limitations that could be easily lifted, like the ability to accept different predicates on the same column.

Today adding a new operator requires the addition of a lot of glue code and surgical changes accross the CQL layer. Making patch for features such as CASSANDRA-18584 much complex than it should be.

The goal of this ticket is to simplify the {{Relation}} and {{Restriction}} hierarchies and modify operator  class so that adding new operators requires only changes to the {{Operator}} class and ANTLR file.
Bug: CompactionStrategyManagerTest.testAutomaticUpgradeConcurrency is flaky
Status: Triage Needed
Priority: Normal
Created: 07/Feb/24 11:56

The unit test {{CompactionStrategyManagerTest.testAutomaticUpgradeConcurrency}} fails occasionally at least in {{trunk}}:
{code}
unit.framework.AssertionFailedError: expected:<1> but was:<0>
	at org.apache.cassandra.db.compaction.CompactionStrategyManagerTest.testAutomaticUpgradeConcurrency(CompactionStrategyManagerTest.java:227)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
{code}
https://app.circleci.com/pipelines/github/adelapena/cassandra/3427/workflows/126307ce-d465-4736-a71e-82e0b8749598

The failure can be reproduced with:
{code}
.circleci/generate.sh -p \
  -e REPEATED_UTESTS=org.apache.cassandra.db.compaction.CompactionStrategyManagerTest \
  -e REPEATED_UTESTS_COUNT=1000
{code}
I have only tested trunk. I haven't seen it on Butler.

Observed flakiness is ~0.2% for all jobs except for tries, which ramps up to ~2.5%.
Bug: Start UCS docs with examples and use cases
Status: Open
Priority: Normal
Created: 13/Feb/24 02:36

Users interested in UCS are primarily going to be interested in examples of how UCS should be used for certain types of workloads.  We start the current docs by saying it can replace every other compaction strategy, but leave it up to the user to figure out exactly what that means for them.  

Before the docs that explain how it works, let's describe how it should be used.  Users interested in the nuts and bolts can scroll down to learn the details, but that shouldn't be a requirement to switch from an existing compaction strategy to UCS.

A table showing examples of LCS, STCS, and TWCS converted to UCS would suffice for 99% of people's needs.  

More information in this Slack thread: https://the-asf.slack.com/archives/CK23JSY2K/p1707700814330359
Bug: When transitioning to Accord migration it's not safe to read immediately using Accord due to concurrent non-serial writes
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/Feb/24 19:17
Resolved: 13/Nov/24 16:31

Concurrent writes at the same time that migration starts make it unsafe to read from Accord because txn recovery will not be deterministic in the presences of writes not done through Accord.

Migration to Accord needs to be split into two phases, in the first phase we write through Accord and always respect the consistency level and do synchronous commit. This allows Paxos and non-serial writes to continue while Accord's metadata covers everything needed for future reads. Paxos continues to operate as normal since it has enough metadata to allow key migration in the second phase and we need to stay online so something needs to handle LWTs.

Then repair runs and makes it possible for Accord to read any data written non-transactionally and we can then do key migration and route all updates (conditional or blind) through Accord while Paxos repair runs so we can stop doing key migration.
Bug: ResultRetriever should batch clusterings/rows during SAI post-filtering reads
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/Mar/24 23:27
Resolved: 08/Nov/24 20:43

SAI currently creates and executes a {{SinglePartitionReadCommand}} for every {{PrimaryKey}} the index produces to read the corresponding row for post-filtering. Informed by the limits present in the read command itself, it should be possible to batch those reads w/ a {{ClusteringIndexNamesFilter}} in many fewer {{SinglePartitionReadCommands}}. When we have a handful of matches in a large partition, this seems like would involve many fewer seeks, less unnecessary object creation, etc.
Bug: NativeTransportEncryptionOptionsTest fails often in CI
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 22/Jan/24 08:51
Resolved: 01/Nov/24 10:08

NativeTransportEncryptionOptionsTest seems to be quite flaky. I happen to see a lot of these in CI recently for trunk.

org.apache.cassandra.distributed.test.NativeTransportEncryptionOptionsTest

testEndpointVerificationEnabledIpNotInSAN
testOptionalMtlsModeDoNotAllowNonSSLConnections
unencryptedNativeConnectionNotlisteningOnTlsPortTest
testEndpointVerificationEnabledIpNotInSAN

This is the error basically for all of them
{noformat}
        junit.framework.AssertionFailedError: Forked Java VM exited abnormally. Please note the time in the report does not reflect the time until the VM exit.
                at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)

{noformat}

They do pass locally just fine.
Bug: Add support for providing nvdDatafeedUrl to OWASP
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/Mar/24 22:00
Resolved: 22/Mar/24 20:53

This allows you to point to a mirror that is faster and doesn’t require an API key.

This is kind of painful to make work in {{ant}} because you can't specify the property at all if you want to use the API and I couldn't find a way to get {{ant}} to conditionally supply the property without having a dedicated invocation of the {{dependency-check}} task with/without the parameter {{nvdDataFeedUrl}} specified.