av9ash/GitBugs huggingface dataset:
Bug: Test Failure: VectorUpdateDeleteTest fails with heap_buffers
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 05/Dec/23 08:20
Resolved: 23/Feb/24 19:23

When {{memtable_allocation_type}} is set to {{heap_buffers}}, {{updateTest}} fails with
{code}
junit.framework.AssertionFailedError: Result set does not contain a row with pk = 0
	at org.apache.cassandra.index.sai.cql.VectorTypeTest.assertContainsInt(VectorTypeTest.java:133)
	at org.apache.cassandra.index.sai.cql.VectorUpdateDeleteTest.updateTest(VectorUpdateDeleteTest.java:308)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
{code}
Bug: Streaming appears to be incompatible with different storage_compatibility_mode settings
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 30/Nov/23 14:25
Resolved: 23/Jan/24 15:49

In particular, SSTableLoader appears to be incompatible with storage_compatibility_mode: NONE, which manifests as a failure of {{org.apache.cassandra.distributed.test.SSTableLoaderEncryptionOptionsTest}} when the flag is turned on (found during CASSANDRA-18753 testing). Setting {{storage_compatibility_mode: NONE}} in the tool configuration yaml does not help (according to the docs, this setting is not picked up).

This is likely a bigger problem as the acceptable streaming version for C* 5 is 12 only in legacy mode and 13 only in none, i.e. two C* 5 nodes do not appear to be able to stream with each other if their setting for the compatibility mode is different.
Bug: Repair fuzz tests fail with paxos_variant: v2
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Nov/23 12:23
Resolved: 26/Mar/24 20:05

Adding {{paxos_variant: v2}} to the test yaml causes all fuzz repair tests to fail with
{code}
java.lang.NullPointerException: null
	at org.apache.cassandra.gms.EndpointStateSerializer.serializedSize(EndpointState.java:337)
	at org.apache.cassandra.gms.EndpointStateSerializer.serializedSize(EndpointState.java:300)
	at org.apache.cassandra.service.paxos.cleanup.PaxosStartPrepareCleanup$RequestSerializer.serializedSize(PaxosStartPrepareCleanup.java:176)
	at org.apache.cassandra.service.paxos.cleanup.PaxosStartPrepareCleanup$RequestSerializer.serializedSize(PaxosStartPrepareCleanup.java:147)
	at org.apache.cassandra.net.Message$Serializer.payloadSize(Message.java:1067)
	at org.apache.cassandra.net.Message.payloadSize(Message.java:1114)
	at org.apache.cassandra.net.Message$Serializer.serializedSize(Message.java:750)
	at org.apache.cassandra.net.Message.serializedSize(Message.java:1094)
...
{code}

This happens for all three options of {{paxos_state_purging}} and both with and without {{storage_compatibility_mode: NONE}}.

Tests still fail if {{PaxosStartPrepareCleanup}} is changed to use {{EndpointState.nullableSerializer}}.
Bug: Rewrite AbstractReplicationStrategy#calculateNaturalReplicas(Token searchToken, ClusterMetadata clusterMetadata) to use read placements
Status: Triage Needed
Priority: Normal
Created: 11/Dec/23 13:57

This method is a remnant of the previous implementation and it should be possible to make it non-abstract and replace the various subclasses' versions with a simple lookup into the current {{DataPlacement}}.
Bug: Avoid flushing on every append in the LocalLog
Status: Triage Needed
Priority: Normal
Created: 01/Dec/23 08:35

Right now, we are performing flush on every transformation that is appended to the local log. While this does make _some_ sense, it may not be what we always want to do. We have initially added this flush as a way to remedy node bounces following schema changes, but this should no longer be necessary.
Bug: Specify "since" in all Deprecated annotations
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 05/Oct/23 13:30
Resolved: 18/Oct/23 18:31

It would be great if we introduced in 5.0 a change in Deprecated annotations like this:

{code}
@Deprecated(since = "4.0")
{code}

or 

{code}
@Deprecated(since = "3.11")
{code}

The reasoning behind this is that as of now, it is pretty cumbersome to figure out what can be removed on the next major version. It has to be, basically, done manually every time.

There is also this parameter available:

{code}
@Deprecated(forRemoval = true / false)
{code}

which indicates whether the annotated element is subject to removal in a future version so we do not need to think about this every time if it is eligible for deletion in a next major or not.

We could then have a check which would ensure that we are not releasing a next major with some deprecations introduced two majors before.
Bug: Test Failure: org.apache.cassandra.index.sai.virtual.IndexesSystemViewTest.testVirtualTableThroughIndexLifeCycle-system_keyspace_directory_jdk11_x86_64 (
Status: Open
Priority: Normal
Created: 05/Dec/23 22:33

 

https://ci-cassandra.apache.org/job/Cassandra-trunk/1801/testReport/org.apache.cassandra.index.sai.virtual/IndexesSystemViewTest/testVirtualTableThroughIndexLifeCycle_system_keyspace_directory_jdk11_x86_64/
{code:java}
Invalid value for row 0 column 3 (is_queryable of type boolean), expected <false> but got <true>

Stacktrace
junit.framework.AssertionFailedError: Invalid value for row 0 column 3 (is_queryable of type boolean), expected <false> but got <true> at org.apache.cassandra.cql3.CQLTester.assertRows(CQLTester.java:1777) at org.apache.cassandra.index.sai.virtual.IndexesSystemViewTest.testVirtualTableThroughIndexLifeCycle(IndexesSystemViewTest.java:74) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
{code}
Bug: cqlsh breaks with Python 3.12
Status: Open
Priority: Normal
Created: 17/Dec/23 20:11

Inside cassandra directory, on running ./bin/cqlsh gives ModuleNotFoundError when run using python 3.12.1.

./bin/cqlsh works well for python versions like 3.9.18.

 

The below image shows the error that occured while using python 3.12.1

!image-2023-12-18-01-39-33-839.png!

 

The below image shows how it should normally work, it was ran on python 3.9.18

!image-2023-12-18-01-40-38-974.png!
Bug: Updating the column of a non-existent row in an Accord transaction results in Atomicity violation
Status: Resolved
Priority: Normal
Resolution: Not A Problem
Created: 31/Oct/23 18:29
Resolved: 25/Jun/24 16:48

*System configuration and information:*

Single node Cassandra with Accord transactions enabled running on docker

Built from commit: [a7cd114435704b988c81f47ef53d0bfd6441f38b|https://github.com/apache/cassandra/commit/a7cd114435704b988c81f47ef53d0bfd6441f38b]

CQLSH: [cqlsh 6.2.0 | Cassandra 5.0-alpha2-SNAPSHOT | CQL spec 3.4.7 | Native protocol v5]

 

*Steps to reproduce in CQLSH:*
{code:java}
CREATE KEYSPACE accord WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'} AND durable_writes = true;{code}
{code:java}
CREATE TABLE accord.accounts (
    partition text,
    account_id int,
    balance int,
    PRIMARY KEY (partition, account_id)
) WITH CLUSTERING ORDER BY (account_id ASC);
{code}
{code:java}
BEGIN TRANSACTION
    INSERT INTO accord.accounts (partition, account_id, balance) VALUES ('default', 0, 100);
    INSERT INTO accord.accounts (partition, account_id, balance) VALUES ('default', 1, 100);
COMMIT TRANSACTION;{code}
atomicity bug happens after executing the following statement:

Based on [Cassandra documentation|https://cassandra.apache.org/doc/4.1/cassandra/cql/dml.html#update-statement] regarding the use of UPDATE statements, I expect the result of this transaction to be the insertion of a new account (\{ account_id: 3, balance: 10 }). The total balance across the three (3) accounts should be maintained (200). After executing the below transaction, the total number of accounts remains at two (2) and the total balance drops to 190. Basically, it appears as if only one half of the transaction proceeds.
{code:java}
BEGIN TRANSACTION
    UPDATE accord.accounts
    SET balance -= 10
    WHERE
      partition = 'default'
      AND account_id = 1;
    UPDATE accord.accounts
    SET balance += 10
    WHERE
      partition = 'default'
      AND account_id = 3;
COMMIT TRANSACTION;{code}
Bug / Error:
======================================================================

The result of performing a table read after executing the buggy transaction is:
{code:java}
 partition | account_id | balance
-----------+------------+---------
   default |          0 |     100
   default |          1 |      90
{code}
{color:#172b4d}Note that the above transactions are not possible without a transaction block because only counter type columns can be updated with += or -= syntax in normal (non-transactional) cql statements. Using counter type columns also results in a separate, related bug: [CASSANDRA-18987|https://issues.apache.org/jira/browse/CASSANDRA-18987]{color}

{color:#172b4d}This was found while testing Accord transactions with [~henrik.ingo] and team.{color}
Bug: commitlog_test.py::TestCommitLog::test_stop_failure_policy failed
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Nov/23 10:17
Resolved: 20/Jun/24 17:00

https://app.circleci.com/pipelines/github/jacek-lewandowski/cassandra/1121/workflows/fce907b1-0526-4d4d-beb5-b6620737a5f3/jobs/50905/tests

{noformat}
AssertionError: Cannot find the commitlog failure message in logs
assert []
self = <commitlog_test.TestCommitLog object at 0x7f1bf1cb3f50>

    def test_stop_failure_policy(self):
        """
        Test the stop commitlog failure policy (default one)
        """
        self.prepare()
    
        self._provoke_commitlog_failure()
        failure = self.node1.grep_log("Failed .+ commit log segments. Commit disk failure policy is stop; terminating thread")
        logger.debug(failure)
>       assert failure, "Cannot find the commitlog failure message in logs"
E       AssertionError: Cannot find the commitlog failure message in logs
E       assert []

commitlog_test.py:325: AssertionError
{noformat}
Bug: Upgrade ci-cassandra.a.o agents to Ubuntu 22.04.3
Status: Open
Priority: Normal
Created: 27/Dec/23 13:15

All agents are currently Ubuntu 18.04 LTS per the [docs|https://github.com/apache/cassandra-builds/blob/trunk/ASF-jenkins-agents.md#server-requirements].

All agents need to be upgraded to Ubuntu 22.04.3 LTS.
Bug: Test failure: auth_test.TestAuthUnavailable
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Dec/23 10:41
Resolved: 04/Jun/24 07:13

https://app.circleci.com/pipelines/github/jacek-lewandowski/cassandra/1233/workflows/bb617340-f1da-4550-9c87-5541469972c4/jobs/62551/tests
Bug: Reuse native transport-driven futures in Debounce
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/Dec/23 13:18
Resolved: 28/May/24 14:56

Currently, we create a future in Debounce, then create one more future in RemoteProcessor#sendWithCallback. This is further exacerbated by chaining calls, when we first attempt to catch up from peer, and then from CMS.


First of all, we should always only use a native transport timeout driven futures returned from sendWithCallback, since they implement reasonable retries under the hood, and are easy to bulk-configure (ie you can simply change timeout in yaml and have all futures change their behaviour).


Second, we should _chain_ futures and use map or andThen for fallback operations such as trying to catch up from CMS after an unsuccesful attemp to catch up from peer.


This should significantly simplify the code and number of blocked/waiting threads.
Bug: Optimisations to PlacementForRange, improve lookup on r/w path
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 11/Dec/23 13:57
Resolved: 23/Apr/24 08:05

The lookup used when selecting the appropriate replica group for a range or token while peforming reads and writes is extremely simplistic and inefficient. There is plenty of scope to improve {{PlacementsForRange}} to by replacing the current naive iteration with use a more efficient lookup.
Bug: ForceSnapshot transformations should not be persisted in the local log table
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 11/Dec/23 13:57
Resolved: 23/Apr/24 09:57

Per its inline comments, ForceSnapshot is a synthetic transformation whose purpose it to enable the local log to jump missing epochs. A common use for this is when replaying persisted events from the metadata log at startup. The log is initialised with {{Epoch.EMPTY}} but rather that replaying every single entry since the beginning of history, we select the most recent snapshot held locally and start the replay from that point. Likewise, when catching up from a peer, a node may receive a snapshot plus subsequent log entries. In order to bring local metadata to the same state as the snapshot, a {{ForceSnapshot}} with the same epoch as the snapshot is inserted into the {{LocalLog}} and enacted like any other other transformation. These synthetic transformations should not be persisted in the `system.local_metadata_log`, as they do not exist in the distributed metadata log. We _should_ persist the snapshot itself in {{system.metadata_snapshots}} so that we can avoid having to re-fetch remote snapshots (i.e. if a node were to restart shortly after receiving a catchup from a peer).
Bug: Better handle Spark job timeouts/process killing in Analytics tests
Status: Resolved
Priority: Normal
Resolution: Abandoned
Created: 20/Dec/23 17:20
Resolved: 17/May/24 10:18

The Process#destroyForcibly() call in ResiliencyTestBase#bulkWriteData can complete without the process actually exiting (and is documented to do so). We should wait on the process to exit before attempting to read the exit code, which throws if the process hasn’t yet exited.  Otherwise, we can lose the Spark output when the test determines that the job is taking too long.
Bug: IR may leak SSTables with pending repair when coming from streaming
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 07/Dec/23 23:11
Resolved: 01/May/24 21:38

There is a race condition where SSTables from streaming may race with pending repair cleanup in compaction causing us to cleanup the pending repair state in compaction while the SSTables are being added to it; this leads to IR failing in the future when those files get selected for repair.

This problem was hard to track down as the in-memory state was wiped, so we don’t have any details.  To better aid these types of investigation we should make sure the repair vtables get updated when IR session failures are submitted
Bug: CMS: Nodes can restart with new ipaddress already defined in the cluster
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Dec/23 15:29
Resolved: 26/Apr/24 12:47

I am simulating running a cluster in Kubernetes and testing what happens when several pods go down and  ip addresses are swapped between nodes. In 4.0 this is blocked and the node cannot be restarted.



To simulate this I create a 3 node cluster on a local machine using 3 loopback addresses

{code}
127.0.0.1
127.0.0.2
127.0.0.3
{code}

The nodes are created correctly and the first node is assigned as a CMS node as shown:

{code}
bin/nodetool -p 7199 describecms
{code}

Cluster Metadata Service:
{code}
Members: /127.0.0.1:7000
Is Member: true
Service State: LOCAL
{code}

At this point I bring down the nodes 127.0.0.2 and 127.0.0.3 and swap the ip addresses for the rpc_address and listen_address 
 
The nodes come back as normal, but the nodeid has now been swapped against the ip address:

Before:
{code}
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens  Owns (effective)  Host ID                               Rack
UN  127.0.0.3  75.2 KiB   16      76.0%             6d194555-f6eb-41d0-c000-000000000003  rack1
UN  127.0.0.2  86.77 KiB  16      59.3%             6d194555-f6eb-41d0-c000-000000000002  rack1
UN  127.0.0.1  80.88 KiB  16      64.7%             6d194555-f6eb-41d0-c000-000000000001  rack1
{code}

After:
{code}
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load        Tokens  Owns (effective)  Host ID                               Rack
UN  127.0.0.3  149.62 KiB  16      76.0%             6d194555-f6eb-41d0-c000-000000000003  rack1
UN  127.0.0.2  155.48 KiB  16      59.3%             6d194555-f6eb-41d0-c000-000000000002  rack1
UN  127.0.0.1  75.74 KiB   16      64.7%             6d194555-f6eb-41d0-c000-000000000001  rack1
{code}

On previous tests of this I have created a table with a replication factor of 1, inserted some data before the swap.   After the swap the data on nodes 2 and 3 is now missing. 


One theory I have is that I am using different port numbers for the different nodes, and I am only swapping the ip addresses and not the port numbers, so the ip:port still looks unique

i.e. 127.0.0.2:9043 becomes 127.0.0.2:9044
and 127.0.0.3:9044 becomes 127.0.0.3:9043
Bug: Add option for MutualTlsAuthenticator to restrict the certificate validity period
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/Oct/23 18:15
Resolved: 26/Mar/24 00:00

In {{org.apache.cassandra.auth.MutualTlsAuthenticator}}, we validate that a certificate is valid by looking at the identities inside the
certificate and making sure the identity exists in the identity to role table.

In some situations we may want to restrict the certificates
we accept by rejecting certificates older than x amount of days. Some certificates can be generated with long expiration dates,
and this might be undesired when you want to protect against potential certificates being compromised. For that reason, it is
important to add an option, that when configured, we can limit the age of the certificate we accept for mTLS authentication.

When enabled, this will force clients to have to renew certificates more frequently, reducing the exposure of a Cassandra cluster
to leaked certificates.
Bug: SAI query timeouts can cause resource leaks
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 06/Dec/23 15:55
Resolved: 13/Dec/23 12:38

There are several places in the SAI query path where a query timeout can result in a resource not being closed correctly. We need to make sure that wherever QueryContext.checkpoint is called we catch the resulting exception and close any open resources.
Bug: Vector search tests are failing on recall accuracy
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 08/Dec/23 12:48
Resolved: 21/Mar/24 13:11

Vector tests are failing randomly because they do not meet recall assertion values. Currently, the following tests have been reported as failing:

VectorSegmentationTest.testMultipleSegmentsForCompaction
VectorDistributedTest.rangeRestrictedTest
VectorDistributedTest.testPartitionRestrictedVectorSearch

Since the vector searches are approximate and the vectors used in the tests are random, it is unlikely that they will always meet a high recall. The recall assertions are looking for recall values of 0.9 and above. Part of this issue is related to the use of random values in the vectors being tested. We have seen, with other tests, that the vector search performs better with non-random generated datasets like the Glove datasets. As such, there are the following available to fix these tests.
 # Downgrade the assertions to a value that is likely to always pass. The problem is that there is no guarantee that a test will always pass any recall value we give it.
 # Use generated datasets for these tests to see if that improves the recall results.
 # Remove the recall assertions unless they are specifically asked for. We could use a system property to enable recall testing for targeted vector testing.

I don't think option 1 is a viable long-term solution as we can never be certain that it will always work. Option 2 has more promise but it could still result in failures because of the approximate nature of the vector searches. As such, option 3 seems the only viable solution here but means that, in most cases, we are only really testing that we are returning results from the search, not how accurate those results are.
Bug: Preclude irrecoverable log corruption in case split-brain situation during leader election with absent seeds
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/Dec/23 13:12
Resolved: 25/Mar/24 08:29

It should be possible to detect a scenario where two partitioned nodes independently elect themselves as the first CMS nodes in a brand new cluster. In such a case, metadata changes should not be applied so that the conflict can be resolved.
Bug: Fix flaky test: org.apache.cassandra.tools.BulkLoaderTest.testBulkLoader_WithArgs1-_jdk17
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Dec/23 17:01
Resolved: 10/Jan/24 18:35

Seen here: https://app.circleci.com/pipelines/github/driftx/cassandra/1418/workflows/5d108993-69f8-4bb8-a616-93a1d904ff1a/jobs/67177/tests

{quote}
junit.framework.AssertionFailedError: Wrong thread status, active threads unaccounted for: [cluster2-nio-worker-2]
	at org.apache.cassandra.tools.OfflineToolUtils.assertNoUnexpectedThreadsStarted(OfflineToolUtils.java:120)
	at org.apache.cassandra.tools.BulkLoaderTest.testBulkLoader_WithArgs1(BulkLoaderTest.java:97)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
{quote}

Looks like either a thread is hanging around too long or we need to except it.