av9ash/GitBugs huggingface dataset:
Bug: Unexpected NullPointerException in ANN+WHERE when adding rows in another partition
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Feb/24 14:24
Resolved: 29/Feb/24 14:40

* *Bug observed on the Docker image 5.0-beta1*
 * *Bug also observed on latest head of Cassandra repo (as of 2024-02-15)*
 * _*(working fine on vsearch branch of datastax/cassandra, commit hash 80c2f8b9ad5b89efee0645977a5ca53943717c0d)*_

Summary: A query with _ann + where clause on a map + where clause on the partition key_ starts erroring once there are other partitions in the table.

There are three SELECT statements in the repro minimal code below - the third is where the error is triggered.
{code:java}
// reproduced with Dockerized Cassandra 5.0-beta1 on 2024-02-15

/////////
// SCHEMA
/////////

CREATE TABLE ks.v_table (
    pk int,
    row_v vector<float, 2>,
    metadata map<text, text>,
    PRIMARY KEY (pk)
);
CREATE CUSTOM INDEX v_md
    ON ks.v_table (entries(metadata))
    USING 'StorageAttachedIndex';
CREATE CUSTOM INDEX v_idx
    ON ks.v_table (row_v)
    USING 'StorageAttachedIndex';


/////////////////////////////
// SELECT WORKS (empty table)
/////////////////////////////

SELECT * FROM ks.v_table
    WHERE metadata['map_k'] = 'map_v'
        AND pk = 0
    ORDER BY row_v ANN OF [0.1, 0.2]
    LIMIT 4;


//////////////
// ADD ONE ROW
//////////////

INSERT INTO ks.v_table (pk, metadata, row_v)
VALUES
    (0, {'map_k': 'map_v'}, [0.11, 0.19]);



/////////////////////////////////////////////
// SELECT WORKS (table has queried partition)
/////////////////////////////////////////////

SELECT * FROM ks.v_table
    WHERE metadata['map_k'] = 'map_v'
        AND pk = 0
    ORDER BY row_v ANN OF [0.1, 0.2]
    LIMIT 4;


//////////////////////////////////
// ADD ONE ROW (another partition)
//////////////////////////////////

INSERT INTO ks.v_table (pk, metadata, row_v)
VALUES
    (10, {'map_k': 'map_v'}, [0.11, 0.19]);


/////////////////////////////////////////////////
// SELECT BREAKS (table gained another partition)
/////////////////////////////////////////////////

SELECT * FROM ks.v_table
    WHERE metadata['map_k'] = 'map_v'
        AND pk = 0
    ORDER BY row_v ANN OF [0.1, 0.2]
    LIMIT 4; {code}
The error has this appearance in CQL Console:
{code:java}
ReadFailure: Error from server: code=1300 [Replica(s) failed to execute read] message="Operation failed - received 0 responses and 1 failures: UNKNOWN from /172.17.0.2:7000" info={'consistency': 'ONE', 'required_responses': 1, 'received_responses': 0, 'failures': 1, 'error_code_map': {'172.17.0.2': '0x0000'}} {code}
And the Cassandra logs have this to say:
{code:java}
java.lang.NullPointerException: Cannot invoke "org.apache.cassandra.index.sai.iterators.KeyRangeIterator.skipTo(org.apache.cassandra.index.sai.utils.PrimaryKey)" because "this.nextIterator" is null {code}
Bug: fix "if" condition for mx4j tool in cassandra-env.sh
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/Feb/24 18:40
Resolved: 21/Feb/24 11:54

There is this in cassandra-env.sh

{code}
if [[ "$MX4J_ADDRESS" == \-Dmx4jaddress* ]]; then
{code}

(similar for port)

This is wrong for /bin/sh shell (our shebang in bin/cassandra) and this does not work, probably in bash only, because /bin/sh does not understand what "[[" is nor it understand what "==" is.

The reason this was never detected so far is that the logic will never come there when MX4J_ADDRESS and / or MX4J_PORT is commented out couple lines above.
Bug: system_views.settings incorrectly handle array types
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Mar/24 12:12
Resolved: 19/Mar/24 14:06

4.1+ gives
{noformat}
cqlsh> select value from system_views.settings where name = 'data_file_directories'; 

 value
------------------------------
 [Ljava.lang.String;@21b4c4bb
 {noformat}
 

should be 

 
{noformat}
cqlsh> select value from system_views.settings where name = 'data_file_directories'; 

 value
----------------------------------------------------------------------------
 [/home/fermat/dev/cassandra/cassandra-instaclustr/cassandra-4.1/data/data]
 {noformat}
Bug: Fix cqlsh_tests/test_cqlsh.py
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Mar/24 11:43
Resolved: 15/Mar/24 17:21

https://app.circleci.com/pipelines/github/driftx/cassandra/1522/workflows/f1cd6321-9a48-4a43-92b3-aed0b610fe5e/jobs/77610/tests
Bug: deprecate dual ports support (native_transport_port_ssl) 
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/Feb/24 12:42
Resolved: 14/Feb/24 15:18

We decided (1) to deprecate dual ports support in 5.0 (and eventually remove it in trunk). This ticket will track the work towards the deprecation for 5.0.

(1) https://lists.apache.org/thread/dow196gspwgp2og576zh3lotvt6mc3lv
Bug: Commitlog with direct io fails test_change_durable_writes
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 14/Mar/24 22:23
Resolved: 25/Mar/24 11:37

With the commitlog_disk_access_mode set to direct, and the improved configuration_test.py::TestConfiguration::test_change_durable_writes from CASSANDRA-19465, this fails with either:

{noformat}
 AssertionError: Commitlog was written with durable writes disabled
{noformat}

Or what appears to be the original exception reported in CASSANDRA-19465:

{noformat}
  node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,465 StorageService.java:631 - Stopping native transport
  node1: ERROR [MutationStage-5] 2024-03-14 17:16:08,465 StorageProxy.java:1670 - Failed to apply mutation locally :
  java.lang.IllegalArgumentException: newPosition > limit: (1048634 > 1048576)
        at java.base/java.nio.Buffer.createPositionException(Buffer.java:341)
        at java.base/java.nio.Buffer.position(Buffer.java:316)
        at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516)
        at java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:321)
        at java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:73)
        at org.apache.cassandra.db.commitlog.CommitLogSegment.allocate(CommitLogSegment.java:216)
        at org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard.allocate(CommitLogSegmentManagerStandard.java:52)
        at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307)
        at org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99)
        at org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53)
        at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:612)
        at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:497)
        at org.apache.cassandra.db.Mutation.apply(Mutation.java:244)
        at org.apache.cassandra.db.Mutation.apply(Mutation.java:264)
        at org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1664)
        at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2624)
        at org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163)
        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:833)
  node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,470 StorageService.java:636 - Stopping gossiper
{noformat}
Bug: Update rpm and debian shell includes
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 22/Jan/24 16:32
Resolved: 08/Feb/24 12:15

While working on CASSANDRA-19001, it was identified that there are differences between bin/cassandra.in.sh and redhat/cassandra.in.sh, and it seems the debian diff on 5.0 was updated once in 2020 since it was created in 2019.

CC [~brandon.williams]
Bug: Fix tests to work with ULID SSTable identifiers to enable uuid_sstable_identifiers_enabled in cassandra-latest.yaml
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 07/Mar/24 09:28
Resolved: 12/Mar/24 12:52

In CASSANDRA-18753 we identified that we want to also set uuid_sstable_identifiers_enabled to true, while running a CI with it turned on, it failed (1).

Errors do not seem to be serious, it is just the test suite we have is not prepared for the case when uuid_sstable_identifiers_enabled is set to true by default.

We need to fix all these tests so we can have cassandra-latest.yaml containing that property.

https://app.circleci.com/pipelines/github/blambov/cassandra/609/workflows/aef2b936-0551-4f3b-9d86-a49451c83947
Bug: Make concurrent_index_builders configurable at runtime
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 11/Jan/24 19:33
Resolved: 25/Jan/24 09:27

CASSANDRA-17781 introduced the YAML option \{{concurrent_index_builders}} to allow us to configure index building independently of compaction. We need to be able to adjust this after index building starts without a restart. (In some cases, we might do it in conjunction w/ \{{nodetool stop}}, but it depends on the level of ongoing disruption.)
Bug: Add cli log level to circle config
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 31/Jan/24 22:23
Resolved: 02/Feb/24 13:28

We try to set the [log cli level|https://github.com/apache/cassandra/blob/trunk/.circleci/config_template.yml#L3538] so that dtest logging is captured to stdout, but we don't do it [everywhere|https://github.com/apache/cassandra/blob/trunk/.circleci/config_template.yml#L3063] so we miss some logs.
Bug: if max_hints_size_per_host < max_hints_file_size then it will write hints after max_hints_size_per_host is reached
Status: Resolved
Priority: Normal
Resolution: Resolved
Created: 21/Mar/24 09:42
Resolved: 25/Mar/24 13:36

there is one problem in the current solution being that if we have this config

{noformat}
max_hints_size_per_host: 2MiB
max_hints_file_size: 128MiB
{noformat} 

basically, max size > size per host, then it will not stop it from writing hints after 2MiB for a particular node, because HintsDescriptor is added among dispatching ones after writer is closed, which happens after there is 128MiB written (all logic in HintsStore), so it will not be included into total sizes, it will be 0 until at least one hints file for that node is written to disk.

I consider this to be the flaw of CASSANDRA-17142, however it is questionable if this is serious enough to deal with in the first place, I don't think somebody would set it up in practice to these values, normally one puts there like max per host is few gigs so this problem is not so visible but it shows in tests almost instantly and it is technically just wrong, regardless of the probability this would happen in real ... 

cc [~yifanc]
Bug: StorageService.getRangeToEndpointMap() MBean operation is running into NPE for LocalStrategy keysapces
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 09/Jan/24 17:21
Resolved: 22/Mar/24 10:49

When the StorageService's MBean operation getRangeToEndpointMap is called for LocalStrategy keyspaces, then it is running into NPE. It is working in earlier major version, but failing in trunk. It can be reproduced in local using JConsole or using a tool like `jmxterm` (unfortunately these tools are not giving full stacktrace). Observed the same behavior with getRangeToEndpointWithPortMap operation too.
Bug: [Analytics] Report additional bulk analytics job stats for instrumentation
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Feb/24 19:51
Resolved: 22/Mar/24 23:23

Currently, the Cassandra bulk analytics library supports a "dialHome" API to publish some initial job metadata, which in its current form, is redirected to a log. The intention behind this is to allow custom implementations that can utilize these summarized stats for instrumentation or reporting of client behavior.

This task is meant to enhance this API to allow for additional job metadata to be published both at the Spark executor level and at the task levels to gather stats such as "success/failure", "number of rows written/read", "failure reason" etc.
Bug: Test failure: org.apache.cassandra.fuzz.ring.ConsistentBootstrapTest.coordinatorIsBehindTest
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 08/Mar/24 07:55
Resolved: 21/Mar/24 14:28

Looks like this was broken by CASSANDRA-18275

Timeouts in the log and then:
{code}
java.lang.IllegalStateException: Can't use shutdown instances, delegate is null

	at org.apache.cassandra.distributed.impl.AbstractCluster$Wrapper.delegate(AbstractCluster.java:283)
	at org.apache.cassandra.distributed.impl.DelegatingInvokableInstance.transfer(DelegatingInvokableInstance.java:49)
	at org.apache.cassandra.distributed.api.IInvokableInstance.runsOnInstance(IInvokableInstance.java:45)
	at org.apache.cassandra.distributed.api.IInvokableInstance.runOnInstance(IInvokableInstance.java:46)
	at org.apache.cassandra.distributed.shared.ClusterUtils.unpauseCommits(ClusterUtils.java:548)
	at org.apache.cassandra.fuzz.ring.ConsistentBootstrapTest.coordinatorIsBehindTest(ConsistentBootstrapTest.java:227)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
	at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)
{code}
Bug: The repository 'https://debian.cassandra.apache.org 311x Release' does not have a Release file.
Status: Resolved
Priority: Normal
Resolution: Invalid
Created: 19/Mar/24 10:50
Resolved: 19/Mar/24 14:14

The debian repo for 311 can't be used anymore; apt update fails with

{color:#000000}Ign:10 [https://debian.cassandra.apache.org|https://debian.cassandra.apache.org/] 311x InRelease {color}
Err:12 [https://debian.cassandra.apache.org|https://debian.cassandra.apache.org/] 311x Release 
 server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none 
Reading package lists... Done{color:#b26818} {color}{color:#000000} {color}
E: The repository 'https://debian.cassandra.apache.org 311x Release' does not have a Release file. 
N: Updating from such a repository can't be done securely, and is therefore disabled by default. 
N: See apt-secure(8) manpage for repository creation and user configuration details.
Bug: Deprecate Python 3.7 and earlier, but allow cqlsh to run with Python 3.6+
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 12/Mar/24 20:50
Resolved: 20/Mar/24 04:50

In 4.0, we introduced Python 3.6 support. In CASSANDRA-19245, for the 5.0 release, we dropped Python 3.6 and 3.7 support since Python 3.8 was the minimum supported version for version 3.29.0 of the Python driver, which we needed to support vectors. Unfortunately, that may break existing users of RHEL 7, which natively only supports Python 3.6.

We should deprecate Python 3.6-3.7 (which are all either EOL or will be soon) and mention this in NEWS.txt/appropriate warnings at {{cqlsh}} startup, but we don't need to have cqlsh refuse to start entirely with 3.6-3.7 until our 6.0 release.
Bug: Improve Could not perform commit after 4089/10 tries log message
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 31/Jan/24 12:18
Resolved: 19/Mar/24 13:54

Improve retry logging for the local processor when using deadline and indefinite strategies:

Currently, it is sometimes possible to get into the situation where we get the following log message:

{code}
java.lang.IllegalStateException: Can not commit transformation: "SERVER_ERROR"(Could not perform commit after 4089/10 tries. Time remaining: 0ms). 
at org.apache.cassandra.tcm.ClusterMetadataService.lambda$commit$6(ClusterMetadataService.java:470) 
at org.apache.cassandra.tcm.ClusterMetadataService.commit(ClusterMetadataService.java:514) 
at org.apache.cassandra.tcm.ClusterMetadataService.commit(ClusterMetadataService.java:467) 
at org.apache.cassandra.tcm.Startup.initializeAsFirstCMSNode(Startup.java:139) 
at org.apache.cassandra.tcm.migration.Election.finish(Election.java:141) 
at org.apache.cassandra.tcm.migration.Election.nominateSelf(Election.java:94) 
at org.apache.cassandra.tcm.ClusterMetadataService.upgradeFromGossip(ClusterMetadataService.java:345) 
at org.apache.cassandra.tcm.CMSOperations.initializeCMS(CMSOperations.java:65) 
at org.apache.cassandra.tools.nodetool.InitializeCMS.execute(InitializeCMS.java:37) 
{code}
Bug: Latency Spike on NTR startup
Status: Triage Needed
Priority: Normal
Created: 15/Mar/24 00:34

Firstly you need the patch from https://issues.apache.org/jira/browse/CASSANDRA-18845 to solve consistency query errors on startup. With that patch there is still a further issue we see on some clusters where the latency spikes too high when initially starting. I see pending compactions and hints metrics increased during this time.

I tried lowering the hint delivery threshold across the cluster thinking it was overloading the node starting up, but this didn't resolve the issue. So at this time I am not sure what the root cause (I still think its combination of the compactions and hints).

As workaround I have this small code change:
{code:java}
            int START_NATIVE_DELAY = Integer.getInteger("cassandra.start_native_transport_delay_secs", 120);
            if (START_NATIVE_DELAY > 0)
            {
                logger.info("Waiting an extra {} seconds before enabling NTR", START_NATIVE_DELAY);
                Uninterruptibles.sleepUninterruptibly(START_NATIVE_DELAY, TimeUnit.SECONDS);
            }
            startNativeTransport();
 {code}
Where wait an configurable time before starting native transport. Delaying NTR startup resolved the issue.

A better solution would be to wait for hints/compactions or whatever is root cause to complete.
Bug: Register the measurements of the bootstrap as Dropwizard metrics
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Feb/24 19:41
Resolved: 13/Mar/24 16:17

Currently, we can view the node's bootstrap state in the following ways:
- via the nodetool cli tool, e.g. by running the "resume" command;
- querying the bootstrapped column of the system_veis.local virtual table;

In addition, we can also expose the status and state of the node's bootstrap via JMX. This is used by third-party tools that rely entirely on the JMX API and don't have access to the CQL interface. The operator will be able to get all the information they need from the dashboards without having to use CLIs.
Bug: LIST SUPERUSERS cql command
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Feb/24 15:03
Resolved: 13/Mar/24 12:02

Developing a new CQL command LIST SUPERUSERS to return list of roles with superuser privilege. This includes roles who acquired superuser privilege in the hierarchy. 

Context: LIST ROLES cql command lists roles, their membership details and displays super=true for immediate superusers. But there can be roles who acquired superuser privilege due to a grant. LIST ROLES command won't display super=true for such roles and the only way to recognize such roles is to look for atleast one row with super=true in the output of LIST ROLES OF <role name> command. While this works to check is a given role has superuser privilege, there may be services (for example, Sidecar) working with C* and may need to maintain list of roles with superuser privilege. There is no existing command/tool to retrieve such roles details. Hence developing this command which returns all roles having superuser privilege.