av9ash/GitBugs huggingface dataset:
Bug: CCM nodes share the same environment variable map breaking upgrade tests
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 10/Apr/24 19:53
Resolved: 16/Apr/24 16:22

In {{node.py}} {{__environment_variables}} is generally always set with a map that is passed in from {{cluster.py}} so it is [shared between nodes|https://github.com/riptano/ccm/blob/ac264706c8ca007cc584871ce907d48db334d36d/ccmlib/node.py#L151] and if nodes modify the map, such as in {{start}} when [updating the Java version|https://github.com/riptano/ccm/blob/ac264706c8ca007cc584871ce907d48db334d36d/ccmlib/node.py#L860] then when {{get_env}} runs it will [overwrite the Java version|https://github.com/riptano/ccm/blob/ac264706c8ca007cc584871ce907d48db334d36d/ccmlib/node.py#L244] that is selected by {{update_java_version}}.

This results in {{nodetool drain}} failing when upgrading from 3.11 to 4.0 in some of the upgrade tests because after the first node upgrades to 4.0 it's not longer possible for the subsequent nodes to select a Java version that isn't 11 because it's overridden by  {{__environment_variables}}.

I'm not even 100% clear on why the code in {{start}} should update {{__environment_variables}} at all if we calculate the correct java version on every invocation of other tools.
Bug: Nodetool to get/set guardrails configurations
Status: Triage Needed
Priority: Normal
Created: 11/Apr/24 05:42

Currently guardrails are only configurable through JMX / cassandra.yaml

This provides a nodetool command to interact with all the getters/setters for guardrails.

 

4.1 PR: [https://github.com/apache/cassandra/pull/3243]

trunk PR: [https://github.com/apache/cassandra/pull/3244]
Bug: Typo in Cassandra Testing Webpage
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/Apr/24 17:49
Resolved: 04/Apr/24 19:45

In `Stress and FQLTool tests` Section:
{code:java}
To run the tests for those tools, first, build jar artifacts for them but calling: {code}
"but" -> "by".
Bug: Docs should mention auth caching post-12988
Status: Open
Priority: Normal
Created: 11/Apr/24 14:45

After CASSANDRA-12988, a performance impact may be seen (see CASSANDRA-19529) by the increased consistency level which can be mitigated with caching.  The [documentation|https://cassandra.apache.org/doc/stable/cassandra/operating/security.html#authentication] should mention this.
Bug: Vector search should be able to restrict on clustering keys when filtering isn't required
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 08/Apr/24 22:08
Resolved: 10/Apr/24 22:29

With a table that has {{primary key((a,b),c,d)}}

a restriction on only the partition works,
e.g. {{where a=. and b=. order by . ann of .}}

but a restriction that also includes a forward sequence of clustering keys (i.e. a clustering key restriction that wouldn't require filtering) does not currently work.
e.g. {{where a=. and b=. and c=. order by . ann of .}}


It appears that StatementRestriction:321 is a little too greedy.
Bug: 4.1.4 Cassandra fails on bootstrap on arm64 due to JNA 5.6.0
Status: Triage Needed
Priority: Normal
Created: 09/Apr/24 23:47

https://issues.apache.org/jira/browse/CASSANDRA-17019 only fixed Cassandra 4.0.0. 
Cassandra 4.0.12 and 4.1.4 are still using JNA 5.6.0. Causing those versions to fail on Apple M1/M2.

Currently, you can work around by replacing the jar in the lib folder. For example, if you are using ccm on mac:
```

cp ~/.ccm/repository/4.0.0/lib/jna-5.13.0.jar ~/.ccm/repository/4.1.4/lib/jna-5.13.0.jar

```
Bug: Optionally avoid hint transfer during decommission - port from 5.0
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/Apr/24 15:07
Resolved: 08/Apr/24 20:36

This ticket is to port the changes already made for https://issues.apache.org/jira/browse/CASSANDRA-17808 to 4.0 and 4.1

This will allow the option to turn off the transferring of hints during decommission (specifically unbootstrap) 

This also allows the hints to be transferred at a higher rate during decommission, as the hinted_handoff_throttle is not divided by the number of nodes in the cluster for the unbootstrap process.
Bug: Optionally enable TLS in the server and client for Analytics testing
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/Apr/24 17:37
Resolved: 08/Apr/24 21:34

All integration tests today run without SSL, which is generally fine because they run locally. However, it would be helpful to be able to start up the sidecar with SSL enabled in the integration test framework so that third-party tests could connect via secure connections for testing purposes.
Bug: [Analytics] Use a classloader to isolate in-jvm dtest classes in integration test
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/Apr/24 17:51
Resolved: 04/Apr/24 23:19

We need to isolate the in-jvm dtest classes (coming from the shaded dtest jar) from the classes in Cassandra Analytics when running integration tests. The isolation will allow us to test the Analytics code in the {{AppClassloader}} without any class pollution from classes coming from the shaded dtest jar. In some cases, the pollution can spill to classes that are shaded in the analytics project (metrics, json serialization, guava, etc).

With the isolation in place we can be more confident that the code we are testing is the code that will be running in a Spark environment.
Bug: Improve I/O during slicing
Status: Triage Needed
Priority: Normal
Created: 04/Apr/24 21:04

We currently pull one chunk at a time off disk to perform partition slices.  This is pretty inefficient with larger partitions, doing 1 system call per chunk, and is wasteful on systems with either higher latency disks or ones with IOPS limits, and is generally wasteful as we could be doing significantly fewer copies from kernel to userland.

As an example, I've create a table with 2MB partitions and selected 5K rows.  There's over 50 read operations to read a total of 810KB.  Aggressively, we could read this into a 1MB buffer, and conservatively, into a 256KB buffer.  There's not really a good reason to perform multiple queries if we know we're going to need all the data except to reuse the buffer, something we might want to do anyways with a buffer pool.  

If we know we need 5K rows, we should at least read all (or some reasonable number of) the chunks we know we're going to need into a buffer that we read from rather than doing to disk 50x.  Even using the 256KB buffer would reduce this query to 4 system calls, a 12x improvement that should help reduce latency, increase throughput and improve density.

Attached is the log of requests to the Data files as collected by:


{noformat}
$ xfsslower -p $(cassandra-pid) 0 | grep Read | tee /mnt/cassandra/artifacts/io2.txt{noformat}
This is a natural follow-up to CASSANDRA-15452 and CASSANDRA-19494
Bug: Test Failure: testIncompleteState
Status: Open
Priority: Normal
Created: 03/Apr/24 23:25

[https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2680/workflows/8b1c0d0a-7458-4b43-9bba-ac96b9bfe64f/jobs/58956/tests#failed-test-0]
{code:java}
junit.framework.AssertionFailedError at org.apache.cassandra.gms.NewGossiperTest.testIncompleteState(NewGossiperTest.java:141) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
Bug: Test Failure: testMixedVersionBlockDecom
Status: Open
Priority: Normal
Created: 03/Apr/24 23:23

[https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2680/workflows/8b1c0d0a-7458-4b43-9bba-ac96b9bfe64f/jobs/58954/tests#failed-test-0]
{code:java}
java.lang.RuntimeException: java.util.concurrent.TimeoutException at org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:79) at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:546) at org.apache.cassandra.distributed.impl.AbstractCluster.close(AbstractCluster.java:1098) at org.apache.cassandra.distributed.test.ring.DecommissionTest.testMixedVersionBlockDecom(DecommissionTest.java:173) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) Caused by: java.util.concurrent.TimeoutException at org.apache.cassandra.utils.concurrent.AbstractFuture.get(AbstractFuture.java:253) at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:538){code}
Bug: Test Failure: test_simple_decommission
Status: Open
Priority: Normal
Created: 03/Apr/24 23:21

Locally it fails for me:
{code:java}
test_simple_decommission failed; it passed 0 out of the required 1 times.
<class 'ccmlib.node.ToolError'>
Subprocess ['nodetool', '-h', 'localhost', '-p', '7200', 'decommission'] exited with non-zero status; exit status: 1; 
stdout: nodetool: Can not commit transformation: "INVALID"(Not enough live nodes to maintain replication factor after decommission.).
See 'nodetool help' or 'nodetool help <command>'.{code}

It failed in CircleCI in a different way once and I cannot reproduce it anymore in CircleCI. It passed many times there:
[https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2680/workflows/8b1c0d0a-7458-4b43-9bba-ac96b9bfe64f/jobs/58929/tests#failed-test-5]
{code:java}
failed on teardown with "Unexpected error found in node logs (see stdout for full details). Errors: [[node2] 'ERROR [GossipTasks:1] 2024-04-03 19:44:52,473 MessagingService.java:501 - Cannot send the message (from:/127.0.0.2:7000, type:GOSSIP verb:GOSSIP_DIGEST_SYN) to /127.0.0.1:7000, as messaging service is shutting down']" Unexpected error found in node logs (see stdout for full details). Errors: [[node2] 'ERROR [GossipTasks:1] 2024-04-03 19:44:52,473 MessagingService.java:501 - Cannot send the message (from:/127.0.0.2:7000, type:GOSSIP verb:GOSSIP_DIGEST_SYN) to /127.0.0.1:7000, as messaging service is shutting down']{code}
Bug: Website fails to pull docker image
Status: Open
Priority: Normal
Created: 03/Apr/24 20:56

When executing "run.sh build website":

{noformat}
Step 14/48 : RUN npm i -g @antora/cli@2.3 @antora/site-generator-default@2.3 @djencks/asciidoctor-openblock
 ---> Running in 4fa9fc4d4357
/usr/local/bin/node: 1: ELF����@���@8: not found
/usr/local/bin/node: 5: Syntax error: "(" unexpected
The command '/bin/sh -c npm i -g @antora/cli@2.3 @antora/site-generator-default@2.3 @djencks/asciidoctor-openblock' returned a non-zero code: 2
Error: No such object: apache/cassandra-website:latest

Executing docker command:
docker run --rm --name website_content  -v /home/drift/cassandra/cassandra-website:/cassandra-website -v /home/drift/cassandra/cassandra-website/site-ui/build/ui-bundle.zip:/ui-bundle.zip  -e ANTORA_CONTENT_SOURCES_CASSANDRA_WEBSITE_URL=/cassandra-website -e ANTORA_UI_BUNDLE_URL=/ui-bundle.zip apache/cassandra-website:latest  build-site
Unable to find image 'apache/cassandra-website:latest' locally
docker: Error response from daemon: pull access denied for apache/cassandra-website, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.
{noformat}
Bug: [Analytics] Migrate remaining integration tests to the single dtest cluster per class model
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Apr/24 17:27
Resolved: 03/Apr/24 18:08

In https://issues.apache.org/jira/browse/CASSANDRA-19251, the majority of the tests were moved to the single in-jvm dtest cluster per class. There are a few remaining tests that are still using the {{CassandraIntegrationTest}} annotation. However, we'd like to consolidate in a single approach for testing for the following reasons:

# Consolidate testing code in a single place to reduce maintenance burden whenever we are touching the testing framework code
# Have a unique and consistent developer experience for integration tests

As part of this effort, we can remove the unused code left from the refactor.
Bug: [Analytics] Refactor Cassandra bridge
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/Apr/24 20:18
Resolved: 02/Apr/24 23:27

I have a proposal to refactor the bridge implementation by splitting the actual implementation and the shaded {{cassandra-all}} library. This separation allows us for better integration of a different {{cassandra-all}} implementation. Additionally, it better separates the actual bridge code from the Cassandra code.
Bug: Analytics Writer should support all valid partition/clustering key types
Status: Triage Needed
Priority: Normal
Created: 01/Apr/24 15:01

Today, the Analytics Writer supports a subset of all valid CQL types as partition/clustering key values as it cannot properly serialize all of them in the `Tokenizer` class, which means they can’t be used as part of the partition/clustering key when sorting data in Spark. We should support all valid partition/clustering key types, including but not limited to frozen UDTs and collections.
Bug: Consider using XOR filters or Ribbon filters instead of bloom filters
Status: Open
Priority: Normal
Created: 01/Apr/24 07:35

See https://lemire.me/blog/2019/12/19/xor-filters-faster-and-smaller-than-bloom-filters/
It seems to use less memory (1.23 vs 1.44) for the same false-positive rate at a cost of immutability.
Bug: Enhance documentation on storage engine with leading summary
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 17/May/24 12:43
Resolved: 02/Jan/25 15:40

The storage engine [documentation|https://github.com/apache/cassandra/blob/trunk/doc/modules/cassandra/pages/architecture/storage-engine.adoc]  would benefit from an abstract or summary which mentions key points that it uses a Log-structured merge (LSM) tree design, is write-oriented, and relies upon bloom filters (not B-trees) to optimize the read path.
Bug: Nodetool tablestats: add keyspace space used
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 30/May/24 18:50
Resolved: 07/Oct/24 23:40

Nodetool tabestats reports the space used live and total per table, but not for the entire keyspace.  This would be useful information.

Also, in the table level stats, it would be useful to have the read/write ratio. This metric is important in choosing compaction strategies such as LCS.
Bug: Flaky test: StartupChecksTest#testKernelBug1057843Check() on Non-Linux OS
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 20/May/24 14:48
Resolved: 20/May/24 17:57

Flaky test: StartupChecksTest#testKernelBug1057843Check() cannot pass in my MacOs(maybe Windows OS). Just skip this test when tested on Non-Linux OS
Bug: When a table attempts to clean up metrics, it was cleaning up all global table metrics
Status: Resolved
Priority: Urgent
Resolution: Fixed
Created: 02/Sep/20 00:17
Resolved: 02/Dec/20 23:20

when i use
{code:java}
bin/nodetool tablestats xxxx{code}
i get this error
{code:java}
error: org.apache.cassandra.metrics:type=Table,keyspace=xxxx,scope=feed,name=WriteLatency
-- StackTrace --
javax.management.InstanceNotFoundException: org.apache.cassandra.metrics:type=Table,keyspace=xxxx,scope=feed,name=WriteLatency
{code}
if i restart the node and check it works but if i write something then i get the error again
 * Cassandra 4.0-beta1
 * Ubuntu 20
Bug: Potential race condition in creating hard link when incremental backup is turned on
Status: Open
Priority: Urgent
Created: 13/Aug/20 16:01

It seems that there is a race condition in creating hard link if incremental backup is turned on.

The following screenshot was captured in a production cluster running Cassandra 3.0.15 after turning on incremental backup. When this {{NoSuchFileException}} happens, due to the {{FSWriteError}} and the default disk failure policy, the JVM will be shutdown, so it's a pretty critical bug.
 !incremental_backup_hardlink_exception.jpg!

Due to the risk of causing production database downtime (if similar issue happens on multiple nodes in a short time frame), and same exception causing JVM shutdown multiple times already, incremental backup had to be turned off for now, but this is not an ideal situation.

!incremental_backup_hardlink_exception1.jpg!

The deployment is on a public cloud environment with EBS-like disks that are backed by SSD with decent latency, throughput and IOPS, so it is hard to think the culprit being in the OS and IO layer. Based on the second screenshot above, this is a low flush traffic {{system.size_estimates}} table, so compaction of the source SSTable doesn't seem to be at play here.
Bug: Build target for shaded in-JVM dtest jar
Status: Resolved
Priority: High
Resolution: Fixed
Created: 03/Sep/20 13:19
Resolved: 18/Sep/20 20:40

Several small changes that are required as a prerequisite for releasing [Harry|https://issues.apache.org/jira/browse/CASSANDRA-15348]:

1. Update snakeYaml in Cassandra
2. Add a shade maven target and packaging script for shaded dtest artifacts

This patch is only a bridge for everyone to test out Harry with trunk/4.0, before we're ready to build in-JVM dtest jars for every version, and stop depending on java-driver (and, subsequently, on Netty and Guava) in Harry.
Bug: Backport CASSANDRA-14970 to 2.1 - New releases must supply SHA-256 and/or SHA-512 checksums
Status: Resolved
Priority: High
Resolution: Fixed
Created: 28/Aug/20 08:48
Resolved: 28/Aug/20 10:28

Backport CASSANDRA-14970 to cassandra-2.1 branch to make cutting a release easier (and consistent with other branches and the docs).
Bug: Add Tool to Perform Ring Membership Changes
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 23/Sep/20 14:25
Resolved: 20/Feb/25 13:53

It would be nice to provide a specific tool (ie. ringtool) to allow operators to perform safe ring membership operations (CASSANDRA-16139) such as removing failed nodes, aborting failed ring update proposals, adding or removing tokens to a host, moving tokens etc.

While we could provide this functionality via nodetool as done currently, I think it would be better to have an exclusive tool for this to separate concerns. Furthermore we can add specific permissions to this tool that are different from nodetool to allow only authorized operators to perform ring membership changes.

We should support the current methods available on nodetool (such as nodetool move, nodetool join, nodetool bootstrap resume, etc) and deprecate them on nodetool asking users to use the new tool moving forward.
Bug: Refactor Local Ring Management
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 23/Sep/20 14:19
Resolved: 20/Feb/25 13:53

Token ring management is one of the most critical parts of Cassandra, yet one of the most overlooked. Some of the problems include but are not limited to:
* Complexity (ie. [pending range calculation|https://github.com/apache/cassandra/blob/8ba163f25a56cb507e621b89b6928c2aef0ecc57/src/java/org/apache/cassandra/locator/TokenMetadata.java#L878]) 
* Inefficiency (ie. [pending range calculation|https://github.com/apache/cassandra/blob/8ba163f25a56cb507e621b89b6928c2aef0ecc57/src/java/org/apache/cassandra/locator/TokenMetadata.java#L878], [AbstractReplicationStrategy.getAddressReplicas|https://github.com/apache/cassandra/blob/33eada06a6dd3529da644377dba180795f522176/src/java/org/apache/cassandra/locator/AbstractReplicationStrategy.java#L233])
* Prone to race conditions (ie. [here|https://github.com/apache/cassandra/blob/8ba163f25a56cb507e621b89b6928c2aef0ecc57/src/java/org/apache/cassandra/locator/ReplicaLayout.java#L198])
* Poor modularity and consistency (ie. natural replicas computed from [NetworkTopologyStrategy|https://github.com/apache/cassandra/blob/8ba163f25a56cb507e621b89b6928c2aef0ecc57/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java] and pending replicas computed from [TokenMetadata|https://github.com/apache/cassandra/blob/8ba163f25a56cb507e621b89b6928c2aef0ecc57/src/java/org/apache/cassandra/locator/TokenMetadata.java#L1271])
* Insufficient testing (due to complexity and poor modularity)

These limitations make it difficult to reliably fix bugs like properly supporting node replacement with the same IP address (CASSANDRA-12344), add improvements such as safe ring membership changes, support for networking via identity instead of IP (CASSANDRA-15823) or add new features such as dynamic virtual nodes.

This ticket aims at refactoring the ring management sub-module (namely TokenMetadata and related classes) to address most of its current limitations in order to support further improvements and new features.

Some of the requirements of the proposed refactoring are:
# Make node-local ring representation fully immutable and snapshottable.
# Add content-based versioning to uniquely identify a ring snapshot throughout the cluster.
# Make token ring management vnode-centric to support membership operations on individual tokens and simplify token assignment calculations.
# Primarily identify ring endpoints by node ID to decouple a node’s identity from its IP address.
# Add a local publish/subscribe mechanism for ring change notifications, so other modules can subscribe to it and receive the newest snapshot of the ring after membership changes.
# Add testing framework to verify correctness of ring membership operations.
# Ensure the refactored sub-module does not change current behavior via comprehensive testing.
Bug: Improve Ring Membership Management
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 23/Sep/20 14:17
Resolved: 20/Feb/25 13:52

This is an umbrella ticket to group tasks to improve the status quo of token ring management.

The plan is to refactor and enhance the current ring management sub-module to ultimately support advanced ring membership operations such as reliable concurrent ring membership changes and dynamic virtual nodes.

A Cassandra Enhancement Proposal (CEP) will be submitted to the community with a detailed proposal of the tickets contained in this epic.
Bug: Fix support for adding UDT fields to clustering keys
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/Jul/20 16:04
Resolved: 22/Sep/20 09:40

Adding UDT fields to clustering keys is broken in all versions, however slightly differently.

In 4.0, there will be a brief moment while schema changes are propagated during which we won’t be able to decode and compare byte sequences. Unfortunately, it is unclear what we should do in such cases, since we can’t just come up with a comparator, and we can’t ignore non-null trailing values, since this will lead to cases where compare for tuples `a;1` and `a;2` would return 0, effectively making them equal, and we don’t know how to compare unknown trailing values. Probably we should reject such query since we can’t sort correctly, but we should make the error message more descriptive than just "Index 1 out of bounds for length 1”. The only problem is that we get this exception only on flush right now, so data already propagates to the node by that time.

In 3.0, the problem is a bit worse than that, since in 3.0 we do not ignore trailing nulls, so some of the values, written before `ALTER TYPE .. ADD` become inaccessible. Both old values, and the new ones should always be accessible.