av9ash/GitBugs huggingface dataset:
Bug: Secondary indexes either violate consistency or become unavailable when post-streaming index builds fail
Status: Open
Priority: Normal
Created: 07/Jul/23 22:53

Back in 2015, we identified in CASSANDRA-10130 a case where failures in 2i builds after SSTable streaming could leave indexes in a partially built state, even after a restart, requiring manual operator intervention. There, and in CASSANDRA-13725, we made an attempt to remedy this situation, ensuring that indexes would at least be rebuilt on restart after this kind of failure. However, there are some difficulties the solution there does not address.

Let's look at a simple example...

Suppose an SSTable has been streamed to a node, and that node arrives in {{CassandraStreamReceiver#finished()}}. We'll call {{finishTransaction()}} to make the presence of the new SSTables durable, and then we'll call {{ColumnFamilyStore#addSStables()}}, which add the table to the {{Tracker}}, making it available for reads. We then notify listeners about the new SSTable, among them the {{SecondaryIndexManager}}, which will do a blocking index build for the new SSTable. Conceptually, at this point, we already have a problem (if a transient one), as there are live SSTables that have not been indexed.

What if the 2i build fails, though? Let's assume it fails because of a disorderly (or orderly!) node shutdown. Some index implementations (SASI, SAI) might be able to rebuild incrementally, but the legacy 2i has no way of doing this right now. A full index rebuild on a large table could take a very long time (days, weeks, etc.) and is ultimately not a viable way to proceed. Let's say we were able to build incrementally though, and we had an SAI index that did exactly this on node restart. We would still have a gap in availability, because on startup, {{ColumnFamilyStore}} (see constructor) does not block on its calls to {{SecondaryIndexManager#addIndex()}}, which, via {{createIndex()}} actuate the index building process.

Of course, SAI implements a notion of "queryability" that would quickly take the node out of rotation for queries across the cluster. Once its initialization task runs on restart, the indexes in question would immediately be marked non-queryable. SAI builds incrementally, and might be able to block startup to do so in this case. Legacy 2i cannot reasonably do this though.
Bug: Detect token-ownership mismatch
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Aug/23 18:39
Resolved: 05/Nov/24 22:11

*Problem Statement*

As we know, Cassandra exchanges important topology and token-ownership-related details over Gossip. Cassandra internally maintains the following two separate caches that have the token-ownership information maintained: 1) Gossip cache and 2) Storage Service cache. The first Gossip cache is updated on a node, followed by the storage service cache. In the hot path, ownership is calculated from the storage service cache. Since two separate caches maintain the same information, then inconsistencies are bound to happen. It could be very well feasible that the Gossip cache has up-to-date ownership of the Cassandra cluster, but the service cache does not, and in that scenario, inconsistent data will be served to the user.

Currently, there is no mechanism in Cassandra that detects and fixes these two caches. 

*Long-term solution*
We are going with the long-term transactional metadata ([https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-21]) to handle such inconsistencies, and that’s the right thing to do.

*Short-term solution*
But CEP-21 might take some time, and until then, there is a need to *detect* such inconsistencies. Once we detect inconsistencies, then we could have two options: 1) restart the node or 2) Fix the inconsistencies on-the-fly.

This JIRA is providing a short-term solution. Please review the pull request (on 4.1): [https://github.com/apache/cassandra/pull/3548]
Bug: Set right client auth for creating SSL context in mTLS optional mode
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 30/Aug/23 23:12
Resolved: 19/Dec/23 20:55

Adding a new value `optional` for require_client_auth in Encryption options. when require_client_auth is optional, the SSL context that is created will allow client connections that provide a client certificate along with the client connections that do not provide certificates.
Bug: Node sends multiple inflight echos
Status: Review In Progress
Priority: Normal
Created: 19/Sep/23 01:51

CASSANDRA-18854 rolled back the changes from CASSANDRA-18845. In particular, 18845 had change to only allow 1 inflight ECHO request at a time. As per 18854 some tests have an error rate due to this change. Creating this ticket to discuss this further. As the current state also does not have retry logic, it just allowing multiple ECHO requests inflight at the same time so less likely that all ECHO will timeout or get lost.

With the change from 18845 adding in some extra logging to track what is going on, I do see it retrying ECHOs. Likewise, I patched a node to drop ECHO requests from a node and also see it retrying ECHOs when it doesn't get a reply.

Therefore, I think the problem is more specific than the dropping of one ECHO request. Yes there no retry logic for failed ECHO requests, but this is the case even both before and after 18845. ECHO requests are only sent via gossip verb handlers calling applyStateLocally. In these failed tests I therefore assuming their cases where it won't call markAlive when other nodes consider the node UP but its marked DOWN by a node.
Bug: Document grant permission for all tables in a keyspace
Status: Triage Needed
Priority: Normal
Created: 19/Sep/23 19:12

Update https://cassandra.apache.org/doc/4.1/cassandra/cql/security.html to include the new DCL added by CASSANDRA-17027.

{code:sql}
GRANT ALTER ON ALL TABLES IN KEYSPACE ks TO user;
{code}
Bug: Make unit tests compatible with the tmp.dir property
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 14/Aug/23 13:35
Resolved: 28/Aug/23 14:50

Several unit tests hard-code file paths under the "/tmp" directory, which means they do not honor the {{tmp.dir}} ant build property. These should be updated so that when the user specifies {{tmp.dir}}, they can be certain that any files or directories created by the unit tests will be in their specified location.
Bug: Update Chronicle bytes
Status: Changes Suggested
Priority: Normal
Created: 31/Jul/23 11:14

[https://github.com/OpenHFT/Chronicle-Bytes/pull/485] fixes test failures of cassandra on s390x.

This patch is merged and available in chronicle-bytes-2.24ea7 and later releases.

possible to update Chronicle bytes and related pkg versions in Cassandra (https://github.com/apache/cassandra/blob/trunk/.build/parent-pom-template.xml)?
Bug: snappy-java-1.1.10.1 vulnerability: CVE-2023-43642
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 27/Sep/23 13:11
Resolved: 27/Sep/23 19:57

https://nvd.nist.gov/vuln/detail/CVE-2023-43642

Another DoS which is probably not a huge deal, but we can upgrade.
Bug: Test failure: junit.framework.TestSuite.org.apache.cassandra.distributed.test.CASMultiDCTest-.jdk11 
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 28/Jul/23 21:53
Resolved: 05/Oct/23 07:30

Seen here:

[https://ci-cassandra.apache.org/job/Cassandra-trunk/1650/testReport/junit.framework/TestSuite/org_apache_cassandra_distributed_test_CASMultiDCTest__jdk11/]
h3.  
{code:java}
Error Message
Schema agreement not reached. Schema versions of the instances: [ef1c8e05-a06d-388d-a46d-53cc22a94762, 6c386108-1805-3985-b48e-8016012a0207, 6c386108-1805-3985-b48e-8016012a0207, ef1c8e05-a06d-388d-a46d-53cc22a94762]

Stacktrace
java.lang.IllegalStateException: Schema agreement not reached. Schema versions of the instances: [ef1c8e05-a06d-388d-a46d-53cc22a94762, 6c386108-1805-3985-b48e-8016012a0207, 6c386108-1805-3985-b48e-8016012a0207, ef1c8e05-a06d-388d-a46d-53cc22a94762] at org.apache.cassandra.distributed.impl.AbstractCluster$ChangeMonitor.waitForCompletion(AbstractCluster.java:907) at org.apache.cassandra.distributed.impl.AbstractCluster.lambda$schemaChange$8(AbstractCluster.java:836) at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96) at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61) at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829)
{code}
Bug: Add CASSANDRA-14227 to 5.0 new features page
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 16/Aug/23 09:25
Resolved: 17/Aug/23 08:40

The [page|https://cassandra.apache.org/doc/trunk/cassandra/new/index.html] listing the 5.0 new features is not mentioning the extended TTL implemented in CASSANDRA-14227
Bug: netty-handler vulnerability: CVE-2023-4586
Status: Resolved
Priority: Normal
Resolution: Invalid
Created: 30/Aug/23 13:34
Resolved: 28/Mar/24 19:35

This is failing OWASP:

{noformat}
Dependency-Check Failure:
One or more dependencies were identified with vulnerabilities that have a CVSS score greater than or equal to '1.0': 
netty-handler-4.1.96.Final.jar: CVE-2023-4586
{noformat}
Bug: Baseline Diagnostic vtables for Accord
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 07/Aug/23 15:36
Resolved: 28/May/24 21:55

In addition to JMX-based metrics, there are bits of diagnostic information for Accord that we should consider exposing through vtables:

1.) We should ensure that coordinator-level CQL transactions and the local reads and writes they spawn are visible to the existing {{QueriesTable}} vtable.

The first may already just work. We may need to make some tweaks to {{TxnNamedRead}} and {{TxnWrite}} for the local operations though. ({{CommandStore}} tasks are out of scope here, as they would probably be more confusing than useful in {{QueriesTable}}?)

2.) A new vtable for pending commands for a key.

- Disable SELECT */require a partition key
- Might require partial back-port of stringifying table/partition key from Accord to be correct
- ex. {{SELECT timestamps FROM myawesometable where ks=? and table=? and partition_key=?}}
- Clustering can be the Accord timestamp elements, no further normal columns.

3.) A new vtable for command store-specific cache stats

- Gather via {{Store.execute()}} for correctness.
- Store id should be partition key (see {{AccordCommandStore}})
- hits, misses, total (maybe just throw out the keyspaces and coalesce ranges?)

4.) (Requires [~aweisberg]'s outstanding work) A new vtable for live migration state

- {{TableMigrationState}} could be flattened into a row
- Is this already persisted? If so, why a new vtable?

5.) A vtable to expose {{accord.local.Node#coordinating()}} as a map

- ex. {{SELECT txn_id, type}}
Bug: Expand CQLSSTableWriter to write SSTable-attached secondary indexes
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 31/Jul/23 22:49
Resolved: 22/Jan/24 12:16

{{CQLSSTableWriter}} currently has no way of writing any secondary indexes inline as it writes the core SSTable components. With SAI, this has become tractable problem, and we should be able to enhance both it and {{SSTableImporter}} to handle cases where we might want to write SSTables somewhere in bulk (and in parallel) and then import them without waiting for index building on import. It would require the following changes:

1.) {{CQLSSTableWriter}} must accept 2i definitions on top of its current table schema definition. Once added to the schema, any {{ColumnFamilyStore}} instances opened will have those 2i defined in their index managers.

2.) All {{AbstractSSTableSimpleWriter}} instances must register index groups, allowing the proper {{SSTableFlushObservers}} to be attached to {{SSTableWriter}}. Once this is done, SAI (and any other SSTable-attached indexes) components will be built incrementally along w/ the SSTable data file, and will be finalized when the newly written SSTable is finalized.

3.) Provide an example (in a unit test?) of how a third-party tool might, assuming access to the right C* JAR, validate/checksum SAI components outside C* proper.

4.) {{SSTableImporter}} should have two new options:
    a.) an option that fails import if any SSTable-attached 2i must be built (i.e. has not already been built and brought along w/ the other new SSTable components)
    b.) an option that allows us to bypass full checksum validation on imported/already-built SSTable-attached indexes (assuming they have just been written by {{CQLSSTableWriter}})
Bug: Update cassandra-stress to use Apache Commons CLI
Status: Open
Priority: Normal
Created: 11/Jul/23 15:17

The Apache Commons CLI library provides an API for parsing command line options with the package org.apache.commons.cli and this is already used by a dozen of existing Cassandra utilities including:
{quote}SSTableMetadataViewer, StandaloneScrubber, StandaloneSplitter, SSTableExport, BulkLoader, and others.
{quote}
However, cassandra-stress is an outlier which uses its own custom classes to parse command line options with classes such as OptionsSimple.  In addition, the options syntax for username, password, and others are not aligned with the format used by CQLSH.

Currently, there are > 5K lines of code in 'settings' which appears to just process command line args.

This suggestion is to:
 
a) Upgrade cassandra-stress to use Apache Commons CLI (no new dependencies are required as this library is already used by the project)
 
b) Align the cassandra-stress CLI options with those in CQLSH, 
 
{quote}For example, using the new syntax like CQLSH:
{quote}
 

cassandra-stress -username foo -password bar
{quote}and replacing the old syntax:
{quote}
cassandra-stress -mode username=foo and password=bar
 
This will simplify and unify the code base, eliminate code and reduce the confusion between similar named classes such as org.apache.cassandra.stress.settings.\{Option, OptionsMulti, OptionsSimple} and org.apache.commons.cli.{Option, OptionGroup, Options)
 

Note: documentation will need to be updated as well
Bug: Waiting for gossip to settle on live endpoints
Status: Triage Needed
Priority: Normal
Created: 13/Sep/23 01:54

This is a follow up to CASSANDRA-18543

Although that ticket added ability to set cassandra.gossip_settle_min_wait_ms this is tedious and error prone. On a node just observed a 79 second gap between waiting for gossip and the first echo response to indicate a node is UP.

The problem being that do not want to start Native Transport until gossip settles otherwise queries can fail consistency such as LOCAL_QUORUM as it thinks the replicas are still in DOWN state.

Instead of having to set gossip_settle_min_wait_ms I am proposing that (outside single node cluster) wait for UP message from another node before considering gossip as settled. Eg.
{code:java}
            if (currentSize == epSize && currentLive == liveSize && liveSize > 1)
            {
                logger.debug("Gossip looks settled.");
                numOkay++;
            } {code}
Bug: Backport CASSANDRA-16418: Cleanup behaviour during node decommission caused missing replica
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 06/Sep/23 08:22
Resolved: 07/Feb/24 14:20

Node decommission triggers data transfer to other nodes. While this transfer is in progress,
receiving nodes temporarily hold token ranges in a pending state. However, the cleanup process currently doesn't consider these pending ranges when calculating token ownership.
As a consequence, data that is already stored in sstables gets inadvertently cleaned up.

STR:
 * Create two node cluster
 * Create keyspace with RF=1
 * Insert sample data (assert data is available when querying both nodes)
 * Start decommission process of node 1
 * Start running cleanup in a loop on node 2 until decommission on node 1 finishes
 * Verify of all rows are in the cluster - it will fail as the previous step removed some of the rows

It seems that the cleanup process does not take into account the pending ranges, it uses only the local ranges - [https://github.com/apache/cassandra/blob/caad2f24f95b494d05c6b5d86a8d25fbee58d7c2/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L466].

There are two solutions to the problem.

One would be to change the cleanup process in a way that it start taking pending ranges into account. Even thought it might sound tempting at first it will require involving changes and a lot of testing effort.

Alternatively we could interrupt/prevent the cleanup process from running when any pending range on a node is detected. That sounds like a reasonable alternative to the problem and something that is relatively easy to implement.

The bug has been already fixed in 4.x with CASSANDRA-16418, the goal of this ticket is to backport it to 3.x.
Bug: Cassandra 4.0 - JNA 5.6.0 does not support macOS M1 arm64
Status: Patch Available
Priority: Normal
Created: 28/Sep/23 13:38

As discussed on Slack: [https://the-asf.slack.com/archives/CJZLTM05A/p1684745250901489]

Created this ticket as clone of CASSANDRA-17019, to ask for considering a JNA library upgrade in Cassandra 4.0, so that we could utilize ARM-based AWS Gravition instances e.g. m7g already with Cassandra 4.0.

From linked ticket:
"Cassandra depends on net.java.dev.jna.jna version 5.6.0 to do the native binding into the C library. JNA 5.6.0 does not support arm64 architecture (Apple M1 devices), causing cassandra to fail on bootstrap."
Bug: CASTing a float to decimal adds wrong digits
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/Jul/23 21:27
Resolved: 12/Jul/23 10:30

If I create a table with a *float* (32-bit) column, and cast it to the *decimal* type, the casting wrongly passes through the double (64-bit) type and picks up extra, wrong, digits. For example, if we have a column e of type "float", and run

INSERT INTO tbl (p, e) VALUES (1, 5.2)

SELECT CAST(e AS decimal) FROM tbl WHERE p=1

The result is the "decimal" value 5.199999809265137, with all those extra wrong digits. It would have been better to get back the decimal value 5.2, with only two significant digits.

It appears that this happens because Cassandra's implementation first converts the 32-bit float into a 64-bit double, and only then converts that - with all the silly extra digits it picked up in the first conversion - into a "decimal" value.

Contrast this with CAST(e AS text) which works correctly - it returns the string "5.2" - only the actual digits of the 32-bit floating point value are converted to the string, without inventing additional digits in the process.
Bug: Optionally fail when a non-partition-restricted query is issued against a storage-attached index
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Aug/23 19:27
Resolved: 05/Jan/24 10:02

With LCS, we will have potentially thousands of SSTables for a given user table. Storage-attached also means SSTable-attached, and searching thousands of attached indexes is not going to scale well at all locally, due to the sheer number of searches and amount of postings list merging involved. We should have a guardrail to prohibit this by default.

Partition-restricted queries, the use-case SAI is broadly designed for, should be very efficient.

UPDATE: The consensus from the discussion below is that we'll add two guardrails here. One is simply for whether we'll allow non-partition-restricted 2i queries. The other is a warn/fail threshold pair for the number of SSTable indexes read locally on a query.
Bug: Test failure: junit.framework.TestSuite.org.apache.cassandra.distributed.test.CASMultiDCTest.testLocalSerialLocalCommit
Status: Open
Priority: Normal
Created: 14/Sep/23 06:07

See CASSANDRA-18707 Where this test is [proven|https://issues.apache.org/jira/browse/CASSANDRA-18707?focusedCommentId=17761803&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17761803] flaky

failure in testLocalSerialLocalCommit:

{noformat}
junit.framework.AssertionFailedError: numWritten: 2 < 3
	at org.apache.cassandra.distributed.test.CASMultiDCTest.testLocalSerialCommit(CASMultiDCTest.java:111)
	at org.apache.cassandra.distributed.test.CASMultiDCTest.testLocalSerialLocalCommit(CASMultiDCTest.java:121)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

{noformat}
Bug: Allow CQL client certificate authentication to work without sending an AUTHENTICATE request
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Sep/23 21:31
Resolved: 31/Jan/24 01:06

Currently when using {{MutualTlsAuthenticator}} or {{MutualTlsWithPasswordFallbackAuthenticator}} a client is prompted with an {{AUTHENTICATE}} message to which they must respond with an {{AUTH_RESPONSE}} (e.g. a user name and password).  This shouldn't be needed as the role can be identified using only the certificate.

To address this, we could add the capability to authenticate early in processing of a {{STARTUP}} message if we can determine that both the configured authenticator supports certificate authentication and a client certificate was provided.  If the certificate can be authenticated, a {{READY}} response is returned, otherwise an {{ERROR}} is returned.

This change can be done done in a fully backwards compatible way and requires no protocol or driver changes;  I will supply a patch shortly!
Bug: Leakage of references to SSTable on unsuccessful operations
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 12/Sep/23 12:21
Resolved: 15/Sep/23 07:28

This is a little bit tricky to describe correctly as I can talk about the symptoms only. I hit this issue when testing CASSANDRA-18781.

In a nutshell, when we go to bulkload an SSTable, it opens it in SSTableLoader. If bulkloading fails on server side and exception is propagated to the client, on releasing of references, it fails on this assert (1). This practically means that we are leaking resources as something still references that SSTable but it was not tidied up (on failure). On a happy path, it is all de-referenced correctly.

I think that this might have implications beyond SSTable loading, e.g. this could happen upon streaming too.

(1) https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/sstable/SSTableLoader.java#L245
Bug: Remove dependency on archived project metrics-reporter-config
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 10/Aug/23 22:52
Resolved: 12/Sep/23 14:32

metrics-reporter-config has been archived for over 3 years, and has received no updates in nearly 6 years:
https://github.com/addthis/metrics-reporter-config

metrics-reporter-config processes YAML unsafely: https://github.com/addthis/metrics-reporter-config/blob/master/reporter-config-base/src/main/java/com/addthis/metrics/reporter/config/AbstractReporterConfig.java#L39C34-L39C45

This is related to CASSANDRA-18614, which is focused on migrating to safe YAML processing for the main configuration files.