av9ash/GitBugs huggingface dataset:
Bug: Restarting a node causes MV unavailable exceptions afterward
Status: Open
Priority: Normal
Created: 09/Feb/24 12:21

If a cluster is started with materialized views enabled and a client is constantly writing to one, when a node is restarted shortly after startup it may throw UEs:

{quote}
INFO  [GossipStage:1] 2024-02-09 06:17:10,073 Gossiper.java:1419 - Node /10.5.11.102:7000 has restarted, now UP
ERROR [MutationStage-2] 2024-02-09 06:17:10,074 Keyspace.java:650 - Unknown exception caught while attempting to update MaterializedView! stresstest.transport_orders
org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level ONE
        at org.apache.cassandra.exceptions.UnavailableException.create(UnavailableException.java:37)
        at org.apache.cassandra.locator.ReplicaPlans.assureSufficientLiveReplicas(ReplicaPlans.java:169)
        at org.apache.cassandra.locator.ReplicaPlans.assureSufficientLiveReplicasForWrite(ReplicaPlans.java:112)
        at org.apache.cassandra.locator.ReplicaPlans.forWrite(ReplicaPlans.java:353)
        at org.apache.cassandra.locator.ReplicaPlans.forWrite(ReplicaPlans.java:344)
        at org.apache.cassandra.locator.ReplicaPlans.forWrite(ReplicaPlans.java:338)
        at org.apache.cassandra.service.StorageProxy.wrapViewBatchResponseHandler(StorageProxy.java:1417)
        at org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:1077)
        at org.apache.cassandra.db.view.TableViews.pushViewReplicaUpdates(TableViews.java:169)
        at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:645)
        at org.apache.cassandra.db.Keyspace.applyFuture(Keyspace.java:476)
        at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:223)
        at org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
        at org.apache.cassandra.net.InboundSink.lambda$new$0(InboundSink.java:78)
        at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:97)
        at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:45)
        at org.apache.cassandra.net.InboundMessageHandler$ProcessMessage.run(InboundMessageHandler.java:430)
        at org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:142)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:750)
{quote}

It's not immediately clear what the impact of this is since the error is on restart and not communicated back to the client.
Bug: Guardrail to warn clients about possible transient incorrect responses for filtering queries against multiple mutable columns
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 22/Mar/24 18:59
Resolved: 03/Apr/24 21:49

Given we may not have time to fully resolve CASSANDRA-19007 before we release 5.0, it would still be helpful to have, at the very minimum, a client warning for cases where a user filters on two or more mutable (static or regular) columns at consistency levels that require coordinator reconciliation. We may also want the option to fail these queries outright, although that need not be the default.

The only art involved in this is deciding what we want to say in the warning/error message. It's probably reasonable to mention there that this only happens when we have unrepaired data. It's also worth noting that SAI queries are no longer vulnerable to this after the resolution of CASSANDRA-19018.
Bug: Skinny dev circle workflow
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/Feb/24 10:13
Resolved: 28/Feb/24 07:03

CircleCi CI runs are getting pretty heavy. During dev iterations we trigger many CI pre-commit jobs which are just an overkill.

This ticket has the purpose to purge from the pre-commit workflow all variations of the test matrix but the vanilla one. That should enable us for a quick and cheap to iterate *during dev*, this is not a substitute for pre-commit . This ticket's work will serve as the basis for the upcoming changes being discussed [atm|https://lists.apache.org/thread/qf5c3hhz6qkpyqvbd3sppzlmftlc0bw0]
Bug: support legacy [plain_text_auth] section in credentials file removed unintentionally
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Mar/24 13:55
Resolved: 14/May/24 09:50

The pylib/cqlshlib/cqlshmain.py code reads data from the credentials file, however, it is immediately ignored.

[https://github.com/apache/cassandra/blob/c9625e0102dab66f41d3ef2338c54d499e73a8c5/pylib/cqlshlib/cqlshmain.py#L2070]
{code:java}
    if not options.username:
        credentials = configparser.ConfigParser()
        if options.credentials is not None:
            credentials.read(options.credentials)        # use the username from credentials file but fallback to cqlshrc if username is absent from the command line parameters
        options.username = username_from_cqlshrc    if not options.password:
        rawcredentials = configparser.RawConfigParser()
        if options.credentials is not None:
            rawcredentials.read(options.credentials)        # handling password in the same way as username, priority cli > credentials > cqlshrc
        options.password = option_with_default(rawcredentials.get, 'plain_text_auth', 'password', password_from_cqlshrc)
        options.password = password_from_cqlshrc{code}
These corrections have been made in accordance with https://issues.apache.org/jira/browse/CASSANDRA-16983 and https://issues.apache.org/jira/browse/CASSANDRA-16456.

The documentation does not indicate that AuthProviders can be used in the cqlshrc and credentials files.

I propose to return the ability to use the legacy option of specifying the user and password in the credentials file in the [plain_text_auth] section.

It is also required to describe the rules for using the credentials file in the documentation.

I can make a corresponding pull request.

EDIT by Stefan Miklosovic:

specifying username and password in credentials file works, it is just that [plain_text_auth] section does not work in credentials file anymore. This was working with CASSANDRA-16983 but it stopped to work by CASSANDRA-16456. Both tickets were firstly introduced in 4.1.0 (for the public). I do not think that it was ever an intention to stop to support that when CASSANDRA-16456 was merged and it was most probably just overlooked.
Bug: Nodetool import expects directory structure
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Feb/24 09:48
Resolved: 02/May/24 08:56

According to the [documentation|https://cassandra.apache.org/doc/4.1/cassandra/operating/bulk_loading.html] the nodetool import should not rely on the folder structure of the imported sst files:

{quote}
Because the keyspace and table are specified on the command line for nodetool import, there is not the same requirement as with sstableloader, to have the SSTables in a specific directory path. When importing snapshots or incremental backups with nodetool import, the SSTables don’t need to be copied to another directory.
{quote}

However when importing old cassandra snapshots, we figured out, that sstables still need to be in a directory called like $KEYSPACE/$TABLENAME files, even when keyspace and table name are already present as parameters for the nodetool import call.

Call we used:

{code}
nodetool import --copy-data mykeyspace mytable /full_path_to/test1
{code}

Log:

{code}
INFO  [RMI TCP Connection(21)-127.0.0.1] 2024-02-15 10:41:06,565 SSTableImporter.java:72 - Loading new SSTables for mykeyspace/mytable: Options{srcPaths='[/full_path_to/test1]', resetLevel=true, clearRepaired=true, verifySSTables=true, verifyTokens=true, invalidateCaches=true, extendedVerify=false, copyData= true}
INFO  [RMI TCP Connection(21)-127.0.0.1] 2024-02-15 10:41:06,566 SSTableImporter.java:173 - No new SSTables were found for mykeyspace/mytable
{code}

However, when we move the sstables (.db-Files) to {{alternative/mykeyspace/mytable}}

and import with

{code}
nodetool import --copy-data mykeyspace mytable /fullpath/alternative/mykeyspace/mytable
{code}

the import works

{code}
INFO  [RMI TCP Connection(23)-127.0.0.1] 2024-02-15 10:43:36,093 SSTableImporter.java:177 - Loading new SSTables and building secondary indexes for mykeyspace/mytable: [BigTableReader(path='/mnt/ramdisk/cassandra4/data/mykeyspace/mytable-561a12d0cbe611eead78fbfd293cee40/me-2-big-Data.db'), BigTableReader(path='/mnt/ramdisk/cassandra4/data/mykeyspace/mytable-561a12d0cbe611eead78fbfd293cee40/me-1-big-Data.db')]
INFO  [RMI TCP Connection(23)-127.0.0.1] 2024-02-15 10:43:36,093 SSTableImporter.java:190 - Done loading load new SSTables for mykeyspace/mytable
{code}


We experienced this in Cassandra 4.1.3 on Java 11 (Linux)
Bug: Refactor SAI so the selection of the index type is not scattered to multiple places
Status: Resolved
Priority: Normal
Resolution: Won't Fix
Created: 05/Feb/24 18:34
Resolved: 01/May/24 16:23

For Accord we want to write an internal index and finding plugging into SAI is a bit more channeling than it could be… we need to find multiple places where the SAI code “infer” the index type so it can delegate… this logic should be done once and made pluggable so custom SAI indexes can be defined
Bug: Add foundation for independent parsing of junit based output for CI reporting to cassandra-builds
Status: Triage Needed
Priority: Normal
Created: 22/Mar/24 19:03

PR attached.

For doing CI ourselves, it's useful to have a single pane of glass report where you have a summary of results for all your suites as well as inlined failures. This should be agnostic to any xunit based output; so long as we co-locate the xunit data in directories adjacent to one another, the script in the PR will generate an in-memory representation of the xunit results as well as inline failures to an existing .html file.

The contents will need to be tweaked a bit to generate the top level branch + sha + checkstyle + summaries information, but the vast majority of that is already parsed and easily available within the script and can be extended pretty trivially.

Opening up a pr to pull this into [cassandra-builds](https://github.com/apache/cassandra-builds) since [~mck] is actively working on that and needs these primitives. I'd expect the contents in ci_parser to be massaged to become a more finalized, full solution before we start to use it but no harm in the incremental merge.
Bug: Update access of ClearSnapshotStrategy
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/Feb/24 21:07
Resolved: 26/Feb/24 22:54

Want to update access of ClearSnapshotStrategy added to Cassandra Analytics library to allow setting TTL for snapshots created by bulk reader. Currently the access of the class is package private, for plugging in custom implementation, we need access outside the package.
Bug: Fix type issues and provide tests for type compatibility between 4.1 and 5.0
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Mar/24 16:07
Resolved: 18/Apr/24 11:05

This is a part of CASSANDRA-14476 - we should verify whether the type compatibility matrix is upgradable from 4.0 and 4.1 to 5.0, and if not, fix the remaining issues.

The implemented tests verify the following:
- assumed compatibility between primitive types
- equals method symmetricity
- freezing/unfreezing
- value compatibility by using a serializer of one type to deserialize a value serialized using a serializer of another type
- serialization compatibility by serializing a row with a column of one type as a column of another type for simple and complex cells (multicell types)
- (comparison) compatibility by comparing serialized values of one type using a comparator of another type; for multicell types - build rows and compare cell paths of a complex type using a cell path comparator of another complex type
- verify whether types that are (value/serialization/comparison) compatible in a previous release are still compatible with this release
- store the compatibility matrix in a compressed JSON file so that we can copy it to future releases to assert backward compatibility (similar approach to LegacySSTableTest)
- verify that type serializers are different for non-compatible type pairs which use custom comparisons

Additionally:
- the equals method in {{TupleType}} and {{UserType}} was fixed to be symmetric. Previously, comparing two values gave a different outcome when inverted.
- fixed a condition in comparison method of {{AbstractCompositeType}}
- ported a fix for composite and dynamic composite types which adds a distinct serializers for them so that the serializers for those types and for {{BytesType}} are considered different; similar thing was done for {{LexicalUUIDType}} to make its serializer different to {{UUIDType}} serializer (see https://the-asf.slack.com/archives/CK23JSY2K/p1712060572432959)
- fixed a problem with DCT builder - in 5.0+ the {{DynamicCompositeType}} generation has a problem with inverse alias-type mapping which makes it vulnerable to problems when the same type has two different aliases
Bug: Optionally fail writes when SAI refuses to index a term value exceeding a configured maximum size
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/Mar/24 16:36
Resolved: 17/Apr/24 16:34

SAI currently emits a client warning when we try to index a text value larger than {{{}cassandra.sai.max_string_term_size{}}}. It might be nice to have a hard limit as well, above which we can reject the mutation entirely.
Bug: JMXStandardsTest fails when running with codecoverage
Status: Triage Needed
Priority: Normal
Created: 08/Feb/24 02:04

JMXStandardsTest is added in unit test since 4.1. I recently run this test with codecoverage and it fails:
{code:java}
ant codecoverage -Dtaskname=testsome -Dtest.name=org.apache.cassandra.tools.JMXStandardsTest {code}
{code:java}
[junit-timeout] ------------- ---------------- --------------- [junit-timeout] Testcase: interfaces(org.apache.cassandra.tools.JMXStandardsTest):      FAILED [junit-timeout] Errors detected while validating MBeans [junit-timeout] Error at signature parameter; type java.lang.invoke.MethodHandles.Lookup is not in the supported set of types, method method 'private static boolean[] org.apache.cassandra.service.StorageServiceMBean.$jacocoInit(java.lang.invoke.MethodHandles$Lookup,java.lang.String,java.lang.Class)' [junit-timeout] Error at signature parameter; type java.lang.Class is not in the supported set of types, method method 'private static boolean[] org.apache.cassandra.service.StorageServiceMBean.$jacocoInit(java.lang.invoke.MethodHandles$Lookup,java.lang.String,java.lang.Class)' [junit-timeout] junit.framework.AssertionFailedError: Errors detected while validating MBeans [junit-timeout] Error at signature parameter; type java.lang.invoke.MethodHandles.Lookup is not in the supported set of types, method method 'private static boolean[] org.apache.cassandra.service.StorageServiceMBean.$jacocoInit(java.lang.invoke.MethodHandles$Lookup,java.lang.String,java.lang.Class)' [junit-timeout] Error at signature parameter; type java.lang.Class is not in the supported set of types, method method 'private static boolean[] org.apache.cassandra.service.StorageServiceMBean.$jacocoInit(java.lang.invoke.MethodHandles$Lookup,java.lang.String,java.lang.Class)' [junit-timeout]         at org.apache.cassandra.tools.JMXStandardsTest.interfaces(JMXStandardsTest.java:156) [junit-timeout]         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit-timeout]         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [junit-timeout]         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
{code}
This `$jacocoInit` method was included in the test, which should be excluded
Bug: enrich system_views.pending_hints with hints sizes
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Mar/24 12:57
Resolved: 10/Apr/24 10:42

I am happy to close this if somebody shows me how to get total size of all hints or all hints per particular node via JMX.

I could find StorageMetrics.totalHints but that is how many hints there are, not their sizes. We also have

org.apache.cassandra.metrics:type=HintedHandOffManager name=<MetricName>
org.apache.cassandra.metrics:type=HintsService name=<MetricName>

But that is again showing other metrics not sizes.

I would add two methods into HintsServiceMBean returning this. Seems to be very easy to do once we do CASSANDRA-19477.
Bug: org.apache.cassandra.tcm.ClusterMetadataService#commit does not catch up when rejected
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 10/Jan/24 21:49
Resolved: 19/Mar/24 15:45

This was found in the cep-15-accord branch (CASSANDRA-18804).  The test that found this was a simple benchmark test.

1) deploy a 6 node cluster
2) create a table
3) in parallel launch many accord transactions

When accord gets a transaction it needs to make sure the table is “managed” by accord which uses TCM for this bookkeeping, this is just a List<TableId> in ClusterMetadata.  We found that we detect that the table isn’t managed so we try to add it, we get a reject and the TCM epoch has not moved forward!

Debugging this it looks like org.apache.cassandra.tcm.RemoteProcessor#commit is the root cause as it only seems to try to catch up if there is a messaging error and not a TCM rejection!  Given that the caller to TCM is not able to find the epoch to “wait” on I feel that this is a TCM issue as TCM normally tries to make sure success/rejects are blocking, but in this one case it appears not to be so
Bug: Analytics writer should support writing UDTs
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Jan/24 18:59
Resolved: 02/Apr/24 16:58

{{CQLSSTableWriter}} supports writing UDTs. We should support converting from Map-type fields or nested Spark structs to UDTs in the Analytics Bulk Writer, similar to the way Bulk Reader converts UDTs to maps.
Bug: [Analytics] Fix bulk reads of multiple tables that potentially have the same data file name
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Mar/24 22:41
Resolved: 01/Apr/24 19:12

When reading multiple data frames using bulk reader from different tables, it is possible to encounter a data file name being retrieved from the same Sidecar instance. Because the {{SSTable}}s are cached in the {{SSTableCache}}, it is possible that the {{org.apache.cassandra.spark.reader.SSTableReader}} is using the incorrect {{SSTable}} if it was cached with the same {{#hashCode}}.
Bug: (Accord) Cassandra bootstrap no longer using the range txn and instead uses the sync point empty txn for reads
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 28/Mar/24 21:56
Resolved: 01/Apr/24 17:20

Ephemeral reads made a change to ReadData which caused Bootstrap to no longer use the range txn it generates and instead uses the empty txn from the sync point.  This was not detected in Accord due to ListRead supporting ranges, and failed in Cassandra as we don’t support range reads.
Bug: Test failure: org.apache.cassandra.distributed.test.jmx.JMXFeatureTest
Status: Open
Priority: Normal
Created: 10/Jan/24 22:16

Two of the tests in the class are flaky as seen here:
https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra?branch=pre-18688-JMXFeatureTest

This is from trunk, but I have a suspicion it would be nice to check all branches where the tests were added
Bug: Repair causes out of memory
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Jan/24 13:45
Resolved: 08/Feb/24 11:46

CASSANDRA-14096 introduced {{repair_session_space}} as a limit for the memory usage for Merkle tree calculations during repairs. This limit is applied to the set of Merkle trees built for a received validation request ({{{}VALIDATION_REQ{}}}), divided by the replication factor so as not to overwhelm the repair coordinator, who will have requested RF sets of Merkle trees. That way the repair coordinator should only use {{repair_session_space}} for the RF Merkle trees.

However, a repair session without {{{}-pr-{}}}/{{{}-partitioner-range{}}} will send RF*RF validation requests, because the repair coordinator node has RF-1 replicas and is also the replica of RF-1 nodes. Since all the requests are sent at the same time, at some point the repair coordinator can have up to RF*{{{}repair_session_space{}}} worth of Merkle trees if none of the validation responses is fully processed before the last response arrives.

Even worse, if the cluster uses virtual nodes, many nodes can be replicas of the repair coordinator, and some nodes can be replicas of multiple token ranges. It would mean that the repair coordinator can send more than RF or RF*RF simultaneous validation requests.

For example, in an 11-node cluster with RF=3 and 256 tokens, we have seen a repair session involving 44 groups of ranges to be repaired. This produces 44*3=132 validation requests contacting all the nodes in the cluster. When the responses for all these requests start to arrive to the coordinator, each containing up to {{repair_session_space}}/3 of Merkle trees, they accumulate quicker than they are consumed, greatly exceeding {{repair_session_space}} and OOMing the node.