av9ash/GitBugs huggingface dataset:
Bug: Add support for vectors on UDFs
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/Jun/23 12:54
Resolved: 12/Jul/23 10:55

CASSANDRA-18504 will add a new vector type, but [it won't be supported on UDFs|https://github.com/apache/cassandra/blob/5027e688da006e5d5bf9bfdf4719caddbf85986a/test/unit/org/apache/cassandra/cql3/validation/operations/CQLVectorTest.java#L248-L271]. The goal of this ticket is to add that support.

This will require adding a new {{o.a.c.cql3.functions.types.TypeCodec}} for vectors. Those codecs are mostly duplicates of the codecs on the Java driver. They are used for UDFs instead of the regular {{AbstractType}} to prevent pulling too many internal dependencies. The driver's vector codec has recently been added by [JAVA-3060|https://datastax-oss.atlassian.net/browse/JAVA-3060].
Bug: Investigate the sizing of BlockPackedReader and MonotonicBlockPackedReader
Status: Triage Needed
Priority: Normal
Created: 02/May/23 17:39

BlockPackedWriter uses a block size of 128 bytes and MonotonicBlockPackedWriter uses a block size of 16384.

There is no documentation about the reasons for choosing these particular sizes.

We should add a micro benchmark to test these classes with different block sizes to show impact of changing them on read and write performance.
Bug: Use trie for literal index building
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 14/Jun/23 15:57
Resolved: 20/Dec/23 15:53

The BlockBalancedTreeRamBuffer for numerics indexes uses a trie for building indexes in memory before flushing to disk. The RAMStringIndexer for literal indexes does not do this. It uses lucene classes to build the in memory index before flushing.

We should combine both of the in-memory index builders and have them both use the trie.
Bug: Add option to sstablepartitions for aggregating stats
Status: Triage Needed
Priority: Normal
Created: 04/May/23 14:56

The {{sstablepartitions}} offline tool added by CASSANDRA-8720 processes sstables one by one. If a partition is spread across multiple sstables, the tool will separately report the metrics of the partition fragment on each sstable. For example, if we are looking for partitions larger than 100MiB, and a partition has a 50MiB fragment in one table and 50MiB in another table, the partition won't be found.

It would be useful to add a kind of {{--aggregate}} option to {{sstablepartitions}} so it aggregates the results of all the analyzed sstables that belong to the same table.
Bug: CEP-30 ANN Vector Search with SAI
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 31/May/23 02:56
Resolved: 21/Dec/23 18:04

[CEP-30|https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-30%3A+Approximate+Nearest+Neighbor%28ANN%29+Vector+Search+via+Storage-Attached+Indexes]
 # Implement approximate nearest neighbor (ANN) vector search capability in Apache Cassandra using storage-attached indexes (SAI).
 # Support a vector of float32 embeddings as a new CQL type.
 # Add ANN search to work with normal Cassandra data flow (insertion, updating, and deleting rows). The implementation should support adding a new vector in log(N) time, and ANN queries in M log(N) time where N is the number of vectors and M is the number of sstables.
 # Compose with other SAI predicates.
 # Scatter/gather across replicas, combining topK from each to get global topK.
Bug: Update BlockBalancedTreeWriter to take advantage of Lucene improvements
Status: Triage Needed
Priority: Normal
Created: 14/Jun/23 15:51

There have been improvements to the Lucene BKDWriter implementation for the one-dimensional case that could be used to improve the BlockBalancedTreeWriter. Specifically [LUCENE-9358|https://issues.apache.org/jira/browse/LUCENE-9358] removes the need for the rotateToTree method by writing the balanced tree in the correct format while it is being built.
Bug: Cassandra Catalyst Program
Status: Resolved
Priority: Normal
Resolution: Done
Created: 08/Jun/23 17:17
Resolved: 29/Nov/23 21:26

The 5.0 has a lot of exciting new features the world should know about. We would like to revive the Cassandra MVP program as a community-run program to let people know about Cassandra 5.0 and how to use/contribute to Cassandra in general.
Bug: Introduce IndexTermType to replace TypeUtils in SAI
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 13/Jun/23 10:25
Resolved: 18/Nov/23 01:03

From a review comment by [~adelapena] on the [CASSANDRA-18067|https://issues.apache.org/jira/browse/CASSANDRA-18067] PR
{quote}Not directly related to this ticket, but it seems to me that the methods on TypeUtil do a lot of repeated work in the hot path just to determine the data type of the caller. These operations include finding the base type of reversed multiple times, multiple instanceof calls, etc. All these operations on each index are called with the same IndexContext#getValidator argument, needlessly repeating work for every column value.

I understand that most of the methods on TypeUtil are things that would normally be part of AbstractType, so each particular data type can provide its own implementation. But we don't want to couple the generic data types with SAI, so we have this class instead.

I think that we could add a kind of TermType class decorating AbstractType, and subclass it for every data type that gets special treatment (LONG, VARINT, DECIMAL and INET). Then, IndexContext would hold the adequate instance of TermType. This instance would be able to provide the same operations as the current TypeUtil, but without the type checks.{quote}
Bug: Expose decommission state to nodetool info
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/May/23 04:48
Resolved: 20/Jun/23 20:40

Currently, when a node is being decommissioned and if any failure happens, then an exception is thrown back to the caller.

But Cassandra's decommission takes considerable time ranging from minutes to hours to days. There are various scenarios in that the caller may need to probe the status again:
 * The caller times out
 * It is not possible to keep the caller hanging for such a long time

And If the caller does not know what happened internally, then it cannot retry, etc., leading to other issues.

So, in this ticket, I am going to add a new nodetool/JMX command that can be invoked by the caller anytime, and it will return the correct status.

It might look like a smaller change, but when we need to operate Cassandra at scale in a large-scale fleet, then this becomes a bottleneck and require constant operator intervention.
Bug: Inconsistent permissions on new empty node in cluster
Status: Triage Needed
Priority: Normal
Created: 25/May/23 08:59

h2. To reproduce
h3. Initial Setup
 # Setup a 3-rack and 6-node cluster on Kubernetes, with disk persistency. Use 2 nodes per rack.
 # Set the Replication Factor of the system_auth keyspace to 3.
 # Set the authorizer to `org.apache.cassandra.auth.CassandraAuthorizer`.
 # Create a `mytest` keyspace with RF=3, and a `myuser` user with read/write permissions on the `mytest` keyspace.
 # Connect to the DB using the `myuser` login/password. Auth works normally.

*Perturbation*
 # Remove the PVC associated to one node (let's call it a-0, for "rack a, node 0").
 # Delete the pod of node a-0.
 # Let the pod restart, as part of the StatefulSet, with a new PVC / disk. A new node is created in Cassandra, so that zone A is internally considered to have 3 nodes by Cassandra, 2 of which are really running.
 # When the pod is ready, connect to it using the `myuser` login/password.
 # Run a `CREATE TABLE` in the `mytest` keyspace.

{*}--> ISSUE{*}: the statement fails with Unauthorized, it should not.
h2. Investigation and resolution
 # Connect to the (new) a-0 pod using the `cassandra` superuser, with cqlsh.
 # Execute `LIST ALL PERMISSIONS;`. You can see the `mytest` user has no permissions on the `mytest` keyspace.
 # Execute `SELECT ALL FROM system_auth.role_permissions;`. The table auto-repairs and now you see the `mytest`-related permissions.
 # Connect to the node using the `myuser` login/password. It now works.
Bug: Move format-specific sstable options into the format configuration
Status: In Progress
Priority: Normal
Created: 18/May/23 09:27

This mainly concerns cassandra yaml settings:
- {{column_index_size}}, which should also be renamed to {{row_index_granularity}}
- {{column_index_cache_size}}
- {{index_summary_capacity}}
- {{index_summary_resize_interval}}

and possibly
- {{key_cache_size}}, {{key_cache_keys_to_save}}, {{key_cache_save_period}}, {{key_cache_migrate_during_compaction}}
- {{sstable_preemptive_open_interval}}

Existing settings should be deprecated but still picked up if defined.

At this point we will not consider table-level options that make better sense as format parameters ({{min/max_index_interval}}, {{bloom_filter_fp_chance}}, {{crc_check_chance}} and possibly {{compression}}), because we do not yet support per-table format selection/configuration.
Bug: Make sstable format configurable per table
Status: Patch Available
Priority: Normal
Created: 18/May/23 09:41

Some SSTable format settings need to be configurable per table for better efficiency. This includes:
 - {{row_index_granularity}}
 - {{bloom_filter_fp_chance}}
 - {{crc_check_chance}}
 - {{min/max_index_interval}}

Some of these are currently configurable using direct properties of tables. Having them as format properties makes better sense and should also support specifying useable combinations of settings, e.g.
{code:java}
CREATE TABLE ... WITH sstable_format = "bti-fast";
CREATE TABLE ... WITH sstable_format = "bti-small";
{code}
where {{bti-fast}} and {{bti-small}} can be defined in {{cassandra.yaml}} e.g. as
{code:java}
sstable.format.options:
  - bti-fast:
      row_index_granularity: 1kiB
      bloom_filter_fp_chance: 0.01
  - bti-small:
      row_index_granularity: 32kiB
      bloom_filter_fp_chance: 0.1
{code}
Bug: Refactor cqlsh On/Off switch implementation and make the output consistent
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/May/23 03:02
Resolved: 27/Sep/23 10:44

This change refactors the On/Off switch implemented in the class SwitchCommand and subclass  SwitchCommandWithValue of cqlshmain.py to use an Enum with static methods instead of custom classes.

The body of on_off_switch + enum definition requires just 15 lines of code vs 33 in SwitchCommand.

The existing code is hard to read, including the usage in the code, which instantiates a SwitchCommand object in-order to invoke the execute method:

 
{code:java}
self.tracing_enabled = SwitchCommand("TRACING", "Tracing").execute(self.tracing_enabled, parsed, self.printerr){code}
this can be replaced by a more familiar direct function call:
{code:java}
self.tracing_enabled = self.on_off_toggle("TRACING", "Tracing", self.tracing_enabled, parsed.get_binding('switch')){code}
 

The refactoring also updates the command output for consistency. Instead of the current:
{code:java}
> tracing on
Now Tracing is enabled
> paging on
Query paging is already enabled. Use PAGING OFF to disable.
> expand on
Now Expanded output is enabled
{code}
replace with more succinct and consistent, using 'ON/OFF' instead of enabled/disabled and removing the redundant 'Now':
{code:java}
> tracing on
TRACING set to ON
> paging on
PAGING is already ON
> expand on
EXPAND set to ON
{code}
Bug: CEP-15: (C*) Improve the chaos generation for Burn Tests: slow/flakey connections and dropped messages
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 14/Apr/23 19:51
Resolved: 13/Sep/23 22:53

Burn test should be enhanced to add the following:

1) message dropping from one node to another (this is different than the current partition set logic in accord.impl.basic.Cluster#partitionSet)
2) for messages with callbacks, trigger failure case
3) redundant message delivery

Related work:
* Simulator’s org.apache.cassandra.simulator.systems.SimulatedAction#applyToMessage
- Figures out what delivery action to perform via org.apache.cassandra.simulator.FutureActionScheduler#shouldDeliver
-- timeout if dropPartition[from] != dropPartition[to] // either to/from is in drop partition, but not both
-- config asked to override and deliver
-- 50/50 chance to deliver, after that 50/50 to deliver w/ timeout, after that cause a failure
- in C* failure is an enum with Timeout and Unknown
- knows the schedule time and the message expire time, and can promote a DELIVER event to DELIVER_AND_TIMEOUT
- triggers the timeout
Bug: Baseline Metrics for Accord Transactions
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 09/Jun/23 19:22
Resolved: 10/Oct/23 10:12

Based on some conversations w/ [~benedict] and [~dcapwell], this is the initial set of metrics that seem both feasible to implement and useful as we monitor the health of a cluster performing Accord transactions:

1.) Basic latency metrics for transactions up to the point of COMMIT and rate metrics for preemption, failure, and timeouts at the coordinator.

This has already been implemented and split into read and write-specific metrics. Our position for now is that metrics around preemption should be useful in place of a more difficult-to-define metric around how many transactions are completed via recovery.

2.) Global cache stats/metrics (i.e. aggregated for all command stores)

We could, at some point, build metrics scoped to a specific {{CommandStore}}, but they might be awkward in MBean/JMX space, as command stores would have to be identified by ID or key range…the latter possibly being able to change across epochs. (An alternative would be just publishing command store-specific stats on-demand to a virtual table instead.)

3.) Something like a decaying histogram of the number of dependencies per transaction (or per partial transaction).

If this is getting worse over time, it could be useful to know/be a way for us to detect that contention is increasing. We should be able to hook this up to {{ProgressLog}} notifications. Recording for PartialDeps/PartialTxn (which ProgressLog gives us at pre-accept) seems acceptable, given this is a directional metric.
Bug: Test failure: test_putget – dtest.putget_test.TestPutGet
Status: Resolved
Priority: Normal
Resolution: Cannot Reproduce
Created: 18/Apr/23 01:27
Resolved: 19/Sep/23 13:58

Seen on current trunk, here:

[https://ci-cassandra.apache.org/blue/organizations/jenkins/Cassandra-devbranch/detail/Cassandra-devbranch/2416/tests]
{code:java}
Error failed on teardown with "Unexpected error found in node logs (see stdout for full details). Errors: [[node3] 'ERROR [PendingRangeCalculator:1] 2023-04-08 00:54:06,468 JVMStabilityInspector.java:68 - Exception in thread Thread[PendingRangeCalculator:1,5,PendingRangeCalculator]\njava.lang.AssertionError: Unknown keyspace system_traces\n\tat org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:325)\n\tat org.apache.cassandra.db.Keyspace.lambda$open$0(Keyspace.java:163)\n\tat org.apache.cassandra.utils.concurrent.LoadingMap.blockingLoadIfAbsent(LoadingMap.java:105)\n\tat org.apache.cassandra.schema.Schema.maybeAddKeyspaceInstance(Schema.java:251)\n\tat org.apache.cassandra.db.Keyspace.open(Keyspace.java:163)\n\tat org.apache.cassandra.db.Keyspace.open(Keyspace.java:152)\n\tat org.apache.cassandra.service.PendingRangeCalculatorService.lambda$new$1(PendingRangeCalculatorService.java:58)\n\tat org.apache.cassandra.concurrent.SingleThreadExecutorPlus$AtLeastOnce.run(SingleThreadExecutorPlus.java:60)\n\tat org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.lang.Thread.run(Thread.java:750)']" Stacktrace Unexpected error found in node logs (see stdout for full details). Errors: [[node3] 'ERROR [PendingRangeCalculator:1] 2023-04-08 00:54:06,468 JVMStabilityInspector.java:68 - Exception in thread Thread[PendingRangeCalculator:1,5,PendingRangeCalculator]\njava.lang.AssertionError: Unknown keyspace system_traces\n\tat org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:325)\n\tat org.apache.cassandra.db.Keyspace.lambda$open$0(Keyspace.java:163)\n\tat org.apache.cassandra.utils.concurrent.LoadingMap.blockingLoadIfAbsent(LoadingMap.java:105)\n\tat org.apache.cassandra.schema.Schema.maybeAddKeyspaceInstance(Schema.java:251)\n\tat org.apache.cassandra.db.Keyspace.open(Keyspace.java:163)\n\tat org.apache.cassandra.db.Keyspace.open(Keyspace.java:152)\n\tat org.apache.cassandra.service.PendingRangeCalculatorService.lambda$new$1(PendingRangeCalculatorService.java:58)\n\tat org.apache.cassandra.concurrent.SingleThreadExecutorPlus$AtLeastOnce.run(SingleThreadExecutorPlus.java:60)\n\tat org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.lang.Thread.run(Thread.java:750)']{code}
Bug: Unify CQLTester#waitForIndex and SAITester#waitForIndexQueryable
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 11/May/23 09:56
Resolved: 22/May/23 16:33

It has been [mentioned|https://github.com/maedhroz/cassandra/pull/11#discussion_r1190166765] in CASSANDRA-18217 that {{CQLTester#waitForIndex}} and {{SAITester#waitForIndexQueryable}} can be unified into a single method. 

The same discussion mentions that it's easy to forget to call those methods after calling {{CQLTester#createIndex}}. So that method can be renamed to {{createIndexAsync}}, and we can create a new method that creates the index and waits for it ({{createIndexSync}}, {{createIndexAndWaitForQueryable}}, or something like that).
Bug: Test failure: org.apache.cassandra.index.sasi.SASICQLTest#testPagingWithClustering
Status: Open
Priority: Normal
Created: 06/Apr/23 16:36

{noformat}
junit.framework.AssertionFailedError: expected:<20> but was:<10>
	at org.apache.cassandra.index.sasi.SASICQLTest.testPagingWithClustering(SASICQLTest.java:83)
{noformat}

https://app.circleci.com/pipelines/github/driftx/cassandra/965/workflows/ca4f7fcf-0022-43e0-804d-8a8b9fbec8ca/jobs/19802
Bug: Test failure: org.apache.cassandra.net.ConnectionTest.testTimeout
Status: Open
Priority: Normal
Created: 30/Jun/23 00:17

[https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2404/workflows/df5b69e6-78c0-41fc-9f37-3191a2578f6f/jobs/28590/tests]
{code:java}
org.awaitility.core.ConditionTimeoutException: Assertion condition defined as a org.apache.cassandra.Util sent count values don't match Expected: <1L> but: was <3L> within 5 seconds. at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:165) at org.awaitility.core.AssertionCondition.await(AssertionCondition.java:119) at org.awaitility.core.AssertionCondition.await(AssertionCondition.java:31) at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:895) at org.awaitility.core.ConditionFactory.untilAsserted(ConditionFactory.java:679) at org.apache.cassandra.Util.spinAssertEquals(Util.java:691) at org.apache.cassandra.net.ConnectionUtils$OutboundCountChecker.lambda$check$0(ConnectionUtils.java:99) at org.apache.cassandra.net.ConnectionUtils$OutboundCountChecker.doCheck(ConnectionUtils.java:120) at org.apache.cassandra.net.ConnectionUtils$OutboundCountChecker.check(ConnectionUtils.java:99) at org.apache.cassandra.net.ConnectionTest.lambda$testTimeout$28(ConnectionTest.java:538) at org.apache.cassandra.net.ConnectionTest.lambda$doTest$8(ConnectionTest.java:246) at org.apache.cassandra.net.ConnectionTest.doTestManual(ConnectionTest.java:266) at org.apache.cassandra.net.ConnectionTest.doTest(ConnectionTest.java:244) at org.apache.cassandra.net.ConnectionTest.test(ConnectionTest.java:233) at org.apache.cassandra.net.ConnectionTest.testTimeout(ConnectionTest.java:507) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) Caused by: java.lang.AssertionError: sent count values don't match Expected: <1L> but: was <3L> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) at org.apache.cassandra.Util.lambda$spinAssertEquals$0(Util.java:691) at org.awaitility.core.AssertionCondition.lambda$new$0(AssertionCondition.java:53) at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:222) at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:209) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833){code}
Bug: mTLS based client and internode authenticators
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/May/23 16:34
Resolved: 28/Jul/23 20:59

Cassandra currently doesn't have any certificate based authenticator for both client connections and internode connections. If one wants to use certificate based authentication protocol like TLS, in which clients send their certificates for the TLS handshake, we can leverage the information from the client certificate to identify a client. Using this authentication mechanism one can avoid the pain of password generations, sharing and rotation.

Introducing following certificate based mTLS authenticators for internode and client connections
MutualTlsAuthenticator (client authentication)
MutualTlsInternodeAuthenticator (internode authentication)
MutualTlsWithPasswordFallbackAuthenticator (for optional mode operation for client authentication)

An implementation of MutualTlsCertificateValidator called SpiffeCertificateValidator whose identity is SPIFFE that is embedded in SAN of the client certificate. One can implement their own CertificateValidator to match their needs and configure it in Cassandra.yaml
Bug: Make cassandra-stress able to read all credentials from a file
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 23/May/23 13:05
Resolved: 26/May/23 16:53

If there is username/password required in order to connect via JMX, there is currently no way how to do this as NodeProbe constructor in cassandra-stress does not accept any credentials. 

The way how we are reading the credentials should be refactored. Currently, if there are credentials on the command line, they will be visible in the logs. Making it visible on the command line for JMX credentials is not ideal either.

What I would like to see is to read all credentials which are necessary for cassandra-stress from ONE file. CQL and JMX combined. 

Because there is already some logic in place and it would be cool to have this backward compatible, we may still support command line credentials for CQL but we would make it deprecated and we would remove it in 6.0 so cassandra-stress will be reading credentials from the file only.

I looked into the implementation and I have an idea how to "inject" credentials where necessary so they would be used even we do not use them on the command line but I have not coded up anything yet.