av9ash/GitBugs huggingface dataset:
Bug: Restore optimization in MultiCBuilder around building one clustering
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 06/Dec/24 20:44
Resolved: 11/Dec/24 03:08

Back in CASSANDRA-10409, we optimized the single clustering case for {{MultiCBuilder}} to produce less garbage. In CASSANDRA-19341, we de-facto removed that optimization, and I'm seeing increased allocations in trunk, specifically at least double the number of {{BTree}} builders. In the write-heavy workload I'm profiling, this jumps up to become the most common TLAB allocation. I'm attaching JFR screenshots from 4.1 and trunk that show the codepaths that touch this on both versions. We should consider reintroducing something like the old {{OneClusteringBuilder}}.

I used {{WriteBench}} to explore things here, mostly w/ the TRUNCATE {{EndOp}}, and made sure commitlog writes, view updates, and {{CassandraTableWriteHandler}} tracing were disabled to reduce the noise.
 
[~blerer]
Bug: Add flag to avoid invalidating key cache on file deletion
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 08/Nov/24 12:50
Resolved: 11/Dec/24 13:45

CASSANDRA-17056 changed the way we invalidate key cache when deleting sstables after compaction. It now iterates over all the keys in the keycache and deletes the key if it is for the current sstable.

It can potentially take several seconds per deleted sstable to do this.

Patch attached which disables this by default, but can be turned on if anyone relies on this.
Bug: Unable to start apache cassandra 5.0.2 on jdk17 
Status: Triage Needed
Priority: Normal
Created: 11/Dec/24 10:23

Error opening zip file or JAR manifest missing : ./../lib/jamm-0.3.2.jar
JVMJ9TI064E Agent initialization function Agent_OnLoad failed for library instrument, return code -1
JVMJ9VM015W Initialization error for library j9jvmti29(-3): JVMJ9VM009E J9VMDllMain failed

 

java -version
openjdk version "17.0.10" 2024-01-16
IBM Semeru Runtime Open Edition 17.0.10.0 (build 17.0.10+7)
Eclipse OpenJ9 VM 17.0.10.0 (build openj9-0.43.0, JRE 17 Linux amd64-64-Bit Compressed References 20240116_670 (JIT enabled, AOT enabled)
OpenJ9   - 2c3d78b48
OMR      - ea8124dbc
JCL      - 2aad089841f based on jdk-17.0.10+7)
Bug: Avoid prepared statement invalidation race when committing schema changes
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Nov/24 07:41
Resolved: 11/Dec/24 09:13

We currently invalidate the prepared statement cache before making the schema change “live", creating a small window where we might re-prepare a statement with the old schema. This causes us to use the wrong epoch when coordinating a request, making replicas think the coordinator is behind and failing the request until the node is bounced (or prepared statement cache invalidated)
Bug: TCM's Retry.Deadline#retryIndefinitely is dangerous if used with RemoteProcessor as the deadline does not impact message retries
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 06/Nov/24 19:05
Resolved: 10/Dec/24 23:28

{code}
public static Deadline retryIndefinitely(long timeoutNanos, Meter retryMeter)
{
    return new Deadline(Clock.Global.nanoTime() + timeoutNanos,
                        new Retry.Jitter(Integer.MAX_VALUE, DEFAULT_BACKOFF_MS, new Random(), retryMeter))
    {
        @Override
        public boolean reachedMax()
        {
            return false;
        }

        @Override
        public long remainingNanos()
        {
            return timeoutNanos;
        }

        public String toString()
        {
            return String.format("RetryIndefinitely{tries=%d}", currentTries());
        }
    };
}
{code}

Sample usage pattern (example is in Accord, but same pattern exists in RemoteProcessor.commit)

{code}
Promise<LogState> request = new AsyncPromise<>();
List<InetAddressAndPort> candidates = new ArrayList<>(log.metadata().fullCMSMembers());
sendWithCallbackAsync(request,
                      Verb.TCM_RECONSTRUCT_EPOCH_REQ,
                      new ReconstructLogState(lowEpoch, highEpoch, includeSnapshot),
                      new CandidateIterator(candidates),
                      retryPolicy);
return request.get(retryPolicy.remainingNanos(), TimeUnit.NANOSECONDS);
{code}

The issue here is that the networking retry has no clue that we gave up waiting on the request, so we will keep retrying until success!  The reason for this is “reachedMax” is used to see if its safe to run again, but it isn’t as the deadline has passed!
Bug: Decouple reading and writing when compacting
Status: Triage Needed
Priority: Normal
Created: 15/Nov/24 18:00

I'm examining a 5.0 cluster, mostly default settings except these overrides:
{noformat}
cluster_name: "density"
num_tokens: 4
concurrent_reads: 64
concurrent_writes: 64
trickle_fsync: true
endpoint_snitch: "Ec2Snitch"
compaction_throughput: 0MiB/s
key_cache_migrate_during_compaction: false
{noformat}
Here's a flame graph of compaction taken with these settings:
{noformat}
asprof -e wall -X '*Unsafe.park*' -X '*Native.epollWait' -I '*compaction*' 
{noformat}
[^image-2024-11-15-09-50-41-854.png]

and in reverse:

[^image-2024-11-15-09-45-51-450.png]

It's clear we can get an easy win by moving the writing portion of compaction to a dedicated thread (ExecutorService.newSingleThreadExecutor() ?), instead of blocking in the same one. For this use case, it would improve compaction throughput by about 2x.
Bug: Duplicit values for uuid column in cassandra-stress
Status: Triage Needed
Priority: Normal
Created: 09/Dec/24 14:31

In case of command
{code:java}
cassandra-stress user profile=abc.yaml ...{code}
and yaml file:
{code:java}
table_definition: | 
  CREATE TABLE prftest1.process_data (
  id uuid,
  id2 uuid,
  data blob,
  PRIMARY KEY (id, id2)

columnspec: 
  - name: id 
    population: seq(1..10) 
  - name: id2 
    population: seq(1..10)
  - name: data 
    size: uniform(5..10) 
    population: seq(1..100){code}
the values for +id2+ (with type {+}uuid{+}) are all the same (not random). 

 

NOTE: in case of type +bigint+ for +id+ and {+}id2{+}, the values for +id2+ are random items. 

It seems as possible issue in case of uuid column type see attached picture  !image-2024-12-09-15-30-45-321.png!
Bug: Investigate removal of custom ScannerList construction
Status: Triage Needed
Priority: Normal
Created: 09/Dec/24 09:06

We have a pretty complex mechanism in place to make it possible for `LeveledCompactionStrategy` to create custom scanner lists that concatenate scanners of non-overlapping sstables.

This mechanism was quite beneficial when the merge of multiple iterators involved popping and pushing back into a priority queue, where simply adding an additional iterator (even an initially exhausted one) caused an increase in the complexity of all iterator advances. Since CASSANDRA-8915 this is not the case, and the maximum cost that can be paid for any number of additional non-overlapping iterators is one key comparison per partition.

We should measure the effect of this optimization and remove the associated complexity if it is not beneficial.
Bug: Fix RequestFailureReason.READ_TOO_MANY_INDEXES in trunk 
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/Dec/24 17:21
Resolved: 05/Dec/24 20:39

RequestFailureReason.READ_TOO_MANY_INDEXES - the code for it is 7 in [cassandra-5.0|https://github.com/apache/cassandra/blob/cassandra-5.0/src/java/org/apache/cassandra/exceptions/RequestFailureReason.java#L31-L40]. That has changed probably by mistake or because of merge conflict in [trunk|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/exceptions/RequestFailureReason.java#L32-L44]. Now the code is 10. This is breaking change. The fix will be just assign code 7 to RequestFailureReason.READ_TOO_MANY_INDEXES and move down all the new reasons added to their current trunk code+1 in trunk. This change is OK because we haven't had a release from trunk yet.
Bug: Accord simulation test failing with "ClassNotFoundException: WARN  [AccordExecutor[1,0]"
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/Dec/24 19:03
Resolved: 04/Dec/24 23:47

{code}
org.apache.cassandra.simulator.SimulationException: Failed on seed 0xe4d9629ed622b9aa
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: WARN  [AccordExecutor[1,0]
	at org.apache.cassandra.simulator.paxos.AbstractPairOfSequencesPaxosSimulation$1.performSimple(AbstractPairOfSequencesPaxosSimulation.java:161)
	at org.apache.cassandra.simulator.Action.performAndRegister(Action.java:468)
	at org.apache.cassandra.simulator.Action.perform(Action.java:486)
	at org.apache.cassandra.simulator.ActionSchedule.next(ActionSchedule.java:384)
	at org.apache.cassandra.simulator.paxos.PaxosSimulation$2.next(PaxosSimulation.java:257)
	at org.apache.cassandra.simulator.paxos.PaxosSimulation.run(PaxosSimulation.java:229)
	at org.apache.cassandra.simulator.paxos.AbstractPairOfSequencesPaxosSimulation.run(AbstractPairOfSequencesPaxosSimulation.java:293)
	at org.apache.cassandra.simulator.paxos.PairOfSequencesAccordSimulation.run(PairOfSequencesAccordSimulation.java:64)
	at org.apache.cassandra.simulator.SimulationRunner$Run.run(SimulationRunner.java:397)
	at org.apache.cassandra.simulator.paxos.AccordSimulationRunner$Run.run(AccordSimulationRunner.java:50)
	at org.apache.cassandra.simulator.paxos.AccordSimulationRunner$Run.run(AccordSimulationRunner.java:40)
	at org.apache.cassandra.simulator.SimulationRunner$BasicCommand.run(SimulationRunner.java:377)
	at org.apache.cassandra.simulator.paxos.AccordSimulationRunner$Run.run(AccordSimulationRunner.java:40)
	at org.apache.cassandra.simulator.paxos.AccordSimulationRunner.main(AccordSimulationRunner.java:103)
	at org.apache.cassandra.simulator.test.ShortAccordSimulationTest.simulationTest(ShortAccordSimulationTest.java:32)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Caused by: java.lang.ClassNotFoundException: WARN  [AccordExecutor[1,0]
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:315)
	at org.apache.cassandra.simulator.paxos.AbstractPairOfSequencesPaxosSimulation$1.performSimple(AbstractPairOfSequencesPaxosSimulation.java:157)
 
{code}
Bug: Accord: Share DurableBefore between CommandStores
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Oct/24 15:19
Resolved: 04/Dec/24 09:33

DurableBefore is a global collection, representing the minimum bounds known for the whole cluster. Currently it is stored per-CommandStore, which is effectively multiple times per table. This is very inefficient, and for large schemas potentially destabilising.
Bug: Accord: Use ExclusiveSyncPoints to join a new topology
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 05/Nov/24 10:40
Resolved: 04/Dec/24 09:33

For correctness, the dependencies we adopt on joining a new topology must exclude the possibility of respondents accepting additional transactions with a lower TxnId, so proxying on the existing `ExclusiveSyncPoint` mechanisms is logical for the time-being. This patch removes the `FetchMajorityDeps` logic in favour of simply waiting for a suitable `ExclusiveSyncPoint` to be proposed.
Bug: Accord Test Fixes: SystemKeyspaceStorageTest, SnapshotTest, "Home key must be contained in the route" exception
Status: Triage Needed
Priority: Normal
Created: 25/Oct/24 19:59

Full trace: 
{code}
java.lang.IllegalArgumentException: Home key must be contained in the route: 1b255f4d-ef25-40a6-0000-000000000009:9223372036854775806 not in [1b255f4d-ef25-40a6-0000-000000000009:[(9223372036854775806,+Inf]]] at accord.primitives.Ranges.toRoute(Ranges.java:200) at accord.primitives.Ranges.toRoute(Ranges.java:37) at accord.local.Node.computeRoute(Node.java:706) at accord.local.Node.computeRoute(Node.java:699) at accord.impl.MajorityDepsFetcher$ShardScheduler.lambda$start$5(MajorityDepsFetcher.java:172) at accord.local.Node.withEpoch(Node.java:380) at accord.impl.MajorityDepsFetcher$ShardScheduler.start(MajorityDepsFetcher.java:161) at accord.impl.MajorityDepsFetcher$ShardScheduler.lambda$scheduleAt$0(MajorityDepsFetcher.java:146) at org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at 
{code}
Bug: Accord:  "modification statements may only mutate a single partition" on a no-op query
Status: Triage Needed
Priority: Normal
Created: 14/Nov/24 19:03

{code}
java.lang.IllegalArgumentException: When running withing a transaction, modification statements may only mutate a single partition

	at org.apache.cassandra.cql3.statements.ModificationStatement.getTxnUpdate(ModificationStatement.java:845)
	at org.apache.cassandra.cql3.statements.ModificationStatement.getTxnWriteFragment(ModificationStatement.java:894)
	at org.apache.cassandra.cql3.statements.TransactionStatement.createWriteFragments(TransactionStatement.java:299)
	at org.apache.cassandra.cql3.statements.TransactionStatement.createUpdate(TransactionStatement.java:318)
	at org.apache.cassandra.cql3.statements.TransactionStatement.createTxn(TransactionStatement.java:352)
	at org.apache.cassandra.cql3.statements.TransactionStatement.execute(TransactionStatement.java:389)
	at org.apache.cassandra.cql3.statements.TransactionStatement.executeLocally(TransactionStatement.java:457)
	at org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:483)
	at org.apache.cassandra.cql3.CQLTester.executeFormattedQuery(CQLTester.java:1721)
	at org.apache.cassandra.cql3.CQLTester.execute(CQLTester.java:1700)
{code}

{code}
@Test
public void repro()
{
    createTable("CREATE TABLE " + KEYSPACE + ".basic_sai0 (pk0 varint,pk1 tinyint,ck0 smallint,ck1 smallint,ck2 tinyint,ck3 timestamp,ck4 ascii,static0 timestamp static,static1 uuid static,regular0 text,regular1 text,regular2 ascii,regular3 smallint,regular4 inet,regular5 decimal,regular6 float,regular7 double,regular8 int, PRIMARY KEY ((pk0,pk1), ck0, ck1, ck2, ck3, ck4)) WITH transactional_mode = 'full' AND CLUSTERING ORDER BY (ck0 ASC,ck1 DESC,ck2 DESC,ck3 ASC,ck4 DESC);");

    execute("BEGIN TRANSACTION \n" +
            "DELETE FROM " + KEYSPACE + ".basic_sai0 WHERE pk0 = ? AND pk1 = ? AND ck0 = ? AND ck1 >= ? AND ck1 <= ?;" +
            "COMMIT TRANSACTION;",
            -9223288646978895870L,(byte)94,(short)16769,(short)29708,(short)-22673);
}
{code}
Bug: sstableloader fails for tables containing vector columns but return code is 0
Status: Open
Priority: Normal
Created: 20/Nov/24 15:41

For Apache Cassandra 5.0, sstableloader fails to load data into a table which has vector columns but still returns 0.
{code:java}
ERROR 07:34:38,796 Error parsing schema for table cycling.comments_vs: Cluster.getMetadata().getKeyspace("cycling").getTable("comments_vs") will be missing or incomplete
java.lang.IllegalArgumentException: Could not parse type name vector<float, 5>
	at com.datastax.driver.core.DataTypeCqlNameParser.parse(DataTypeCqlNameParser.java:233)
	at com.datastax.driver.core.TableMetadata.build(TableMetadata.java:311)
	at com.datastax.driver.core.SchemaParser.buildTables(SchemaParser.java:302)
	at com.datastax.driver.core.SchemaParser.buildKeyspaces(SchemaParser.java:203)
	at com.datastax.driver.core.SchemaParser.refresh(SchemaParser.java:109)
	at com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:417)
	at com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:324)
	at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:235)
	at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:109)
	at com.datastax.driver.core.Cluster$Manager.negotiateProtocolVersionAndConnect(Cluster.java:1813)
	at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1726)
	at com.datastax.driver.core.Cluster.init(Cluster.java:214)
	at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:387)
	at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:366)
	at com.datastax.driver.core.Cluster.connect(Cluster.java:311)
	at org.apache.cassandra.utils.NativeSSTableLoaderClient.init(NativeSSTableLoaderClient.java:70)
	at org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:197)
	at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:92)
	at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:58){code}
Summary shows 0 bytes transferred.
{code:java}
Summary statistics: 
   Connections per host    : 1         
   Total files transferred : 0         
   Total bytes transferred : 0B        
   Total duration          : 3278 ms   
   Average transfer rate   : 0.000B/s 
   Peak transfer rate      : 0B/s  {code}
 

Update 11/27/2024
Important Observation:

When I tried the above bulkload, there was another table in my cassandra-cluster in which one of the columns was a UDT containing a vector type. After dropping that table and UDT, bulkload worked.
Bug: Some benchmark tests leak LocalLog-related thread
Status: Triage Needed
Priority: Normal
Created: 27/Nov/24 17:02

I am running a benchmark, like CacheLoaderBench and there are these messages when iteration is over

{code}
<JMH had finished, but forked VM did not exit, are there stray running threads? Waiting 9 seconds more...>

Non-finished threads:

Thread[GlobalLogFollower,5,system]
  at java.base@11.0.12/jdk.internal.misc.Unsafe.park(Native Method)
  at java.base@11.0.12/java.util.concurrent.locks.LockSupport.park(LockSupport.java:323)
  at app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:290)
  at app//org.apache.cassandra.utils.concurrent.WaitQueue$Standard$AbstractSignal.await(WaitQueue.java:283)
  at app//org.apache.cassandra.tcm.log.LocalLog$Async$AsyncRunnable.run(LocalLog.java:816)
  at app//org.apache.cassandra.concurrent.InfiniteLoopExecutor.loop(InfiniteLoopExecutor.java:121)
  at app//org.apache.cassandra.concurrent.InfiniteLoopExecutor$$Lambda$233/0x0000000800393840.run(Unknown Source)
  at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
  at java.base@11.0.12/java.lang.Thread.run(Thread.java:829)
Thread[DestroyJavaVM,5,main]
{code}

I noticed this is its teardown

{code}
    @TearDown(Level.Trial)
    public void teardown()
    {
        CQLTester.tearDownClass();
        CommitLog.instance.stopUnsafe(true);
        CQLTester.cleanup();
    }
{code}

when I do this it will be OK

{code}
   @TearDown(Level.Trial)
    public void teardown()
    {
        CQLTester.tearDownClass();
        CommitLog.instance.stopUnsafe(true);
        ClusterMetadataService.instance().log().close();
        CQLTester.cleanup();
    }
{code}

Worth to say that benchark _eventually_ finishes. It is not stuck, JMH just waits and checks for 30 seconds and then it seems to "kill it".
Bug:  IAE on SAI query when using Accord
Status: Triage Needed
Priority: Normal
Created: 25/Nov/24 11:23

{code}
Caused by: java.lang.IllegalArgumentException: 1b255f4d-ef25-40a6-0000-00000000000b:4459194184563723815 >= 1b255f4d-ef25-40a6-0000-00000000000b:4459194184563723815
	at accord.primitives.Range.<init>(Range.java:212)
	at accord.primitives.Range$EndInclusive.<init>(Range.java:44)
	at org.apache.cassandra.service.accord.TokenRange.<init>(TokenRange.java:43)
	at org.apache.cassandra.service.accord.txn.TxnRangeRead.<init>(TxnRangeRead.java:127)
	at org.apache.cassandra.service.StorageProxy.readWithAccord(StorageProxy.java:2194)
	at org.apache.cassandra.service.reads.range.RangeCommandIterator.query(RangeCommandIterator.java:219)
	at org.apache.cassandra.service.reads.range.RangeCommandIterator.sendNextRequests(RangeCommandIterator.java:261)
	at org.apache.cassandra.service.reads.range.RangeCommandIterator.computeNext(RangeCommandIterator.java:131)
	at org.apache.cassandra.service.reads.range.RangeCommandIterator.computeNext(RangeCommandIterator.java:66)
	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
	at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:90)
	at org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:1045)
	at org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:629)
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:603)
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:401)
	at org.apache.cassandra.distributed.impl.Coordinator.lambda$executeWithPagingWithResult$5(Coordinator.java:178)
	at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
{code}

Repro:
{code}
cluster.coordinator(1).execute("SELECT * FROM distributed_test_keyspace.basic_sai0 WHERE pk0 = ? AND pk1 = ? AND pk2 = ? AND pk3 = ? AND pk4 = ? AND pk5 = ? AND pk6 = ? AND pk7 = ? AND ck6 = ? AND regular4 > ? AND static1 < ? ALLOW FILTERING;", ConsistencyLevel.QUORUM, ")?䊌操?뗘螇??ꎹ񱳽??᭘","hjlgnyj",(int)680309240,new java.util.Date(4653332152890845609L),909237048714349026L,(float)0.8014085,(short)4035,(float)0.029493034,"bjxmch",new java.util.Date(3153130488766643L),UUID.fromString("000f7fb9-87f4-48bb-b6cd-666843b44928"));
{code}