av9ash/GitBugs huggingface dataset:
Bug: Dropwizard Meter causes timeouts when infrequently used
Status: Patch Available
Priority: Normal
Created: 26/Jan/24 21:03

Observed instances of timeouts on clusters with long uptime and infrequently used tables and possibly just request paths such as not using CAS for large fractions of a year.

CAS seems to be more severely impacted because it has more metrics in the request path such as latency measurements for prepare, propose, and the read from the underlying table.

Tracing showed ~600-800 milliseconds for these operations in between the “appending to memtable” and “sending a response” events. Reads had a delay between finishing the construction of the iterator and sending the read response.

Stack traces dumped every 100 milliseconds using {{sjk}} shows that in prepare and propose a lot of time was being spent in {{{}Meter.tickIfNecessary{}}}.
{code:java}
Thread [2537] RUNNABLE at 2024-01-25T21:14:48.218 - MutationStage-2
com.codahale.metrics.Meter.tickIfNecessary(Meter.java:71)
com.codahale.metrics.Meter.mark(Meter.java:55)
com.codahale.metrics.Meter.mark(Meter.java:46)
com.codahale.metrics.Timer.update(Timer.java:150)
com.codahale.metrics.Timer.update(Timer.java:86)
org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
org.apache.cassandra.service.paxos.PaxosState.prepare(PaxosState.java:92)
Thread [2539] RUNNABLE at 2024-01-25T21:14:48.520 - MutationStage-4
com.codahale.metrics.Meter.tickIfNecessary(Meter.java:72)
com.codahale.metrics.Meter.mark(Meter.java:55)
com.codahale.metrics.Meter.mark(Meter.java:46)
com.codahale.metrics.Timer.update(Timer.java:150)
com.codahale.metrics.Timer.update(Timer.java:86)
org.apache.cassandra.metrics.LatencyMetrics.addNano(LatencyMetrics.java:159)
org.apache.cassandra.service.paxos.PaxosState.propose(PaxosState.java:127){code}
{{tickIfNecessary}} does a linear amount of work proportional to the time since the last time the metric was updated/read/created and this can actually take a measurable amount of time even in a tight loop. On my M2 MBP it was 1.5 milliseconds for a day, ~200 days took ~74 milliseconds. Before it warmed up it was 140 milliseconds.

A quick fix is to schedule a task to read all the meters once a day so it isn’t done in the request path and we have a more incremental amount to process at a time.

Also observed that {{tickIfNecessary}} is not 100% thread safe in that if it takes longer than 5 seconds to run the loop it can end up with multiple threads attempting to run the loop at once and then they will concurrently run {{EWMA.tick}} which probably results in some ticks not being performed.

This issue is still present in the latest version of {{Metrics}} if using {{{}EWMA{}}}, but {{SlidingWindowTimeAverages}} looks like it has a bounded amount of work required to tick.  Switching would change how our metrics work since the two don't have the same behavior.
Bug: CommitlogArchiver only has granularity to seconds for restore_point_in_time
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 28/Feb/24 16:00
Resolved: 15/Sep/24 18:28

Commitlog archiver allows users to backup commitlog files for the purpose of doing point in time restores.  The [configuration file|https://github.com/apache/cassandra/blob/trunk/conf/commitlog_archiving.properties] gives an example of down to the seconds granularity but then asks what whether the timestamps are microseconds or milliseconds - defaulting to microseconds.  Because the [CommitLogArchiver uses a second based date format|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java#L52], if a user specifies to restore at something at a lower granularity like milliseconds or microseconds, that means that the it will truncate everything after the second and restore to that second.  So say you specify a restore_point_in_time like this:
restore_point_in_time=2024:01:18 17:01:01.623392
it will silently truncate everything after the 01 seconds.  So effectively to the user, it is missing updates between 01 and 01.623392.

This appears to be a bug in the intent.  We should allow users to specify down to the millisecond or even microsecond level. If we allow them to specify down to microseconds for the restore point in time, then it may internally need to change from a long.
Bug: Test Failure: org.apache.cassandra.fuzz.ring.ConsistentBootstrapTest.coordinatorIsBehindTest
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 30/Jan/24 16:55
Resolved: 25/Mar/24 12:27

{code:java}
java.lang.IllegalStateException: Can't use shutdown instances, delegate is null at org.apache.cassandra.distributed.impl.AbstractCluster$Wrapper.delegate(AbstractCluster.java:283) at org.apache.cassandra.distributed.impl.DelegatingInvokableInstance.transfer(DelegatingInvokableInstance.java:49) at org.apache.cassandra.distributed.api.IInvokableInstance.runsOnInstance(IInvokableInstance.java:45) at org.apache.cassandra.distributed.api.IInvokableInstance.runOnInstance(IInvokableInstance.java:46) at org.apache.cassandra.distributed.shared.ClusterUtils.unpauseCommits(ClusterUtils.java:548) at org.apache.cassandra.fuzz.ring.ConsistentBootstrapTest.coordinatorIsBehindTest(ConsistentBootstrapTest.java:227) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}

https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2636/workflows/93adbf3e-acf8-4a62-a1f6-baf4f4689347/jobs/54912/tests
Bug: Read repair through Accord needs to only route the read repair through Accord if the range is actually migrated/running on Accord
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/Feb/24 16:15
Resolved: 14/Aug/24 17:17

This is because the read repair will simply fail if Accord doesn't manage that range. Not only does it need to be routed through Accord but if it races with topology change it needs to retry and not surface an error.
Bug: Accord: RejectBefore must be up-to-date on joining nodes before ready to coordinate
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Jan/24 11:10
Resolved: 05/Aug/24 15:53

The exclusive sync point used to join the shard will be known by a majority of the existing replicas, but in the event the quorum changes and the new replica has not recorded the exclusive sync point this might in principle lead to failing to reject a TxnId that should be rejected.

Simple fix, but introduce tests to corroborate this issue, and see if can reproduce in burn test.
Bug: Incorrect error type on oversized compound partition key
Status: Triage Needed
Priority: Normal
Created: 14/Jan/24 17:12

Cassandra limits key lengths (partition and clustering) to 64 KB. If a user attempts to INSERT data with a partition key or clustering key exceeding that size, the result is a clear InvalidRequest error with a message like "{{{}Key length of 66560 is longer than maximum of 65535{}}}".

There is one exception: If you have a *compound* partition key (i.e., two or more partition key components) and attempt to write one of them larger than 64 KB, then instead of an orderly InvalidRequest like you got when there was just one component, now you get a NoHostAvailable  with the message: "{{{}error("'H' format requires 0 <= number <= 65535")}){}}}". This is not only uglier, it can also confuse the Cassandra driver to retry this request - because it doesn't realize that the request itself is broken and there is no point to repeat it.

Interestingly, if there are multiple clustering key columns, this problem doesn't happen: we still get a nice InvalidRequest if any one of these is more than 64 KB.
Bug: Test Failure: org.apache.cassandra.index.sai.cql.RandomIntersectionTest.randomIntersectionTest[Small partition restricted high high]-system_keyspace_directory_jdk17
Status: Resolved
Priority: Normal
Resolution: Not A Problem
Created: 25/Jan/24 19:01
Resolved: 04/Mar/24 16:10

Seen here: 
[https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2629/workflows/75f57272-f299-40e1-8e4f-fdb75bca2f7c/jobs/53821/tests]

The tests were run with _memtable_allocation_type: heap_buffers_ (By default we run them now with offheap_objects, to be changed in CASSANDRA-19326)
{code:java}
junit.framework.AssertionFailedError: Got less rows than expected. Expected 16 but got 0 at org.apache.cassandra.cql3.CQLTester.assertRows(CQLTester.java:1880) at org.apache.cassandra.index.sai.cql.RandomIntersectionTest.lambda$runRestrictedQueries$3(RandomIntersectionTest.java:118) at org.apache.cassandra.cql3.CQLTester.beforeAndAfterFlush(CQLTester.java:2269) at org.apache.cassandra.index.sai.cql.RandomIntersectionTest.runRestrictedQueries(RandomIntersectionTest.java:104) at org.apache.cassandra.index.sai.cql.RandomIntersectionTest.randomIntersectionTest(RandomIntersectionTest.java:95) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36){code}
CC [~maedhroz]
Bug: CompactionStress tool write data cannot work because of new TCM feature
Status: Open
Priority: Normal
Created: 07/Feb/24 07:51

CompactionStress tool write data cannot work anymore because of new TCM feature
{code:java}
./compaction-stress write -d /tmp/compaction -g 5 -p ../cqlstress-example.yaml -t 4 

Cannot write anything to the disk{code}
It was introduced by the Patch: CEP-21. After applying that patch, this tool cannot work

anymore
{code:java}
Implementation of Transactional Cluster Metadata as described in CEP-21
Hash: ae084237
{code}
I attach some key code related diff for further debugging
Bug: StreamingTombstoneHistogramBuilder.DataHolder does not merge histogram points correctly on overflow
Status: In Progress
Priority: Normal
Created: 08/Feb/24 23:12

The algorithm tries to merge the two nearest points in the histogram and create a new point that is in between the two merged points based on the weight of each point. This can overflow long arithmetic with the code that is currently there  and the work around is pick one of the points and just put it there.

This can be worked around by changing the midpoint calculation to not overflow.
Bug: Switch memtable_allocation_type from offheap_objects to heap_buffers in test/conf/cassandra.yaml
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 25/Jan/24 16:33
Resolved: 31/Jan/24 16:34

By default we use heap_buffers in cassandra.yaml, but the unit tests were switched for testing to offheap_objects. This needs to be reverted. We should be testing with heap_buffers on all branches, we do it only in cassandra-3.0 at the moment.
Bug: Test Failure: dtest-upgrade.upgrade_tests.upgrade_through_versions_test.*
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 16/Feb/24 16:23
Resolved: 27/Feb/24 06:26

Failing in Jenkins:
 * [dtest-upgrade-novnode-large.upgrade_tests.upgrade_through_versions_test.TestProtoV4Upgrade_AllVersions_EndsAt_Trunk_HEAD.test_parallel_upgrade_with_internode_ssl|https://ci-cassandra.apache.org/job/Cassandra-5.0/170/testReport/junit/dtest-upgrade-novnode-large.upgrade_tests.upgrade_through_versions_test/TestProtoV4Upgrade_AllVersions_EndsAt_Trunk_HEAD/test_parallel_upgrade_with_internode_ssl/]
 * [dtest-upgrade-novnode-large.upgrade_tests.upgrade_through_versions_test.TestProtoV4Upgrade_AllVersions_RandomPartitioner_EndsAt_Trunk_HEAD.test_parallel_upgrade_with_internode_ssl|https://ci-cassandra.apache.org/job/Cassandra-5.0/170/testReport/junit/dtest-upgrade-novnode-large.upgrade_tests.upgrade_through_versions_test/TestProtoV4Upgrade_AllVersions_RandomPartitioner_EndsAt_Trunk_HEAD/test_parallel_upgrade_with_internode_ssl/]
 * [dtest-upgrade-novnode.upgrade_tests.upgrade_through_versions_test.TestProtoV3Upgrade_AllVersions_EndsAt_Trunk_HEAD.test_parallel_upgrade|https://ci-cassandra.apache.org/job/Cassandra-5.0/170/testReport/junit/dtest-upgrade-novnode.upgrade_tests.upgrade_through_versions_test/TestProtoV3Upgrade_AllVersions_EndsAt_Trunk_HEAD/test_parallel_upgrade/]
 * [dtest-upgrade.upgrade_tests.upgrade_through_versions_test.TestProtoV4Upgrade_AllVersions_RandomPartitioner_EndsAt_Trunk_HEAD.test_parallel_upgrade|https://ci-cassandra.apache.org/job/Cassandra-5.0/170/testReport/junit/dtest-upgrade.upgrade_tests.upgrade_through_versions_test/TestProtoV4Upgrade_AllVersions_RandomPartitioner_EndsAt_Trunk_HEAD/test_parallel_upgrade/]
 * [dtest-upgrade.upgrade_tests.upgrade_through_versions_test.TestProtoV4Upgrade_AllVersions_RandomPartitioner_EndsAt_Trunk_HEAD.test_parallel_upgrade_with_internode_ssl|https://ci-cassandra.apache.org/job/Cassandra-5.0/170/testReport/junit/dtest-upgrade.upgrade_tests.upgrade_through_versions_test/TestProtoV4Upgrade_AllVersions_RandomPartitioner_EndsAt_Trunk_HEAD/test_parallel_upgrade_with_internode_ssl/]
Bug: Fix a few leftovers from previous versions on the java17.adoc page for 5.0+
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Jan/24 14:59
Resolved: 29/Jan/24 23:51

- Java17 is experimental for now for both JDK11 and JDK17 builds
 - the text refers to the 4.0 source code when it should be 5.0
 - The JDK 11 build section points to output that says we build with non-default JDK version 11. This is probably from the before JDK8 drop times.  (we probably need to check and update other example screens, too)
Bug: Do not go to disk to get HintsStore.getTotalFileSize
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 16/Mar/24 20:30
Resolved: 25/Mar/24 13:35

When testing a cluster with more requests than it could handle, I noticed significant CPU time (25%) spent in HintsStore.getTotalFileSize.  Here's what I'm seeing from profiling:

10% of CPU time spent in HintsDescriptor.fileName which only does this:

 
{noformat}
return String.format("%s-%s-%s.hints", hostId, timestamp, version);{noformat}
At a bare minimum here we should create this string up front with the host and version and eliminate 2 of the 3 substitutions, but I think it's probably faster to use a StringBuilder and avoid the underlying regular expression altogether.

12% of the time is spent in org.apache.cassandra.io.util.File.length.  It looks like this is called once for each hint file on disk for each host we're hinting to.  In the case of an overloaded cluster, this is significant.  It would be better if we were to track the file size in memory for each hint file and reference that rather than go to the filesystem.

These fairly small changes should make Cassandra more reliable when under load spikes.

CPU Flame graph attached.

I only tested this in 4.1 but it looks like this is present up to trunk.
Bug: Cassandra 4.1.4 floods logs with "Completed 0 uncommitted paxos instances for"
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Feb/24 14:44
Resolved: 10/Jun/24 09:15

Hello,

On our cluster logs are flooded with: 
{code:java}
INFO  [OptionalTasks:1] 2024-02-27 14:27:51,213 PaxosCleanupLocalCoordinator.java:185 - Completed 0 uncommitted paxos instances for XXXXXXXXX on ranges [(9210458530128018597,-9222146739399525061], (-9222146739399525061,-9174246180597321488], (-9174246180597321488,-9155837684527496840], (-9155837684527496840,-9148981328078890812], (-9148981328078890812,-9141853035919151700], (-9141853035919151700,-9138872620588476741], {code}
I cannot find anything in doc regarding this longline. Also this are huge log payloads that heavy flood system.log.
Bug: Non-transactional schema updates can interfere with Accord transaction execution
Status: Triage Needed
Priority: Normal
Created: 21/Feb/24 21:27

While Accord can handle topology changes correctly it can’t handle non-transaction schema updates because those execute outside of Accord. When Accord tries to execute a transaction against the schema in the epoch the transaction is supposed to execute in then it is possible for different nodes to see different schemas when reading or writing data as part of a transaction.

Dropping a needed a column or table is the most likely issue as we don't support altering column types.

Because commit is async it is possible for a table or to be dropped before the writes can be propagated after it was acknowledged instead of signaling an error. While the table was dropped it's possible the client needed the error to know that the request was processed improperly or that it needed to take some other action client side.

Or add table where the original coordinator can't read the table, but the recovery coordinator can and might apply different results to different replicas.
Bug: Link in docs to Achilles Java Driver links to malicious site
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 07/Feb/24 10:47
Resolved: 26/Jun/24 19:50

https://cassandra.apache.org/doc/4.1/cassandra/getting_started/drivers.html#java

The Achilles link looks dangerous. I tried it and it looked like the link has been taken over by a malicious user.
Bug: Accord: Asynchronous reads may be unsafe
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 23/Jan/24 16:04
Resolved: 18/Jun/24 18:10

In principle we should invalidate asynchronous reads before they complete if the data they read may be invalid, but this anyway causes faults when we permit them to occur in accord-core. We can and perhaps should simply ensure the reads are issued against an sstable/memtable snapshot taken by the command store, as this is lower cost and more robust. Otherwise we should investigate what issue asynchronous reads cause.
Bug: Rework json testing
Status: Open
Priority: Normal
Created: 21/Mar/24 16:20

The only real json testing we have is in [this dtest|https://github.com/apache/cassandra-dtest/blob/trunk/json_test.py] which is full of magic like running from the tests' docstrings which makes other things very difficult, like adding any kind of version-based variation.  I propose rewriting this, either as sane python or a java dtest.
Bug: Simplify metadata log implementation using custom partitioner
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/Mar/24 14:26
Resolved: 11/Apr/24 08:09

The distributed metadata log table can be simplified by leveraging the fact that replicas are all responsible for the entire token range. Given this assumption, we can then use {{ReversedLongLocalPartitioner}} introduced in CASSANDRA-19391 to make it much easier to append to/read from the tail of the log, effectively removing the need for the {{Period}} construct. This will also improve apply to the local metadata log used at startup.
Bug: Revert switch to approximate time in Dispatcher to avoid mixing with nanoTime() in downstream timeout calculations
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 01/Mar/24 22:47
Resolved: 07/Mar/24 22:23

CASSANDRA-15241 changed {{Dispatcher}} to use the {{approxTime}} implementation of {{MonotonicClock}} rather than {{nanoTime()}}, but clock drift between the two, can potentially cause queries to time out more quickly. We should be able to revert the {{Dispatcher}} to use {{nanoTime()}} again and similarly change {{QueriesTable} to {{nanoTime()}} as well for consistency.
Bug: Upgrade Python driver to 3.29.0
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/Jan/24 12:01
Resolved: 19/Jan/24 17:40

The version of the Python driver that is used by cqlsh (3.28.0) doesn't correctly support vectors of variable-width types. Wrong deserialization can either fail with an exception or silently provide wrong results. For example:
{code:java}
cqlsh:k> CREATE TABLE t (k int PRIMARY KEY, v vector<varint, 2>);
cqlsh:k> INSERT INTO t(k, v) VALUES (1, [23452352356235654634567437463767365783768, 3]);
cqlsh:k> SELECT * FROM t;
 k | v
---+--------------------------
 1 | [289729430, -1001073214]}}
{code}
The most recent driver at the moment (3.29.0) still doesn't support this kind of vector, but it always fails with a proper message instead of dangerously providing wrong results, thanks to [PYTHON-1371|https://datastax-oss.atlassian.net/browse/PYTHON-1371].
Bug: Test Failure: org.apache.cassandra.distributed.test.RemoveNodeTest.testAbort
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 30/Jan/24 16:52
Resolved: 12/Feb/24 08:55

{code:java}
junit.framework.AssertionFailedError: "RemovalStatus: No removals in progress. " does not contain MID_LEAVE at org.apache.cassandra.distributed.test.RemoveNodeTest.testAbort(RemoveNodeTest.java:54) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}

https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra?branch=testAbort-trunk
Bug: Accord: PreLoadContext must properly and consistently support ranges
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 01/Feb/24 14:06
Resolved: 29/Mar/24 17:12

There are some mechanisms for ensuring range transactions are loaded for range transactions, but they do not currently work properly (having several race conditions), are potentially costly in terms of memory consumption, and are inconsistent with how they work for key transactions.