av9ash/GitBugs huggingface dataset:
Bug: Row cache inconsistency issue: A read can put stale data into row cache in a race condition
Status: Open
Priority: Normal
Created: 06/Apr/23 21:25

We found the issue in our production system which has the version 3.11.6.  When we did an update and then read immediately after update successfully, we may read the stale data sometimes.  Same issue for writeAll + readOne consistency and writeQuorm+readQuorum. The issue is gone once we disabled the row cache.

The config for row cache: 

caching = \{'keys': 'ALL', 'rows_per_partition': 'ALL'}

 

After some investigations, we think there is a race condition during read/write path. Problems:

When two threads are reading and writing the same partition (for example, two rows with same partition key) at same time, the read thread may load the stale data into row cache for the row which is being updated.
{{}}
{panel:title=The steps of write-thread inserting a row to partition p}
{{W-Step }}{{{}1{}}}{{{}: inserts the value v1 to memtable.{}}}
{{W-Step }}{{{}2{}}}{{{}: invalidates the row cache using partition key.{}}}
{panel}
{{}}
{panel:title=The steps of read-thread reading a row from partition p}
{{R-Step }}{{{}1{}}}{{{}: Checks row cache and finds whether the row is not present in cache. If not, goes to '{}}}{{{}R-Step {}}}{{{}2'{}}}{{{}.{}}}
{{R-Step }}{{{}2{}}}{{{}: Insert a sentinel (timestamp) as the row value into row cache to tell other read threads should skip the row cache.{}}}
{{R-Step }}{{{}3{}}}{{{}: Read from storage layer and get value v0 which can be older than v1.{}}}
{{R-Step }}{{{}4{}}}{{{}: Insert v0 to row cache {}}}{{for}} {{the row by checking }}{{if}} {{the row doesn't exist or it has the same sentinel. *The inconsistency is caused by this step. Should not insert the stale value if the sentinel doesn't exist in row cache any more.*}}
{panel}
{{}}
{panel:title=The sequence to reproduce the issue}
{{R-Step }}{{1}}
{{R-Step }}{{2}}
{{R-Step }}{{3}}
{{W-Step }}{{1}}
{{W-Step }}{{2}}
{{R-Step }}{{4}}
{panel}
{{}}
Bug: Provide checkstyle in ant scripts on all branches; improve documentation on recommended JDK env for dev
Status: Triage Needed
Priority: Normal
Created: 07/Apr/23 14:01

See CASSANDRA-18262 comment thread for discussion leading up to this

# We should checkstyle only on the lowest supported JDK for our builds in CI
# Having checkstyle be part of the local {{ant jar}} target for all supported JDKs would be nice ergonomics for folks who are working with another supported JDK version so they don't get surprised in CI
# We should tighten up our docs in terms of recommendations on which JDK to build/run with locally and help smooth out that process for folks
Bug: Test failure: pdtest: dtest-novnode.ttl_test.TestDistributedTTL.test_ttl_is_respected_on_delayed_replication
Status: Resolved
Priority: Normal
Resolution: Cannot Reproduce
Created: 04/Apr/23 20:04
Resolved: 05/Apr/23 15:25

{{Failed 1 times in the last 30 runs. Flakiness: 3%, Stability: 96%}}

{code}
Error Message
AssertionError: Expected [[1, 1, None, None], [2, 2, None, None]] from SELECT * FROM ttl_table;, but got [[2, 2, None, None]]
Stacktrace
self = <ttl_test.TestDistributedTTL object at 0x7fce080d8c10>

    def test_ttl_is_respected_on_delayed_replication(self):
        """ Test that ttl is respected on delayed replication """
        self.prepare()
        self.node2.stop()
        self.session1.execute("""
            INSERT INTO ttl_table (key, col1) VALUES (1, 1) USING TTL 5;
        """)
        self.session1.execute("""
            INSERT INTO ttl_table (key, col1) VALUES (2, 2) USING TTL 1000;
        """)
>       assert_all(
            self.session1,
            "SELECT * FROM ttl_table;",
            [[1, 1, None, None], [2, 2, None, None]]
        )

ttl_test.py:480: 
{code}

At face value this didn't look like a classic "oops I timed out", but now that I read that error message I'm not as sure. So I'd check that first and just close out if it's a timeout by another name until we get the infra stabilized.
Bug: CEP-21 Add debounce to log replay
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Apr/23 08:30
Resolved: 04/Apr/23 08:00

To avoid overload we should debounce log replay requests between non-CMS nodes and CMS members. If a non-CMS node already has a replay request inflight, any subsequent requests should wait for the inflight response and return that. As soon as a response has been received, that request should be considered stale and a new one initiated.
Bug: CEP-21 Dereference TableMetadata in simple partition builder
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Apr/23 08:35
Resolved: 04/Apr/23 08:00

In some places we retain static {{TableMetadata}} instances and use them to construct a {{PartitionUpdate}}.  For instance, TraceKeyspace contains Sessions & Events static members which are created at startup when the current epoch is Epoch.EMPTY. These are later used to construct mutations when tracing is enabled and when the mutations are serialised and sent between replica & coordinator the epoch comparisons in PartitionUpdate deserializer trigger an IncompatibleSchemaException. Ultimately we should not expose static {{TableMetadata}} instances in this way.
Bug: CEP-21 Ensure that global log replication factor is maintained after decommission
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Apr/23 08:34
Resolved: 04/Apr/23 08:00

If a CMS node is decommissioned, another node should automatically become a CMS member to take its place. If there are no non-CMS peers, the decommission should be rejected.
Bug: CEP-21 Remove paranoid check which fails due to pre-existing mismatch on replica vs query range
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Apr/23 08:33
Resolved: 04/Apr/23 08:00

If short read protection is activated, a replica plan is constructed to fetch additional partitions from a single replica in the original set. As additional partitions are requested, the data range of the query is reduced but the original \{{Replica}} instance is reused, resulting in a mismatch between the range of the query and that of the replica. This is not new to TCM and is present in trunk. It is called out in a comment, but thought to be harmless as the narrowed range should be a subset of the original data range which did match the replica's. Early TCM patches introduced an addition check here on the two ranges and so rejects SRP queries which trunk does not.
Bug: CEP-21 During multi step operations, defer token map update until completion of final step
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Apr/23 08:35
Resolved: 04/Apr/23 08:00

Updating the token map, used by {{UniformRangePlacement}} to calculate placements, too early can cause stream plans to be calculated incorrectly. The eager update of the token map was intended to ensure that subsequent (and possibly concurrent) range movements would remain compatible with the operation's end state, but this is not necessary as the affected ranges are locked and intersecting operations are rejected.
Bug: CEP-21 Improve seedlist inspection at startup
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Apr/23 08:31
Resolved: 04/Apr/23 08:00

The intention is to only force a new node directly into becoming the first CMS node in local, single node installs or CCM cluster. Otherwise, we should always attempt to discover peers to determine an existing CMS or to initiate a new election by proposing ourself.
Bug: CEP-21 Add request failure reason to indicate invalid routing of a read or write
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Apr/23 08:34
Resolved: 04/Apr/23 08:00

In the context of a read or write request, this would indicate that a node does not consider itself a replica of the specified token or range. This implies that one or both of the coordinator and replica has an outdated view of cluster metadata and may result in the operation failing if the specified consistency level can no longer be satisfied using the coordinator's initial replica plan.
Bug: CEP-21 Various fixes to schema related in-jvm dtests
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Apr/23 08:32
Resolved: 04/Apr/23 08:00

* JVMDTestTest
    schema change with stopped node, must not be CMS
* SSTableIdGenerationTest
    This test revolves around creating sstables with sequential ids, then
    restarting the node after enabling sstable uuids, creating more sstables
    and then counting the number of each type of sstable. It seems to rely
    on the local host id being unavailable when the node restarts, which
    causes the mutations in the commitlog, which have already been flushed
    to be replayed as the null local host id doesn't match the actual one in
    the sstable metadata. In turn this leads to an extra sstable being
    created after the restart, duplicating (some of) what was in the earlier
    sstables.
* MigrationCoordinatorTest
    deleted
Bug: CEP-21 Make StorageService treatment of bootstrapping nodes consistent with previous implementation
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Apr/23 08:33
Resolved: 04/Apr/23 08:00

Correctly return true from {{isBootstrapMode()}} if the node is bootstrapping. Also, {{getJoiningNodes()}} should exclude nodes which have registered, but are yet to start joining.
Bug: CEP-21 Always use Paxos.v2 for global log reads/writes regardless of cluster configuration
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Apr/23 08:32
Resolved: 04/Apr/23 08:00

As there is no legacy use of the global log keyspace and it is purely a system keyspace, we should always use paxos V2 for SERIAL reads and writes there.
Bug: CEP-21 Implement retries for log replay on CMS members
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Apr/23 08:33
Resolved: 04/Apr/23 08:00

This should enable {{AbstractLocalProcessor}} implementations to backoff and retry when executing distributed reads of the consistent log, in much the same way that {{RemoteProcessor}} does when making replay requests to CMS members.
Bug: Update generate-idea-files for J17
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 19/Apr/23 18:19
Resolved: 05/Jul/23 14:43

There was a discussion in CASSANDRA-18258 how to update generate-idea-files.

The final agreement was to create one target to cover both Java 11 and Java 17.

It will be good to figure out CASSANDRA-18263 and reshuffle arguments and tasks based on what we decide to use as gc in testing for both Java 11 and Java 17.
Bug: Remove legacy command line options from cassandra-stress
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 15/May/23 16:31
Resolved: 09/Aug/23 21:25

The cassandra-stress *mode* option allows specifying options for native protocol and cql3, but these don't seem useful as there would seem to be no other valid options now that cql3 is the standard and thrift no longer supported. 

    -mode native cql3 user=cassandra password=xxxxxx

can be simplified to:

    -mode user=cassandra password=xxxxxx

Also, the readme.txt in tools/stress states "cassandra-stress supports benchmarking any Cassandra cluster of version 2.0+" but maybe should be updated to a supported Cassandra version, i.e., 3.11.x.
Bug: [Analytics] Fix example documentation 
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 07/Jun/23 18:58
Resolved: 27/Jun/23 20:41

The documentation for the example job for Cassandra Analytics [link|https://github.com/apache/cassandra-analytics/blob/trunk/cassandra-analytics-core-example/README.md] configures Sidecar to use the local CCM cluster. This documentation is however broken since some feedback during the review of the Sidecar PR changed one of the yaml options from {{uploads_staging_dir}} to {{staging_dir}}.

We need to update the documentation to reflect that change.
Bug: In-tree documentation takes a long time to generate
Status: In Progress
Priority: High
Created: 03/Apr/22 15:33

To generate the in-tree documentation the website-content tooling will execute the following steps for each branch.
 # Copy or clone the Cassandra repository depending on the URL supplied
 # Checkout the branch
 # Select the version of Java based on the Cassandra version
 # Compile Cassandra
 # Generate ADOC files for Nodetool and the Cassandra YAML configuration
 # Commit to repository

Once the generated ADOC files are committed to the repository, Antora is executed to generate the HTML.

The majority of the time taken to execute the above steps is in Step 4 where Cassandra is compiled. This time could be reduced by downloading the precompiled binaries for branches and tags.
Bug: Add  utests_system_keyspace_directory to pre-commit tests workflow in CircleCI
Status: Resolved
Priority: High
Resolution: Fixed
Created: 19/May/22 20:26
Resolved: 31/May/22 18:22

utests_system_keyspace_directory tests get broken every now and then but we don't have them added in Jenkins and they are not mandatory pre-commit in CircleCI.

A quick first step to have a better monitoring on those is to add them at least being mandatory ran before a commit in CircleCI
Bug: Throw exception on wrong config boundaries
Status: Open
Priority: Normal
Created: 22/Apr/22 15:01

While working on CASSANDRA-15234 we noticed usage of negative values where they are not supposed to be used. We fixed that for the parameters in scope - type duration, data storage and data rate but as [~brandon.williams] pointed - there are other examples from the rest of the config that is good to be fixed too.

This ticket should handle:
- check the rest of the parameters, where negatives shouldn't be allowed they shouldn't be allowed

- ensure that whatever validations we apply to parameters during startup (check the DatabaseDescriptor) are applied also in the respective setters for those parameters.
Bug: Add support for big endian architecture (s390x)
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 30/Jun/22 10:27
Resolved: 29/Jul/23 08:48

Noticed that some of the Cassandra tests are failing on big endian architectures (s390x). Please find the attached code patch which corrects the data handling for big endian architecture. It also corrects the byte ordering when dealing with sun.misc.Unsafe methods. After the patch following tests are passing for s390x which were failing earlier:

TTLTest
ScrubTest
LegacySSTableTest
SSTableExportSchemaLoadingTest
SSTableMetadataViewerTest
StandaloneUpgraderOnSStablesTest
StandaloneVerifierOnSSTablesTest

The code change ensures that Cassandra for little endian architectures remain unaffected.
Bug: DOC - tools/nodetool is untitled and might be affecting search results
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Apr/22 02:04
Resolved: 02/Jan/25 09:15

Reported by [~mck] in [ASF Slack #cassandra-website|https://the-asf.slack.com/archives/C01RY1LUPD5/p1650446747714449]:

{quote}I see that the nodetool doc pages have the title {{Untitled | …}}  ( i think this impacting search results){quote}
Bug: Text value containing "/*" interpreted as multiline comment in cqlsh
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/May/22 07:22
Resolved: 24/Sep/24 13:17

I use CQLSH command line utility to load some DDLs. The version of utility I use is this:
{noformat}
[cqlsh 6.0.0 | Cassandra 4.0.0.47 | CQL spec 3.4.5 | Native protocol v5]{noformat}
Command that loads DDL.cql:
{noformat}
cqlsh -u username -p password cassandra.example.com 65503 --ssl -f DDL.cql
{noformat}
I have a line in CQL script that breaks the syntax.
{noformat}
INSERT into tablename (key,columnname1,columnname2) VALUES ('keyName','value1','/value2/*/value3');{noformat}
{{/*}} here is interpreted as start of multi-line comment. It used to work on older versions of cqlsh. The error I see looks like this:
{noformat}
SyntaxException: line 4:2 mismatched input 'Update' expecting ')' (...,'value1','/value2INSERT into tablename(INSERT into tablename (key,columnname1,columnname2)) VALUES ('[Update]-...) SyntaxException: line 1:0 no viable alternative at input '(' ([(]...)
{noformat}
Same behavior while running in interactive mode too. {{/*}} inside a CQL statement should not be interpreted as start of multi-line comment.

With schema:
{code:java}
CREATE TABLE tablename ( key text primary key, columnname1 text, columnname2 text);{code}
Bug: Option to disable write path during streaming for CDC enabled tables
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/May/22 20:47
Resolved: 07/Jun/22 18:32

For the CDC-enabled tables, a special write path is employed during streaming. The mutations streamed are written into commit log first. 

There are scenarios that the commit logs can accumulate, which lead to failure of streaming and blocking writes. 

I'd like to propose adding a dynamic toggle to disable the special write path for CDC during streaming. 

Please note that the toggle is a trade-off. Because the special write path is there in the hope to ensure data consistency. Turning it off allows the streaming to pass, but in some extreme scenarios, the downstream CDC consumers may have holes in the stream, depending on how they consumes the commit logs.
Bug: "Total TrueDiskSpaceUsed" computation does not include manifest and schema files
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Apr/22 21:25
Resolved: 24/Jul/24 15:49

This is the output of {{nodetool listsnapshots}}:

{noformat}
Snapshot Details:
Snapshot name                  Keyspace name Column family name True size Size on disk Creation time            Expiration time
test                           ks            indexed_table      1.1 KiB   21.22 KiB    2022-04-26T19:13:20.102Z
test                           ks            my_table           960 bytes 10.76 KiB    2022-04-26T19:13:20.102Z
dropped-1650997415751-my_table ks            my_table           989 bytes 989 bytes    2022-04-26T18:23:35.751Z

Total TrueDiskSpaceUsed: 0 KiB
{noformat}

Total TrueDiskSpaceUsed does not match the sum of "true size on disk" of all snapshots.

The reason for this is because {{Total TrueDiskSpaceUsed}} computation does not include the size of snapshot schema and manifest files.

We should fix the {{Total TrueDiskSpaceUsed}} computation to include the the size of snapshot schema and manifest files, so it will look like this:

{noformat}
Snapshot Details:
Snapshot name                  Keyspace name Column family name True size Size on disk Creation time            Expiration time
test                           ks            indexed_table      1.1 KiB   21.22 KiB    2022-04-26T19:13:20.102Z
test                           ks            my_table           960 bytes 10.76 KiB    2022-04-26T19:13:20.102Z
dropped-1650997415751-my_table ks            my_table           989 bytes 989 bytes    2022-04-26T18:23:35.751Z

Total TrueDiskSpaceUsed: 3 KiB
{noformat}
Bug: Test failure: org.apache.cassandra.db.RangeTombstoneTest.testRangeTombstoneCompaction
Status: Resolved
Priority: Normal
Resolution: Cannot Reproduce
Created: 25/Apr/22 16:38
Resolved: 13/Sep/23 12:42

https://ci-cassandra.apache.org/job/Cassandra-trunk/1099/testReport/org.apache.cassandra.db/RangeTombstoneTest/testRangeTombstoneCompaction_compression_2/

{code}
junit.framework.AssertionFailedError: Expecting open marker, got Row: name=8 | val=8
	at org.apache.cassandra.db.RangeTombstoneTest.testRangeTombstoneCompaction(RangeTombstoneTest.java:561)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
{code}
Bug: Prevent JVM Humongous allocations
Status: Open
Priority: Normal
Created: 23/May/22 17:20

Several articles on Cassandra note that certain workloads, especially large partitions, can cause humongous allocations in Java, which can be reduced, but not always eliminated by increasing RegionSize.  RegionSize max is 32MB.  Humongous allocations are an undesirable performance issue because new objects get directly allocated in old space.

Below is a sample message seen in the gc.log

2022-05-23T18:46:58.743+0800: 3758624.170: [GC pause (G1 Humongous Allocation) (young) (initial-mark) 3758624.171: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 56725, predicted base time: 39.83 ms, remaining time: 160.17 ms, target pause time: 200.00 ms]

Instead of Cassandra users tuning RegionSize manually, a better solution would be for Cassandra to limit the size of allocations, or consider moving humongous allocations off heap.

 

References

[https://docs.oracle.com/en/java/javase/11/gctuning/garbage-first-g1-garbage-collector1.html#GUID-D74F3CC7-CC9F-45B5-B03D-510AEEAC2DAC]

[https://stevenlacerda.medium.com/identifying-and-fixing-humongous-allocations-in-cassandra-bf46444cec41]

https://krzysztofslusarski.github.io/2020/11/10/humongous.html
Bug: Backport CASSANDRA-17466 and CASSANDRA-17663 to 3.0 and 3.11
Status: Open
Priority: Normal
Created: 20/May/22 20:37

CASSANDRA-17466 was fixed in 4.0+ but not 3.x.  This bug would close SSTables leaving the cluster in a bad state and needing to be restarted in order to recover
Bug: DOC - Fix Dynamo paper links to have working urls
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 01/Apr/22 07:39
Resolved: 04/Jan/24 01:47

References to Dynamo paper at the Architecture [Overview|https://cassandra.apache.org/doc/latest/cassandra/architecture/overview.html] and [Dynamo|https://cassandra.apache.org/doc/latest/cassandra/architecture/dynamo.html] pages refer to an URL that gives 404 error.

We need to update the URL to point to another copy of the same paper.

Before: [http://courses.cse.tamu.edu/caverlee/csce438/readings/dynamo-paper.pdf]
After: [https://www.cs.cornell.edu/courses/cs5414/2017fa/papers/dynamo.pdf]
Bug: MacOS install should include cql.html local help file
Status: Open
Priority: Normal
Created: 07/Jun/22 01:06

On MacOS, /usr/share is not writable – even by root. Instead, /usr/local/share is the preferred location.

The current share directory path is /usr/share/doc/cassandra/CQL.html.  This would just add an alternative path.

Using a list comprehension would simplify the current code.  E.g.,     
{quote}import pathlib

HTML_FALLBACK = 'https://cassandra.apache.org/doc/latest/cql/index.html'

loc = [ f"\{pkgpath}/doc/cql3/CQL.html', "/usr/share/doc/cassandra/CQL.html", "{color:#4c9aff}/usr/local/share{color}/doc/cassandra/CQL.html" ]

item = next((x.as_uri() for x in loc if Path( x ).exists()), HTML_FALLBACK)
{quote}
Homebrew doesn't install the doc files, so this would likely need either

a) added to homebrew in [https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/cassandra.rb]

or

b) installed with pip's setup.cql and accessed with Python's [importlib.resources|https://docs.python.org/3/library/importlib.resources.html]
Bug: 'if not exists' clause should be produced on normal DESCRIBE cqlsh command
Status: Resolved
Priority: Normal
Resolution: Won't Do
Created: 08/Apr/22 18:56
Resolved: 12/Apr/22 10:23

IF NOT EXISTS is saved when one takes, for example, a snapshot and it is added in schema.cql. However, I noticed that if one creates a table (or any other CQL entity for that matter), on DESCRIBE, it will not create the output with IF NOT EXISTS. I believe this is a discrepancy and it should be targetted.