av9ash/GitBugs huggingface dataset:
Bug: [Analytics] Use XXHash32 for digest calculation of SSTables
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 05/Feb/24 19:45
Resolved: 14/Feb/24 01:47

During bulk writes, Cassandra Analytics calculates the MD5 checksum of every SSTable it produces. During SSTable upload to Cassandra Sidecar, Cassandra Analytics includes the {{content-md5}} header as part of the upload request. This information is used by Cassandra Sidecar to validate the integrity of the uploaded SSTable and prevent issues with bit flips and corrupted SSTables.

Recently, Cassandra Sidecar introduced [support for additional checksum validations|https://issues.apache.org/jira/browse/CASSANDRASC-97] during SSTable upload. Notably the XXHash32 digest support was added which offers for more performant checksum calculations. This support now allows Cassandra Analytics to use a more efficient digest algorithm that is friendlier on the CPU usage of Sidecar and spark resources.
Bug: Expose mode of authentication in system_views.clients, nodetool clientstats, and ClientMetrics
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 05/Feb/24 15:37
Resolved: 13/Feb/24 11:18

CASSANDRA-18554 added support for mTLS-authenticated clients. Part of this contribution introduced {{{}MutualTlsWithPasswordFallbackAuthenticator{}}}, which enables Cassandra to support either password and mTLS-authenticated connections.

As an operator, it would be useful to know which connections are mTLS authenticated, and which are password authenticated, as a possible mode of operation is migrating users from one from of authentication to another. It would also be useful to know if that if authentication attempts are failing which mode of authentication is unsuccessful.

Proposing to add the following:
 * Add a {{mode: string}} and {{metadata: map<string, string>}} to {{{}AuthenticatedUser{}}}. Update existing {{IAuthenticator}} implementations to pass {{mode}} (e.g. {{password}} , {{{}mtls{}}}), and optionally pass a {{metadata}} map (e.g. this can include the extracted {{identity}} from a client certificate for {{mtls}} authentication).
 * Update nodetool clientstats to add a new option flag {{{}--metadata{}}}, which when passed exposes these new fields on {{{}AuthenticatedUser{}}}. (Not added to existing output to maintain compatibility, much like {{-client-options}} did.
 * Update {{system_views.clients}} to include columns for these new fields.
 * Add new metrics to {{{}ClientMetrics{}}}:
 ** Track authentication success and failures by mode. (Note: The metrics present by authentication mode scope are contextual based on the Authenticator used (e.g. only {{scope=Password}} will be present for {{{}PasswordAuthenticator{}}})

{noformat}
Existing:

org.apache.cassandra.metrics:name=AuthSuccess,type=Client
org.apache.cassandra.metrics:name=AuthFailure,type=Client

New:

org.apache.cassandra.metrics:name=AuthSuccess,scope=MutualTls,type=Client
org.apache.cassandra.metrics:name=AuthSuccess,scope=Password,type=Client

org.apache.cassandra.metrics:name=AuthFailure,scope=MutualTls,type=Client
org.apache.cassandra.metrics:name=AuthFailure,scope=Password,type=Client
{noformat}
 * 
 ** Track connection counts by mode:

{noformat}
Existing:
org.apache.cassandra.metrics:name=ConnectedNativeClients,type=Client
org.apache.cassandra.metrics:name=connectedNativeClients,type=Client (previously deprecated but still maintained)

New:
org.apache.cassandra.metrics:name=ConnectedNativeClients,scope=MutualTls,type=Client
org.apache.cassandra.metrics:name=ConnectedNativeClients,scope=Password,type=Client
{noformat}
 * 
 ** A metric to track encrypted vs. non-encrypted connections:

{noformat}
org.apache.cassandra.metrics:name=ConnectedNativeClients,scope=Encrypted,type=Client
org.apache.cassandra.metrics:name=ConnectedNativeClients,scope=Unencrypted,type=Client
{noformat}
Bug: Flaky Host replacement tests and shrink tests
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 17/Jan/24 22:35
Resolved: 22/Jan/24 20:10

During Circle CI runs there are some flaky integration tests, some noticed are
 * HostReplacementMultiDCTest
 * HostReplacementMultiDCFailureTest
 * HostReplacementFailureTest
 * LeavingSingleFailureTest

Some of the error message I see in these tests are e.g.

`java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Failed to bind port 42611 on 127.0.0.2.`

On repeated run, these tests pass.
Bug: Minor corrections to TCM Implementation documentation
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/Jan/24 17:14
Resolved: 12/Feb/24 08:54

In the "[Bootstrap: InProgressSequence, PrepareJoin, BootstrapAndJoin|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/tcm/TCM_implementation.md#bootstrap-inprogresssequence-preparejoin-bootstrapandjoin]” section of TCM Implementation doc, InProgressSequence transformations are mentioned as "{_}InProgressSequence`, holding the three transformations (`PrepareJoin`, `MinJoin` `FinishJoin`){_}”. As per [PrepareJoin|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/tcm/transformations/PrepareJoin.java#L154], these are {*}StartJoin{*}, Mi{*}d{*}Join & FinishJoin. Doc needs to be updated.
Bug: BEGIN TRANSACTIONS throws error using index
Status: Triage Needed
Priority: Normal
Created: 09/Feb/24 19:44

I'm testing BEGIN TRANSACTION...END TRANSACTION. The only examples I've seen use a single PRIMARY KEY.

I decided to test using a single PRIMARY KEY and an indexed column, which seems reasonable. I only get errors. Perhaps my syntax is incorrect, but I'm not sure how. 

When I make that indexed column part of the PRIMARY KEY, the transactions work just fine. 

I've attached the CQL I'm using and the runs I've made with errors.
Bug: Glossary lists UCS with definition of TWCS
Status: Triage Needed
Priority: Normal
Created: 09/Feb/24 01:40

The docs for UCS have the definition for TWCS.

[https://cassandra.apache.org/_/glossary.html#unifiedcompactionstrategy-ucs]

Also, the page mentions "DataStax" 4 times, including the title.  I believe this is just an artifact from porting the DS docs to OSS.
Bug: Range queries performed with NODE_LOCAL update read request metrics instead of range request metrics
Status: Triage Needed
Priority: Normal
Created: 08/Feb/24 10:52

Queries like {{SELECT * FROM foo.bar}} update regular read metrics when executed with CL.NODE_LOCAL instead of range metrics (what happens with other CL).

I understand that NODE_LOCAL is different, but I still think it's better to update range metrics for consistency's sake.
This was not fixed in -- CASSANDRA-17052{-}, but I assume this was an accidental omission, not a conscious decision (at least the ticket doesn't mention{-} that).
Actually, the test in CASSANDRA-17052 does explicitly expect bumping read metrics rather than range metrics.

[~aleksey] , [~ifesdjeen] could you please weigh in on that?
Bug: Startup Validation Failures when Checking Sidecar Connectivity (Analytics)
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 07/Feb/24 23:59
Resolved: 08/Feb/24 00:10

We have experienced repeated startup validation failures caused by Sidecar health checks for some jobs with a large number of Spark executors.
It is worth increasing the overall timeout for Sidecar health checks from current 30 seconds to 5 minutes and making it configurable.
Bug: Data loss during decommission possible due to a delayed and unsynced pending ranges calculation
Status: Triage Needed
Priority: Normal
Created: 05/Feb/24 10:59

This possible issue has been discovered while inspecting flaky tests of CASSANDRA-18824. Pending ranges calculation is executed asynchronously when the node is decommissioned. If the data is inserted during decommissioning, and pending ranges calculation is delayed for some reason (it can be as it is not synchronous), we may end up with partial data loss. That can be just a wrong test. Thus, I perceive this ticket more like a memo for further investigation or discussion. 

Note that this has obviously been fixed by TCM.

The test in question was:

{code:java}
        try (Cluster cluster = init(builder().withNodes(2)
                                             .withTokenSupplier(evenlyDistributedTokens(2))
                                             .withNodeIdTopology(NetworkTopology.singleDcNetworkTopology(2, "dc0", "rack0"))
                                             .withConfig(config -> config.with(NETWORK, GOSSIP))
                                             .start(), 1))
        {
            IInvokableInstance nodeToDecommission = cluster.get(1);
            IInvokableInstance nodeToRemainInCluster = cluster.get(2);

            // Start decomission on nodeToDecommission
            cluster.forEach(statusToDecommission(nodeToDecommission));
            logger.info("Decommissioning node {}", nodeToDecommission.broadcastAddress());

            // Add data to cluster while node is decomissioning
            int numRows = 100;
            cluster.schemaChange("CREATE TABLE IF NOT EXISTS " + KEYSPACE + ".tbl (pk int, ck int, v int, PRIMARY KEY (pk, ck))");
            insertData(cluster, 1, numRows, ConsistencyLevel.ONE); // <------------------- HERE - when PRC is delayed, we get there only ~50% of inserted rows

            // Check data before cleanup on nodeToRemainInCluster
            assertEquals(100, nodeToRemainInCluster.executeInternal("SELECT * FROM " + KEYSPACE + ".tbl").length);
    }
{code}
Bug: Not using min/max column name meta data from CASSANDRA-8180 when the LIMIT clause is used
Status: Open
Priority: Normal
Created: 30/Jan/24 21:37

{code:java}
CREATE TABLE tracking_details (
    tracker_public_id text,
    updated_at timeuuid,
    details text,
    PRIMARY KEY (tracker_public_id, updated_at)
) WITH CLUSTERING ORDER BY (updated_at DESC){code}
{code:java}
INSERT INTO tracking_details  (tracker_public_id, updated_at, details) VALUES('trk_e0544f077f7f4140bf1b961493fa99fe', now(), '{"details":[{...1-3KB }]}');{code}
{code:java}
nodetool flush trackingdata tracking_details{code}
{code:java}
INSERT INTO tracking_details  (tracker_public_id, updated_at, details) VALUES('trk_e0544f077f7f4140bf1b961493fa99fe', now(), '{"details":[{...1-3KB }]}');

INSERT INTO tracking_details  (tracker_public_id, updated_at, details) VALUES('trk_e0544f077f7f4140bf1b961493fa99fe', now(), '{"details":[{...1-3KB }]}');{code}
{code:java}
nodetool flush trackingdata tracking_details{code}
{code:java}
Compaction is disabled- 
 {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'enabled': 'false', 'max_threshold': '32', 'min_threshold': '4'}{code}
{code:java}
show version
[cqlsh 6.0.0 | Cassandra 4.0.5 | CQL spec 3.4.5 | Native protocol v5]{code}

_Each node is reading from both sstables - *Merged data from memtables and 2 sstables.*_ 
I thought this was fixed - https://issues.apache.org/jira/browse/CASSANDRA-8180_
_But somehow  C* doesn't optimizes the read path_

_Dev Cluster Info - 3 Nodes, RF is 3 and CL is LOCAL_QUORUM_

_Tracing attached_
Bug: Quickstart commands are slightly wrong
Status: Triage Needed
Priority: Normal
Created: 01/Feb/24 19:26

I'm new to Cassandra. I tried [the quickstart|https://cassandra.apache.org/_/quickstart.html] and noticed some things that I figured I'd fix.
 # The csqlsh versions are inconsistent between commands, resulting in the second command (step 5) failing.
 # The prompt message is slightly out of date, I figured I'd update it while I was here
 # In step 7, the insert forgets to include a timestamp. I assume that was accidental, so I fixed that too.
Bug: Accord: Never recover read-only transactions; simply invalidate
Status: Triage Needed
Priority: Normal
Created: 01/Feb/24 14:12

read-only transactions do not need to be recovered to supply client responses or for other transactions to make progress. The only situation that might require a read to be recovered is for recovery of a write transaction that needs to know whether the read might have witnessed or not-witnessed it at a specific `executeAt`. This can be special-cased, either to run recovery in this circumstance, or to simply compute the necessary recovery information to decide whether it is possible or not.
Bug: Accord: Harden Node.Id handling: graceful restart for left nodes, and ensure don’t cause problems with IP reuse
Status: Triage Needed
Priority: Normal
Created: 01/Feb/24 14:09

We rely on TCM for mapping node.id to replicas/IPs, but TCM is not Accord-epoch-aware, so it might erase a mapping before Accord is finished with it (and so, after a reboot Accord may not be able to find it again), but also might permit an IP to be re-used for a new Node.Id when Accord is still using it for an older epoch.
Bug: [Analytics] No longer need to synchronize on Schema.instance after Cassandra 4.0.12
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 31/Jan/24 22:34
Resolved: 31/Jan/24 23:12

We no longer need to synchronize on the {{Schema.instance}} in Analytics after the release of Cassandra 4.0.12, that includes a synchronization fix in https://issues.apache.org/jira/browse/CASSANDRA-18317. We need to clean up the TODOs pending on that code being released
Bug: [Analytics] Improve logging for bulk writes and on task failures
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/Jan/24 18:28
Resolved: 30/Jan/24 23:32

Builds on top of the resiliency changes to add additional logging around token-range metadata fetched from C* during bulk-writes, and also specifically for case when the spark task fails when the cluster is resized.
Bug: [Analytics] Upgrade to Cassandra 4.0.12 and remove RowBufferMode and BatchSize options
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Jan/24 19:53
Resolved: 31/Jan/24 18:54

In cassandra-all:4.0.12, improvements were made for the CQLSSTableWriter. The sorted writer now can produce size-capped SSTables. It replaces the need for the unsorted sstable writer, which has to buffer and sort data on flushing. The dataset to write in the spark application is already sorted. By avoiding using the unsorted writer, it prevents wasting CPU time on sorting the sorted data. Since the sorted sstable writer does not need to buffer data, its size estimation is more accurate than the unsorted one, meaning the produced sstables files are closer to the expectation.

By removing the unsorted sstable writer, it no longer requires the RowBufferMode option.
By supporting size-capping in sorted writer, it no longer requires the BatchSize option.
Bug: Repair can trigger the disk_failure_policy
Status: Open
Priority: Normal
Created: 26/Jan/24 13:47

While a cluster was running a repair, we observed Insufficient disk space would unexpectedly shut down gossip instead of skipping the repair and give the below error.  It would seem the right behavior would be to skip and go on to the next one, as nodetool garbagecollect and cleanup would do.  Instead the two nodes that were experiencing the error continued to stay in UP-PARTIAL state with no gossip running.

 
{quote}2024-01-20 18:37:50,716 [ERROR] [CompactionExecutor:2] cluster_id=9 ip_address=10.0.0.1 CassandraDaemon.java:581 - Exception in thread Thread[CompactionExecutor:2,1,main]
org.apache.cassandra.io.FSDiskFullWriteError: java.io.IOException: Insufficient disk space to write 625503268755 bytes into the foobar keyspace
    at org.apache.cassandra.db.Directories.getWriteableLocation(Directories.java:443)
    at org.apache.cassandra.db.Directories.getWriteableLocationAsFile(Directories.java:339)
    at org.apache.cassandra.db.compaction.CompactionManager.antiCompactGroup(CompactionManager.java:1556)
    at org.apache.cassandra.db.compaction.CompactionManager.doAntiCompaction(CompactionManager.java:1522)
    at org.apache.cassandra.db.compaction.CompactionManager.performAnticompaction(CompactionManager.java:788)
    at org.apache.cassandra.db.compaction.CompactionManager$9.runMayThrow(CompactionManager.java:702)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: Insufficient disk space to write 625503268755 bytes into the foobar keyspace
    at org.apache.cassandra.io.FSDiskFullWriteError.<init>(FSDiskFullWriteError.java:27)
    ... 13 common frames omitted
{quote}
Bug: Timeuuid compare is broken
Status: Resolved
Priority: Normal
Resolution: Invalid
Created: 31/Jan/24 15:35
Resolved: 31/Jan/24 16:21

{{I have stumbled over a wired problem on my pc.}}

{{When i turn on my wifi interface, then some of my integration test are failing.}}

{{The mac part(lsb) of the timeuuids become changed in our Uuid implementation.}}

{{These uuids are used for the cassandra insertions and queries.}}

 

{{TestSetup with "broken" Uuids:}}
{code:java}
CREATE TABLE object_comment (
    object timeuuid,
    comment timeuuid,
    value blob,
    PRIMARY KEY (object, comment)
)

INSERT INTO object_comment (object, comment , value) VALUES (95278adc-c03f-11ee-ab43-bb35e932d536, cf9e6440-c01e-11ee-847b-34cff6b1be80, 0x01);
INSERT INTO object_comment (object, comment , value) VALUES (95278adc-c03f-11ee-ab43-bb35e932d536, cf9f75b0-c01e-11ee-847b-34cff6b1be80, 0x02);

// cf9f75b0-c01e-11ee-847b-34cff6b1be7f is lsb-1 and the same timestamp
SELECT * FROM object_comment where object = 95278adc-c03f-11ee-ab43-bb35e932d536 AND comment <= cf9f75b0-c01e-11ee-847b-34cff6b1be7f; object                               | comment                              | value
--------------------------------------+--------------------------------------+-------
 95278adc-c03f-11ee-ab43-bb35e932d536 | cf9e6440-c01e-11ee-847b-34cff6b1be80 |  0x01
 95278adc-c03f-11ee-ab43-bb35e932d536 | cf9f75b0-c01e-11ee-847b-34cff6b1be80 |  0x02(2 rows)
 {code}
 

 

The second row must not be present. The Only row expected is : 
{code:java}
95278adc-c03f-11ee-ab43-bb35e932d536 | cf9e6440-c01e-11ee-847b-34cff6b1be80 |  0x01{code}
 

I think i have found the cause of the issue.

The Methods `org.apache.cassandra.utils.TimeUUID#compareTo` and `org.apache.cassandra.db.marshal.TimeUUIDType#compareCustom` return different results.

Test pseudocode:
{code:java}
var id = UUID.fromString("cf9f75b0-c01e-11ee-847b-34cff6b1be80");
var idDecrementInLsb = UUID.fromString("cf9f75b0-c01e-11ee-847b-34cff6b1be7f");

// java.util.UUID#compareTo
assertThat(idDecrementInLsb.compareTo(id)).isEqualTo(-1);

var timeUuidDec = org.apache.cassandra.utils.TimeUUID.fromUuid(idDecrementInLsb);
var timeUuidId = org.apache.cassandra.utils.TimeUUID.fromUuid(id);

// org.apache.cassandra.utils.TimeUUID#compareTo
assertThat(timeUuidDec.compareTo(timeUuidId)).isEqualTo(-1);

// org.apache.cassandra.db.marshal.TimeUUIDType.compareCustom
assertThat(org.apache.cassandra.db.marshal.TimeUUIDType.compareCustom(idDecrementInLsb, id1)).isEqualTo(-1); // This fails
 {code}
Bug: [Analytics] Fix range split and use open-closed range notation consistently
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 25/Jan/24 01:35
Resolved: 30/Jan/24 20:08

The ranges used in the analytics library do not have the consistent notation with Cassandra. The analytics library, as in the cassandra ecosystem, should use the open-closed range notation consistently, to avoid potential bugs in implementation.

Besides that, during write process, the split sub-ranges are unordered. It does not seem to affect correctness, but can be confusing.
Bug: Make system_views.thread_pools mutable for core_pool_size and max_pool_size
Status: Patch Available
Priority: Normal
Created: 29/Jan/24 15:47

In CASSANDRA-19289, we added core_pool_size, max_pool_size and max_tasks_queued to the output of system_views.thread_pools. From operational perspective, if an operator looks into this table, it would be very handy if she is able to change these pool parameters directly on that table instead of setting that up via JMX and respective attributes / setters, to avoid context switching / not leaving shell. It would look like:

{noformat}
update system_views.thread_pools SET core_pool_size = 4, max_pool_size = 4 WHERE name = 'SecondaryIndexExecutor';
{noformat}

From the implementation point of view, this is possible to do as we have AbstractMutableVirtualTable however we would probably need to introduce some simple register for created thread pools as vtable is currently based on metrics through which executors themselves are not accessible to call setCorePoolSize or similar.
Bug: Expose core_pool_size, max_pool_size and max_tasks_queued in nodetool tpstats / system_views.thread_pools
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 23/Jan/24 16:16
Resolved: 29/Jan/24 15:21

It would be nice to expose this information so it is visible from cql(sh) / nodetool tpstats. I briefly checked the complexity of this and it should be quite easy, just add few columns in both and put the sizes to metrics so we can read that.

Without this, I think the only way to know the sizes is to read it from JMX. I just prefer some more handy and faster way to check the sizes so I can adjust them accordingly when necessary.