av9ash/GitBugs huggingface dataset:
Bug: Joining nodes simultaneously with auto_bootstrap:false can cause token collision
Status: Open
Priority: Normal
Created: 19/Dec/20 22:53

While raising a 6-node ccm cluster to test 4.0-beta4, 2 nodes chosen the same tokens using the default {{allocate_tokens_for_local_rf}}. However they both succeeded bootstrap with colliding tokens.

We were familiar with this issue from CASSANDRA-13701 and CASSANDRA-16079, and the workaround to fix this is to avoid parallel bootstrap when using {{allocate_tokens_for_local_rf}}.

However, since this is the default behavior, we should try to detect and prevent this situation when possible, since it can break users relying on parallel bootstrap behavior.

I think we could prevent this as following:
1. announce intent to bootstrap via gossip (ie. add node on gossip without token information)
2. wait for gossip to settle for a longer period (ie. ring delay)
3. allocate tokens (if multiple bootstrap attempts are detected, tie break via node-id)
4. broadcast tokens and move on with bootstrap
Bug: Fix architecture/storage_engine documentation format
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 22/Dec/20 17:24
Resolved: 04/Jan/24 01:34

While reading the docs for cassandra I've took the time to fix and improve the formatting used in [https://cassandra.apache.org/doc/latest/architecture/storage_engine.html.]

It had inconsistent indentation for the options and their default values.

Formatting of the *NOTE* sections was also made consistent, removing extra {{*}} characters.

And other small typos on architecure/dynamo and architecture/overview.
Bug: Make TokenMetadata's ring version increments atomic
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Nov/20 21:18
Resolved: 01/Feb/21 17:47

The update semantics of the ring version in {{TokenMetadata}} are not clear. The instance variable itself is {{volatile}}, but it is still incremented by a non-atomic check-and-set, and not all codepaths do that while holding the {{TokenMetadata}} write lock. We could make this more intelligible by forcing the external callers to use both the write when invalidating the ring and read lock when reading the current ring version. Most of the readers of the ring version (ex. compaction) don't need it to be fast, but it shouldn't be a problem even if they do. If we do this, we should be able to avoid a situation where concurrent invalidations don't produce two distinct version increments.
Bug: Fix flaky test test_disk_balance_after_boundary_change_lcs - disk_balance_test.TestDiskBalance
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 06/Oct/20 22:16
Resolved: 26/Oct/20 04:23

https://app.circleci.com/pipelines/github/dcapwell/cassandra/622/workflows/adcd463c-156a-43c7-a9bc-7f3e4938dbe8/jobs/3514

{code}
        error_message = '' if 'error_message' not in kwargs else kwargs['error_message']
        assert vmin > vmax * (1.0 - error) or vmin == vmax, \
>           "values not within {:.2f}% of the max: {} ({})".format(error * 100, args, error_message)
E       AssertionError: values not within 10.00% of the max: (8022760, 9192165, 4575645, 9235566, 9091014) (node2)

tools/assertions.py:206: AssertionError
{code}

Marking as distinct issue after chat in CASSANDRA-14030
Bug: CRC32 is inefficient on x86
Status: Patch Available
Priority: Normal
Created: 17/Dec/20 10:19

The client/server protocol specifies CRC24 and CRC32 as the checksum algorithm (cql_protocol_V5_framing.asc). Those however are expensive to compute; this affects both the client and the server.

 

A better checksum algorithm is CRC32C, which has hardware support on x86 (as well as other modern architectures).
Bug: Cassandra version above 3.11.0 failing for ARM64 
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Oct/20 12:22
Resolved: 16/Nov/20 09:00

Cassandra versions above 3.11.0 are failing on ARM64 platform with below issue:

java.lang.UnsatisfiedLinkError: /tmp/jna-3506402/jna3214742498288082263.tmp: *Error loading shared library ld-linux-aarch64.so.1: No such file or directory (needed by /tmp/jna-3506402/jna3214742498288082263.tmp)*
Bug: [jamm] Consider implementing ClusteringComparator without a lambda
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 27/Nov/20 02:30
Resolved: 26/Jul/23 18:29

Using lambdas forces jamm to do things that can easily break. It might be safer to implement things like ClusteringComparator directly as classes or as an enum

{noformat}
Unexpected exception during request (org.apache.cassandra.transport.messages.ErrorMessage)
java.lang.UnsupportedOperationException: can't get field offset on a hidden class: private final org.apache.cassandra.db.ClusteringComparator org.apache.cassandra.db.ClusteringComparator$$Lambda$165/0x000000010028ab60.arg$1
	at jdk.unsupported/sun.misc.Unsafe.objectFieldOffset(Unknown Source)
	at org.github.jamm.MemoryLayoutSpecification.sizeOfInstanceWithUnsafe(MemoryLayoutSpecification.java:108)
	at org.github.jamm.MemoryLayoutSpecification.sizeOfWithUnsafe(MemoryLayoutSpecification.java:89)
	at org.github.jamm.MemoryMeter.measure(MemoryMeter.java:217)
	at org.github.jamm.MemoryMeter.measureDeep(MemoryMeter.java:259)
	at org.apache.cassandra.utils.ObjectSizes.measureDeep(ObjectSizes.java:155)
	at org.apache.cassandra.cql3.QueryProcessor.storePreparedStatement(QueryProcessor.java:454)
	at org.apache.cassandra.cql3.QueryProcessor.prepare(QueryProcessor.java:424)
	at org.apache.cassandra.cql3.QueryProcessor.prepare(QueryProcessor.java:408)
	at org.apache.cassandra.transport.messages.PrepareMessage.execute(PrepareMessage.java:114)
	at org.apache.cassandra.transport.Message$Request.execute(Message.java:253)
	at org.apache.cassandra.transport.Message$Dispatcher.processRequest(Message.java:725)
	at org.apache.cassandra.transport.Message$Dispatcher.lambda$channelRead0$0(Message.java:630)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:119)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Unknown Source)
{noformat}
Bug: Consistency can be violated when bootstrap or decommission is resumed after node restart
Status: Open
Priority: Normal
Created: 19/Nov/20 18:58

Since CASSANDRA-12008, successfully transferred ranges during decommission are saved on the {{system.transferred_ranges}} table. This allow skipping ranges already transferred when a failed decommission is retried with {{nodetool decommission}}.

If instead of resuming the decommission, an operator restarts the node, waits N minutes and then performs a new decommission, the previously transferred ranges will be skipped during streaming, and any writes received by the decommissioned node during these N minutes will not be replicated to the new range owner, what violates consistency.

This issue is analogous to the issue mentioned [on this comment|https://issues.apache.org/jira/browse/CASSANDRA-8838?focusedCommentId=16900234&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16900234] for resumable bootstrap (CASSANDRA-8838).

In order to prevent consistency violations we should clear the {{system.transferred_ranges}} state during node restart, and maybe a system property to disable it. While we're at this, we should change the default of {{-Dcassandra.reset_bootstrap_progress}} to {{true}} to clear the {{system.available_ranges}} state by default when a bootstrapping node is restarted.
Bug: cqlsh help command displays a broken url
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 11/Dec/20 18:37
Resolved: 15/Aug/23 07:12

When running cqlsh the help command outputs a web page which returns 404
{code}
cqlsh> help create_table
*** No browser to display CQL help. URL for help topic create_table : https://cassandra.apache.org/doc/cql3/CQL-3.2.html#createTableStmt
{code}
Bug: Unexpected warning "Ignoring Unrecognized strategy option" for NetworkTopologyStrategy when restarting
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 05/Nov/20 21:08
Resolved: 13/Nov/20 13:57

During restarting, bunch of warning messages like "AbstractReplicationStrategy.java:364 - Ignoring Unrecognized strategy option {datacenter2} passed to NetworkTopologyStrategy for keyspace distributed_test_keyspace" are logged. 
The warnings are not expected since the mentioned DC exist. 
It seems to be caused by the improper order during startup, so that when opening keyspaces it is unaware of DCs. 

The warning can be reproduced using the test below. 

{code:java}
@Test
public void testEmitsWarningsForNetworkTopologyStategyConfigOnRestart() throws Exception {
    int nodesPerDc = 2;
    try (Cluster cluster = builder().withConfig(c -> c.with(GOSSIP, NETWORK))
                                    .withRacks(2, 1, nodesPerDc)
                                    .start()) {
        cluster.schemaChange("CREATE KEYSPACE " + KEYSPACE +
                             " WITH replication = {'class': 'NetworkTopologyStrategy', " +
                             "'datacenter1' : " + nodesPerDc + ", 'datacenter2' : " + nodesPerDc + " };");
        cluster.get(2).nodetool("flush");
        System.out.println("Stop node 2 in datacenter 1");
        cluster.get(2).shutdown().get();
        System.out.println("Start node 2 in datacenter 1");
        cluster.get(2).startup();
        List<String> result = cluster.get(2).logs().grep("Ignoring Unrecognized strategy option \\{datacenter2\\}").getResult();
        Assert.assertFalse(result.isEmpty());
    }
}
{code}
Bug: CEP-28: Reading and Writing Cassandra Data with Spark Bulk Analytics
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 22/Oct/20 16:46
Resolved: 19/May/23 22:44

*Description:*
This ticket introduces the Spark-Cassandra Bulk Analytics library and associated Cassandra Sidecar endpoints.

 

Please see [CEP-28|https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-28%3A+Reading+and+Writing+Cassandra+Data+with+Spark+Bulk+Analytics] for more details, motivation, and available source code links.
Bug: materialized view can return stale data even with R+W>N
Status: Triage Needed
Priority: Normal
Created: 02/Oct/20 15:12

This is a similar issue as CASSANDRA-8272.

With a key-value table on a 2-node cluster with RF2 and there is a MV that put `value` as partition key.

Insert with ConsistencyLevel.ONE twice with different data, assuming there is a network partition between 2 nodes:
- Node 1 received: pk="a" -> value=1 @ts1
- Node 2 received: pk="a" -> value=2 @ts2 where ts2 > ts1

When querying `value`=1 on MV with ConsistencyLevel.ALL, we got stale row: "a" -> 1.
Bug: batch_size_fail_threshold_in_kb does not account for LWTs
Status: Open
Priority: Normal
Created: 28/Oct/20 14:53

Subject probably says it all, but basically there's a batch_size_warn_threshold_in_kb and batch_size_fail_threshold_in_kb in the config but these don't apply to LWTs.

So of course I found a situation in which devs were doing LWT operations of many megabytes :D

Can we make it possible to forbid these, the same way it can be done for batches?
Bug: ViewWriteMetrics do not use ClientRequestMetrics members
Status: Open
Priority: Normal
Created: 10/Nov/20 18:23

Observed while adding ClientRequestMetrics for CASSANDRA-16183

While ViewWriteMetrics inherits from ClientRequestMetrics, none of the additional members from that class are used. 

We should figure out if those should be marked somehow, or if it would it be worth deprecating ViewWriteMetrics.timeouts, etc and making that class derive directly from LatencyMetrics in a later release.

[additional discussion in comments|https://issues.apache.org/jira/browse/CASSANDRA-16183?focusedCommentId=17229398&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17229398]
Bug: During bootstrap streaming, skipping write path may create orphan MV rows 
Status: Triage Needed
Priority: Normal
Created: 02/Oct/20 15:20

CASSANDRA-13065 improves the speed of bootstrap streaming by skipping write path for base table with MV.

Unfortunately during bootstrapping, the bootstrapping node may receive a base write that is already deleted or shadowed on other replicas, but the bootstrapping node didn't receive all sstables yet thus bootstapping node may create an orphan view row based on received sstables. Without write path, the newer version data will not create tombstone to shadow orphan view row.

For example, node-A has a base sstable containing: "k=1, v=1@2019", but bootstrapping node didn't receive it yet. A write "k=1, v=0@2018" hitting on bootstrapping node will create an orphan view row: "v=0@2018, k=1". Applying streaming sstables from bootstrap or rebuild through write path can help, but that's something we want to get rid of.
Bug: Minimal 4.0 COMPACT STORAGE backport
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/Oct/20 16:54
Resolved: 08/Dec/20 07:25

There are several behavioural changes related to compact storage, and these differences are larger than most of us have anticipated: we first thought there’ll be that “appearing column”, but there’s implicit nulls in clusterings thing, and row vs column deletion.

Some of the recent issues on the subject are: CASSANDRA-16048, which allows to ignore these differences. The other one was trying to improve user experience of anyone still using compact storage: CASSANDRA-15811.

Easily reproducible differernces are:

(1) hidden columns show up, which breaks SELECT * queries
 (2) DELETE v and UPDATE v WITH TTL would result into row removals in non-dense compact tables (CASSANDRA-16069)
 (3) INSERT allows skipping clusterings, which are filled with nulls by default.

Some of these are tricky to support, as 15811 has shown. Anyone on OSS side who might want to upgrade to 4.0 while still using compact storage might be affected by being forced into one of these behaviours.

Possible solutions are to document these behaviours, or to bring back a minimal set of COMPACT STORAGE to keep supporting these.

It looks like it is possible to leave some of the functionality related to DENSE flag and allow it to be present in 4.0, but only for these three (and potential related, however not direrclty visible) cases.

[~e.dimitrova] since you were working on removal on compact storage, wanted to reassure that this is not a revert of your patch. On contrary: your patch was instrumental in identifying the right places.

cc [~slebresne] [~aleksey] [~benedict] [~marcuse]

|[patch|https://github.com/apache/cassandra/pull/785]|[ci|https://app.circleci.com/pipelines/github/ifesdjeen/cassandra?branch=13994-followup]|
Bug: Remove Windows scripts
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/Oct/20 15:08
Resolved: 02/Dec/20 16:52

As per the email thread in cassandra-dev mailing list[1], remove windows scripts from Cassandra 4.0 onwards, due to the lack of maintenance and tests.

1: https://www.mail-archive.com/dev@cassandra.apache.org/msg15583.html
Bug: 4.0 Quality: Coordination & Replication Fuzz Testing
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 10/Nov/20 23:07
Resolved: 16/Feb/22 08:02

CASSANDRA-16180, CASSANDRA-16181, and CASSANDRA-15977 have largely focused on auditing the existing tests around coordination, replication, and read-repair, respectively. We've expanded existing test cases, added coverage around components that we've refactored along the way, and added in-JVM dtest upgrade tests where possible.

What remains is verifying the distributed read and write paths in the face of common operational events, namely node restarts, bootstrapping, decommission, and cleanup. If we can find a way to simulate these events, [Harry|https://github.com/apache/cassandra-harry] seems like a good candidate to host the verification logic itself.

To keep things simple initially, I would propose that we start by testing simple read-only and write-only workloads (the former without read repair).
Bug: JaCoCo reports on all unit test jobs on ci-cassandra.a.o
Status: Open
Priority: Normal
Created: 24/Dec/20 13:35

Enable the  jenkins [jacoco plugin|https://plugins.jenkins.io/jacoco/] and wrap all test runs with jacoco.

Patch at https://github.com/apache/cassandra-builds/compare/trunk...thelastpickle:mck/tests_jacoco

Example results are:
 - CI build: https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch-test/149/
 - CI JaCoCo report: https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch-test/149/jdk=jdk_1.8_latest,label=cassandra/jacoco/
Bug: Add Unavailables metric for CASWrite in the docs
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 16/Dec/20 10:57
Resolved: 19/Oct/22 18:03

"Unavailables" is a metric that is available for all types of operations:
 * CASRead, Read, CASWrite, Write

In the documentation, the Unavailables for CASWrite and it should be added.

[https://cassandra.apache.org/doc/latest/operating/metrics.html]
Bug: Replica failure causes timeout on multi-DC write
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 11/Dec/20 11:49
Resolved: 26/Oct/21 17:33

Inserting a mutation larger than {{max_mutation_size_in_kb}} correctly throws a write error on a single DC keyspace with RF=3:
{noformat}
cassandra.WriteFailure: Error from server: code=1500 [Replica(s) failed to execute write] message="Operation failed - received 0 responses and 3 failures: UNKNOWN from /127.0.0.3:7000, UNKNOWN from /127.0.0.2:7000, UNKNOWN from /127.0.0.1:7000" info={'consistency': 'LOCAL_ONE', 'required_responses': 1, 'received_responses': 0, 'failures': 3}
{noformat}
The same insert wrongly causes a timeout on a keyspace with 2 dcs (RF=3 each):
{noformat}
cassandra.WriteTimeout: Error from server: code=1100 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'consistency': 'LOCAL_ONE', 'required_responses': 1, 'received_responses': 0}
{noformat}
Reproduction steps:
{noformat}
# Setup cluster
ccm create -n 3:3 test
for i in {1..6}; do echo 'max_mutation_size_in_kb: 1000' >> ~/.ccm/test/node$i/conf/cassandra.yaml; done
ccm start

# Create schema
ccm node1 cqlsh
CREATE KEYSPACE test WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': 3, 'dc2': 3};
CREATE TABLE test.test (key int PRIMARY KEY, val blob);
exit;

# Insert data
python
from cassandra.cluster import Cluster
cluster = Cluster()
session = cluster.connect('test')
blob = f = open("2mbBlob", "rb").read().hex()
session.execute("INSERT INTO test (key, val) VALUES (1, textAsBlob('" + blob + "'))")
{noformat}
Reproduced in 3.0, 3.11, 4.0, trunk.
Bug: Avoid removing batch when it's not created during view replication
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/Oct/20 17:46
Resolved: 07/Sep/21 15:02

When the base replica is also a view replica we don't write a local batchlog, but they are unnecessarily removed when the view write is successful, what creates (and persists) a tombstone into the system.batches table.
Bug: DOC - Incorrect link for DevCenter in Data Modeling Tools page
Status: In Progress
Priority: Normal
Created: 08/Oct/20 21:53

The link to download DevCenter in the [Data Modeling Tools|https://cassandra.apache.org/doc/latest/data_modeling/data_modeling_tools.html] page is incorrect:

bq. DataStax DevCenter is a tool for managing schema, executing queries and viewing results. While the tool is no longer actively supported, it is still popular with many developers and is available as a [free download|https://academy.datastax.com/downloads].
Bug: Improve performance when calculating StreamTasks with optimised streaming
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/Nov/20 10:57
Resolved: 01/Dec/20 11:42

The way stream tasks are calculated currently is quite inefficient, improve that.

Also, we currently try to distribute the streaming nodes evenly, this creates many more sstables than necessary - instead we should try to stream everything from a single peer, this should reduce the number of sstables created on the out-of-sync node.
Bug: Improve compaction param "provide_overlapping_tombstones" handling
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 14/Dec/20 12:28
Resolved: 13/Apr/21 08:56

We currently have no cqlsh autocompletion for {{provide_overlapping_tombstones}}. We should also improve the validation for the parameter given as it is currently quite unhelpful:

{code}
cqlsh:x> create table z (id int primary key, x int) with compaction = {'class': 'SizeTieredCompactionStrategy', 'provide_overlapping_tombstones':'xyz'};
NoHostAvailable:
{code}

and an exception in the logs;
{code}
ERROR [Native-Transport-Requests-1] 2020-12-14 13:25:46,575 QueryMessage.java:121 - Unexpected error during query
java.lang.IllegalArgumentException: No enum constant org.apache.cassandra.schema.CompactionParams.TombstoneOption.XYZ
        at java.lang.Enum.valueOf(Enum.java:238)
        at org.apache.cassandra.schema.CompactionParams$TombstoneOption.valueOf(CompactionParams.java:59)
        at org.apache.cassandra.schema.CompactionParams.create(CompactionParams.java:98)
        at org.apache.cassandra.schema.CompactionParams.fromMap(CompactionParams.java:255)
        at org.apache.cassandra.cql3.statements.schema.TableAttributes.build(TableAttributes.java:98)
        at org.apache.cassandra.cql3.statements.schema.TableAttributes.validate(TableAttributes.java:58)
        at org.apache.cassandra.cql3.statements.schema.CreateTableStatement.builder(CreateTableStatement.java:145)
        at org.apache.cassandra.cql3.statements.schema.CreateTableStatement.apply(CreateTableStatement.java:104)
        at org.apache.cassandra.schema.Schema.transform(Schema.java:587)
        at org.apache.cassandra.schema.MigrationManager.lambda$announce$2(MigrationManager.java:221)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)

{code}
Bug: Fix flaky test incompletePropose - org.apache.cassandra.distributed.test.CASTest
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Dec/20 21:40
Resolved: 17/Feb/21 13:20

https://app.circleci.com/pipelines/github/dcapwell/cassandra/853/workflows/0766c0de-956e-4831-aa40-9303748a2708/jobs/5030

{code}
junit.framework.AssertionFailedError: Expected: [[1, 1, 2]]
Actual: []

	at org.apache.cassandra.distributed.shared.AssertUtils.fail(AssertUtils.java:193)
	at org.apache.cassandra.distributed.shared.AssertUtils.assertEquals(AssertUtils.java:163)
	at org.apache.cassandra.distributed.shared.AssertUtils.assertRows(AssertUtils.java:63)
	at org.apache.cassandra.distributed.test.CASTest.incompletePropose(CASTest.java:124)
{code}