av9ash/GitBugs huggingface dataset:
Bug: Migrate dependency handling from maven-ant-tasks to resolver-ant-tasks
Status: Resolved
Priority: High
Resolution: Fixed
Created: 16/Jan/21 20:53
Resolved: 02/Apr/21 08:28

Cassandra resolves dependencies and generates maven pom files through the use of [maven-ant-tasks|http://maven.apache.org/ant-tasks/]. This is no longer a supported project.

The recommended upgrade is to [resolver-ant-tasks|http://maven.apache.org/resolver-ant-tasks/]. It follows similar APIs so shouldn't be too impactful a change.

The existing maven-ant-tasks has caused [some headaches already|https://issues.apache.org/jira/browse/CASSANDRA-16359] with internal super poms referencing insecure http:// central maven repository URLs that are no longer supported.

We should also take the opportunity to 
 - define the "test" scope (classpath) for those dependencies only used for tests (currently we are packaging test dependencies into the release binary artefact),
 - remove the jar files stored in the git repo under the "lib/" folder.

These two above points have to happen in tandem, as the jar files under {{lib/}} are those that get bundled into the {{build/dist/lib/}} and hence the binary artefact. That is, all jar files under {{lib/}} are the project's "compile" scope, and all other dependencies defined in build.xml are either "provided" or "test" scope. These different scopes for dependencies are currently configured in different maven-ant-tasks poms. See https://github.com/apache/cassandra/commit/d43b9ce5092f8879a1a66afebab74d86e9e127fb#r45659668
Bug: BLOG - Post about Cassandra & Kubernetes - SIG Update #2
Status: Resolved
Priority: High
Resolution: Implemented
Created: 24/Mar/21 22:42
Resolved: 13/Jul/21 12:55

We're getting another update up for the Cassandra & Kubernetes SIG 

 

[https://github.com/Anant/cassandra-website/blob/master/src/_posts/2021-03-26-cassandra-and-kubernetes-sig-update.markdown]
Bug: high and critical CVEs io.netty to 4.1.42.Final to fix critical and high vulnerabilities
Status: Resolved
Priority: High
Resolution: Invalid
Created: 19/Feb/21 21:03
Resolved: 23/Feb/21 15:53

Repository | Tag | CVE ID | Severity | Packages | Source Package | Package Version | Fix Status
-- | -- | -- | -- | -- | -- | -- | --
datastax/cassandra-mgmtapi-3_11_7 | v0.1.22 | CVE-2019-20445 | critical | io.netty_netty-all |   | 4.0.44.Final | fixed in 4.1.44
datastax/cassandra-mgmtapi-3_11_7 | v0.1.22 | CVE-2019-20444 | critical | io.netty_netty-all |   | 4.0.44.Final | fixed in 4.1.44
datastax/cassandra-mgmtapi-3_11_7 | v0.1.22 | CVE-2019-16869 | high | io.netty_netty-all |   | 4.0.44.Final | fixed in 4.1.42.Final
Bug: Group By breaks range tombstone closer
Status: Open
Priority: Normal
Created: 08/Feb/21 17:26

This is caused by the same piece of GROUP BY code that has caused CASSANDRA-16307: counting. Since we have to look one row ahead in order to “finish” the group, we’ll check out the next row but will return null to transformations (including RT closer one). RT Closer will not see the row and, since the iterator is considered done, it’ll complain that that’s not right.

Similar result can be achieved without paging, but with LIMIT 1.

Minimal repro:
{code:java}
        try (Cluster cluster = init(builder().withNodes(1).start()))
        {
            cluster.schemaChange("CREATE TABLE distributed_test_keyspace.table_5 " +
                                 "(pk0 bigint,pk1 bigint,ck0 bigint,ck1 bigint, PRIMARY KEY ((pk0,pk1), ck0, ck1)) " +
                                 "WITH  CLUSTERING ORDER BY (ck0 ASC,ck1 ASC);");
            long pk1 = 1;
            long pk2 = 1;
            cluster.coordinator(1).execute("DELETE FROM distributed_test_keyspace.table_5 USING TIMESTAMP 100000 WHERE pk0=? AND pk1=? AND ck0>?;", ConsistencyLevel.QUORUM, pk1, pk2, 2L);
            cluster.coordinator(1).execute("INSERT INTO distributed_test_keyspace.table_5 (pk0,pk1,ck0,ck1) VALUES (?,?,?,?) USING TIMESTAMP 100001;", ConsistencyLevel.QUORUM, pk1, pk2, 1L, 1L);
            cluster.coordinator(1).execute("INSERT INTO distributed_test_keyspace.table_5 (pk0,pk1,ck0,ck1) VALUES (?,?,?,?) USING TIMESTAMP 100001;", ConsistencyLevel.QUORUM, pk1, pk2, 3L, 1L);
            cluster.coordinator(1).executeWithPaging("SELECT pk0,pk1,ck0,ck1 FROM distributed_test_keyspace.table_5 WHERE pk0=? AND pk1=? GROUP BY pk0,pk1,ck0;",
                                                     ConsistencyLevel.QUORUM,
                                                     1,
                                                     pk1, pk2);
        }
{code}
Bug: Cluster topology change may produce false unavailable for queries
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Mar/21 23:09
Resolved: 15/Apr/21 15:56

When the coordinator processes a query, it first gets the {{ReplicationStrategy}} (RS) from the keyspace to decide the peers to contact. Again, it gets the RS to perform the liveness check for the requested CL. 

The RS is a volatile filed in Keyspace, and it is possible that those 2 getter calls return different RS values in the presence of cluster topology changes, e.g. add a node, etc. 

In such scenario, the check at the second step can throw an unexpected unavailable. From the perspective of the query, the cluster can satisfy the CL. 

We should use a consistent view of RS during the peer selection and CL liveness check. In other word, both steps should reference to the same RS object. It is also more clear and easier to reason about to the clients. Such queries are made before the topology change.
Bug: Python dtests to use ccm tag instead of the `cassandra-test` branch
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/Jan/21 16:21
Resolved: 14/Jan/21 18:44

The python dtests (cassandra-dtest repo) creates its clusters using ccm.

The version of ccm it uses is the HEAD of the {{cassandra-test}} branch.  This is referenced in the [requirements.txt|https://github.com/apache/cassandra-dtest/blob/trunk/requirements.txt#L3].

The history for why a separate branch of ccm is used by dtests is explained in https://github.com/apache/cassandra-dtest/pull/13 

Long story short: the separate branch avoids two headaches
- the 'latest commit to master' broke all the c* builds surprises', and 
- the 'i have to cut a release of ccm just to get a ccm change into use by dtests'

But the two branches: {{master}} and {{cassandra-test}}; have effectively been treated as two separate repositories, with (non-fast-forward) merges happening in both directions. This makes the git history of both branches messy and difficult to use, and it makes the merge strategy confusing. Bi-directional merging between branches is considered a poor practice by many (Laura Wingerd's 'The Flow of Change' [presentation|https://www.perforce.com/sites/default/files/flow-change-wingerd.pdf] and [book|https://books.google.no/books?id=mlm61wb2v3MC&pg=PT191&lpg=PT191&dq=%22don%27t+drive+through+hedges%22&source=bl&ots=I_rYBRJwTD&sig=ACfU3U1iKLORDQii5uiTveaKPOpa3cFqng&hl=en&sa=X&ved=2ahUKEwju96-DpZnuAhWytYsKHeW6D5EQ6AEwAHoECAEQAg#v=onepage&q=%22don't%20drive%20through%20hedges%22&f=false] refers to this as "don't drive through hedges" and encourages the "merge down, copy up" approach against the "tofu scale: firm above, soft below"). 

To date, AFAIK no merges between the branches have occurred since January 2018.

A possible improvement to this process is to replace the {{cassandra-test}} branch with a floating tag (of the same name).

That way new commits to {{master}} are not automatically used by the python dtests. And changes made to ccm and intended/needed to be used by the dtests can be put in use by re-tagging {{cassandra-test}} to master's HEAD. 

The re-tagging approach is
{code}
git tag -a -f cassandra-test
git push origin :refs/tags/cassandra-test
git push origin --tags
{code}
Bug: Upgrade to logback package 1.2.0 or later fix high vulnerabilities
Status: Resolved
Priority: Normal
Resolution: Won't Fix
Created: 19/Feb/21 21:18
Resolved: 01/Mar/21 15:06

Tag | Distro | CVE ID | Severity | Packages | Source Package | Fix Package Version
-- | -- | -- | -- | -- | -- | --
v0.1.22 | Ubuntu-bionic | CVE-2017-5929 | critical | ch.qos.logback_logback-core | 1.1.3 | fixed in 1.2.0
Bug: Unsafe to run nodetool cleanup during bootstrap or decommission
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Feb/21 09:15
Resolved: 23/Jan/23 22:45

What we expected: Running a cleanup is a safe operation; the result of running a query after a cleanup should be the same as the result of running a query before a cleanup.

What actually happened: We ran a cleanup during a decommission. All the streamed data was silently deleted, the bootstrap did not fail, the cluster's data after the decommission was very different to the state before.

Why: Cleanups do not take into account pending ranges and so the cleanup thought that all the data that had just been streamed was redundant and so deleted it. We think that this is symmetric with bootstraps, though have not verified.

Not sure if this is technically a bug but it was very surprising (and seemingly undocumented) behaviour.
Bug: concurrent.LongSharedExecutorPoolTest.testPromptnessOfExecution broken
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 05/Mar/21 22:25
Resolved: 22/Mar/21 16:23

{noformat}
junit.framework.AssertionFailedError
	at org.apache.cassandra.concurrent.LongSharedExecutorPoolTest.testPromptnessOfExecution(LongSharedExecutorPoolTest.java:169)
	at org.apache.cassandra.concurrent.LongSharedExecutorPoolTest.testPromptnessOfExecution(LongSharedExecutorPoolTest.java:102)
{noformat}

https://ci-cassandra.apache.org/job/Cassandra-trunk/308/testReport/org.apache.cassandra.concurrent/LongSharedExecutorPoolTest/testPromptnessOfExecution/
Bug: DataOutputBuffer#scratchBuffer can use off-heap or on-heap memory as a means to control memory allocations
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 25/Feb/21 23:45
Resolved: 13/Aug/22 03:04

While running workflows to compare 3.0 with trunk we found that allocations and GC are significantly higher for a write mostly workload (22% read, 3% delete, 75% write); below is what we saw for a 2h run

Allocations
30: 1.64TB
40: 2.99TB

GC Events
30: 7.39k events
40: 13.93k events

When looking at the allocation output we saw the follow for memory allocations

!https://issues.apache.org/jira/secure/attachment/13021238/Screen%20Shot%202021-02-25%20at%203.34.28%20PM.png!

Here we see that org.apache.cassandra.io.util.DataOutputBuffer#expandToFit is around 52% of the memory allocations.  When looking at this logic I see that allocations are on-heap and constantly throw away the buffer (as a means to allow GC to clean up).

With the patch, allocations/gc are the following

Allocations
30: 1.64TB
40 w/ patch: 1.77TB
40: 2.99TB

GC Events
30: 7.39k events
40 w/ patch: 8k events
40: 13.93k events


With the patch only 0.8% allocations

!https://issues.apache.org/jira/secure/attachment/13021239/Screen%20Shot%202021-02-25%20at%204.14.19%20PM.png!
Bug: nodetool bootstrap resume returns success even if there is an error during bootstrap
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 05/Mar/21 18:50
Resolved: 28/Oct/22 22:45

"nodetool bootstrap resume” prints a relevant error message if the operation fails, but it then proceeds to return a normal error code. It ignores the ProgressEventType.ERROR message that arrives before COMPLETE. This also happens when we handle failed connections. BootstrapMonitor should at least track whether or not it has seen an error, which would allow us to throw when one occurs after signaling, and therefore to inform callers.

Trunk PR: [https://github.com/apache/cassandra/pull/1949]
Bug: CMS is deprecated
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 08/Mar/21 17:42
Resolved: 08/May/23 17:51

When running under Java 11, you will see:

{quote}
[junit-timeout] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
{quote}

Per the JEP: http://openjdk.java.net/jeps/291 " The G1 garbage collector is intended, in the long term, to be a replacement for most uses of CMS."

We don't need to act on this immediately at the time of writing this ticket, but it is something that will eventually happen so we should get ahead of it.
Bug: Remove ViewMutationStage as its unused
Status: Resolved
Priority: Normal
Resolution: Won't Fix
Created: 31/Mar/21 01:05
Resolved: 12/Apr/23 16:32

I been testing and tracing the code and as far as I can determine ViewMutationStage is unused. Looking at StorageProxy::mutateMV if the MV mutation is on same endpoint as the base table endpoint it calls mutation.apply(writeCommitLog); which happens on MutationStage.

If the endpoint is a different endpoint it ends with call to asyncWriteBatchedMutations(wrappers, localDataCenter, Stage.VIEW_MUTATION); which in turn calls sendToHintedReplicas. But this method only uses the passed in stage if executing locally, which in case of the MV mutations is never true in my testing.

So it passes message with MUTATION_REQ which gets handled on the receiving node via MutationStage.

So at no point does the ViewMutationStage get used. I even tried to execute some of the View tests that check for pending tasks on this stage. Well there never even a completed task on that stage so these tests seem to wait on ViewMutationStage for no reason as best I can tell.

Even if the stage is used somewhere, its not used for all view mutations. So this stage is currently in my opinion code bloat and just confusing for the system operator.

 

An alternative solution to this ticket would be to implement VIEW_MUTATION_REQ etc and have everything MV mutation related run on the ViewMutationStage but that no easy change to make. See CounterMutation for example.
Bug: Track the duration of setting up components during Cassandra startup
Status: Triage Needed
Priority: Normal
Created: 15/Feb/21 19:13

We should track the duration during startup in order to quantitatively measure how fast/slow each step is doing and to provide a better insight in order to make decisions about potential improvements. 
The timings, e.g. time to load sstables, time to finish bootstrapping, time to establish ssl connections, etc., can be stored in the system table.
Bug: Add ability to ttl snapshots
Status: In Progress
Priority: Normal
Created: 15/Feb/21 21:22

It should be possible to add a TTL to snapshots, after which it automatically cleans itself up.

This will be useful together with the {{auto_snapshot}} option, where you want to keep an emergency snapshot in case of accidental drop or truncation but automatically remove it after a specified period when it's no longer useful. So in addition to allowing a user to specify a snapshot ttl on {{nodetool snapshot}} we should have a {{auto_snapshot_ttl}} option that allows a user to set a ttl for automatic snaphots on drop/truncate.
Bug: Add Plugin Support for CQLSH
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Feb/21 07:24
Resolved: 02/Sep/22 13:32

Currently the Cassandra drivers offer a plugin authenticator architecture for the support of different authentication methods. This has been leveraged to provide support for LDAP, Kerberos, and Sigv4 authentication. Unfortunately, cqlsh, the included CLI tool, does not offer such support. Switching to a new enhanced authentication scheme thus means being cut off from using cqlsh in normal operation.

We should have a means of using the same plugins and authentication providers as the Python Cassandra driver.

Here's a link to an initial draft of [CEP|https://docs.google.com/document/d/1_G-OZCAEmDyuQuAN2wQUYUtZBEJpMkHWnkYELLhqvKc/edit?usp=sharing].
Bug: Expose application_name and application_version in virtual table system_views.clients
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 11/Jan/21 13:27
Resolved: 08/Mar/22 10:16

Recent java-driver's [com.datastax.oss.driver.api.core.session.SessionBuilder|https://docs.datastax.com/en/drivers/java/4.9/com/datastax/oss/driver/api/core/session/SessionBuilder.html] respects properties [ApplicationName|https://docs.datastax.com/en/drivers/java/4.9/com/datastax/oss/driver/api/core/session/SessionBuilder.html#withApplicationName-java.lang.String-] and [ApplicationVersion|https://docs.datastax.com/en/drivers/java/4.9/com/datastax/oss/driver/api/core/session/SessionBuilder.html#withApplicationVersion-java.lang.String-].

It would be helpful to expose this information via virtual table {{system_views.clients}} and with {{{}nodetool clientstats{}}}.

+Additional information for newcomers:+

The drivers can send as part of the {{STARTUP MESSAGE}} the {{APPLICATION_NAME}} and {{APPLICATION_VERSION}} options. Two new volatile fields {{applicationName}} and {{applicationVersion}} need to be added to {{ClientState}} in a similar way to {{driverName}} and {{{}driverVersion{}}}.
The {{APPLICATION_NAME}} and {{APPLICATION_VERSION}} options need to be retrieved in {{StartupMessage#execute}} and passed to the {{{}ClientState{}}}. 
The new {{application_name}} and {{application_version}} columns need to be added to the {{system_views.clients}} represented by the {{ClientsTable}} class. The data then need to be retrieved from the {{ClientState}} through {{{}ConnectedClient{}}}.
Some unit tests similat to {{SettingsTableTest}} should be added.
Bug: Add startup check for large read_ahead_kb setting
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 09/Feb/21 19:06
Resolved: 03/Jan/22 21:41

A well known production tuning recommendation is to dial back the {{read_ahead_kb}} Linux setting from the default of 4096 in most distributions as this can cause high IO usage and cache churn on read-intensive workloads.

We should add a startup warning to detect when a high {{read_ahead_kb}} is used and recommend the user to dial back to a reasonable value, which varies according to the workload but a general guideline is to tune this to 8kb.
Bug: cqlsh encoding error with unicode in multi-line statement
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 25/Mar/21 20:53
Resolved: 06/May/21 21:33

{noformat}
CREATE TABLE test.users (
    user_id int PRIMARY KEY,
    name text,
    state text
)
{noformat}

Multiline cql with unicode characters will fail with an encoding error:
{noformat}
cqlsh> insert into test.users ( user_id, name, state ) values (
   ... 6,
   ... 'Bonne',
   ... 'Année');
'ascii' codec can't encode character u'\xe9' in position 74: ordinal not in range(128)
{noformat}

This is only when running Python 2.7 and when python3 is not present in the environment.
Bug: Provide a nodetool way of invalidating auth caches
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 25/Jan/21 08:18
Resolved: 03/Sep/21 07:35

We currently have nodetool commands to invalidate certain caches like KeyCache, RowCache and CounterCache. 
Being able to invalidate auth caches as well can come in handy in situations where, critical backend auth changes may need to be in effect right away for all the connections, especially in configurations where cache validity is chosen to be for a longer duration. An example can be that an authenticated user "User1" is no longer authorized to access a table resource "table1" and it is vital that this change is reflected right away, without having to wait for cache expiry/refresh to trigger.
Bug: Make ReadCommand::toCQLString return valid CQL
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 10/Mar/21 13:44
Resolved: 06/Jul/21 12:24

The method {{ReadCommand::toCQLString}} doesn't always return queries that are valid CQL. For example, queries for a table without clustering columns always add the {{WHERE}} keyword even if there isn't a restriction following it:
{code:java}
CREATE TABLE t (k int PRIMARY KEY, v int);
SELECT * FROM t; 
-- toCQLString 3.0:        SELECT * FROM k.t WHERE () = () LIMIT 100
-- toCQLString 3.11/trunk: SELECT * FROM k.t WHERE  LIMIT 100
{code}
Also, case sensitive names and text values are not properly quoted:
{code:java}
CREATE TABLE "T" ("K" text, "C" text, "V" text, PRIMARY KEY("K", "C"));
SELECT * FROM "T" WHERE "K" = 'a' AND "C" = 'b' AND "V" = 'c' ALLOW FILTERING;
-- toCQLString 3.0/3.11/trunk: SELECT * FROM k.T WHERE K = a AND V = c AND (C) = (b) LIMIT 100
{code}
Similar problems can be found on selections of collection items:
{code:java}
CREATE TABLE k.t (k int, c int, s set<text>,  m map<text, text>, PRIMARY KEY(k, c));

SELECT s['a'] FROM t;
-- toCQLString trunk: SELECT s[a] FROM k.t LIMIT 100

SELECT * FROM t WHERE m['a'] = 'b' ALLOW FILTERING;
-- toCQLString 3.0/3.11/trunk: SELECT * FROM k.t WHERE m[a] = b LIMIT 100
{code}
Some similar problems with more impact than the described above are already being addressed in CASSANDRA-16483 and CASSANDRA-16479.

I think we should add more exhaustive JUnit tests for {{ReadCommand::toCQLString}}, making sure that its output is parseable and equivalent to the original command. Also, we probably should make sure that we have {{toCQLString}} methods in all the query components that are used by {{ReadCommand::toCQLString}} (selection, filters, etc.). This way we can use them instead of generic {{toString}} methods, making it clear that they are supposed to return valid CQL.
Bug: SinglePartitionSliceCommandTest.testPartitionDeletionRangeDeletionTie is flaky
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 11/Mar/21 18:10
Resolved: 16/Mar/21 00:07

Although we cannot see that in regular CI, this test is flaky and I can see that in our private test runs. The problem is that it assumes two consecutive queries are executed in the exact same second. A similar test case {{testPartitionDeletionRowDeletionTie}} has been already fixed in CASSANDRA-16443. We need almost identical fix for this one as well.