av9ash/GitBugs huggingface dataset:
Bug: Add JDK17 option to cassandra-builds (build-scripts and jenkins dsl) and on jenkins agents
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 30/Aug/22 07:21
Resolved: 24/Apr/23 20:06

Add JDK17 option to cassandra-builds build-scripts, they only currently support options {{8}} and {{11}}.

Add JDK17 to the matrix axes in the jenkins dsl.

Ensure JDK17 is installed on all the jenkins agents.
Bug: DESCRIBE output does not quote column names using reserved keywords
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 22/Sep/22 19:55
Resolved: 11/May/23 14:16

The DESCRIBE output of the column names that using reserved keywords are not quoted for UDTs. The following test reproduces. Reading the code, it looks like that the such columns names are not quoted in materialized view, UDF and user defined aggregation. 
The impact of the bug is that schema described cannot be imported due to the usage of reserved keywords as column names. 
 
{code:java}
    @Test
    public void testUsingReservedInCreateType() throws Throwable
    {
        String type = createType(KEYSPACE_PER_TEST, "CREATE TYPE %s (\"token\" text, \"desc\" text);");               assertRowsNet(executeDescribeNet(KEYSPACE_PER_TEST, "DESCRIBE TYPE " + type),
                row(KEYSPACE_PER_TEST, "type", type, "CREATE TYPE " + KEYSPACE_PER_TEST + "." + type + " (\n" +
                        "    \"token\" text,\n" +
                        "    \"desc\" text\n" +
                        ");"));
    } {code}
+Additional information for newcomers:+
 * Unit tests for DESCRIBE statements are in {{DescribeStatementTest}}
 * The statement implementation is in {{DescribeStatement and fetch the create statement from the different schema element using  SchemaElement.toCqlString}}
Bug: Remove dependency on Maven Ant Tasks
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 12/Jul/22 21:19
Resolved: 25/Aug/22 05:41

Apache Cassandra depends on Maven Ant Tasks (MAT) during build, for declaring dependencies and generating POM files from within build.xml. MAT has long been retired (no commits since maintenance in 2015), has registered CVEs in dependencies (CVE-2017-1000487), and encourages migration to its successor, Maven Artifact Resolver Ant Tasks (MARAT).

As part of CASSANDRA-16391 <https://issues.apache.org/jira/browse/CASSANDRA-16391>, mck migrated dependency resolution to MARAT, but MAT is still included in our build for generating POMs since MARAT does not have an alternative to the writepom task provided by MAT. I have a patch ready that removes MAT completely, with a workaround for POM generation.

I am not advocating for any kind of migration away from Ant to an alternative like Gradle or Maven, just to be extra clear.
Bug: CQL data masking functions
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 30/Sep/22 11:08
Resolved: 27/Jan/23 16:17

Add native CQL functions to mask the contents of a column, as described in [CEP-20|https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-20%3A+Dynamic+Data+Masking].
Bug: Resolve pylint issues in pylexotron.py and improve readability
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Jul/22 02:19
Resolved: 09/Aug/22 14:01

Add comments to clarify behavior

Convert short form constants to long form (e.g., re.I == re.IGNORECASE)

Camel case class names

 

 

[Original pylint] code has been rated at 6.71/10
Bug: Add guardrail to disallow DROP KEYSPACE commands for non superuser accounts
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Jul/22 18:38
Resolved: 22/Jul/22 16:22

Similar to CASSANDRA-17558, we should have the ability to ban DROP KEYSPACE commands cluster-wide for non superuser accounts.

The goal as with blocking DROP/TRUNCATE TABLE is to help prevent users from inadvertently throwing away data and providing a cleaner way for operators to just disable this cluster-wide than role management / runbook / etc.
Bug: IndexOutOfBoundsException in Paging State Version Inference (V3 State Received on V4 Connection)
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 19/Aug/22 17:10
Resolved: 24/Aug/22 19:28

In {{PagingState.java}}, {{index}} is an integer field, and we add long values to it without a {{Math.toIntExact}} check. While we’re checking for negative return values returned by {{getUnsignedVInt}}, there's a chance that the value returned by it is so large that addition operation would cause integer overflow, or the value itself is large enough to cause overflow.
Bug: Nested selection of reversed collections fails
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Sep/22 16:31
Resolved: 14/Apr/23 10:16

The following test fails caused by the fact we make reversed a type that wraps the underline type
{code:java}
@Test
    public void testMapsReversed() throws Throwable
    {
        createTable("CREATE TABLE %s (" +
                    "   k int, " +
                    "   c frozen<map<text, int>>, " +
                    "   v int, " +
                    "   PRIMARY KEY(k, c)" +
                    ") WITH CLUSTERING ORDER BY (c DESC)");

        execute("SELECT c['testing'] FROM %s");
    }
{code}
With the error
{code:java}
org.apache.cassandra.exceptions.InvalidRequestException: Invalid element selection: c is of type frozen<map<text, int>> is not a collection
{code}
When you look in a debugger you see that the real type is ReversedType(MapType(…)), so the current checks fail.

{+}Additional information for newcomers{+}:
 * The problem impact element selection as well as range selection that where introduced in CASSANDRA-7396 as part of 4.0. So the patch will need to be done for 4.0.
 * The checks that needs to be modified is in {{Selectable.WithElementSelection}} and {{Selectable.WithSliceSelection}}
 * The unit tests should be added to {{CollectionsTest}}
Bug: Extend nodetool verify with functionality to detect partitions with duplicate keys
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 01/Aug/22 18:30
Resolved: 04/Aug/22 16:44

We've seen instances where, during upgrades across major versions and in mixed version clusters, we end up with data in a table that has duplication of primary keys. This is obviously an illegal state and being able to quickly scan for this and log partition keys for which we're seeing this state has proven quite useful.

It's a fairly small change to verify/scrub to integrate this extra functionality.
Bug: Test Failure: org.apache.cassandra.db.commitlog.CommitLogInitWithExceptionTest.testCommitLogInitWithException-compression
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Sep/22 15:20
Resolved: 09/Nov/22 18:03

[Link|https://ci-cassandra.apache.org/job/Cassandra-4.1/169/testReport/org.apache.cassandra.db.commitlog/CommitLogInitWithExceptionTest/testCommitLogInitWithException_compression/]

Failed 1 times in the last 14 runs. Flakiness: 7%, Stability: 92%

Stacktrace
{code:java}
java.lang.NullPointerException
	at org.apache.cassandra.db.commitlog.CommitLogInitWithExceptionTest.testCommitLogInitWithException(CommitLogInitWithExceptionTest.java:93)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.lang.Thread.run(Thread.java:829)
{code}



{code:java}
Standard Output
INFO  [main] 2022-09-25 11:43:16,512 Reflections.java:219 - Reflections took 1221 ms to scan 8 urls, producing 1756 keys and 6922 values
INFO  [main] 2022-09-25 11:43:17,480 Reflections.java:219 - Reflections took 907 ms to scan 8 urls, producing 1756 keys and 6922 values
INFO  [main] 2022-09-25 11:43:17,573 YamlConfigurationLoader.java:104 - Configuration location: file:////home/cassandra/cassandra/build/test/cassandra.compressed.yaml
DEBUG [main] 2022-09-25 11:43:17,574 YamlConfigurationLoader
...[truncated 35568 chars]...
.apache.cassandra.db.commitlog.CommitLogInitWithExceptionTest$MockCommitLogSegmentMgr.createSegment(CommitLogInitWithExceptionTest.java:106)
	at org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager$AllocatorRunnable.run(AbstractCommitLogSegmentManager.java:155)
	at org.apache.cassandra.concurrent.InfiniteLoopExecutor.loop(InfiniteLoopExecutor.java:121)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
{code}
Bug: Argparse module migration as the Python Optparse is deprecated
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Sep/22 19:07
Resolved: 25/Jul/23 08:48

[Deprecated since version 2.7: The optparse module is deprecated and will not be developed further; development will continue with the argparse module.|https://docs.python.org/2/library/optparse.html]

Argparse is described in [PEP 389 – argparse - New Command Line Parsing Module|https://peps.python.org/pep-0389/]

 

A partial upgrade path from [{{optparse}}|https://docs.python.org/3/library/optparse.html#module-optparse] to [{{argparse}}|https://docs.python.org/3/library/argparse.html#module-argparse]:

https://docs.python.org/3/library/argparse.html#upgrading-optparse-code
 * Replace all [{{optparse.OptionParser.add_option()}}|https://docs.python.org/3/library/optparse.html#optparse.OptionParser.add_option] calls with [{{ArgumentParser.add_argument()}}|https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser.add_argument] calls.

 * Replace {{(options, args) = parser.parse_args()}} with {{args = parser.parse_args()}} and add additional [{{ArgumentParser.add_argument()}}|https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser.add_argument] calls for the positional arguments. Keep in mind that what was previously called {{{}options{}}}, now in the [{{argparse}}|https://docs.python.org/3/library/argparse.html#module-argparse] context is called {{{}args{}}}.

 * Replace [{{optparse.OptionParser.disable_interspersed_args()}}|https://docs.python.org/3/library/optparse.html#optparse.OptionParser.disable_interspersed_args] by using [{{parse_intermixed_args()}}|https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser.parse_intermixed_args] instead of [{{parse_args()}}|https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser.parse_args].

 * Replace callback actions and the {{callback_*}} keyword arguments with {{type}} or {{action}} arguments.

 * Replace string names for {{type}} keyword arguments with the corresponding type objects (e.g. int, float, complex, etc).

 * Replace {{optparse.Values}} with [{{Namespace}}|https://docs.python.org/3/library/argparse.html#argparse.Namespace] and {{optparse.OptionError}} and {{optparse.OptionValueError}} with {{{}ArgumentError{}}}.

 * Replace strings with implicit arguments such as {{%default}} or {{%prog}} with the standard Python syntax to use dictionaries to format strings, that is, {{%(default)s}} and {{{}%(prog)s{}}}.

 * Replace the OptionParser constructor {{version}} argument with a call to {{{}parser.add_argument('--version', action='version', version='<the version>'){}}}.
Bug: WEBSITE – in-tree trunk should be marked as `prerelease: true`
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 16/Aug/22 21:37
Resolved: 26/Aug/22 10:20

this will avoid the "A newer version of this documentation is available." banner displaying on the version docs for our latest released version.

ref: https://docs.antora.org/antora/latest/component-prerelease/#true

change is to be applied to https://github.com/apache/cassandra/blob/trunk/doc/antora.yml
Bug: Fix org.apache.cassandra.simulator.test.ShortPaxosSimulationTest.simulationTest
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 22/Jul/22 14:18
Resolved: 27/Jul/22 20:20

Fix org.apache.cassandra.simulator.test.ShortPaxosSimulationTest.simulationTest failing on current trunk:
https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/1795/workflows/bc8619bb-7301-4428-9662-bc24e5f38b87/jobs/13335/tests#failed-test-0

and also in this run 

https://app.circleci.com/pipelines/github/driftx/cassandra/551/workflows/ba8cadb9-025f-46f9-b663-69d9e956d30d/jobs/6596/tests
Bug: BulkLoader tool initializes schema unnecessarily via streaming
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 06/Jul/22 16:26
Resolved: 05/Aug/22 20:51

Changes to streaming setup code for CASSANDRA-17663 mean that {{BulkLoader}} initializes the schema/system keyspace, which is not what we want in a tool. Initialization is due to a call to {{SystemKeyspace.getPreferredIP}} from the {{BulkLoader}} when it starts to transmit the SSTables from the bulk loader to the Cassandra instance.

{noformat}
getPreferredIP:1063, SystemKeyspace (org.apache.cassandra.db)
sendMessage:213, StreamingMultiplexedChannel (org.apache.cassandra.streaming.async)
sendControlMessage:191, StreamingMultiplexedChannel (org.apache.cassandra.streaming.async)
sendControlMessage:1033, StreamSession (org.apache.cassandra.streaming)
startStreamingFiles:1257, StreamSession (org.apache.cassandra.streaming)
prepareSynAck:802, StreamSession (org.apache.cassandra.streaming)
messageReceived:622, StreamSession (org.apache.cassandra.streaming)
run:76, StreamDeserializingTask (org.apache.cassandra.streaming)
run:30, FastThreadLocalRunnable (io.netty.util.concurrent)
run:748, Thread (java.lang)
{noformat}

The existing {{BulkLoaderTest}} fails to detect this as it doesn't actually connect to anything so does not initialize streaming.

Affects 4.1 and trunk, and may affect 4.0, although the 4.0 patch for CASSANDRA-17663 is different than 4.1+, and may require different mitigation.
Bug: Fix incorrect resource name in LIST PERMISSION output
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 22/Aug/22 17:20
Resolved: 06/Dec/22 03:02

When producing the resource name, it seems to assume that the content in the `[]` is the function's input type, where it could also be part of the function name, as long as it is quoted. Here is an example to reproduce. In cqlsh,

{code:java}
> CREATE FUNCTION test."admin_created_udf[org.apache.cassandra.db.marshal.LongType]"(input int) RETURNS NULL ON NULL INPUT RETURNS int LANGUAGE java AS 'return 42;';

> LIST EXECUTE OF user;
 role  | username | resource                                | permission
-------+----------+-----------------------------------------+------------
 user  |    user  | <function test.admin_created_udf(long)> |    EXECUTE

(1 rows)
{code}

The input should be "int", but in the output, it says "long". 

If the content enclosed by "[]" is not a valid class, the LIST PERMISSION request always fails for the user with "ConfigurationException: Unable to find abstract-type class".

The bug is discovered by Piotr Sarna.
Bug: Confusing error message when using ? with functions
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Sep/22 22:59
Resolved: 10/Nov/22 19:45

{code}
INSERT INTO %S (a, b, c) VALUES (? + 1, ?, ?)
{code}

Errors saying

{code}
Ambiguous '+' operation with args ? and 1: use type casts to disambiguate
{code}

Now, if you google “type casts CQL” you get https://docs.datastax.com/en/dse/5.1/cql/cql/cql_reference/refCqlFunction.html which says to do

{code}
CAST( selector AS to_type )
{code}

But this also fails!

{code}
InvalidRequestException: Ambiguous call to function system.castAsFloat (can be matched by following signatures: system."castAsFloat" : (bigint) -> float, system."castAsFloat" : (counter) -> float, system."castAsFloat" : (double) -> float, system."castAsFloat" : (int) -> float, system."castAsFloat" : (tinyint) -> float, system."castAsFloat" : (varint) -> float, system."castAsFloat" : (decimal) -> float, system."castAsFloat" : (smallint) -> float): use type casts to disambiguate
{code}

What we have to do is 

{code}
INSERT INTO %S (a, b, c) VALUES ((int) ? + 1, ?, ?)
{code}
We should improve the error message to show the expected syntax (or fix CAST to work in this case).
Bug: Add ability to generate a one-time heap dump output to a file on Unhandled Exceptions
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/Aug/22 19:51
Resolved: 15/Aug/22 14:51

We should add the optional hot-proppable / in .yaml ability to collect a heap dump when any uncaught exception is handled by the {{JVMStabilityInspector}}, assuming it's not an OOM or something that marks the JVM unstable.

This has come in handy in a variety of prod debugging scenarios and would be good to have baked into the DB proper for operators.
Bug: Fix race condition on updating cdc size and advancing to next segment
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/Aug/22 15:41
Resolved: 16/Aug/22 22:32

org.apache.cassandra.distributed.test.cdc.ToggleCDCOnRepairEnabledTest is a bit flaky on [trunk. 

As [this run|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra?branch=17666] shows it was flaky since it was introduced a month ago as part of

CASSANDRA-17666 but the flakiness is so low that if we don't run it in a loop it is hard to hit it. 

Both tests in the test class can fail with the same exception:
{code:java}
org.apache.cassandra.distributed.shared.ShutdownException: Uncaught exceptions were thrown during test at org.apache.cassandra.distributed.impl.AbstractCluster.checkAndResetUncaughtExceptions(AbstractCluster.java:1056) at org.apache.cassandra.distributed.impl.AbstractCluster.close(AbstractCluster.java:1042) at org.apache.cassandra.distributed.test.cdc.ToggleCDCOnRepairEnabledTest.testCDCOnRepairEnabled(ToggleCDCOnRepairEnabledTest.java:95) at org.apache.cassandra.distributed.test.cdc.ToggleCDCOnRepairEnabledTest.testCDCOnRepairIsEnabled(ToggleCDCOnRepairEnabledTest.java:40) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) Suppressed: java.lang.NullPointerException at org.apache.cassandra.db.commitlog.CommitLogSegmentManagerCDC$CDCSizeTracker.recalculateOverflowSize(CommitLogSegmentManagerCDC.java:390) at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:81) at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:47) at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:57) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829){code}
CC [~ycai] , [~jmckenzie]
Bug: Repair coordinator can get stuck in an infinite loop if it gets a FailSession after it's marked a session FINALIZED
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Aug/22 16:52
Resolved: 22/Aug/22 16:52

If the repair coordinator gets a FailSession message after it has marked the session as FINALIZED it will start an infinite loop of FailSession messages. This is unique to 4.0+ in {{CoordinatorSession.java}}
Bug: IllegalArgumentException in Gossiper#order due to concurrent mutations to elements being applied
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/Sep/22 18:44
Resolved: 11/Oct/22 00:46

org.apache.cassandra.gms.Gossiper#order was added due to sub-systems depending on a happens-before relationship of events even though Gossip does not provide such ordering… to help out we “order” the events discovered by gossip so we apply them in a predictable order… This has been found to have issues where we add the elements into the global state and under some conditions they get mutated causing ordering to fail

{code}
ERROR 2022-09-01T19:04:24,789 [GossipStage:1] org.apache.cassandra.service.CassandraDaemon:234 - Exception in thread Thread[GossipStage:1,5,main]
java.lang.IllegalArgumentException: Comparison method violates its general contract!
at java.util.TimSort.mergeLo(TimSort.java:777) 
at java.util.TimSort.mergeAt(TimSort.java:514) 
at java.util.TimSort.mergeForceCollapse(TimSort.java:457) 
at java.util.TimSort.sort(TimSort.java:254) 
at java.util.Arrays.sort(Arrays.java:1512) 
at java.util.ArrayList.sort(ArrayList.java:1464) 
at java.util.Collections.sort(Collections.java:177) 
at org.apache.cassandra.gms.Gossiper.order(Gossiper.java:1327) 
at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1334) 
at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49) 
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:70) 
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:116) 
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_345]
{code}
Bug: Revise timeout handling on queries triggering timeout introduced by CASSANDRA-7392
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 11/Aug/22 16:13
Resolved: 26/Sep/22 17:58

In situations where a replica times out with the code introduced in CASSANDRA-7392, they drop the timed-out query on the floor without notifying the coordinator; this doesn't become apparent coordinator-side until we go to serialize the response. While the results of this (i.e. missed data) should mostly be fixed by short read protection on repeat requests, the coordinator can in some cases throw an exception like {{IllegalStateException: UnfilteredRowIterator for keyspace.table has an open RT bound as its last item}} which is decidedly unfriendly.

We should introduce a specific exception to allow replicas to notify a coordinator something has timed out and been dropped on the floor so the coordinator can appropriately handle that scenario and abort the query at that time.
Bug: Scrubber falls into infinite loop when the last partition is broken in data file
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 25/Aug/22 17:56
Resolved: 26/Aug/22 15:48

Scrubber cannot deal with the situation when the last partition in a data file is broken but the index is fine. When compression is enabled, scrubber falls into infinite loop.

This affects all 3.0, 3.11, 4.0, 4.1 and trunk.
Bug: Include GitSHA in nodetool version output
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/Jul/22 21:08
Resolved: 25/Oct/22 19:42

It can be useful to see specifically which Git SHA a running instance of Cassandra was built with, especially when running clusters in development for soak testing.

 

I have a patch ready for this, and am preparing it now.