av9ash/GitBugs huggingface dataset:
Bug: NullPointerException on SnapshotLoader.loadSnapshots
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 23/Mar/23 21:08
Resolved: 26/Mar/23 21:37

Node startup fail with on 4.1.1:

{noformat}
INFO [main] 2023-03-23 18:13:13,585 MigrationCoordinator.java:257 - Starting migration coordinator and scheduling pulling schema versions every PT1M
ERROR [main] 2023-03-23 18:13:13,592 CassandraDaemon.java:898 - Exception encountered during startup
java.lang.NullPointerException: null
	at org.apache.cassandra.service.snapshot.SnapshotLoader$Visitor.preVisitDirectory(SnapshotLoader.java:106)
	at org.apache.cassandra.service.snapshot.SnapshotLoader$Visitor.preVisitDirectory(SnapshotLoader.java:77)
	at java.base/java.nio.file.Files.walkFileTree(Files.java:2732)
	at org.apache.cassandra.service.snapshot.SnapshotLoader.loadSnapshots(SnapshotLoader.java:162)
	at org.apache.cassandra.service.snapshot.SnapshotManager.loadSnapshots(SnapshotManager.java:114)
	at org.apache.cassandra.service.snapshot.SnapshotManager.start(SnapshotManager.java:88)
	at org.apache.cassandra.service.StorageService.startSnapshotManager(StorageService.java:1050)
	at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:1043)
	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:842)
	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:775)
	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:425)
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:752)
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:876)
{noformat}
Bug: LongBTreeTest times out after btree improvements from CASSANDRA-15510
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Jan/23 19:45
Resolved: 03/Apr/23 13:22

Happening in both ci-cassandra.a.o and circleci.

LongBTreeTest is timing out on 4.0, 4.1, trunk branches.

Started back in mid April (https://github.com/apache/cassandra/commit/018c8e0d5e and https://github.com/apache/cassandra/commit/596daeb7f08). 

Nightlies shows when the failures started, evident by the 'jdk=jdk_1.8_latest,label=cassandra,split=7/' subfolder missing in the following… 
- https://nightlies.apache.org/cassandra/trunk/Cassandra-trunk-test-burn/1254/Cassandra-trunk-test-burn/
- https://nightlies.apache.org/cassandra/cassandra-4.0/Cassandra-4.0-test-burn/343/
Bug: die disk failure policy will not kill jvm as documented
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 01/Mar/23 23:02
Resolved: 09/Mar/23 16:07

After Cassandra has successfully starts up with disk_failure_policy die, when encounter a file system error, Cassandra server will only throw exception instead of shut down gossip and client transport and kill JVM. Document: [https://cassandra.apache.org/doc/latest/cassandra/configuration/cass_yaml_file.html#disk_failure_policy]

 

The reason for this is the default FS error handler is not handing policy die correctly. Instead of shutting down gossip and native transport, it throws an error.

 

The JVMStabilityInspectorTest is passing because the error handler is not set so no exception is thrown.
Bug: Remove Scripted UDFs from trunk to enable easier JDK17 testing on current trunk
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 09/Feb/23 15:55
Resolved: 12/Feb/23 15:40

CASSANDRA-17281 was created to do the work around removing Scripted UDFs 

Seems like adding hooks with the current Sandbox might be non-trivial, considering that it was the reason for removing the hooks for other scripted languages.

Also, Security Manager is deprecated already in Java 17, the current sandbox had also some performance implications. With all that said, this ticket is meant to blindly remove the scripted UDFs and any related tests and CASSANDRA-17281 will be used for any UDF refactoring and addition of scripted UDFs hooks. This will help us to speed up the incremental addition of JDK17 builds and enable people to test directly on trunk instead of having their own modified branches. Feature branch was considered but considering all moving parts that we have to maintain a few different branches in different repos and states it is too time-consuming to maintain.

Supportive ML thread where I will also post the result of this ticket for final confirmation:

[https://lists.apache.org/thread/c39yrbdszgz9s34vr17wpjdhv6h2oo61]

*WARNING: The API for UDFs might change as part of CASSANDRA-17281 where hooks will be introduced for JavaScript UDFs and some additional refactoring is going to happen. This ticket primary goal is to clean the internals and enable people to easily test with JDK17*
Bug: Add CircleCI config files for J11+J17
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 08/Feb/23 18:34
Resolved: 20/Mar/23 17:15

Based on the direction of [this discussion|https://lists.apache.org/thread/hchv59c1sntgb74clynj0zfd8jvwdmgy], I would like to propose CircleCI config files which can be used to test current trunk with JDK 17 (after I blindly remove the scripted UDFs in another ticket, to be opened soon)
Bug: Switch checkstyle running only with JDK8 to be run with JDK11
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/Feb/23 21:33
Resolved: 30/Mar/23 10:25

Currently we run checkstyle, eclipse warnings (and probably anything else?) only with JDK8.

In preparation to remove JDK8 we need to switch those to run with JDK11.
Bug: Nodetool info should report on the new networking cache
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 31/Mar/23 18:00
Resolved: 24/May/23 15:11

CASSANDRA-15229 separated the chunk and network cache, creating a new network_cache_size parameter.

However, *nodetool info* does not report the in-use size of this cache or a hit ratio as it does for key, row, counter and chunk cache.  
{quote}Exceptions : 4

Key Cache : entries 2852, size 297.59 KiB, capacity 100 MiB, 2406561 hits, 2409424 requests, 0.999 recent hit rate, 14400 save period in seconds

Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds

Counter Cache : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds

Chunk Cache : entries 1118, size 55.02 MiB, capacity 992 MiB, 4695794 misses, 7179421 requests, 0.346 recent hit rate, 24.145 microseconds miss latency

Percent Repaired : 0.0%
{quote}
However, when its full, it will be logged:
{quote}[INFO ] [epollEventLoopGroup-5-12] cluster_id=1 ip_address=127.1.1.1  NoSpamLogger.java:92 - Maximum memory usage reached (128.000MiB), cannot allocate chunk of 8.000MiB
{quote}
It should report a line similar to the above:
{quote}Network Cache : entries ?, size ? MiB, capacity ? MiB, ? misses, ? requests, ? recent hit rate
{quote}
Also, not sure why the above show NaN for row and counter cache, is there is a divide by zero error when the entries are zero?
Bug: Remove git hook for pre-push as it is redundant and causes issues when merging to mainline
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 09/Mar/23 00:00
Resolved: 09/Mar/23 23:21

{code}
[cep-15-accord][~/repos/apache-cassandra]$ git push origin cep-15-accord
Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
Entering 'modules/accord'
Username for 'https://github.com':
{code}

This is caused by .build/git/git-hooks/pre-push/100-push-submodules.sh logic

{code}
  local -r cmd='
branch="$(git rev-parse --abbrev-ref HEAD)"
[[ "$branch" == "HEAD" ]] && exit 0

default_remote="$(git config --local --get branch."${branch}".remote || true)"
remote="${default_remote:-origin}"

git push --atomic "$remote" "$branch"
'
  git submodule foreach --recursive "$cmd"
{code}

This logic was to make sure that the submodule is pushed before you push your changes, but this is slightly redundant as .build/git/git-hooks/pre-commit/100-verify-submodules-pushed.sh will not allow you to commit the submodule SHA until it can confirm its on GitHub.
Bug: consistent timeout: dtest-upgrade.upgrade_tests.cql_tests.cls.test_cql3_non_compound_range_tombstones on trunk
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 30/Mar/23 19:50
Resolved: 31/Mar/23 15:44

Failed 30 times in the last 30 runs. Flakiness: 0%, Stability: 0%

link: https://ci-cassandra.apache.org/job/Cassandra-trunk/1511/testReport/dtest-upgrade.upgrade_tests.cql_tests/cls/test_cql3_non_compound_range_tombstones/
Error message: failed on setup with "Failed: Timeout >900.0s"
Bug: 'Maximum memory usage reached' chunk cache log message doesn't specify which cache is exhausted
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/Jan/23 19:28
Resolved: 02/Feb/23 17:49

With Cassandra 4.0.x, we are seeing this cassandra.log message very frequently on the majority of our Cassandra 4.0 clusters:

    _[INFO ] [epollEventLoopGroup-5-3] cluster_id=99 ip_address=127.0.0.50  NoSpamLogger.java:92 - Maximum memory usage reached (128.000MiB), cannot allocate chunk of 8.000MiB_

It took me several weeks to track down what it means, until I saw this start-up message

    _BufferPools.java:49 - Global buffer pool limit is 2.000GiB for chunk-cache and 128.000MiB for networking_

This maximum memory usage warning would benefit from clarifying that its the {*}network cache{*}, not the *off-heap chunk* *cache* which is exhausted.  With 'chunk cache' in both warning messages, they're too easily confused.
Bug: mockito-inline causes tests to fail beacause o.a.c.distributed.mock.nodetool.InternalNodeProbe spies on StorageServiceMBean
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 12/Jan/23 13:54
Resolved: 17/Jan/23 08:31

While working on CASSANDRA-14361, when we included mockito-inline into the build to test the new functionality, unrelated tests in CI started to fail. (1)

This is happening because mockito, together with stuff which enables static mocking, just does not play together with our way of doing things in dtest framework.

The workaround is consisting of removing Mockito from InternalNodeProbe, it tries to spy on StorageService to not send any notifications back. This might be workarounded so we do not need Mockito hence tests are fixed and mocking of static methods is possible without any other tests failing.

(1) https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/2168/#showFailuresLink

see also: [https://github.com/mockito/mockito/issues/2203]
Bug: Upgrade Zstd to 1.5.4
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 11/Feb/23 18:20
Resolved: 15/Feb/23 12:50

Zstd 1.5.4 was released yesterday. There is a lot of non-trivial performance improvements. 

Upgrade zstd-jni 1.5.0 to 1.5.4.

(1) https://github.com/facebook/zstd/releases/tag/v1.5.4
Bug: Node Draining Should Abort All Current SSTables Imports
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 28/Mar/23 17:37
Resolved: 07/Apr/23 20:14

SS tables imports that will end up being ignored due to the node draining should fail instead of succeeding.
Each active SS tables import should periodically check for the node status and see whether or not it is `DRAINING`.
In case the node starts draining the import should abort immediately by throwing an `InterruptedException`.
Bug: JDK17 - fix nodetool_test.TestNodetool.test_sjk
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 19/Mar/23 19:28
Resolved: 23/Mar/23 19:11

 
[https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2321/workflows/de1f521d-c5cb-4ddd-bc45-9ec71b577bf3/jobs/19923/tests]
 
{code:java}
AssertionError: Expected 'SJK hh' output assert False == True self = <nodetool_test.TestNodetool object at 0x7f9ab6d95908> @since('4.0') def test_sjk(self): """ Verify that SJK generally works. """ cluster = self.cluster cluster.populate([1]).start() node = cluster.nodelist()[0] out, err, _ = node.nodetool('sjk --help') logger.debug(out) hasPattern = False for line in out.split(os.linesep): if " ttop [Thread Top] Displays threads from JVM process" == line: hasPattern = True assert hasPattern == True, "Expected help about SJK ttop" out, err, _ = node.nodetool('sjk') logger.debug(out) hasPattern = False for line in out.split(os.linesep): if " ttop [Thread Top] Displays threads from JVM process" == line: hasPattern = True assert hasPattern == True, "Expected help about SJK ttop" out, err, _ = node.nodetool('sjk hh -n 10 --live') logger.debug(out) hasPattern = False for line in out.split(os.linesep): if re.match('.*Instances.*Bytes.*Type.*', line): hasPattern = True > assert hasPattern == True, "Expected 'SJK hh' output" E AssertionError: Expected 'SJK hh' output E assert False == True nodetool_test.py:482: AssertionError{code}
Bug: Gossip stateMapOrdering does not have correct ordering when both EndpointState are in the bootstrapping set
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 28/Feb/23 21:03
Resolved: 01/Mar/23 21:42

There is a bug when stateMapOrdering sees two or more EndpointState that are both in the bootstrapping set, this may cause the ordering to change causing Collections.sort to fail
Bug: Cassandra start up with ERROR of sstable_formats
Status: Resolved
Priority: Normal
Resolution: Not A Problem
Created: 09/Mar/23 10:06
Resolved: 09/Mar/23 11:06

Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: Invalid yaml. Please remove properties [sstable_formats] from your cassandra.yaml
Invalid yaml. Please remove properties [sstable_formats] from your cassandra.yaml
ERROR [main] 2023-03-09 18:03:34,899 CassandraDaemon.java:911 - Exception encountered during startup: Invalid yaml. Please remove properties [sstable_formats] from your cassandra.yaml
Bug: CustomClassLoader does not load jars rendering triggers from JARs broken
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Feb/23 21:22
Resolved: 17/Feb/23 15:55

A user had to downgrade to 4.0.7 from 4.1.0 because they hit a problem with CustomClassLoader for triggers. 

User says that in Apache Cassandra 4.1.0 the trigger mechanism does not work, not their trigger, but the possibility of loading any trigger in Cassandra.

In the Cassandra 4.1.0 version of CustomClassLoader (https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/triggers/CustomClassLoader.java) the code is changed in such a way that when copying the JAR Cassandra uses java.nio.file.Files, while earlier versions (cassandra 4.0.X or 3.X) used Guava com.google.common.io.Files to copy the JAR file.

The difference between one and the other is that Guava by default overwrites the file if it already exists and user has permissions to do so, and in Java by default it does not overwrite.

Copying is done here (1) from inputJar to out. However, the problem is that we are getting temporary file from here (2) and the implementation loops unless it succeeds to create an empty file. (3) - But that fails to copy the file to out because copying does not work when the target file already exists.

(1) https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/triggers/CustomClassLoader.java#L86

(2) https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/triggers/CustomClassLoader.java#L81

(3) https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/io/util/FileUtils.java#L152
Bug: [CircleCI] Add JDK 11 Java and Python upgrade tests in trunk
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 25/Feb/23 19:31
Resolved: 14/Jun/23 14:24

Currently we test upgrades with Java 8, in preparation to drop it we need first to ensurer we test with 11. -This has to be added not only in trunk,, but also 4.1-
This ticket will be used for trunk. A separate one  will be opened for 4.0 and 4.1.
I believe [~mck] has this ready to push in Jenkins at some point so this ticket should accommodate the addition of those upgrade tests in CircleCI for 4.1 and trunk.
Bug: Upgrade maven-shade-plugin to fix shaded dtest JAR build
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 05/Jan/23 19:29
Resolved: 14/Mar/23 21:00

Could not build shaded dtest JAR with ./build-shaded-dtest-jar.sh due to:
{code:java}
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project cassandra-dtest-shaded: Error creating shaded jar: Problem shading JAR ~/Repos/apache/cassandra/target/cassandra-dtest-shaded-4.0.1-SNAPSHOT.jar entry net/openhft/chronicle/wire/YamlWire$TextValueIn.class: org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class net/openhft/chronicle/wire/YamlWire$TextValueIn.class: 65536 -> [Help 1] {code}
 

Tried on both Java 8 and Java 11, included ant clean / realclean / unlinking the entire ~/.m2/repository.

 

Fixed by upgrading maven-shade-plugin in relocate-dependencies.pom:
{code:java}
                 <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-shade-plugin</artifactId>
-                <version>3.2.1</version>
+                <version>3.4.1</version>{code}
Bug: Update build.xml for experimental JDK17 use
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Jan/23 21:33
Resolved: 12/Feb/23 19:57

To support JDK17 we need to make changes to build.xml and our scripts. There is a preliminary patch which makes a switch from JDK8+JDK11 to JDK11+JDK17 but it will need a change considering CASSANDRA-18133

CC [~mck]
Bug: Remove WaitingOnFreeMemtableSpace metric
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Mar/23 17:41
Resolved: 24/Jul/23 10:55

It doesn't seem like this metric is [ever updated|https://github.com/apache/cassandra/search?q=WaitingOnFreeMemtableSpace], or ever was.

While we're at it we should remove any other unused metric.
Bug: Accord: transition Node.Id from long to int
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/Jan/23 15:34
Resolved: 06/Feb/23 14:40

Trimmed down version of the patch, swapping long for int, but still keeping Node.Id class. Delaying the rest (elimination of Node.Id as a class) until immutability work lands, to ease rebase pain.
Bug: Add feature flag for dynamic data masking
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 09/Mar/23 17:54
Resolved: 16/Mar/23 20:57

Dynamic data masking ([CEP-20|https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-20%3A+Dynamic+Data+Masking]) is a new feature, so it will need a feature flag in {{cassandra.yaml}}. Something like:
{code}
# If enabled, dynamic data masking allows to attach CQL masking functions to the columns of a table.
# Users without the UNMASK permission will see an obscured version of the values of the columns with an attached mask.
# If dynamic data masking is disabled it won't be allowed to create new column masks, although it will still be possible
# to drop any previously existing masks. Also, any existing mask will be ignored at query time, so all users will see
# the clear values of the masked columns.
dynamic_data_masking_enabled: false
{code}
Bug: Feature documentation lost / moved out of focus
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 06/Mar/23 14:08
Resolved: 09/Mar/23 20:02

Documentation added with CASSANDRA-17344 wasn't moved with CASSANDRA-17976 and now the documentation on the website page "Cassandra/Operating/Virtual tables" shows an outdated version for [4.1|https://cassandra.apache.org/doc/4.1/cassandra/operating/virtualtables.html] and [latest|https://cassandra.apache.org/doc/latest/cassandra/operating/virtualtables.html].
Bug: Fix tests post JDK-8210522 (rewrite reflection of "modifiers" field)
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 19/Jan/23 23:58
Resolved: 30/Jan/23 22:25

 

From JDK-8210522:
{code:java}
Core reflection has a filtering mechanism to hide security and integrity sensitive fields and methods from Class getXXXField(s) and getXXXMethod(s). The filtering mechanism has been used for several releases to hide security sensitive fields such as System.security and Class.classLoader.
This CSR proposes to extend the filters to hide fields from a number of highly security sensitive classes in java.lang.reflect and java.lang.invoke.
{code}
We are using at a few places in our tests 
{code:java}
Field.class.getDeclaredField("modifiers");{code}
This breaks as expected when tests are run with JDK17, example:
 
{code:java}
java.lang.RuntimeException: java.lang.NoSuchFieldException: modifiers
 at org.apache.cassandra.transport.MessagePayloadTest.makeCqlQueryHandlerAccessible(MessagePayloadTest.java:79)
 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.lang.reflect.Method.invoke(Method.java:568)
 at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
 at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
 at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
 at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
 at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
 at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
 at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
 at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
 at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:221)
 at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54) 
Caused by: java.lang.NoSuchFieldException: modifiers at java.base/java.lang.Class.getDeclaredField(Class.java:2610) 
at org.apache.cassandra.transport.MessagePayloadTest.makeCqlQueryHandlerAccessible(MessagePayloadTest.java:70) 
... 15 more{code}