av9ash/GitBugs huggingface dataset:
Bug: Add the ability to disable bulk loading of SSTables on a node
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Aug/23 00:15
Resolved: 19/Sep/23 11:20

Currently, Cassandra database users can use sstableloader to bulk load data into Cassandra. However, for a Cassandra operator, there is no way to forcibly block this behavior. Additionally, there is no metric indicating whether the bulk load is being used on the server side. If a client is using sstableloader, they will also need to upgrade the sstableloader code to the new major version. This lack of control and visibility can become a blocker during a major version upgrade.

 

1. Can we add a config to disable bulk load feature? Or it falls into https://issues.apache.org/jira/browse/CASSANDRA-8303

2. Can we add metrics for bulk load used on server end?
Bug: Fix logging / exception output on mismatched cache and schema version
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Sep/23 06:59
Resolved: 18/Sep/23 19:34

Sometimes I see this in the logs and Cassandra just continues to start as nothing happened:

{code}
INFO  [loadSavedCache:1] 2023-09-18 08:53:05,395 AutoSavingCache.java:214 - Reading saved cache: /home/fermat/dev/cassandra/cassandra-instaclustr/cassandra/data/saved_caches/KeyCache-g.db, /home/fermat/dev/cassandra/cassandra-instaclustr/cassandra/data/saved_caches/KeyCache-g.crc, /home/fermat/dev/cassandra/cassandra-instaclustr/cassandra/data/saved_caches/KeyCache-g.metadata
INFO  [loadSavedCache:1] 2023-09-18 08:53:05,403 AutoSavingCache.java:277 - Harmless error reading saved cache /home/fermat/dev/cassandra/cassandra-instaclustr/cassandra/data/saved_caches/KeyCache-g.db
java.lang.RuntimeException: Cache schema version e40b993e-88c0-314c-8a5e-c76871f0a144 does not match current schema version f1884cc9-7d66-3b5f-85e5-4bc0d647c885
	at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:228)
	at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
	at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)
{code}

Well if the log says it is harmless, why do we need to log the exception? There is no reason to print that stacktrace, it is just enough to print the mismatching versions. It should be also changed from INFO to WARN, or maybe better to DEBUG? I mean ... it is harmless, right? Why should I see, by default, that something harmless has happened?
Bug: CIDR permission cache probably does not work
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 18/Sep/23 12:32
Resolved: 18/Sep/23 12:38

CIDR permission cache code:

{code:java}
    /**
     * Invalidate a role from CIDR permissions cache
     * @param roleName role for which to invalidate the cache
     * @return boolean returns true if given role found in the cache and invalidated, otherwise returns false
     */
    public boolean invalidateCidrPermissions(String roleName)
    {
        if (cache.getIfPresent(roleName) == null)
            return false;

        invalidate(RoleResource.role(roleName));
        return true;
    }
{code}

passes {{String}} role name to the cache and if it returns {{null}} it just returns {{false}}. It will always return {{null}} because cache expects objects of type {{RoleResource}}.
Bug: Optimize chunk cache
Status: Triage Needed
Priority: Normal
Created: 15/Sep/23 11:02

1. Chunk cache hashcode computation can be costly as it is on a hot path, would be good to precompute it (as well as hashcode of some other immutable classes).

2. Invalidation of a file can be costly if the cache is really large. We should rather rely on the automatic eviction of unused entries.
Bug: Test failure: org.apache.cassandra.tools.StandaloneSplitterWithCQLTesterTest.testNoSnapshotOption-.jdk17
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 28/Jul/23 22:02
Resolved: 11/Aug/23 20:04

Seen here:

[https://ci-cassandra.apache.org/job/Cassandra-trunk/1650/testReport/junit/org.apache.cassandra.tools/StandaloneSplitterWithCQLTesterTest/testNoSnapshotOption__jdk17/]
h3.  
{code:java}
Error Message
java.lang.reflect.InaccessibleObjectException: Unable to make field private final sun.nio.fs.UnixFileSystem sun.nio.fs.UnixPath.fs accessible: module java.base does not "opens sun.nio.fs" to unnamed module @1f28c152 at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354) at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297) at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178) at java.base/java.lang.reflect.Field.setAccessible(Field.java:172) at org.apache.cassandra.utils.concurrent.Ref.getFields(Ref.java:656) at org.apache.cassandra.utils.concurrent.Ref$Visitor.traverse(Ref.java:613) at org.apache.cassandra.utils.concurrent.Ref$Visitor.run(Ref.java:568) at org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:833)
{code}
Bug: Investigate suppressed streaming error
Status: Open
Priority: Normal
Created: 06/Sep/23 13:49

In CASSANDRA-18815, a [previously suppressed exception|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/streaming/StreamSession.java#L700] began to surface, requiring the tests to ignore it.  However, the exception does not appear on the 4.1 branch even though it looks like it should.  We should investigate and solve this.
Bug: Test failure: org.apache.cassandra.db.RangeTombstoneTest#overlappingRangeTest
Status: Open
Priority: Normal
Created: 13/Sep/23 12:04

The unit test {{org.apache.cassandra.db.RangeTombstoneTest#overlappingRangeTest}} seems to be flaky at least on 4.1 and 5.0:

* 4.1: https://app.circleci.com/pipelines/github/adelapena/cassandra/3206/workflows/e43a16ff-5849-40e6-b78b-7f732d748b86/jobs/78331/tests
* 5.0: https://app.circleci.com/pipelines/github/adelapena/cassandra/3204/workflows/96c19c84-6acc-4f71-9805-5a451aca9eaf/jobs/77991/tests

{code}
java.lang.NullPointerException: Cannot invoke "org.apache.cassandra.db.rows.Row.hasLiveData(long, boolean)" because the return value of "org.apache.cassandra.db.partitions.Partition.getRow(org.apache.cassandra.db.Clustering)" is null
	at org.apache.cassandra.db.RangeTombstoneTest.overlappingRangeTest(RangeTombstoneTest.java:421)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
{code}
The repeated runs can be generated with:
{code}
.circleci/generate.sh -p -s -e REPEATED_UTESTS=org.apache.cassandra.db.RangeTombstoneTest -e REPEATED_UTESTS_COUNT=1000
{code}
The error seems to have a flakiness under 0.1%, and I haven't seen it on Butler.
Bug: Source location is lost when an exception is thrown from the coordinator in in-jvm dtests
Status: Patch Available
Priority: Normal
Created: 05/Sep/23 10:53

The mechanism is simple - currently we only throw the cause and the enclosing stacktrace is dropped. Therefore, it is usually difficult to locate which test line is responsible for throwing the exception.

For example, an induced error caused by an invalid query in {{BooleanTest}} yields the following message:
{noformat}
org.apache.cassandra.exceptions.InvalidRequestException: Undefined column name dk in table distributed_test_keyspace.tbl
	at org.apache.cassandra.schema.TableMetadata.getExistingColumn(TableMetadata.java:402)
	at org.apache.cassandra.cql3.SingleColumnRelation.newEQRestriction(SingleColumnRelation.java:184)
	at org.apache.cassandra.cql3.Relation.toRestriction(Relation.java:143)
	at org.apache.cassandra.cql3.restrictions.StatementRestrictions.<init>(StatementRestrictions.java:205)
	at org.apache.cassandra.cql3.restrictions.StatementRestrictions.<init>(StatementRestrictions.java:145)
	at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:1276)
	at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:1136)
	at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:1126)
	at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:1098)
	at org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:857)
	at org.apache.cassandra.distributed.impl.Coordinator.unsafeExecuteInternal(Coordinator.java:109)
	at org.apache.cassandra.distributed.impl.Coordinator.unsafeExecuteInternal(Coordinator.java:103)
	at org.apache.cassandra.distributed.impl.Coordinator.lambda$executeWithResult$0(Coordinator.java:66)
	at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
	at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)
{noformat}

It does not provide any information on which line in the {{BooleanTest.java}} caused the problem.
Bug: Fix the way we produce a DTest jar
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 25/Jul/23 14:31
Resolved: 27/Jul/23 08:16

The problem is related to how we produce the {{dtest jar}}. That repackages all dependencies along with our classes in a single jar. Some dependencies might have digital signatures extracted into {{META-INF}} as, say, {{*.SF}} files. We need to remove them because they are invalid for the uber-jar we produce. Therefore, there are exclusions in {{zipgroupfileset}} operations, more or less like this:

{code:xml}
<zipgroupfileset dir="${build.lib}" includes="*.jar" excludes="META-INF/*.SF"/>
{code}

However, this does not work as expected. The {{exclude}} attribute denotes jar files we don't want to repackage rather than filter on the content of the jar file. The new BouncyCastle jars include signatures in SF and DSA files, which all land in the final {{dtest.jar}} regardless of that filter.

See also https://stackoverflow.com/questions/35577351/exclude-jar-from-ant-build-using-zipgroupfileset/35593838#35593838

The issue was discovered and explained by [~jlewandowski] in CASSANDRA-17992
Bug: CircleCI: Fix BASE_BRANCH in generate.sh for cassandra-5.0
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 25/Aug/23 17:31
Resolved: 28/Aug/23 13:36

We have hardcoded BASE_BRANCH in the CircleCI script generate.sh. That one is used for the diff comparison when we detect new/changed tests.

After adding cassandra-5.0, BASE_BRANCH needs to be updated from trunk to cassandra-5.0 in the newly created branch.
Bug: Attempted NTS creation logs even on failure
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Jul/23 16:17
Resolved: 02/Aug/23 15:50

If you try to create a keyspace with a nonexistent DC, you will still see a log like this:

{noformat}
INFO  [Native-Transport-Requests-1] 2023-07-21 11:02:40,062 NetworkTopologyStrategy.java:93 - Configured datacenter replicas are DC2:rf(3), DC1:rf(1)
{noformat}

even if you receive:
{noformat}
ConfigurationException: Unrecognized strategy option {DC2} passed to NetworkTopologyStrategy for keyspace test
{noformat}

which prevented the creation.
Bug: Microbench tests fail on JDK11
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 10/Jul/23 16:03
Resolved: 17/Jul/23 16:41

When I run 
{code:java}
ant microbench -Dbenchmark.name=ReadWriteTest{code}
 using jdk8 on ds-trunk, it finishes successfully, but when I run it on jdk11 on ds-trunk, I get this error. 
{code:java}
Caused by: java.lang.IllegalAccessException: access to public member failed: jdk.internal.ref.Cleaner.clean[Ljava.lang.Object;@5189c629/invokeVirtual, from org.apache.cassandra.io.util.FileUtils (unnamed module @6cbf4322){code}
We suspect that ant target does not use the jvmargs as other test targets do.
Bug: Remove AssertJ annotations from the production code
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Jul/23 11:06
Resolved: 28/Jul/23 15:43

We are using VisibleForTesting annotation from AssertJ in the production code. At the same time, we are not shipping assertj library in the tarball. 

The way it works is that we depend on assertj with "provided" scope and then we will not ship the lib. It is possible, because the default RetentionPolicy for that annotation is CLASS which means that while we compile with the annotation, it does not need to be there in runtime ...

There is VisibleForTesting annotation from Guava which is used extensively. Usage of VisibleForTesting from AssertJ was used by mistake as that probably popped up as the first import to add when doing code completion in IDEA.
Bug: Expose bootstrap failure state via JMX
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/Aug/23 08:47
Resolved: 20/Aug/23 17:20

Similar to how we have exposed a node's decommissioning via JMX as part of CASSANDRA-18555.

This ticket will address the same when a new node joins (i.e., bootstrap). 

When a node is bootstrapped, an error is thrown back to the caller if any failure happens.

But Cassandra's bootstrap takes considerable time ranging from minutes to hours to days. There are various scenarios in that the caller may need to probe the status again:
 * The caller times out
 * It is not possible to keep the caller hanging for such a long time

And If the caller does not know what happened internally, then it cannot retry, etc., leading to other issues.

So, in this ticket, a new nodetool/JMX option will be exposed that can be invoked by the caller anytime, and it will return the correct status.
Bug: Move jflex from runtime to build dependencies
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 12/Jul/23 06:19
Resolved: 13/Jul/23 13:47

JFlex depends on java-cup-runtime which has a kind of exotic license saying that it is compatible with GPL. Therefore, we should investigate that case.

JFlex seems to be used only during the build to generate some sources for SASI indexes. Therefore, it should be possible to move JFlex from the runtime dependencies to the build dependencies and not include it in the distribution.
Bug: Improved DeletionTime serialization
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 05/Jul/23 04:29
Resolved: 26/Jul/23 08:49

DeletionTime.markedForDeleteAt is a long useconds since Unix Epoch. But I noticed that with 7 bytes we can already encode ~2284 years. We can either shed the 8th byte, for reduced IO and disk, or can encode some sentinel values as flags there. [~blerer] suggested starting with DeletionTime.LIVE.

That would mean reading and writing 1 byte instead of 12 (8 mfda long + 4 ldts int). Yes we already avoid serializing DeletionTime (DT) in sstables at _row_ level entirely but not at _partition_ level and it is also serialized at index, metadata, etc.
Bug: Update JMH
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/Jul/23 16:24
Resolved: 09/Aug/23 14:16

[~marianne-manaog] was trying to do some profiling and noticed illegal access exceptions. Our version of JMH is from 2018; as per my recommendation, upgrading to the latest one from the end of 2022 solved her problems. The new version is tested with the JDK versions up to 17.
Bug: Remove 3.x from the versions checked for prepared statement behaviour
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/Jul/23 11:46
Resolved: 28/Jul/23 16:06

CASSANDRA-17248 introduced a different behaviour of prepared statements depending on the minimum cluster version. The old behaviour is used when the min cluster version is below 3.0.26, 3.11.12 or 4.0.2. 

However, from 5.0 we won't support direct upgrades from 3.0 or 3.11, so we can eliminate the checks for those versions.
Bug: Update commons-lang3 to 3.13.0
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 22/Aug/23 23:04
Resolved: 28/Aug/23 13:58

It seems we are on a version that was testing up to Java 11 - 3.11.0
In the latest one the community goes up to JDK21 so I think it is good to update, hopefully also on cassandra-5.0
Below are the release notes:
https://commons.apache.org/proper/commons-lang/changes-report.html#a3.12.0
Bug: BTI format's decision to preload partition index is incorrect
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 19/Jul/23 08:38
Resolved: 03/Aug/23 13:57

On load, BTI format SSTables are supposed to walk through the partition index when the SSTable does not have a bloom filter, to prefetch the index in cache.

This decision [is currently inverted|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/sstable/format/bti/BtiTableReaderLoadingBuilder.java#L125C62-L125C62], causing both an unnecessary preload when a filter is present, and making filterless sstables inefficient.
Bug: Empty keystore_password no longer allowed on encryption_options
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 17/Aug/23 20:53
Resolved: 28/Aug/23 08:15

After CASSANDRA-18124 (introduced in 4.1.2 and 5.0) it is no longer possible to set an empty {{keystore_password}} under {{client_encryption_options}} or {{server_encryption_options}} using the default implementation {{{}DefaultSslContextFactory{}}}.

While keytool does not allow generating keystores with empty passwords, it does support reading them. It is not uncommon to use PKCS12 certificates generated by other tools (eg. openssl) that do not enforce passwords.

The fix for this should be pretty straightforward, which should involve changing [FileBasedSslContextFactory.validatePassword|https://github.com/apache/cassandra/blob/cassandra-4.1.2/src/java/org/apache/cassandra/security/FileBasedSslContextFactory.java#L128-L135] to only disallow null passwords (which would be consistent with previous versions). I will create pull requests against the relevant branches shortly.
{noformat}
Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: Failed to initialize SSL
org.apache.cassandra.exceptions.ConfigurationException: Failed to initialize SSL
	at org.apache.cassandra.config.DatabaseDescriptor.applySslContext(DatabaseDescriptor.java:1155)
	at org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:390)
	at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:204)
	at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:188)
	at org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:804)
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:747)
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:875)
Caused by: java.io.IOException: Failed to create SSL context using Native transport
	at org.apache.cassandra.security.SSLFactory.validateSslContext(SSLFactory.java:405)
	at org.apache.cassandra.config.DatabaseDescriptor.applySslContext(DatabaseDescriptor.java:1150)
	... 6 more
Caused by: java.lang.IllegalArgumentException: 'keystore_password' must be specified
	at org.apache.cassandra.security.FileBasedSslContextFactory.validatePassword(FileBasedSslContextFactory.java:133)
	at org.apache.cassandra.security.FileBasedSslContextFactory.buildKeyManagerFactory(FileBasedSslContextFactory.java:151)
	at org.apache.cassandra.security.AbstractSslContextFactory.createNettySslContext(AbstractSslContextFactory.java:181)
	at org.apache.cassandra.security.SSLFactory.createNettySslContext(SSLFactory.java:168)
	at org.apache.cassandra.security.SSLFactory.validateSslContext(SSLFactory.java:355)
	... 7 more
{noformat}
Bug: netty-all vulnerability: CVE-2023-34462
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 05/Jul/23 10:54
Resolved: 06/Jul/23 15:08

This is failing owasp:

https://nvd.nist.gov/vuln/detail/CVE-2023-34462

{quote}
The `SniHandler` can allocate up to 16MB of heap for each channel during the TLS handshake. When the handler or the channel does not have an idle timeout, it can be used to make a TCP server using the `SniHandler` to allocate 16MB of heap.
{quote}
Bug: CEP-15 (C*): when loading commands that have empty waiting_on, make sure not to loose the partial deps
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Aug/23 19:06
Resolved: 21/Aug/23 20:35

This was found in benchmark clusters

{code}
java.lang.IllegalStateException: Deps do not match; expected {}, {} == [tlp_stress:[(-1182483594468561006,-472993437787424406]]]:{tlp_stress:DecoratedKey(-1000036068852656106, 3030312e302e35383730313035):[[17,1692599889087003,0,1]]}, {}
	accord.utils.Invariants.illegalState(Invariants.java:44)
	accord.utils.Invariants.checkState(Invariants.java:133)
	accord.local.Command$Committed.<init>(Command.java:826)
	accord.local.Command$Committed.<init>(Command.java:817)
	accord.local.Command$Executed.<init>(Command.java:905)
	accord.local.Command$Executed.executed(Command.java:951)
	accord.local.Command$SerializerSupport.executed(Command.java:165)
	org.apache.cassandra.service.accord.AccordKeyspace.loadCommand(AccordKeyspace.java:1197)
{code}

The root cause is that “empty” waiting_on drops the partial deps when we write/load but this becomes an issue for Command.Committed as it double checks partial_deps == waiting_on.deps
Bug: Fix Depends and Build-Depends for Java for Debian packages
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 30/Aug/23 18:40
Resolved: 30/Aug/23 19:51

Same needs to happen for 5.0 and trunk as was done in CASSANDRA-18751.

There is still Java 8 and Java 11 combo. We need to throw away Java 8 and add Java 17.

FYI Debian 12 does not have openjdk Java 11 anymore.