av9ash/GitBugs huggingface dataset:
Bug: Simplify the bind marker and Term logic
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 31/Aug/23 12:43
Resolved: 23/Jan/24 13:19

The current logic around {{Term}} and {{Terms}} classes is confusing specially with {{MultiItemTerminal}} and {{MultiColumnRaw}} that are used to handle different use cases that could be handled simply with the {{Term}} interface.

On top of that IN marker add to the confusion because the are represented as single Term where in practice they are a set of terms. Representing them as a {{Terms}} could simplify  the way we handle IN restrictions.

The goal of this ticket is:
*  to refactor the {{Term}} and {{Terms}} interfaces to simplify the logic
* Represents IN bind marker as {{Terms}} instead of having 2 different representations (a list of terms and a single {{MultiItemTerminal}}.
* Simplify the {{AbstractMarker}} hierachy
Bug: Limit Java runtime in 5.1 to JDK 11 and 17 in scripts; add a flag to opt out of that
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Jul/23 23:42
Resolved: 19/Jan/24 14:11

Currently, we limit our users from building with non-default Java versions in build.xml.

They can easily hack build.xml for test purposes with different versions.

Cassandra–5.1 will be run on JDK11 and JDK17, but on startup, we do not limit people to those two, but only to everything >= 11. We should also put an upper limit of 17 in our Cassandra startup scripts. We can also add a flag to opt-out if someone wants to test with newer versions.
Bug: Make publishing to https://pypi.org/project/cqlsh/ part of Cassandra core
Status: Triage Needed
Priority: Normal
Created: 06/Jul/23 18:38

Myself and [~bschoeni] currently maintain https://github.com/jeffwidman/cqlsh which  repackages the CQLSH from Cassandra it into a standalone Python package that is published to https://pypi.org/project/cqlsh/ for lighter-weight installs.

Given that our project is simply a build/packaging project, we wanted start a conversation about upstreaming this into core Cassandra.

We don't have any differences from the CQLSH source that's in cassandra, instead we point all contributors to contribute back to the Cassandra project. In fact we've made multiple contributions back to `cqlsh` ourselves and have drastically cleaned up the code: https://github.com/search?q=repo%3Aapache%2Fcassandra%20is%3Apr%20author%3Ajeffwidman%20author%3Abschoening&type=pullrequests

Our goal: 
When a Cassandra release happens, the build/release process automatically publishes cqlsh to https://pypi.org/project/cqlsh/

Benefits:
- more efficient because we no longer have to manually copy/paste cqlsh source around, it'd be automatically done by CI.
- easier for folks to test python packaging because they could use this pipeline to build the core source locally into a python package.
- reduced bus factor because the PyPI namespace would be under the ownership of core Cassandra rather than just Jeff/Brad.

We realize the Cassandra team isn't python/PyPI experts, so we'd be more than happy to help wire this up and maintain it.

I also realize that Cassandra has no interest in maintaining lots of build targets... but given that cqlsh is written in Python and PyPI is the cannonical place to publish python packages, this seems like a natural fit for core cassandra rather than a standalone project.

Our one hesitation around this discussion is that we're a little concerned that we might lose the nimbleness we've currently got from having a separate project. Ie, if something is screwed up on PyPI / the build process, we can quickly get it fixed and get a new release out so that users aren't blocked. Would it be possible as part of this process to continue that myself/Brad had commit rights to the build process for PyPI? To be clear, we aren't asking for commit rights to the Java code or anything outside of Python, we just want to be sure that if we go to the trouble of working with you to upstream this that there's a commitment from Cassandra to keeping this build working, or to letting us be able to fix the build. Otherwise there's no point in upstreaming it only for it to go unmaintained and us look on helplessly from the sidelines. We're very flexible here on the solution.

Thoughts?
Bug: (Accord): Bug fixes from CASSANDRA-18675 to better support adding keyspaces
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Aug/23 16:47
Resolved: 10/Jan/24 22:00

This work is forking CASSANDRA-18675 and pulling out the initial work and bug fixes, but does not actually include drop keyspace support, but does include the testing for add keyspaces.
Bug: Update installation steps for Debian bases systems
Status: Triage Needed
Priority: Normal
Created: 16/Sep/23 07:12

The current installation steps contains the following step to trust the cassandra package:
curl https://downloads.apache.org/cassandra/KEYS | sudo apt-key add -

Instead, it would be better to use the following steps:
{code:java}
echo "deb [signed-by=/etc/apt/keyrings/cassandra.gpg] https://debian.cassandra.apache.org 41x main" | sudo tee -a /etc/apt/sources.list.d/cassandra.sources.list {code}
{code:java}
sudo curl https://downloads.apache.org/cassandra/KEYS -o /etc/apt/keyrings/cassandra.asc
{code}
{code:java}
sudo gpg -o /etc/apt/keyrings/cassandra.gpg --dearmor /etc/apt/keyrings/cassandra.asc {code}
{code:java}
sudo apt-get update
{code}
{code:java}
 sudo apt-get install cassandra {code}
Bug: Fix gen-doc task
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 01/Aug/23 17:20
Resolved: 02/Jan/24 20:10

As discussed on CASSANDRA-17687, we see below errors when ant gen-doc is run:
{code:java}
gen-asciidoc: [exec] python3 ./scripts/gen-nodetool-docs.py [exec] python3 ./scripts/convert_yaml_to_adoc.py ../conf/cassandra.yaml ./modules/cassandra/pages/configuration/cass_yaml_file.adoc [exec] Traceback (most recent call last): [exec] File "/home/jlewandowski/dev/cassandra/trunk/doc/./scripts/convert_yaml_to_adoc.py", line 144, in <module> [exec] convert(yaml_file, dest_file) [exec] File "/home/jlewandowski/dev/cassandra/trunk/doc/./scripts/convert_yaml_to_adoc.py", line 61, in convert [exec] with open(dest_file, 'w') as outfile: [exec] FileNotFoundError: [Errno 2] No such file or directory: './modules/cassandra/pages/configuration/cass_yaml_file.adoc' [exec] make: *** [Makefile:26: gen-asciidoc] Error 1 [exec] Result: 2{code}
-This seems to have been broken during the migration from rst to asciidoc.-
Bug: [Analytics] Make bulk writer resilient to cluster resize events
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 14/Sep/23 19:33
Resolved: 20/Dec/23 17:08

Currently, the analytics bulk writer fails-fast if there are any ongoing cluster resizing events such as expansion, shrink, host-replacement or node movements. 

This effort is make the bulk writer resilient to these events by having the writes account for the transient nodes and their corresponding pending token ranges.

Related: https://issues.apache.org/jira/browse/CASSANDRASC-60
Bug: Tab complete datacenter values in cqlsh
Status: Open
Priority: Normal
Created: 11/Sep/23 21:36

cqlsh has a number of great tab completions.  For example, when creating a keyspace it will tab complete the syntax for options and give you options for the replication strategy.  It doesn't show options for the data centers, which would be nice to have.  The server has access to the list of data centers in the cluster.  So there shouldn't be a reason why that couldn't tab complete.
Bug: Add support for vector search in SAI
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 01/Aug/23 15:58
Resolved: 29/Oct/23 11:26

The patch associated with this ticket adds a new vector index to SAI. This introduces the following new elements and changes to SAI:
 * VectorMemtableIndex - the in-memory representation of the vector indexes that writes data to a DiskANN instance
 * VectorSegmentBuilder - that writes a DiskANN graph to the following on-disk components:
 ** VECTOR - contains the floating point vectors associated with the graph
 ** TERMS - contains the HNSW graph on-disk representation written by a HnswGraphWriter
 ** POSTINGS - contains the index postings as written by a VectorPostingsWriter
 * VectorIndexSegmentSearcher - used to search the on-disk DiskANN graph
Bug: Importer should build SSTable indexes successfully before making new SSTables readable
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 14/Jul/23 19:56
Resolved: 24/Jul/23 22:53

Importing SSTables w/ already built SAI components is pretty unproblematic. However, the case where we import SSTables without accompanying index components can be problematic in two ways, given the indexes are built after the SSTables themselves are added to the tracker:

1.) The new SSTables can be seen for reads on the base table before their attached indexes are built and ready to query in the index {{View}}.

2.) If a failure occurs in an index build, the import just leaves the index in a compromised state, and even though a failure to import will be reported to the user, the SSTables will be readable for queries on the base table.

If the index implementation allows it (and SAI does), {{SSTableImporter}} should make sure SSTable indexes are built before making SSTables visible as part of the base table. If an index build fails, the import should fail and it should appear to ongoing reads like it never happened, including the index continuing to be queryable.
Bug: Introduce in-jvm dtest for testing in Analytics project
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Aug/23 20:19
Resolved: 06/Oct/23 20:01

Now that the in-jvm dtest framework supports JMX, we’ve updated the Cassandra sidecar to use that as its base test framework. We should use this support in the Analytics project so we can run tests more quickly and allow for more advanced testing (startup/shutdown/move/decommission/expand)

Work to complete this ticket would be to incorporate the sidecar’s test framework into the analytics project and get the `SampleCassandraJob` to run successfully on an in-jvm 3-node cluster with the Sidecar running. We can then leverage this framework in future tests.
Bug: UnifiedCompactionTask is incorrectly setting keepOriginals
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Aug/23 13:20
Resolved: 23/Nov/23 15:29

{code:java}
super(cfs, txn, gcBefore, strategy.getController().getIgnoreOverlapsInExpirationCheck());{code}
in {{UnifiedCompactionTask}} is calling the base constructor
{code:java}
 public CompactionTask(ColumnFamilyStore cfs, LifecycleTransaction txn, long gcBefore, boolean keepOriginals)
{code}
which can set {{keepOriginals}} to true when it should not be.
Bug: Test failure: org.apache.cassandra.io.DiskSpaceMetricsTest.testFlushSize-.jdk17 (from org.apache.cassandra.io.DiskSpaceMetricsTest-.jdk17)
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 28/Jul/23 22:23
Resolved: 17/Nov/23 12:10

Seen here:

[https://ci-cassandra.apache.org/job/Cassandra-trunk/1644/testReport/org.apache.cassandra.io/DiskSpaceMetricsTest/testFlushSize__jdk17/]
h3.  
{code:java}
Error Message
expected:<7200.0> but was:<1367.83970468544>

Stacktrace
junit.framework.AssertionFailedError: expected:<7200.0> but was:<1367.83970468544> at org.apache.cassandra.io.DiskSpaceMetricsTest.testFlushSize(DiskSpaceMetricsTest.java:119) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
{code}
Bug: Remove deprecated options in CompressionParams
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 10/Aug/23 16:20
Resolved: 16/Oct/23 18:59

These should go away (1). They were deprecated like 8 years ago in CASSANDRA-9712 and CASSANDRA-9839.

We should also remove this (2). That is a little bit more tricky but nothing special I would say ...

(1) https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/schema/CompressionParams.java#L86-L88
(2) https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/schema/CompressionParams.java#L96-L97
Bug: Java's Adaptive IHOP may allow removing server.options -XX:InitiatingHeapOccupancyPercent override
Status: Open
Priority: Normal
Created: 21/Jul/23 17:48

The  jvm17-server.options retains the -XX:InitiatingHeapOccupancyPercent=70 setting vs 45% JVM default.  But Java 11 introduced Adaptive IHOP and this override may not be required anymore, or at least it should be performance tested. 

Java 11 ends active support on Sept 2023, so it may not be important to consider changing this for jvm11-server.options
Bug: Add optional task to run Sonar analysis
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 22/Aug/23 08:45
Resolved: 13/Nov/23 13:52

Add the ability to run the Sonar analysis and submit results to the Sonar Server instance.

Additional Ant tasks are available to set up the Sonar server locally in a docker container. The scripts can configure the local server instance using the REST API, which includes setting up the project, getting the analysis token, and installing the quality profile and quality gate (stored in {{.build/sonar}} directory).

The usage description is available in {{.build/README.md}}
Bug: IsolatedJMX should not release all TCPEndpoints on instance shutdown
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/Aug/23 13:10
Resolved: 24/Aug/23 22:52

In the original implementation of the JMX feature, we fixed some memory leaks by clearing some internal state in Java’s TCPEndpoint. However, that implementation was overly aggressive and cleared the whole map, vs. just removing the endpoints created by the individual instances. This causes issues when you remove a node from the cluster (as all of the endpoints are cleared, not just the ones in use by that instance).
 
In stead, we should check if the endpoint was created by the instance in question and only remove it if it was.
Bug: Add Jepsen's Elle to Accord and Paxos validation
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Sep/23 22:00
Resolved: 07/Nov/23 18:16

Paxos and Accord both use a custom validator to make sure we offer Strict Serializability, but to increase confidence we should use external validators as well, such as Jepson’s Elle.
Bug: Replace intercepting System.exit by a custom SecurityManager with something else
Status: Open
Priority: Normal
Created: 31/Jul/23 06:17

{{SecurityManager}} is currently used to intercept JVM shutdown in several places. Unfortunately, it will be dropped from JDK, and so far, there has been no alternative mechanism implemented in JDK. It already results in many runtime warnings when running Cassandra 5.0 with JDK 17 (those warnings cannot be disabled).

https://openjdk.org/jeps/411

We may wait for something provided by JDK in a future release or maybe implement a custom solution if needed.
Bug: ClientRequestSize metrics should not treat CONTAINS restrictions as being equality-based
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Sep/23 21:44
Resolved: 30/Oct/23 19:35

The following behavior needs to be changed to consider the column restricted by {{CONTAINS}} or {{CONTAINS KEY}} as "read", rather than "provided by the client". We already do this for things like range restrictions, and the current behavior is inconsistent.

{noformat}
@Test
public void shouldRecordReadMetricsForContainsQuery() throws Throwable
{
    createTable("CREATE TABLE %s (pk int, ck int, v set<int>, PRIMARY KEY (pk, ck))");

    executeNet(CURRENT, "INSERT INTO %s (pk, ck, v) VALUES (1, 1, {1, 2, 3} )");
    executeNet(CURRENT, "INSERT INTO %s (pk, ck, v) VALUES (2, 2, {4, 5, 6})");
    executeNet(CURRENT, "SELECT * FROM %s WHERE v CONTAINS 1 ALLOW FILTERING");

    assertEquals(1, ClientRequestSizeMetrics.totalRowsRead.getCount());
    // The filtering term is provided by the client in the request, so we don't consider that column read.
    assertEquals(2, ClientRequestSizeMetrics.totalColumnsRead.getCount());
}
{noformat}

The fix should be literally two lines, one in {{SingleRestriction}} and one in the test above.
Bug: Update org.caffinitas.ohc:ohc-core dependency and remove org.caffinitas.ohc:ohc-core-j8
Status: Changes Suggested
Priority: Normal
Created: 28/Aug/23 17:57

It seems the `org.caffinitas.ohc:ohc-core-j8` is not required anymore and the Cassandra project moved away from JDK8 support. The changes are here: CASSANDRA-18255

We can remove it.
Bug: Support Dynamic Port Allocation for in-jvm dtest framework
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Aug/23 18:33
Resolved: 28/Aug/23 21:58

Currently, {{INodeProvisionStrategy}} supports two strategies {{OneNetworkInterface}} and {{MultipleNetworkInterfaces}}. However the {{seedPort}}, {{storagePorts}}, {{nativeTransportPorts}}, and {{jmxPorts}} are always fixed or a function of the node number.

In order to better support parallel test runs, we need to support dynamic port allocation for the {{seedPort}}, {{storagePorts}}, {{nativeTransportPorts}}, and {{jmxPorts}}. This would enable us to more easily write tests that can run in parallel. This effort is only a stepping stone in what's required to run more tests in parallel, but it allows us to begin somewhere with the in-jvm dtest framework.
Bug: Test failure: Fix Python DTests post CASSANDRA-18676
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 08/Aug/23 19:46
Resolved: 14/Aug/23 17:25

From Butler:
|failling|DTEST|u.cql_tests |[TestCQLNodes3RF3_Upgrade_current_4_1_x_To_indev_trunk.test_negative_timestamp (upgrade) |https://butler.cassandra.apache.org/#/ci/upstream/workflow/Cassandra-trunk/failure/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_4_1_x_To_indev_trunk/test_negative_timestamp]|
|failing|DTEST|u.cql_tests |[TestCQLNodes2RF1_Upgrade_indev_4_0_x_To_indev_trunk.test_negative_timestamp (upgrade) |https://butler.cassandra.apache.org/#/ci/upstream/workflow/Cassandra-trunk/failure/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_indev_4_0_x_To_indev_trunk/test_negative_timestamp]|
|failing|DTEST|u.cql_tests |[cls.test_negative_timestamp (upgrade) |https://butler.cassandra.apache.org/#/ci/upstream/workflow/Cassandra-trunk/failure/upgrade_tests.cql_tests/cls/test_negative_timestamp]|
|failing|DTEST|u.cql_tests |[TestCQLNodes3RF3_Upgrade_indev_4_1_x_To_indev_trunk.test_negative_timestamp (upgrade) |https://butler.cassandra.apache.org/#/ci/upstream/workflow/Cassandra-trunk/failure/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_indev_4_1_x_To_indev_trunk/test_negative_timestamp]|
|failing|DTEST|u.cql_tests |[TestCQLNodes3RF3_Upgrade_indev_4_0_x_To_indev_trunk.test_negative_timestamp (upgrade) |https://butler.cassandra.apache.org/#/ci/upstream/workflow/Cassandra-trunk/failure/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_indev_4_0_x_To_indev_trunk/test_negative_timestamp]|
|failing|DTEST|u.cql_tests |[TestCQLNodes3RF3_Upgrade_current_4_0_x_To_indev_trunk.test_negative_timestamp (upgrade) |https://butler.cassandra.apache.org/#/ci/upstream/workflow/Cassandra-trunk/failure/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_4_0_x_To_indev_trunk/test_negative_timestamp]|
|failing|DTEST|u.cql_tests |[TestCQLNodes2RF1_Upgrade_current_4_0_x_To_indev_trunk.test_negative_timestamp (upgrade) |https://butler.cassandra.apache.org/#/ci/upstream/workflow/Cassandra-trunk/failure/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_4_0_x_To_indev_trunk/test_negative_timestamp]|
|failing|DTEST|u.cql_tests |[TestCQLNodes2RF1_Upgrade_current_4_1_x_To_indev_trunk.test_negative_timestamp (upgrade) |https://butler.cassandra.apache.org/#/ci/upstream/workflow/Cassandra-trunk/failure/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_4_1_x_To_indev_trunk/test_negative_timestamp]|
|failing|DTEST|u.cql_tests |[TestCQLNodes2RF1_Upgrade_indev_4_1_x_To_indev_trunk.test_negative_timestamp (upgrade)|https://butler.cassandra.apache.org/#/ci/upstream/workflow/Cassandra-trunk/failure/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_indev_4_1_x_To_indev_trunk/test_negative_timestamp]|
Bug: Enable unnecessary import from the same package check in IDEs
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Sep/23 20:12
Resolved: 25/Sep/23 08:35

Currently, the unnecessary imports are disabled by the code style and are not displayed by the IDE with the configuration stored in the project root. See the screenshot below. 

It seems the following needs to be done:
- enable this check in the IDE configuration for all supported IDEs;
- update the documentation;
Bug: Fix dtest: TestTopology.test_resumable_decommission
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 07/Sep/23 15:17
Resolved: 12/Sep/23 13:31

Strangely this wasn't failing when CASSANDRA-18815 was done, but it is now for the same cause.

https://ci-cassandra.apache.org/job/Cassandra-5.0/27/testReport/dtest.topology_test/TestTopology/test_resumable_decommission/
Bug: Cassandra 5 Debian package only runs on OpenJDK
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/Sep/23 01:18
Resolved: 19/Sep/23 19:01

A recent change to the Cassandra 5 branch has caused the Cassandra deb package to no longer install on systems that don't run OpenJDK

 

The following line is too restrictive

[https://github.com/apache/cassandra/blob/trunk/debian/control#L14]

It should be modified to use the java11-runtime and java17-runtime virtual packages.
Bug: Test failure: dtest-novnode.transient_replication_test.TestTransientReplicationRepairStreamEntireSSTable.test_optimized_primary_range_repair_with_lcs
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 25/Sep/23 13:40
Resolved: 27/Sep/23 14:27

https://ci-cassandra.apache.org/job/Cassandra-trunk/1718/testReport/dtest-novnode.transient_replication_test/TestTransientReplicationRepairStreamEntireSSTable/test_optimized_primary_range_repair_with_lcs/
{code:java}
Error Message
failed on setup with "TypeError: 'list' object is not callable"

Stacktrace
self = <transient_replication_test.TestTransientReplicationRepairStreamEntireSSTable object at 0x7f4b3a4e4ac0> fixture_dtest_setup = <dtest_setup.DTestSetup object at 0x7f4b3a583250> @pytest.fixture(scope='function', autouse=True) def setup_cluster(self, fixture_dtest_setup): > self.tokens = self.tokens() E TypeError: 'list' object is not callable transient_replication_test.py:206: TypeError
{code}
Bug: Upgrade caffeine to 3.1.8 and fix CIDR permissions cache invalidation
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Aug/23 18:55
Resolved: 03/Oct/23 20:26

3.1.8 is based on Java 11. This version is testing with newer JDK versions, while 2.x versions are based on JDK8, and as I understand, only bug-fix releases are expected.
Bug: TimeWindowCompactionStrategy with unsafe_aggressive_sstable_expiration keeps overlaping SSTable references
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Aug/23 12:48
Resolved: 21/Sep/23 14:26

When {{unsafe_aggressive_sstable_expiration}} is turned on, TWCS should not create or maintain an iterator of overlapping sstables. However, because {{TimeWindowCompactionController}} inherits from {{CompactionController}} and only sets {{ignoreOverlaps}} after the base class has constructed the overlap iterator, it ends up making an overlap iterator and then never updating it.

The end result is that such a compaction keeps references to lots of and likely _all_ other SSTables on the node and thus delays the deletion of obsolete ones by hours or even days.