av9ash/GitBugs huggingface dataset:
Bug: distutils Version classes are deprecated. Use packaging.version instead.
Status: Open
Priority: Low
Created: 10/Mar/23 20:19

Lately I see a lot in Python DTests the below warning:
{code:java}
DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.{code}
Example from running  auditlog_test.py::TestAuditlog::test_archive_on_shutdown with trunk:

 
{code:java}
dtest.py:48
  /cassandra-dtest/dtest.py:48: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    MAJOR_VERSION_4 = LooseVersion('4.0')
 
../../dtest/lib/python3.8/site-packages/ccmlib/common.py:773
auditlog_test.py::TestAuditlog::test_archive_on_shutdown
auditlog_test.py::TestAuditlog::test_archive_on_shutdown
auditlog_test.py::TestAuditlog::test_archive_on_shutdown
auditlog_test.py::TestAuditlog::test_archive_on_shutdown
auditlog_test.py::TestAuditlog::test_archive_on_shutdown
  /dtest/lib/python3.8/site-packages/ccmlib/common.py:773: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    return LooseVersion(match.group(1))
 
auditlog_test.py: 42 warnings
  /dtest/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    other = LooseVersion(other)
 
auditlog_test.py::TestAuditlog::test_archive_on_shutdown
  /cassandra-dtest/conftest.py:437: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    since = LooseVersion(since_str_or_list)
 
auditlog_test.py::TestAuditlog::test_archive_on_shutdown
auditlog_test.py::TestAuditlog::test_archive_on_shutdown
auditlog_test.py::TestAuditlog::test_archive_on_shutdown
auditlog_test.py::TestAuditlog::test_archive_on_shutdown
auditlog_test.py::TestAuditlog::test_archive_on_shutdown
auditlog_test.py::TestAuditlog::test_archive_on_shutdown
auditlog_test.py::TestAuditlog::test_archive_on_shutdown
  /dtest/lib/python3.8/site-packages/ccmlib/common.py:481: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    version = LooseVersion(str(version))
 
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===Flaky Test Report===
{code}
Bug: Move the IntelliJ Idea code style and inspections configuration to the project's root .idea directory
Status: Open
Priority: Low
Created: 22/Feb/23 18:19

h4. Current State

Each IDE supported by Cassandra has its own configuration files, which we store in a special 'ide' directory in the project root. idea - for the IntelliJ IDEA configuration, nbproject - for the NetBeans, and there is currently no directory for the Eclipse configuration exist.

We use ant tasks for initial project setup for a particular IDE:
* generate-idea-files
* generate-eclipse-files

For NetBeans you always choose the ide/nbproject directory as the root of the project, so that the code style configuration for the NetBeans is always shared between developers.

h4. The Issues With Current State

There are a few inconveniences we are currently facing that could be resolved by sharing the IntelliJ IDEA code style settings through VCS:
# Once a new checkstyle lint is implemented, and the IDE's code style configuration is updated they are not immediately available to a developer. The developer must track the git history to use the latest code style changes and manually copy the updated code style files into the .idea directory, or re-run the generate-idea-files task which will erase any locally saved settings (e.g. words dictionaries);
# The code style usually changes by adding new rules one by one, and new checkstyle lints are implemented, so it may differ between release branches. It is a common practice to keep the code style on the same branch so that IntelliJ can use an appropriate configuration for the branch being checked out.
# The IntelliJ IDEA code style configuration format that stored the codeStyleSettings.xml file is outdated, from version 2017.3 the codeStyle directory is used. This prevents us from updating the code style configuration without a lot of tedious manual changes.

The share project settings through VCS﻿ approach is well described in the documentation pages:
https://www.jetbrains.com/help/idea/configure-project-settings.html#share-project-through-vcs

h4. What To Do

# Migrate the IntelliJ code style configuration to the new format, assuming nobody is using such an old version of IntelliJ IDEA;
# Move the code style configuration and inspections configuration files to the project's root .idea directory to address the issues described above;
# Rely on the IntelliJ IDEA code style for backward compatibility instead of supporting it ourselves;
Bug: Extend implicit allow-filtering to clustering keys as well
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 14/Mar/23 19:07
Resolved: 15/Mar/23 12:47

This was overlooked in CASSANDRA-18238. 

We should be able to do selects on vtables not only when selecting on regular columns but also on clustering ones. Currently we can do that on regulars only.
Bug: Add unit tests for per-row TTL and Timestamp usage in CQLSSTableWriter
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 23/Jan/23 18:46
Resolved: 02/Feb/23 14:51

CQLSSTableWriter supports per-row setting of both timestamp and TTL values, but it’s not tested or documented today.  Add tests to cover setting both TTL and Timestamp values for rows using the CQLSSTableWriter.
Bug: Error Unknown column during deserialization missing keyspace and table name
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 20/Mar/23 18:16
Resolved: 06/Jun/23 14:31

The ERROR message generated in ColumnSubselection.java when a column name is not found only prints the column name, not the keyspace and table.  It can be difficult to track down the source when more than one table uses the same name.  E.g., 'id'.
{quote}{{if (column == null)}}
{
{{        column = metadata.getDroppedColumn(name);}}
{{        if (column == null)}}
{{                throw new UnknownColumnException("Unknown column " + UTF8Type.instance.getString(name) + " during deserialization");}}
{{}}}
{quote}
Example:

[ERROR] cluster_id=15 ip_address=192.168.65.10  java.lang.RuntimeException: Unknown column id during deserialization

Proposed:

[ERROR] cluster_id=15 ip_address=192.168.65.10  java.lang.RuntimeException: Unknown column id in table cycling.route during deserialization
Bug: Accumulate all the `docs` PR from GitHub into a single patch
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 22/Jan/23 14:24
Resolved: 26/Jan/23 15:38

In order to make it easier to review the `docs` labeled pull-requests all of them should be accumulated into a single commit as per the [request|https://github.com/apache/cassandra/pull/2062#issuecomment-1385002289] from [~smiklosovic]
Bug: high speculative retries on v4.1.3
Status: Resolved
Priority: Urgent
Resolution: Fixed
Created: 16/Aug/23 14:08
Resolved: 16/Aug/24 18:37

There are up to 10+ times higher speculative retries for reads on 4.1.3 comparing to 4.0.7 and 4.1.2 when using QUORUM and default setting of 99p.

On 4.1.3 after upgrade I see speculative retries for up to 35% of all reads for specific table. Latency for reads is stable around 500 microseconds.

java 1.8.0_382 is used
Bug: Add an optimized default configuration to tests and make it available for new users
Status: Resolved
Priority: Urgent
Resolution: Fixed
Created: 15/Aug/23 09:41
Resolved: 05/Apr/24 15:02

We currently offer only one sample configuration file with Cassandra, and that file is deliberately configured to disable all new functionality and incompatible improvements. This works well for legacy users that want to have a painless upgrade, but is a very bad choice for new users, or anyone wanting to make comparisons between Cassandra versions or between Cassandra and other databases.

We offer very little indication, in the database packaging itself, that there are well-tested configuration choices that can solve known problems and dramatically improve performance. This is guaranteed to paint the database in a worse light than it deserves, and will very likely hurt adoption.

We should find a way to offer a very easy way of choosing between "optimized" and "compatible" defaults. At minimal, we could provide alternate yaml files. Alternatively, we could build on the {{storage_compatibility_mode}} concept to grow it into a setting that not only enables/disables certain settings, but also changes their default values.
Bug: Reduce size of per-SSTable index components
Status: Resolved
Priority: Urgent
Resolution: Fixed
Created: 17/Jul/23 15:29
Resolved: 18/Aug/23 20:02

The current per-SSTable index components are large because the primary keys that are stored in them include the token as part of the byte comparable. The byte comparable puts the token first meaning that we get very little prefix compression from either the trie or the sorted terms store. 

We can fix this by removing the token from the primary key serialization. This would allow us to get the prefix compression from the trie and the sorted terms store.
Bug: Backport CASSANDRA-16905 to older branches
Status: Resolved
Priority: Urgent
Resolution: Fixed
Created: 15/Aug/23 20:54
Resolved: 18/Aug/23 18:01

Recently hit an un-recoverable situation in Cassandra 4.0.10 after dropping a 'map' column and adding it back as a 'blob', which caused corruption that neither offline nor online scrub could fix.
When dropping a 'blob' column and attempting to add it back as a 'map' type, the operation is blocked with:

{code:java}
InvalidRequest: Error from server: code=2200 [Invalid query] message="Cannot re-add previously dropped column 'col1' of type map<int, tinyint>, incompatible with previous type blob"
{code}

We need to do the same going from 'map' to 'blob' to avoid this potentially very bad scenario that can cause data loss.
Bug: Cassandra 4.0.5 nodes fails to start when replacing dead node
Status: Triage Needed
Priority: Urgent
Created: 16/Aug/23 19:11

Trying to replace a down node the new nodes fail to start, using property {_}*replace_address*{_}.

Below message appears continuously in system logs.
{noformat}
WARN  [Messaging-EventLoop-3-2] 2023-08-16 14:18:58,565 NoSpamLogger.java:95 - /xxx.xxx.xxx.xxx:7000->/yyy.yyy.yyy.yyy:7000-URGENT_MESSAGES-[no-channel] dropping message of type GOSSIP_DIGEST_SYN whose timeout expired before reaching the network

INFO  [Messaging-EventLoop-3-2] 2023-08-16 14:19:23,910 NoSpamLogger.java:92 - /xxx.xxx.xxx.xxx->/yyy.yyy.yyy.yyy:7000-URGENT_MESSAGES-[no-channel] failed to connect

io.netty.channel.ConnectTimeoutException: connection timed out: /xxx.xxx.xxx.xxx:7000

    at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$2.run(AbstractEpollChannel.java:576)

    at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)

    at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)

    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)

    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)

    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)

    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)

    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

    at java.lang.Thread.run(Thread.java:748)

{noformat}
 
{noformat}
DEBUG [ScheduledTasks:1] 2023-08-16 19:09:56,919 StorageService.java:2399 - Ignoring application state LOAD from /yyy.yyy.yyy.yyy:7000 because it is not a member in token metadata 

DEBUG [ScheduledTasks:1] 2023-08-16 19:10:56,920 StorageService.java:2399 - Ignoring application state LOAD from /yyy.yyy.yyy.yyy:7000 because it is not a member in token metadata 

 {noformat}

xxx.xxx.xxx.xxx - IP of down node yyy.yyy.yyy.yyy - IP of new node

NO other ERROR/WARNING appears in logs. The node goes into UJ state, but never joins the ring.

While this doesn't happen always, but we are seeing this increased behaviour since upgrading from 3.11.9 to 4.0.5.

Configuration are all fine as to mitigate this we terminate the node and spawn a new one with same configs.
Bug: Upgrade snappy java library
Status: Resolved
Priority: High
Resolution: Fixed
Created: 25/Sep/23 10:30
Resolved: 27/Sep/23 19:17

The snappy java library needs to be updated to fix the latest CVEs.
[https://github.com/xerial/snappy-java/security/advisories/GHSA-55g7-9cwv-5qfv]
Bug: Add 5.0-alpha1 to cassandra-dtest upgrade_manifest.py
Status: Resolved
Priority: High
Resolution: Fixed
Created: 09/Sep/23 10:27
Resolved: 12/Sep/23 12:40

With the release of 5.0-alpha1, that version can be added to the dtest-upgrades.

Unfortunately ccm has a bug in it where it cannot determine the supported jdks for binary installs of 5+

Patches:
 - https://github.com/riptano/ccm/compare/master...thelastpickle:ccm:mck/5.0-alpha1
 - https://github.com/apache/cassandra-dtest/compare/trunk...thelastpickle:cassandra-dtest:mck/5.0-alpha1-bump

CI demonstrating the failure
- 5.0 https://app.circleci.com/pipelines/github/michaelsembwever/cassandra/242/workflows/8eee6b06-46d5-4d42-b137-b696b63a190c/jobs/18955
- trunk https://app.circleci.com/pipelines/github/michaelsembwever/cassandra/243/workflows/d5a12c3d-04d2-4de2-a88f-3c8cae09c341/jobs/18956

CI demonstrating the fix
- 5.0 https://app.circleci.com/pipelines/github/michaelsembwever/cassandra/244/workflows/0d0484c0-82c1-49aa-9656-8d0eb6d6648b/jobs/19051
- trunk https://app.circleci.com/pipelines/github/michaelsembwever/cassandra/245/workflows/c6d2963b-6b2a-4b33-b721-b56e84627c48/jobs/19146


Note this does not fix DSE versions, which was already broken.
Bug: Upgrade the snakeyaml library version
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Sep/23 22:55
Resolved: 01/Dec/23 18:43

Apache cassandra uses 1.26 version of snakeyaml dependency and there are several [vulnerabilities|https://mvnrepository.com/artifact/org.yaml/snakeyaml/1.26#] in this version that can be fixed by upgrading to 2.x version. I understand that this is not security issue as cassandra already uses SafeConstructor and is not a vulnerability under OWASP, so there are no plans to fix it as per  CASSANDRA-18122

 

Cassandra as a open source used and distributed by many enterprise customers and also when downloading cassandra as tar and using it external scanners are not aware of the implementation of SafeConstructor have no idea if it's vulnerable or not. 

Can we consider upgrading the version to 2.x in the next releases as snakeyaml is not something that has a large dependency between the major and minor versions. I am happy to open a PR for this. Please let me know your thoughts on this.
Bug: JDK21 support
Status: Patch Available
Priority: Normal
Created: 08/Sep/23 23:05

This patch builds on the work in CASSANDRA-16895 that added JDK17 to the list of supported Java versions, and extends that work to enable building and running Cassandra under JDK21.

The following commits comprise the changes included in the attached patch:
 - [https://github.com/apache/cassandra/commit/b15d4d6980e787ab5f3405ca8cb17a9c92a4aa47]

 - [https://github.com/apache/cassandra/commit/0c5df38dafe58bfec8924e81507bb604e1543897]

 - [https://github.com/apache/cassandra/commit/6506b7279d98eed4b2b65b71e0c6f41eb78c6913]

 - [https://github.com/apache/cassandra/commit/564cbd534c5a975cda0c629c14c68c8745b41451]
Bug: Create release branch cassandra-5.0
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 28/Jul/23 20:46
Resolved: 16/Aug/23 10:24

Figure out what needs to be done beyond the following…

1. Create release branch
{code:java}
git switch -c cassandra-5.0 trunk
git push --set-upstream origin cassandra-5.0
{code}
2. Bump trunk's version
{code:java}
git switch trunk
# increment version to 5.1
edit build.xml debian/changelog CHANGES.txt NEWS.txt
{code}

2a. Update jvm-dtest supported upgrade paths, and dtest jar generation (both circleci and in-tree scripts), in both cassandra-5.0 and trunk branches:
 - [https://github.com/apache/cassandra/blob/trunk/test/distributed/org/apache/cassandra/distributed/upgrade/UpgradeTestBase.java#L85-L96]
 - [https://github.com/apache/cassandra/blob/trunk/.circleci/config.yml#L2374]
 - [https://github.com/apache/cassandra/blob/trunk/.build/run-tests.sh#L85]

2b. Bump versions in cassandra-5.0 branch
 - doc/antora.yml 
 - build.xml, append "-alpha1" to `base.version`

3. Add `5.0-alpha`, `5.0-alpha1`, `5.0-beta`, `5.0-rc`, `5.0.x` and `5.1` to jira versions
(no existing tickets will be changed - assignees need to change appropriate bugs from 5.x to what is appropriate…)

4. Update docker images to include cassandra-5.0
(Docker images also need to be deployed)

5. Add pipeline to ci-cassandra
[https://github.com/apache/cassandra-builds/blob/trunk/jenkins-dsl/cassandra_job_dsl_seed.groovy#L51]

6. Add dtest version and upgrade paths
 - [https://github.com/apache/cassandra-dtest/blob/trunk/upgrade_tests/upgrade_manifest.py]
 - [https://github.com/apache/cassandra-builds/blob/trunk/build-scripts/cassandra-test.sh#L41]

7. Update how_to_commit documentation
[https://github.com/apache/cassandra-website/blob/trunk/site-content/source/modules/ROOT/pages/development/how_to_commit.adoc]

8. Update website CI to trigger off cassandra-5.0 builds
 - [https://github.com/apache/cassandra-builds/commit/1fc9b5ee71dc37e1145f276ead5c680c6b3fe3db]

9. Update website to build new branches
 - https://github.com/apache/cassandra-website/blob/trunk/site-content/Dockerfile#L93
Bug: Create 5.0 Landing Page
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 28/Aug/23 21:42
Resolved: 02/Jan/25 08:40

As discussed on the dev list, the upcoming release of Apache Cassandra 5.0 would benefit from a public landing page: [https://lists.apache.org/thread/xwb1bxpdcof1bk3x8wnbo5wrlkklq07k]

The landing page would educate users about what is coming up in this important release, highlighting why upgrading will be valuable to them, as well as guiding them into more community activities, such as Town Halls and Contributor Meetings, where they can learn more and become further involved.

The landing page should include:
 * An overview of the release with a brief summary of the major features
 * Links to each of the relevant CEP pages on Confluence
 * CTAs to community platforms and activities - Slack, Meetups, Town Halls, Contributor Meetings, etc.

At the same time, the CEP pages on Confluence should be updated to ensure they include the most current information about the feature and enhanced with any videos or other resources that highlight the work being done.

All of the new and updated content should be written to be accessible and understandable to end users.

I will draft the initial page and add it to this ticket.
Bug: Extend compaction interfaces to provide split points at operation start
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Aug/23 07:57
Resolved: 17/Dec/24 17:20

The current compaction interfaces allow a compaction strategy to split at arbitrary points while it is writing output. In some cases (e.g. UCS) we know in advance where we want to split. Giving this information before the operation starts allows it to operate on multiple segments of the output in parallel, i.e. parallelize within an operation rather than between operations, which can reduce individual operations' duration and significantly improve the DB's chances of keeping up with load, especially on L0.
Bug: Add declarative root CI structure
Status: Open
Priority: Normal
Created: 07/Aug/23 15:35

Currently we have a somewhat declarative structure in .circleci, however there's still quite a bit that's baked in (resource limitations, parallelism, which suites qualify for which pipelines (pre-commit vs. post-commit vs. ???), etc). Further, while CASSANDRA-18133 brings the build scripts in-tree, all these parameters (pipelines, env vars, job definitions, resource constraints) are all still scattered throughout the shell scripts and/or reliant on the {{JenkinsFile}} to determine what suites comprise what pipelines.

This ticket aims to decouple the definition of pipelines and jobs for CI from the implementations themselves. The goal here is to define, establish, and test both the base config and some helper methods to provide _other_ configurations (circle, ASF CI, etc) the tools they need to programmatically inherit base CI config from a purely declarative structure.

Follow-up tickets will involve rewriting the in-tree build scripts and JenkinsFile generation to rely on this structure, as well as integrating the config parsing unit tests into our CI.
Bug: (Accord): C* stores table in Range which will cause ranges to be removed from Accord when DROP TABLE is performed
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 17/Jul/23 20:58
Resolved: 28/Sep/24 06:42

When translating the C* model to Accord we store the keyspace in the Range, this has the side effect that DROP KEYSPACE will cause Accord to see ranges removed between epoch versions.  When this is enabled in BurnTest we see that there are cases where we stop making progress and can loop forever (until we OOM).

Example

{code}
Failed on seed -1330125844737109546
accord.burn.SimulationException: Failed on seed -1330125844737109546
java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.util.HashMap.newNode(HashMap.java:1774)
	at java.util.HashMap.putVal(HashMap.java:632)
	at java.util.HashMap.put(HashMap.java:613)
	at accord.impl.InMemoryCommandStore$InMemorySafeStore.addCommandInternal(InMemoryCommandStore.java:582)
	at accord.impl.InMemoryCommandStore$InMemorySafeStore.addCommandInternal(InMemoryCommandStore.java:557)
	at accord.impl.AbstractSafeCommandStore$$Lambda$516/1241529534.accept(Unknown Source)
	at accord.impl.AbstractSafeCommandStore.getIfLoaded(AbstractSafeCommandStore.java:84)
	at accord.impl.AbstractSafeCommandStore.getInternalIfLoadedAndInitialised(AbstractSafeCommandStore.java:91)
	at accord.local.SafeCommandStore.ifLoadedAndInitialised(SafeCommandStore.java:194)
	at accord.local.Commands.lambda$updateWaitingOn$3(Commands.java:687)
	at accord.local.Commands$$Lambda$514/809300666.accept(Unknown Source)
	at accord.utils.SimpleBitSet.reverseForEach(SimpleBitSet.java:358)
	at accord.local.Command$WaitingOn$Update.forEachWaitingOnCommit(Command.java:1280)
	at accord.local.Commands.updateWaitingOn(Commands.java:685)
	at accord.local.Commands.initialiseWaitingOn(Commands.java:675)
	at accord.local.Commands.apply(Commands.java:481)
	at accord.messages.Apply.apply(Apply.java:121)
	at accord.messages.Apply.apply(Apply.java:34)
{code}