av9ash/GitBugs huggingface dataset:
Bug: Incompatible JNA libraries cause unnecessary failures when using cassandra-all jar.
Status: Open
Priority: Normal
Created: 19/Apr/21 17:06

There are some additional use-cases where lack of JNA libraries is causing issues for users of cassandra-all libraries - cases where the native libraries were not loaded but we should be able to continue should be handled without throwing exceptions. Note that the Cassandra +server+ will not start up without an appropriate/compatible JNA jar available, but users of the cassandra-all libraries will have a more difficult time maintaining compatibility across JNA versions as they deploy to, for example, Spark setups that have newer/older JNA versions.
Proposal:

- Audit usages of `NativeLibrary`
- Wherever possible, handle the -1 return values without failing +if `DatabaseDescriptor.isClientInitialized()` returns true+.
- Maintain existing failure behavior if the system is not in client mode but the call returns -1.
Bug: DOC - Publish drivers compatibility matrix
Status: Open
Priority: Normal
Created: 20/Apr/21 00:30

[~jaid] brought up [on the User ML|https://lists.apache.org/thread.html/r41ab9448a8af2e95996577c82bd5a9ca09e308bc79917b15fad45580%40%3Cuser.cassandra.apache.org%3E] whether a compatibility matrix exists on the Apache Cassandra website similar to [the Java driver matrix published on the DataStax Docs website|https://docs.datastax.com/en/driver-matrix/doc/driver_matrix/javaDrivers.html].

I've logged this ticket to consider including the compatibility matrix from the following _supported_ drivers:
 * Java driver
 * Python driver
 * Node.js driver
 * C# driver driver
 * C++ driver

My intention is to socialise this idea with the devs and provided there are no objections, I'll look into implementing it.
Bug: Development docs are not upto to date
Status: Open
Priority: Normal
Created: 27/Jun/21 04:19

The latest developers guide for unit testing isn't accurate [link|https://cassandra.apache.org/doc/latest/development/testing.html]
 For e.g
{code:java}
ant test -Dtest.name=<simple_classname>
{code}
isn't valid. It's now
{code:java}
ant testsome -Dtest.name=<simple_classname>
{code}
Similarly, we also have `ant long-testsome` & `ant burn-testsome` for e.g. I would be nice to have these documented as well for developers or users.

AC: 
* The website accurately represents all testing scenarios present in build.xml or otherwise.
Bug: Upgrade lz4-java to 1.8.0 to add RH6 support back
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Jun/21 21:32
Resolved: 23/Jun/21 20:13

After upgrading lz4-java the native version is disabled on RHEL 6 hosts as the libc there isn’t new enough; this was reported to lz4-java in https://github.com/lz4/lz4-java/issues/163 and they released 1.8.0 (https://github.com/lz4/lz4-java/releases/tag/1.8.0) to fix this issue, we should upgrade to add RHEL 6 support back.
Bug: Update libthrift jar to at least 0.9.3-1, investigate 0.14.0
Status: Resolved
Priority: Normal
Resolution: Won't Fix
Created: 15/Apr/21 14:14
Resolved: 24/Sep/21 15:30

Cassandra 3.x and 2.x uses libthrift 0.9.2, which has a number of vulnerabilities associated with it which are applicable to Cassandra;

CVE-2015-3254
CVE-2018-1320 (CASSANDRA-15424)
CVE-2019-0205 (CASSANDRA-15420)

Updating to 0.9.3-1 will mitigate these, however that branch suffers CVE-2020-13949. 

To mitigate risks from using out of date libthrift versions, Cassandra should be updated to use 0.14.0
Bug: Flaky ViewComplexTest, ViewFilteringTest and InsertUpdateIfConditionTest
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 17/May/21 10:38
Resolved: 04/Jun/21 04:27

*ViewComplexTest*

Flaky [test|https://ci-cassandra.apache.org/job/Cassandra-4.0/43/testReport/junit/org.apache.cassandra.cql3/ViewComplexTest/testPartialDeleteSelectedColumnWithoutFlush_3_/] and move back away from 'long' section.

*InsertUpdateIfConditionTest* (CASSANDRA-16676)

Fails [here|https://ci-cassandra.apache.org/job/Cassandra-4.0/46/testReport/junit/org.apache.cassandra.cql3.validation.operations/InsertUpdateIfConditionTest/testListItem_2__clusterMinVersion_4_0_0_rc2_SNAPSHOT_/] with a timeout. We can see in the history it takes quite a while in [CI|https://ci-cassandra.apache.org/job/Cassandra-4.0/46/testReport/junit/org.apache.cassandra.cql3.validation.operations/InsertUpdateIfConditionTest/history/] _but_ it takes just 1m locally. Probably due to constrained resources. Looking at the [individual|https://ci-cassandra.apache.org/job/Cassandra-4.0/46/testReport/junit/org.apache.cassandra.cql3.validation.operations/InsertUpdateIfConditionTest/] test cases, for compression i.e., we can see 378 at an average of 1s each it can easily go over the timeout of 240s. Recommendation is to either move to 'long' section of to raise the timeout for the class for CI.

*ViewFilteringTest*

Move back from 'long' section
Bug: Fix flaky testSkipScrubCorruptedCounterRowWithTool
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 06/Apr/21 13:05
Resolved: 10/Aug/22 05:26

See CASSANDRA-16532 where extra flakiness was detected on 3.0 and 3.11 branches for {{testSkipScrubCorruptedCounterRowWithTool}}
Bug: Add an option to nodetool garbagecollect that collects only a fraction of the data
Status: In Progress
Priority: Normal
Created: 27/Jun/21 05:01

nodetool garbagecollect can currently only run across an entire table.

For a very large table, with many use cases, the most likely tables to be full of 'garbage' are the oldest tables. With both LCS and STCS, the tables with the lowest generation number are, under normal operation, going to have the majority of data that is masked by a tombstone or overwritten.

In order to make 'nodetool garbagecollect' more useful for such large tables, I propose that we add an option `--oldest-fraction` that takes a floating point value between 0.00 and 1.00, and only runs 'garbagecollect' over the oldest SSTables that cover at least that fraction of data.

This would mean, for insatnce, that if you ran this with `--oldest-fraction 0.1` every week, that no table would be older than 10 weeks old, and there would exist no data that has been overwritten, TTL'd, or deleted that was originally written more than 10 weeks ago.

In my use case, the oldest LCS table is about 20 months old if the table operates in steady-state on Cassandra 3.11.x, but only 5% of the data in tables that age has not been overwritten. This breaks some of the performance promise of LCS – if your last level is 50% filled with overwritten data, then your chance of finding data only in that level is significantly less than advertised.

'nodetool compact' is extremely expensive, and not conducive to any sort of incremental operation currently. But nodetool garbagecollect run on a fraction of the oldest data would be.
Bug: Add user defined SSTables option for nodetool garbagecollect
Status: In Progress
Priority: Normal
Created: 27/Jun/21 04:40

nodetool garbagecollect does not yet have an option to supply a user defined list of SSTables to process.  

This is unfortunate, because there are many cases where an operator would know which subset of tables are in need of a garbagecollect.  Perhaps with STCS, one would choose to garbagecollect only the largest file.

With a large LCS table, it is typical that the highest levels have the most overwritten data, and it may take a very long time  to run a full garbagecollect or full compaction, but a relatively short time to process a smaller subset of SSTables.
Bug: Add user defined SSTables option for nodetool scrub
Status: Patch Available
Priority: Normal
Created: 27/Jun/21 04:45

nodetool scrub does not currently have an option to supply a user-defined list of SSTables in the way that nodetool compact does. 

This is a very big operational  oversight.  For example, if you have a _single_ SSTable with a single corrupt block, you have to scrub the _entire_ table in order to recover most of that table's data.  On a large table (I have an LCS table with 1.2TB) this takes _forever_ – to the point that we decided that the best practice is to delete the entire file and repair the range that the file contains, rather than scrub and repair only the range of the bad blocks.
Bug: Create release branch cassandra-4.0
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Apr/21 20:53
Resolved: 12/Jul/21 21:02

Figure out what needs to be done beyond the following…

1. Create release branch
{code}
git switch -c cassandra-4.0 trunk
git push --set-upstream origin cassandra-4.0
{code}

2. Bump trunk's version 
{code}
git switch trunk
# increment version to 4.1
edit build.xml debian/changelog CHANGES.txt NEWS.txt
{code}

3. Add pipeline to ci-cassandra
https://github.com/apache/cassandra-builds/blob/trunk/jenkins-dsl/cassandra_job_dsl_seed.groovy#L51

-4. Update website versions-
(it not needed until a release is made)

5. Add dtest version and upgrade paths
 - https://github.com/apache/cassandra-dtest/blob/trunk/upgrade_tests/upgrade_manifest.py
 - https://github.com/apache/cassandra/blob/trunk/.circleci/config.yml#L2374
 - https://github.com/apache/cassandra-builds/blob/trunk/build-scripts/cassandra-test.sh#L31

6. Update how_to_commit documentation
https://github.com/apache/cassandra/blob/trunk/doc/source/development/how_to_commit.rst
Bug: Make SSLContext creation pluggable/extensible
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 12/May/21 23:55
Resolved: 04/Oct/21 07:17

Currently Cassandra creates the SSLContext via SSLFactory.java. SSLFactory is a final class with static methods and not overridable. The SSLFactory loads the keys and certs from the file based artifacts for the same. While this works for many, in the industry where security is stricter and contextual, this approach falls short. Many big organizations need flexibility to load the SSL artifacts from a custom resource (like custom Key Management Solution, HashiCorp Vault, Amazon KMS etc). While JSSE SecurityProvider architecture allows us flexibility to build our custom mechanisms to validate and process security artifacts, many times all we need is to build upon Java's existing extensibility that Trust/Key Manager interfaces provide to load keystores from various resources in the absence of any customized requirements on the Keys/Certificate formats.

My proposal here is to make the SSLContext creation pluggable/extensible and have the current SSLFactory.java implement an extensible interface. 

I contributed a similar change that is live now in Apache Kafka (2.6.0) - https://issues.apache.org/jira/browse/KAFKA-8890 

I can spare some time writing the pluggable interface and run by the required reviewers.

 

Created [CEP-9: Make SSLContext creation pluggable|https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-9%3A+Make+SSLContext+creation+pluggable] 

 

cc: [~dcapwell] [~djoshi]
Bug: Avoid race in AbstractReplicationStrategy endpoint caching
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/May/21 11:45
Resolved: 17/Jan/22 13:20

We should make sure we track which ringVersion we are caching in AbstractReplicationStrategy to avoid a race where we might return the wrong EndpointsForRange.

{code}
Caused by: java.lang.IllegalArgumentException: 9010454139840013625 is not contained within (9223372036854775801,-4611686018427387905]
	at org.apache.cassandra.locator.EndpointsForRange.forToken(EndpointsForRange.java:59)
	at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalReplicasForToken(AbstractReplicationStrategy.java:104)
	at org.apache.cassandra.locator.ReplicaLayout.forTokenReadLiveSorted(ReplicaLayout.java:330)
	at org.apache.cassandra.locator.ReplicaPlans.forRead(ReplicaPlans.java:594)
{code}
Bug: Request-Based Native Transport Rate-Limiting
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 10/May/21 22:15
Resolved: 19/Aug/21 15:58

Together, CASSANDRA-14855, CASSANDRA-15013, and CASSANDRA-15519 added support for a runtime-configurable, per-coordinator limit on the number of bytes allocated for concurrent requests over the native protocol. It supports channel back-pressure by default, and optionally supports throwing OverloadedException if that is requested in the relevant connection’s STARTUP message.

This can be an effective tool to prevent the coordinator from running out of memory, but it may not correspond to how expensive a queries are or provide a direct conceptual mapping to how users think about request capacity. I propose adding the option of request-based (or perhaps more correctly message-based) back-pressure, coexisting with (and reusing the logic that supports) the current bytes-based back-pressure.

_We can roll this forward in phases_, where the server’s cost accounting becomes more accurate, we segment limits by operation type/keyspace/etc., and the client/driver reacts more intelligently to (especially non-back-pressure) overload, _but something minimally viable could look like this_:

1.) Reuse most of the existing logic in Limits, et al. to support a simple per-coordinator limit only on native transport requests per second. Under this limit will be CQL reads and writes, but also auth requests, prepare requests, and batches. This is obviously simplistic, and it does not account for the variation in cost between individual queries, but even a fixed cost model should be useful in aggregate.
 * If the client specifies THROW_ON_OVERLOAD in its STARTUP message at connection time, a breach of the per-node limit will result in an OverloadedException being propagated to the client, and the server will discard the request.
 * If THROW_ON_OVERLOAD is not specified, the server will stop consuming messages from the channel/socket, which should back-pressure the client, while the message continues to be processed.

2.) This limit is infinite by default (or simply disabled), and can be enabled via the YAML config or JMX at runtime. (It might be cleaner to have a no-op rate limiter that's used when the feature is disabled entirely.)

3.) The current value of the limit is available via JMX, and metrics around coordinator operations/second are already available to compare against it.

4.) Any interaction with existing byte-based limits will intersect. (i.e. A breach of any limit, bytes or request-based, will actuate back-pressure or OverloadedExceptions.)

In this first pass, explicitly out of scope would be any work on the client/driver side.

In terms of validation/testing, our biggest concern with anything that adds overhead on a very hot path is performance. In particular, we want to fully understand how the client and server perform along two axes constituting 4 scenarios. Those are a.) whether or not we are breaching the request limit and b.) whether the server is throwing on overload at the behest of the client. Having said that, query execution should dwarf the cost of limit accounting.
Bug: JMXTimer exposes attributes in inconsistent time units
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Jun/21 00:14
Resolved: 16/Jul/21 17:25

JMXTimer objects are constructed with a duration time unit, which is fixed to MICROSECONDS in the codebase. According to that, we should expect the time values returned from the JXMTimer are in micros.

However, the time unit is inconsistent among the JMXTimer attributes. 
Most of the attributes such as percentiles and mean values returned are in micros, except Values and RecentValues. 
Those 2 attributes expose the raw histogram values of the underlying Timer (CodaHale) and the values are fixed to be based on nanos. 

The inconsistency leads to confusion and mis-interpretation of the values, if the end user is not familiar with the implementation details. One may consider the Values and RecentValues are also in micros.
Besides the confusion, given the intention is to record the time values in the micros resolution, we do not need to allocate 165 buckets in the DecayingEstimatedHistogramReservoir. 165 buckets is necessary for nanos, but not for micros. We can only allocate 90 buckets and it should reduce ~50% memory footprint used by the Timers. 

I'd like to propose an approach to scale the values being recorded in the reservoirs used by Timers and reduce the allocation.
Bug: Fix org.apache.cassandra.tools.nodetool.StatusTest.testOutputWhileBootstrapping
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/Jun/21 22:13
Resolved: 04/Jun/21 01:09

[https://jenkins-cm4.apache.org/job/Cassandra-4.0/71/testReport/junit/org.apache.cassandra.tools.nodetool/StatusTest/testOutputWhileBootstrapping_cdc/]
Bug: ProtocolVersion.V4 is still used in places in the code
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Apr/21 00:16
Resolved: 08/May/21 14:21

While working on CASSANDRA-16567, [~adelapena] observed that _ProtocolVersion.V4_ is used in _ViewTest_.

I decided to do a quick grep and observed a list of places where we still refer to V4 and it seems at least in many of the tests that was left not intentionally.

This ticket is to verify the usage of _ProtocolVersion.V4_ in the codebase and bump it to V5 or  default version, similar to what was done in CASSANDRA-16567, wherever there is a need.
Bug: CircleCI - python dtests failing because of ccm issue
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/May/21 19:24
Resolved: 24/May/21 18:39

Circle CI fails to run dtests.

[https://app.circleci.com/pipelines/github/adelapena/cassandra/483/workflows/e9a5f84e-e465-40e6-92f4-08e8632edf47/jobs/4315]
{code:java}
Obtaining ccm from git+
[https://github.com/riptano/ccm.git@cassandra-test#egg=ccm]
 (from -r /home/cassandra/cassandra-dtest/requirements.txt (line 9))
  Updating ./env3.6/src/ccm clone (to revision cassandra-test)
  Running command git fetch -q --tags
WARNING: Discarding git+
[https://github.com/riptano/ccm.git@cassandra-test#egg=ccm]
. Command errored out with exit status 1: git fetch -q --tags Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement ccm (unavailable)
ERROR: No matching distribution found for ccm (unavailable)
WARNING: You are using pip version 21.0.1; however, version 21.1.1 is available.



{code}
Bug: BinLog does not close chronicle queue leaving this to GC to cleanup
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 30/Jun/21 17:06
Resolved: 02/Jul/21 18:33

auditlog_test.py::TestAuditlog::test_archiving_fql  and test_fql_nodetool_options fail from time to time due to the test relying on a race condition; we do not close chronicle queue so rotation may not happen before stopping archiver, the tests fail if rotation happens before stopping archiver (which is done based off GC).
Bug: generate-idea-files taks unnecessarily builds the project
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/May/21 05:36
Resolved: 17/Jun/21 10:15

This is especially painful when the project cannot be built because we are in the middle of solving some merge conflicts, but we want to fetch and update dependencies. It seems be to be unnecessary to build the project or to do other checks when we just want to load the project into our IDE.

From what I've tested, it is enough to limit dependencies of that task to:
{{init,maven-ant-tasks-init,resolver-dist-lib,gen-cql3-grammar,generate-jflex-java,createVersionPropFile}}
Bug: Intermittent failure of SEPExecutorTest.changingMaxWorkersMeetsConcurrencyGoalsTest caused by race condition when shrinking maximum pool size to zero
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/May/21 21:13
Resolved: 21/May/21 18:34

A difficult-to-hit race condition exists in changingMaxWorkersMeetsConcurrencyGoalsTest when changing the maximum pool size from 0 -> 4 which results in the test failing like so:

{{junit.framework.AssertionFailedError: Test tasks did not hit max concurrency goal expected:<true> but was:<false>junit.framework.AssertionFailedError: Test tasks did not hit max concurrency goal expected:<true> but was:<false> at org.apache.cassandra.concurrent.SEPExecutorTest.assertMaxTaskConcurrency(SEPExecutorTest.java:198) at org.apache.cassandra.concurrent.SEPExecutorTest.changingMaxWorkersMeetsConcurrencyGoalsTest(SEPExecutorTest.java:132)}}

I can hit this issue maybe 2/3 times for every 100 invocations of the unit test.

The issue that causes the failure is that if tasks are still enqueued when the maximum pool size is set to zero and if all of the SEPWorker threads enter the STOP state before the pool size is bumped to 4, then no SEPWorker threads will be spun up to service the task queue. This causes the above error.

Why don't we spin up SEPWorker threads when enqueing tasks? Because of the guard logic in addTask: [https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/concurrent/SEPExecutor.java#L113,L121]

In this scenario taskPermits will not be zero (because we have tasks on the queue) so we never call {{maybeStartSpinningWorker()}}.

A trick to make this issue much easier to hit is to insert a {{Thread.sleep(500)}} immediately after setting the pool size to zero. This has the effect of guaranteeing that all SEPWorker threads will be STOP'd before enqueueing more work.

Here's a fix that attempts to spin up an SEPWorker whenever we grow the number of work permits: https://github.com/mfleming/cassandra/commit/071516d29e41da9924af24e8002822d3c6af0e01
Bug: Add 4_0_x to Python DTest's upgrade_manifest (after cassandra-4.0 branch was created)
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/May/21 15:47
Resolved: 07/May/21 08:07

Python DTest upgrade manifest:
 - https://github.com/apache/cassandra-dtest/compare/trunk...thelastpickle:mck/16642--introduce-cassandra-4.0
 - https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest/645/
 - https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest-upgrade/273/
Bug: modify SecondaryIndexManager#indexPartition() to retrieve only columns for which indexes are actually being built
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 30/Jun/21 19:46
Resolved: 10/Aug/21 20:45

Secondary indexes are (for the moment) built as special compaction tasks via {{SecondaryIndexBuilder}}. From a profiling perspective, the fun begins in {{SecondaryIndexManager.indexPartition()}}. The work above it in {{SecondaryIndexBuilder}} is just key iteration.

 !index1.png! 

Two basic things happen in {{indexPartition()}}. First, we read a single partition in its entirety, and then we send individual rows to the {{Indexer}}. When we read these partitions, we use {{ColumnFilter.all()}}, which ends up materializing full rows, even when we’re indexing a single column (or at least fewer columns than we need for all the indexes participating in the build). If we narrowed this to fetch only the necessary columns, we might be able to create less garbage in {{AbstractBTreePartition#searchIterator()}} when we create a copy of the underlying full row from disk.

In some initial testing, I’ve been using a simple schema with fairly narrow rows.

{noformat}
CREATE TABLE tlp_stress.allow_filtering (
    partition_id text,
    row_id int,
    payload text,
    value int,
    PRIMARY KEY (partition_id, row_id)
) WITH CLUSTERING ORDER BY (row_id ASC)
{noformat}

The price of deserializing these rows is still visible, however, in the results of some basic sampling profiling.

 !index2.png! 


The possible optimization above to avoid unnecessary copying of a row’s columns would also narrow cell deserialization only to indexed cells, which would probably be very beneficial for index builds with very wide rows. One minor wrinkle in all of this is that since 3.0, it has been possible to create indexes one entire rows, rather than single columns, so we’d have to keep that case in mind.