av9ash/GitBugs huggingface dataset:
Bug: Bloom filter false positive rate calculation does not take into account true negatives
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/May/20 10:31
Resolved: 18/May/21 19:26

The bloom filter false positive ratio is [currently computed|https://github.com/apache/cassandra/blob/ded62076e7fdfd1cfdcf96447489ea607ca796a0/src/java/org/apache/cassandra/metrics/TableMetrics.java#L738] as:

{{bf_fp_ratio = false_positive_count / (false_positive_count + true_positive_count)}}

However, this calculation doesn't take into account true negatives (false negatives never happen on bloom filters).

In a situation where there are 1000 reads for non existing rows, and there are 10 false positives, the bloom filter false positive ratio will be wrongly calculated as 10/10 = 1.0, while it should be 10/1000 = 0.01.

We should update the calculation to:

{{bf_fp_ratio = false_positive_count / #bf_queries}}

Original jira by [~pauloricardomg]
Bug: cqllib testing not running in CircleCI
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 28/Apr/20 17:30
Resolved: 30/Sep/20 12:43

CASSANDRA-14951 added the ability to run the cqllib tests but didn’t add them to CircleCI and are missing currently.  It would be best to add the matrix of supported cases to CircleCI.
Bug: Tarball contains duplicate entries
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 28/Apr/20 15:31
Resolved: 06/May/20 11:04

The tarball contains a lot of duplicate entries. One example is cassandra-stress.bat:
{code:sh}
tar -tvf /home/map/Downloads/apache-cassandra-3.11.6-bin.tar.gz |grep "cassandra-stress.bat"
-rw-r--r-- 0/0            1097 2020-02-10 23:57 apache-cassandra-3.11.6/tools/bin/cassandra-stress.bat
-rwxr-xr-x 0/0            1097 2020-02-10 23:57 apache-cassandra-3.11.6/tools/bin/cassandra-stress.bat
{code}
Bug: YamlConfigurationLoader swallows a real exception and exposes a misleading exception message
Status: Triage Needed
Priority: Normal
Created: 23/Apr/20 14:08

Method YamlConfigurationLoader.loadConfig(URL) parses a YAML config file:

{code:java}
try {
  ...
  Config result = loadConfig(yaml, configBytes);
  ...
}
catch (YAMLException e) {
  throw new ConfigurationException("Invalid yaml: " + url + SystemUtils.LINE_SEPARATOR + " Error: " + e.getMessage(), false);
}
{code}

In the catch block, the real cause is dropped and a misleading message "Invalid yaml" is given. Instead in my case I have a cause:

{code:none}
java.lang.NoSuchMethodError: com.google.common.collect.Sets.newConcurrentHashSet()Ljava/util/Set;
{code}

This is caused by an incompatible Guava version in classpath. It was impossible to figure out the real root cause without debugging.

One possible fix is to include the caught exception (YAMLException e) in the cause of the thrown exception:

{code:java}
throw new ConfigurationException(msg, e)
{code}

The 'msg' should be changed because it is not necessarily "Invalid yaml".

Another possible fix is to not catch exception here - just let the root cause exception propagate to the caller.
Bug: jvm dtest builder should be provided to the factory and expose state
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Apr/20 20:28
Resolved: 05/Jun/20 17:37

Currently the builder is rather heavy and creates configs plus call the factory with specific fields only, this isn’t that flexible and makes it harder to have custom cluster definitions which require additional fields to be defined.  To solve this we should make the builder be sent to the factory and expose the state so the factory can get all the fields it needs, the factory should also be in charge of creating the configs
Bug: add client request size metrics to netty pipeline
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 07/Apr/20 19:52
Resolved: 10/Apr/20 00:53

We currently lack metrics around client connection incoming / outgoing bytes.  It’s fairly standard to know how many bytes are read and written to the network, but that aggregates client facing and internal cluster traffic to a single number.  This patch will help us understand client overhead more granularly.
Bug: flaky testWithMismatchingPending - org.apache.cassandra.distributed.test.PreviewRepairTest
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/Apr/20 19:08
Resolved: 09/Jul/20 14:38

Observed in: 
https://app.circleci.com/pipelines/github/newkek/cassandra/34/workflows/1c6b157d-13c3-48a9-85fb-9fe8c153256b/jobs/191/tests

Failure:
{noformat}
testWithMismatchingPending - org.apache.cassandra.distributed.test.PreviewRepairTest
junit.framework.AssertionFailedError
	at org.apache.cassandra.distributed.test.PreviewRepairTest.testWithMismatchingPending(PreviewRepairTest.java:97)
{noformat}

[Circle CI|https://circleci.com/gh/dcapwell/cassandra/tree/bug%2FCASSANDRA-15685]
Bug: Only track ideal CL failure when request CL is met
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 06/Apr/20 19:22
Resolved: 07/Apr/20 01:35

When ideal_consistency_level is set (CASSANDRA-13289), we currently increment a counter if a request doesn’t meet the consistency level specified in the configuration (or through JMX).

At the moment, we increment the counter if the query was successful or not. I think it would be slightly better if we only incremented the counter if the ideal CL wasn’t achieved but the query’s CL was met.

The original JIRA, stated the following as an objective:
{quote}If your application writes at LOCAL_QUORUM how often are those writes failing to achieve EACH_QUORUM at other data centers. If you failed your application over to one of those data centers roughly how inconsistent might it be given the number of writes that didn't propagate since the last incremental repair?
{quote}
The main benefit to the JIRA was to set a CL higher than the CL being used, and to track how often we weren’t able to hit that CL despise hitting the underlying CL. We should only increment the counter in a case where we were able to meet the query provided consistency but were unable to meet the ideal consistency level.
Bug: DOC - Add cqlsh Python 3 support in NEWS.txt and tools section of website
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 09/Apr/20 08:27
Resolved: 18/Apr/20 05:51

h2. Background
I found out by accident that CASSANDRA-10190 is not documented in {{NEWS.txt}}.

h2. Scope
* {{NEWS.txt}} - Add info on Python 3 support for cqlsh.
* Docs - include new {{--python}} option
Bug: Security vulnerabilities with dependency jars  of Cassandra 3.11.6
Status: Resolved
Priority: Normal
Resolution: Won't Fix
Created: 04/Jun/20 15:17
Resolved: 24/Sep/21 14:55

The latest release of Cassandra 3.11.6 has few dependency jars which have some security vulnerabilities.

 

Apache Thrift (org.apache.thrift:libthrift:0.9.2) has below mentioned security vulnerabilities reported
|+[https://nvd.nist.gov/vuln/detail/CVE-2016-5397]+|
|+[https://nvd.nist.gov/vuln/detail/CVE-2018-1320]+|
|+[https://nvd.nist.gov/vuln/detail/CVE-2019-0205]+|

 

Netty Project (io.netty:netty-all:4.0.44.Final) has below mentioned security vulnerabilities reported
|+[https://nvd.nist.gov/vuln/detail/CVE-2019-16869]+|
|+[https://nvd.nist.gov/vuln/detail/CVE-2019-20444]+|
|+[https://nvd.nist.gov/vuln/detail/CVE-2019-20445]+|

 

Is there a plan to upgrade these jars in any upcoming release?
Bug: Flaky unit test: SASIIndexTest.testInsertingIncorrectValuesIntoAgeIndex
Status: Resolved
Priority: Normal
Resolution: Auto Closed
Created: 18/Jun/20 12:46
Resolved: 13/Sep/21 17:12

The unit test {{SASIIndexTest.testInsertingIncorrectValuesIntoAgeIndex}} seems to be flaky in 3.11, as it can be seen in [cassandra-ci|https://ci-cassandra.apache.org/view/Cassandra%203.11/job/Cassandra-3.11-test/42/testReport/org.apache.cassandra.index.sasi/SASIIndexTest/testInsertingIncorrectValuesIntoAgeIndex/] and also locally. Trunk doesn't seem to be affected.

{{SASIIndexTest.testIndexMemtableSwitching}} is [also flaky|https://ci-cassandra.apache.org/view/Cassandra%203.11/job/Cassandra-3.11-test/42/testReport/org.apache.cassandra.index.sasi/SASIIndexTest/testIndexMemtableSwitching/], although I haven't been able to reproduce it running it separately, but only when running the entire {{SASIIndexTest}}.

These could have been introduced when merging up CASSANDRA-15778, since they failed for their [CircleCI run|https://app.circleci.com/pipelines/github/ifesdjeen/cassandra/17/workflows/0442eebe-c764-41c5-ba06-6617bcb9fc5f/jobs/2213], and they didn't seem to fail before that, or at least I cannot reproduce them locally with the previous commit.
Bug: Add progress column in percents for system_views.sstable_tasks
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 25/Apr/20 18:58
Resolved: 28/May/20 14:31

It would be very handy to have a percentage column in system_views.sstable_tasks which would say how far a respective task is.

Indeed, there are currently "progress" and "total" columns but honestly, for every day usage, it is rather strange to expect that humans will divide these numbers in head if they want to roughly know what the overall progress is. One just does not have a rough estimation of the task progress when he is presented with two quite big numbers and to estimate the progress from the. 

In the following output, the field "progress_in_percents" is introduced.

 

PR is here [https://github.com/apache/cassandra/pull/566]

 
{code:java}
admin@cqlsh> select * from system_views.sstable_tasks ;

@ Row 1
-----------------------+--------------------------------------
 keyspace_name         | mykeyspace
 table_name            | mytable
 task_id               | 0db5d9b1-8726-11ea-8a6c-b92f3be367bb
 kind                  | secondary index build
 progress              | 19456965
 progress_in_percents  | 8.17
 total                 | 238208674
 unit                  | bytes

@ Row 2
----------------------+--------------------------------------
 keyspace_name        | mykeyspace
 table_name           | mytable.mytable_surname_idx
 task_id              | 1817ee71-8726-11ea-8a6c-b92f3be367bb
 kind                 | compaction
 progress             | 284396233
 progress_in_percents | 75.92
 total                | 374598446
 unit                 | bytes
{code}
Bug: Improve DROP COMPACT STORAGE
Status: Resolved
Priority: Normal
Resolution: Later
Created: 13/May/20 15:47
Resolved: 14/Jan/21 12:36

DROP COMPACT STORAGE was introduced in CASSANDRA-10857 as one of the steps to deprecate Thrift. However, current semantics of dropping compact storage flags from tables reveal several columns that are usually empty (colum1 and value in non-dense case, value for dense columns, and a column with an empty name for super column families). Showing these columns  can confuse application developers, especially ones that have never used thrift and/or made writes that assumed presence of those fields, and used compact storage in 3.x because is has “compact” in the name.

There’s not much we can do in a super column family case, especially considering there’s no way to create a supercolumn family using CQL, but we can improve dense and non-dense cases. We can scan stables and make sure there are no signs of thrift writes in them, and if all sstables conform to this rule, we can not only drop the flag, but also drop columns that are supposed to be hidden. However, this is both not very user-friendly, and is probably not worth development effort. 

An alternative to scanning is to add {{FORCE DROP COMPACT}} syntax (or something similar) that would just drop columns unconditionally. It is likely that people who were using compact storage with thrift know they were doing that, so they'll usually use "regular" {{DROP COMPACT}}, withouot force, that will simply reveal the columns as it does right now.

Since for fixing CASSANDRA-15778, and to allow EmptyType column to actually have data[*] we had to remove empty type validation, properly handling compact storage starts making more sense, but we’ll solve it through not having columns, hence not caring about values instead, or keeping values _and_ data, not requiring validation in this case. EmptyType field will have to be handled differently though.

[*] as it is possible to end up with sstables upgraded from 2.x or written in 3.x before CASSANDRA-15373, which means not every 2.x upgraded or 3.x cluster is guaranteed to have empty values in this column, and this behaviour, even if undesired, might be used by people. 

Open question is: CASSANDRA-15373 adds validation to EmptyType that disallows any non-empty value to be written to it, but we already allow creating table via CQL, and still write data into it with thrift. It seems to have been unintended, but it might have become a feature people rely on. If we simply back port 15373 to 2.2 and 2.1, we’ll change and break behaviour. Given no-one complained in 3.0 and 3.11, this assumption is unlikely though.
Bug: When CDC is disabled bootstrapping breaks
Status: Resolved
Priority: Normal
Resolution: Cannot Reproduce
Created: 07/Apr/20 19:29
Resolved: 13/Apr/21 06:50

Related to CASSANDRA-12697

There is an edge case left over.  If a cluster had enabled CDC on a table then subsequently set cdc=false, subsequent bootstraps break. 

 

This is because the cdc column is false on the existing nodes but null on the bootstrapping node, causing the schema sha to never match.

 

There are a couple possible fixes:

  1.  Since 12697 was only about upgrades we can serialize the cdc column IFF the cluster nodes are all on the same version.

  2.  We can force cdc=false on all tables where it's null.

 

I think #1 is probably simpler. #2 would probably cause more of the same problem if nodes are not all updated with the fix.
Bug: Invalid cdc_raw_directory prevents server startup
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/Apr/20 21:49
Resolved: 14/Apr/20 00:46

If {{cdc_raw_directory}} is set to an invalid directory it prevents startup of the server even when cdc_enabled is set false.

The directory can either be set directly by the {{cdc_raw_directory}} setting in configuration YAML or indirectly via the {{cassandra.storage_dir}} system property, which is how I encountered it.

Easy to reproduce by setting {{cdc_raw_directory}} to {{notadir/notasubdir}}

Additionally while investigating, discovered that {{DatabaseDescriptor.guessFileStore}} can cause a {{NullPointerException}} if it runs out of parent elements
 before it can get a FileStore. It should provide a more useful ConfigurationException providing details on the problematic path.
 {{guessFileStore}} is used for checks on {{commitlog_directory}}, {{cdc_raw_directory}} and {{data_file_directories}}.
Bug: Bootstrap completes Successfully without streaming all the data
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 13/Jun/20 03:50
Resolved: 13/Jun/20 14:35

I am seeing a strange issue where, adding a new node with auto_bootstrap: true is not streaming all the data before it joins the cluster. Don't see any information in the logs about bootstrap failures.

Here is the sequence of logs

 
{code:java}
INFO [main] 2020-06-12 01:41:49,642 StorageService.java:1446 - JOINING: schema complete, ready to bootstrap
INFO [main] 2020-06-12 01:41:49,642 StorageService.java:1446 - JOINING: waiting for pending range calculation
INFO [main] 2020-06-12 01:41:49,643 StorageService.java:1446 - JOINING: calculation complete, ready to bootstrap
INFO [main] 2020-06-12 01:41:49,643 StorageService.java:1446 - JOINING: getting bootstrap token
INFO [main] 2020-06-12 01:42:19,656 StorageService.java:1446 - JOINING: Starting to bootstrap...
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for cfId <UUID>. If a table was just created, this is likely due to the schema not being fully propagated. Please wait for schema agreement on table creation.
INFO [StreamReceiveTask:1] 2020-06-12 02:29:51,892 StreamResultFuture.java:219 - [Stream #f4224f444-a55d-154a-23e3-867899486f5f] All sessions completed INFO [StreamReceiveTask:1] 2020-06-12 02:29:51,892 StorageService.java:1505 - Bootstrap completed! for the tokens
{code}
Cassandra Version: 3.11.3

I am not able to reproduce this issue all the time, but it happened couple of times. Is there any  race condition/corner case, which could cause this issue?
Bug: order by descending on frozen list not working
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/May/20 04:31
Resolved: 23/Jul/20 12:04

By creating a table like the following:
{code:java}
CREATE TABLE IF NOT EXISTS software (
 name ascii,
 version frozen<list<int>>,
 data ascii,
 PRIMARY KEY(name,version)
)
{code}
It works and version is ordered in an ascending order. But when trying to order in descending order:
{code:java}
CREATE TABLE IF NOT EXISTS software (
    name ascii,
    version frozen<list<int>>,
    data ascii,
    PRIMARY KEY(name,version)
) WITH CLUSTERING ORDER BY (version DESC);
{code}
The table is created normally, but when trying to insert a row:
{code:java}
insert into software(name, version) values ('t1', [2,10,30,40,50]); 
{code}
Cassandra throws an error:
{code:java}
InvalidRequest: Error from server: code=2200 [Invalid query] message="Invalid list literal for version of type frozen<list<int>>"
{code}
The goal here is that I would like to get the last version of a software.
Bug: SEPExecutorTest changingMaxWorkersMeetsConcurrencyGoalsTest failure
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 09/Apr/20 05:56
Resolved: 18/Apr/20 17:18

{code}
[junit-timeout] Testsuite: org.apache.cassandra.concurrent.SEPExecutorTest
[junit-timeout] Testsuite: org.apache.cassandra.concurrent.SEPExecutorTest Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.656 sec
[junit-timeout] 
[junit-timeout] Testcase: changingMaxWorkersMeetsConcurrencyGoalsTest(org.apache.cassandra.concurrent.SEPExecutorTest):	FAILED
[junit-timeout] expected:<true> but was:<false>
[junit-timeout] junit.framework.AssertionFailedError: expected:<true> but was:<false>
[junit-timeout] 	at org.apache.cassandra.concurrent.SEPExecutorTest.assertMaxTaskConcurrency(SEPExecutorTest.java:180)
[junit-timeout] 	at org.apache.cassandra.concurrent.SEPExecutorTest.changingMaxWorkersMeetsConcurrencyGoalsTest(SEPExecutorTest.java:110)
[junit-timeout] 
[junit-timeout] 
[junit-timeout] Test org.apache.cassandra.concurrent.SEPExecutorTest FAILED
{code}
Bug: Batch statement preparation fails if multiple tables and parameters are used
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Apr/20 15:21
Resolved: 21/Apr/20 15:31

Batch statement preparation fails with an assertion error if multiple tables and parameters are used.

{{BEGIN BATCH }}
{{ UPDATE tbl1 SET v1 = 1 WHERE k1 = ?}}
{{ UPDATE tbl2 SET v2 = 2 WHERE k2 = ?}}
{{APPLY BATCH}}

The logic for affectsMultipleTables [here|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java#L144] looks inverted
 This later causes an assertion failure [here|https://github.com/apache/cassandra/blob/24c8a21c1c131abd89c6b646343ff098d1b3263b/src/java/org/apache/cassandra/cql3/VariableSpecifications.java#L75]
Bug: ASF CI builds for JDK11
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/May/20 08:14
Resolved: 14/Jul/20 18:22

ASF CI builds today only run on JDK1.8

On the Jenkins cluster JDKs from 1.4 through to 15 are available. See attached screenshot for naming specifics.

This ticket is to add JDK11 compile and test targets on Cassandra-trunk, for parity to CircleCI's workflows. (There is also the question about testing/running on Cassandra-2.2 which needs to support JDK1.7, though Cassandra-2.2 is nearing EOL.)

 

The JDK is specified in the groovy DSL:
[https://github.com/apache/cassandra-builds/blob/master/jenkins-dsl/cassandra_job_dsl_seed.groovy#L11] 

 

Some examples:
 * CircleCI JDK1.8 workflow example. This builds with JDK1.8 and tests with both JDK1.8 and JDK11.
 ** [https://app.circleci.com/pipelines/github/dcapwell/cassandra/259/workflows/bce6fbf9-a4a3-4bfd-be7d-3e7961b440d8]
 * CircleCI JDK11 workflow example. This builds and tests only with JDK11.
 ** [https://app.circleci.com/pipelines/github/dcapwell/cassandra/259/workflows/38e7d77e-1d5e-47f7-a462-277665428306]
 * Jenkins Cqlshlib tests showing matrix builds:
 ** [https://ci-cassandra.apache.org/view/Cassandra%204.0/job/Cassandra-trunk-cqlsh-tests/]

 

Background thread:[https://lists.apache.org/thread.html/rbaeb960901fa53b50227b37d64e5a456b68f749f4229c6e3e086ff85%40%3Cdev.cassandra.apache.org%3E]
Bug: Javadoc fails on java 11: unexported package
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 26/Apr/20 12:19
Resolved: 29/Apr/21 15:16

Javadoc generation fails on java 11 due to an unexported package:
{noformat}
[javadoc] Constructing Javadoc information...
[javadoc] /my/local/cassandra/src/java/org/apache/cassandra/utils/JMXServerUtils.java:311: error: package sun.rmi.registry is not visible
[javadoc]     private static class JmxRegistry extends sun.rmi.registry.RegistryImpl {
[javadoc]                                                     ^
[javadoc]   (package sun.rmi.registry is declared in module java.rmi, which does not export it to the unnamed module)
[javadoc] 1 error{noformat}
The attached patch adds the existing "jdk11-javac-exports" property as an embedded argument to the <javadoc> called in the create-javadoc macro.
Bug: stop and restart fail in Debian 10 buster
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Apr/20 10:03
Resolved: 29/Apr/20 17:06

Debian 10 "buster" has an updated {{start-stop-daemon}} version that causes init script to fail to stop or restart Cassandra daemon. {{stop}} option do not stops first launched instance and depending of previous operations, {{restart}} and {{start}} could launch a second Cassandra daemon instance.

{{start-stop-daemon}} since version 1.19.3 fail to stop matching only a pidfile that can be writen by an unprivileged (non-root) user because it is considered a security risk.

Adding a second matching option (user) solved this.

This fix does not break compatibility with older versions and does not mess with pidfile's mode or permissions.

Related: CASSANDRA-15099

https://github.com/vice/cassandra/tree/debian-buster
Bug: Cassandra driver first  connection get the user's own schema information
Status: Open
Priority: Normal
Resolution: Invalid
Created: 12/Jun/20 10:21
Resolved: 12/Jun/20 14:28

We know that cassandra driver making a conenction with the coordinator node first time , the driver may select all the keyspaces/tables/columns/types from the server and cache the data locally. 
For different users they may have different tables and types ，so It is not suitable to get all the meta data cached , It is fine to just cache the user's own schema information not all.

 And doing this is safe and save first time connection resourse.
Bug: JAVA 8/11: test_resumable_rebuild - rebuild_test.TestRebuild
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 23/Jun/20 02:55
Resolved: 26/Mar/21 08:43

JAVA 11:

test_resumable_rebuild - rebuild_test.TestRebuild

Fails locally and in  

[CircleCI | [https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/222/workflows/11202c7e-6c94-4d4e-bbbf-9e2fa9791ad0/jobs/1338]]
Bug: Cannot read data from a 3-node cluster which has two nodes down
Status: Triage Needed
Priority: Normal
Created: 07/May/20 06:48

I start up a 3 nodes cluster, and write a row with 'replication_factor' : '2'. The consistency level is ONE.
Then I kill two nodes, and try to get the row that I just inserted by cqlsh.
But cqlsh returns NoHostAvailable.
I find this issue in CA 3.11.5, and it can also be exposed in newest 3.11.6.