av9ash/GitBugs huggingface dataset:
Bug: Update AssertJ
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 06/Oct/22 20:49
Resolved: 31/Jul/23 14:09

From the changelog of AssertJ [here,|https://assertj.github.io/doc/#assertj-core-3-23-1-release-notes] it seems JDK17 is tested and relevant issues fixed as per 3.23.1, and we are on 3.15.0

This ticket should accommodate an upgrade to the latest version of AssertJ for trunk.
Bug: Do not leak 2015 memtable synthetic Epoch
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Dec/22 08:41
Resolved: 12/Jan/23 07:26

This [Epoch|https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/db/rows/EncodingStats.java#L48] can [leak|https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/db/Memtable.java#L392] affecting all the timestamps logic.  It has been observed in a production env it can i.e. prevent proper sstable and tombstone cleanup.

To reproduce create the following table:
{noformat}
drop keyspace test;
create keyspace test WITH replication = {'class':'SimpleStrategy', 'replication_factor' : 1};
CREATE TABLE test.test (
    key text PRIMARY KEY,
    id text
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '2', 'tombstone_compaction_interval': '3000', 'tombstone_threshold': '0.1', 'unchecked_tombstone_compaction': 'true'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.0
    AND default_time_to_live = 10
    AND gc_grace_seconds = 10
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE INDEX id_idx ON test.test (id);
{noformat}

And stress load it with:
{noformat}
insert into test.test (key,id) values('$RANDOM_UUID $RANDOM_UUID', 'eaca36a1-45f1-469c-a3f6-3ba54220363f') USING TTL 10
{noformat}

Notice how all inserts have a 10s TTL, the default 10s TTL and gc_grace is also at 10s. This is to speed up the repro:
- Run the load for a couple minutes and track sstables disk usage. You will see it does only increase, nothing gets cleaned up and it doesn't stop growing (notice all this is well past the 10s gc_grace and TTL)
- Running a flush and a compaction while under load against the keyspace, table or index doesn't solve the issue.
- Stopping the load and running a compaction doesn't solve the issue. Flushing does though.
- On the original observation where TTL was around 600s and gc_grace around 1800s we could get GBs of sstables that weren't cleaned up or compacted away after hours of work.
- Reproduction can also happen on plain sstables by repeatedly inserting/deleting/overwriting the same values over and over again without 2i indices or TTL being involved.

The problem seems to be [EncodingStats|https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/db/rows/EncodingStats.java#L48] using a synthetic Epoch in 2015 which plays nice with Vint serialization.  Unfortunately {{Memtable}} is using that to keep track of the {{minTimestamp}} which can leak the 2015 Epoch. This confuses any logic consuming that timestamp. In this particular case purge and fully expired sstables weren't properly detected.
Bug: Restore internode custom tracing
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Oct/22 08:16
Resolved: 18/Dec/22 02:38

Title originally was: Ferrying OpenTelemetry span/context via org.apache.cassandra.net.Message.Header

Lately I endeavored to integrate [Jaeger with Cassandra|https://github.com/smok-serwis/cassandra-jaeger-tracing]. However, the only way to reliably sent messages across modes requires specifying a ParamType, which is limited only to an UUID, which is not enough to sent both the 128-bit spanId and parent, nevermind the baggage.

Is there any other way I can reliably send the data to other node using Message.Header?

Or could I alternatively coordinate with the Cassandra team to reserve an integer for the definite use of ferrying OpenTelemetry data?

I'v detailed my endeavors [here|https://henrietta.com.pl/index.php/2022/10/19/how-do-you-integrate-custom-tracing-with-cassandra/].
Bug: cqlsh - module 're' has no attribute 'sre_parse' - with Python 3.11
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/Dec/22 17:43
Resolved: 20/Jan/23 21:31

User reported an error with cqlsh (Cassandra 4.0.7) on Stack Overflow: [https://stackoverflow.com/questions/74673247/cannot-able-to-run-cqlsh-due-to-python-attribute-error|https://stackoverflow.com/questions/74673247/cannot-able-to-run-cqlsh-due-to-python-attribute-error?noredirect=1#comment131807816_74673247]

 

Found out that the user was using Python 3.11, and I was able to reproduce it with that.

{{% python3.11 bin/cqlsh.py}}
{{Traceback (most recent call last):}}
{{  File "/Users/aaronploetz/local/apache-cassandra-4.0.7/bin/cqlsh.py", line 159, in <module>}}
{{    from cqlshlib import cql3handling, cqlhandling, pylexotron, sslhandling, cqlshhandling}}
{{  File "/Users/aaronploetz/local/apache-cassandra-4.0.7/bin/../pylib/cqlshlib/cql3handling.py", line 19, in <module>}}
{{    from cqlshlib.cqlhandling import CqlParsingRuleSet, Hint}}
{{  File "/Users/aaronploetz/local/apache-cassandra-4.0.7/bin/../pylib/cqlshlib/cqlhandling.py", line 23, in <module>}}
{{    from cqlshlib import pylexotron, util}}
{{  File "/Users/aaronploetz/local/apache-cassandra-4.0.7/bin/../pylib/cqlshlib/pylexotron.py", line 342, in <module>}}
{{    class ParsingRuleSet:}}
{{  File "/Users/aaronploetz/local/apache-cassandra-4.0.7/bin/../pylib/cqlshlib/pylexotron.py", line 343, in ParsingRuleSet}}
{{    RuleSpecScanner = SaferScanner([}}
{{                      ^^^^^^^^^^^^^^}}
{{  File "/Users/aaronploetz/local/apache-cassandra-4.0.7/bin/../pylib/cqlshlib/saferscanner.py", line 91, in _{_}init{_}_}}
{{    s = re.sre_parse.State()}}
{{        ^^^^^^^^^^^^}}

{{AttributeError: module 're' has no attribute 'sre_parse'}}

Appears to be something specific (again) with Python's synchronizing regex engine (SRE).  Works fine with Python 3.10, so there may have been a(nother) breaking change in that the re module with 3.11.
Bug: TRUNCATED data come back after a restart or upgrade
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 09/Dec/22 00:03
Resolved: 26/Apr/23 11:52

When we use the TRUNCATE command to delete all data in the table, the deleted data come back after a node restart or upgrade. This problem happens at the latest releases (2.2.19, 3.0.28, or 4.0.7)
h1. Steps to reproduce
h2. To reproduce it at release (3.0.28 or 4.0.7)

Start up a single Cassandra node. Using the default configuration and execute the following cqlsh commands.
{code:java}
CREATE KEYSPACE IF NOT EXISTS ks WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };
CREATE TABLE  ks.tb (c3 TEXT,c4 TEXT,c2 INT,c1 TEXT, PRIMARY KEY (c1, c2, c3 ));
INSERT INTO ks.tb (c3, c1, c2) VALUES ('val1','val2',1);
CREATE INDEX IF NOT EXISTS tb ON ks.tb ( c3);
TRUNCATE TABLE ks.tb;
DROP INDEX IF EXISTS ks.tb; {code}
Execute a read command
{code:java}
cqlsh> SELECT c2 FROM ks.tb; 

 c2
----

(0 rows) {code}
Then, we flush the node and kill the Cassandra daemon by
{code:java}
bin/nodetool flush
pgrep -f cassandra | xargs kill -9 {code}
We restart the node. When the node has started, perform the same read, and the deleted data comes back again.
{code:java}
cqlsh> SELECT c2 FROM ks.tb; 

 c2
----
  1

(1 rows) {code}
h2. To reproduce it at release (2.2.19)

We don't need to kill the Cassandra daemon. Use bin/nodetool stopdaemon is enough. The other steps are the same as reproducing it at 4.0.7 or 3.0.28.
{code:java}
bin/nodetool -h ::FFFF:127.0.0.1 flush 
bin/nodetool -h ::FFFF:127.0.0.1 stopdaemon{code}
 

I have put the full log to reproduce it for release 4.0.7 and 2.2.19 in the comments.
Bug: running Ant rat targets without git fails
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 19/Oct/22 11:51
Resolved: 21/Oct/22 08:24

This was detected while reviewing CASSANDRA-17753

When trying it build without git command, I am getting this:

{code}
_rat_init:

_build_ratinclude:
    [exec] Execute failed: java.io.IOException: Cannot run program "git" (in directory "/home/smiklosovic/test/cassandra-CASSANDRA-17753"): error=2, No such file or directory
    
BUILD FAILED
/home/smiklosovic/test/cassandra-CASSANDRA-17753/.build/build-rat.xml:38: "Warning: Could not find file /home/smiklosovic/test/cassandra-CASSANDRA-17753/build/.versioned_files to copy
{code}

This was ran on a system without Git and against code downloaded from Github in a zip file.
Bug: Optimize and fix flaky test_network_topology_strategy and test_network_topology_strategy_each_quorum for large containers in CircleCI
Status: Open
Priority: Normal
Created: 15/Nov/22 23:10

As pointed in CASSANDRA-18001, there are two Python Large DTests that fail because of not enough resources with large containers. [~adelapena] looked into them and suggested that the tests themselves can be optimized for running on Large containers instead of using XLarge containers for only two tests from the whole suite. 
{quote}As for the two tests starting nine nodes each, {{test_network_topology_strategy}} and {{{}test_network_topology_strategy_each_quorum{}}}, I think they were added by CASSANDRA-10584 to test multi-DC consistency levels. I wonder if they actually need to start 3 data centers with 3 nodes each. Would two data centers with 3 nodes be enough to test multi-DC consistency levels with multiple data centers? 6 nodes should be doable on Circle with the current resources.
{quote}
EDIT: On further testing it was noticed that the tests also started failing on 3.0 and 3.11 even with XLarge containers and there might be also some timing issue further involved and to be investigated.
Bug: List command output not correct for super user, after grant command
Status: Resolved
Priority: Normal
Resolution: Workaround
Created: 04/Nov/22 17:44
Resolved: 01/Aug/23 05:49

Running local Cassandra with below config:
{noformat}
authenticator: PasswordAuthenticator
authorizer: CassandraAuthorizer
role_manager: CassandraRoleManager
network_authorizer: CassandraNetworkAuthorizer{noformat}

Created a super user and then ran *Grant select* command on a keyspace. 
{noformat}
shaadmin1@cqlsh> CREATE USER 'shaadmin1c1' WITH PASSWORD 'shaadmin1c1' SUPERUSER;
shaadmin1@cqlsh:system_auth> grant select on testk1.t1 to shaadmin1c1;
shaadmin1@cqlsh:system_auth> alter role shaadmin1c1 with access to all datacenters;
{noformat}
 

After this, list permissions command showing only select permission for that role on the resource.
{noformat}
shaadmin1c1@cqlsh> list all permissions of shaadmin1c1;
role | username | resource | permission
----------------------------------------+-----------
shaadmin1c1 | shaadmin1c1 | <table testk1.t1> | SELECT
{noformat}
 


Row in role_permissions table:
{noformat}
role | resource | permissions
------------------------------------------------------------------------------------------
shaadmin1c1 | data/testk1/t1 | {'SELECT'}{noformat}

But insert command by that role on the resource is successful because role is a super user
{noformat}
shaadmin1c1@cqlsh> insert into testk1.t1 (c1, c2) values ('a', 1);
shaadmin1c1@cqlsh> select * from testk1.t1 ;
c1 | c2
---+---
a | 1
(1 rows)
{noformat}
 


The problem is, output of list permissions command, which indicates only select permission on the resource, is misleading. I think list command need to be fixed to show all permissions super user has on the resource. Also grant command for a super user can be either a no-op or throw error, because the role already have requested permissions.

 

Documentation also misleading:
{quote}True automatically grants AUTHORIZE, CREATE and DROP permission on ALL ROLES.
Superusers can only manage roles by default. To manage other resources, {color:#ff0000}you must grant the permission set to that resource. ** {color}For example, to allow access management for all keyspaces: {{{}GRANT ALL PERMISSIONS ON ALL KEYSPACES TO }}\{{{}{*}role_name{*}{}}}.
{quote}
Bug: Reuse of metadata collector can break key count calculation
Status: Open
Priority: Normal
Created: 16/Dec/22 14:19

When flushing a memtable we currently pass a constructed {{MetadataCollector}} to the {{SSTableMultiWriter}} that is used for writing sstables. The latter may decide to split the data into multiple sstables (e.g. for separate disks or driven by compaction strategy) — if it does so, the cardinality estimation component in the reused {{MetadataCollector}} for each individual sstable contains the data for all of them.

As a result, when such sstables are compacted the estimation for the number of keys in the resulting sstables, which is used to determine the size of the bloom filter for the compaction result, is heavily overestimated.

This results in much bigger L1 bloom filters than they should be. One example (which came about during testing of the upcoming CEP-26, after insertion of 100GB data with 10% reads):
(current)
{code}
 		Bloom filter false positives: 22627369
 		Bloom filter false ratio: 0.02257
 		Bloom filter space used: 1848247864
 		Bloom filter off heap memory used: 2338964088
{code}
(fixed)
{code}
 		Bloom filter false positives: 24426545
 		Bloom filter false ratio: 0.02429
 		Bloom filter space used: 1118910096
 		Bloom filter off heap memory used: 1532357432
{code}
Bug: Bug fix for WriteTimeoutException when using Paxos v2 in LWT performance test
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Oct/22 15:27
Resolved: 07/Nov/22 13:09

A WriteTimeoutException is encountered when using Paxos v2 in
an LWT performance test that only has a single datacenter because Paxos was still waiting for a response from another datacenter during the Commit/Acknowledge phase even though we were running with LOCAL_SERIAL.
Bug: Jenkins: Consider using the no-build-test flag
Status: Patch Available
Priority: Normal
Created: 04/Nov/22 16:57

CASSANDRA-16625 and CASSANDRA-18000 added a {{-Dno-build-test=true}} flag to skip the dependencies of the Ant test targets. This is useful to speed up the test targets if the {{build-test}} target has already been previously run, for example as part of {{{}ant jar{}}}.

That was created thinking mainly on CircleCI's multiplexer, where we run the same test target repeatedly. Skipping the already run depended on targets can significantly speed up the tests. The flag however is also useful for all other test jobs because every parallel runner can skip the test building step, and we have hundreds of parallel runners. Saving around 30s on every runner adds up considerable savings.

Maybe this flag can also be used for skipping test builds on Jenkins too, so each parallel test split can benefit from a slight boost. That could be done if either {{build-test}} or {{jar}} have already been run before calling the test target. I'm not familiarized with Jenkins config so I'm not sure whether it makes sense.
Bug: Add SpotBugs to the build
Status: Open
Priority: Normal
Created: 28/Nov/22 22:03

When working on CASSANDRA-17178 I found that several classes were being reported by the Simulator for not defining serializer version when they are Serializable; this may cause issues for the Simulator so felt it would be best to detect these earlier on before merging new ones.

SpotBugs has a large set of checks, some more valuable than others for the project; so we should maintain a curated list of issues to fail the build on, and others to warn on.

This topic was discussed in the following mail threads:

* Should we add?: https://lists.apache.org/thread/1ro1mvkpvt4vr24nw7dbpdlxo82mq3hz
* Should we fix UTF-8 issues? https://lists.apache.org/thread/sokxf46s7hyoxr9q4wm7dv3q2nm19nt3
Bug: Handle sstable metadata stats file getting a new mtime after compaction has finished
Status: Patch Available
Priority: Normal
Created: 15/Dec/22 09:31

Due to a race between compaction finishing and compaction strategies getting reloaded there is a chance that we try to add both the new sstable and the old compacted sstable to the compaction strategy, and in the LCS case this can cause the old sstable to get sent to L0 to avoid overlap. This changes the mtime of the stats metadata file and if the node is shut down before the sstable is actually deleted from disk, we fail starting with the following exception:

{code}
.../mockcf1-392b3ff07c5a11ed8c662f5760cb10b3/nb_txn_compaction_3983c030-7c5a-11ed-8c66-2f5760cb10b3.log
REMOVE:[.../data/TransactionLogsTest/mockcf1-392b3ff07c5a11ed8c662f5760cb10b3/nb-0-big-,1671096247000,5][4003386800]
         ***Unexpected files detected for sstable [nb-0-big-]: last update time [Thu Dec 15 10:24:09 CET 2022] (1671096249000) should have been [Thu Dec 15 10:24:07 CET 2022] (1671096247000)
ADD:[.../data/TransactionLogsTest/mockcf1-392b3ff07c5a11ed8c662f5760cb10b3/nb-2-big-,0,5][319189529]
{code}

A workaround for this (until we properly fix the way compaction strategies get notified about sstable changes) is to ignore the timestamp of the STATS component when cleaning up compaction leftovers on startup.
Bug: On-disk string index with index building and on-disk query path
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Nov/22 15:12
Resolved: 14/Apr/23 14:43

An on-disk index for string (literal) datatypes. This index is used for the following datatypes:
 * UTF8Type
 * AsciiType
 * CompositeType
 * Frozen types

This includes the ability to write the index to disk at index creation, by specific index rebuild and during SSTable compaction. 

Also the ability to query the on-disk index and combine the results with those from the in-memory indexes created by CASSANDRA-18058.
Bug: Allow to use user-defined functions (UDF) as masking functions
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 23/Nov/22 12:53
Resolved: 16/Mar/23 20:57

Allow to attach user-defined functions (UDF) to tables so they can be used as masking functions, as defined by [CEP-20|https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-20%3A+Dynamic+Data+Masking]. This will be similar to the native data masking functions introduced by CASSANDRA-17941 and attached to tables by CASSANDRA-18068.
Bug: Add a new SELECT_MASKED permission
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 23/Nov/22 12:49
Resolved: 16/Mar/23 20:57

Add a table-level SELECT_MASKED permission allowing certain users to query but not see the unmasked columns introduced by CASSANDRA-18068, assuming that they also have the SELECT permission on the table, as defined by [CEP-20|https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-20%3A+Dynamic+Data+Masking]. 

It would look like:
{code}
> CREATE USER unprivileged_user WITH PASSWORD 'xyz';
> CREATE USER privileged_user WITH PASSWORD 'zyx';
 
> GRANT SELECT ON ALL TABLE patients TO unprivileged_user;
> GRANT SELECT ON ALL TABLE patients TO privileged_user;
> GRANT SELECT_MASKED ON ALL TABLE patients TO privileged_user;
 
> LOGIN unprivileged_user
> SELECT name, birth FROM patients WHERE name='alice' ALLOW FILTERING;
 
Unauthorized: Error from server: code=2100 [Unauthorized] message="User has no UNMASK nor SELECT_UNMASK permission on <table k.patients>"
             
> LOGIN privileged_user
> SELECT name, birth FROM patients WHERE name='alice' ALLOW FILTERING;
 
 name    | birth
---------+------------
 alXXXXe | 1900-01-01
{code}
Bug: Add a new UNMASK permission
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 23/Nov/22 12:44
Resolved: 13/Mar/23 10:54

Add a new UNMASK permission allowing users with that permission to see the data masked by the masking functions attached to columns introduced by CASSANDRA-18068, as defined by [CEP-20|https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-20%3A+Dynamic+Data+Masking].

It would look like:
{code}
> CREATE TABLE patients (
      id timeuuid PRIMARY KEY,
      name text MASKED WITH default(),
      birth date MASKED WITH default()
      );
 
> INSERT INTO patients(id, name, birth) VALUES (now(), 'alice', '1982-12-21');
 
> CREATE USER unprivileged_user WITH PASSWORD 'xyz';
> CREATE USER privileged_user WITH PASSWORD 'zyx';
 
> GRANT SELECT ON TABLE patients TO unprivileged_user;
> GRANT SELECT ON TABLE patients TO privileged_user;
> GRANT UNMASK ON TABLE patients TO privileged_user;
 
> LOGIN unprivileged_user
 
> SELECT name, birth FROM patients WHERE id=db2b372f-f91b-4537-b46b-c478f8330c29;
 
 name    | birth
---------+------------
 alXXXXe | 1900-01-01 
             
> LOGIN privileged_user
> SELECT name, birth FROM patients WHERE id=db2b372f-f91b-4537-b46b-c478f8330c29;
 
 name  | birth
-------+------------
 alice | 1982-12-21
{code}
Bug: Bug fix for formula for Clock.waitUntil
Status: Resolved
Priority: Normal
Resolution: Resolved
Created: 06/Dec/22 16:01
Resolved: 07/Dec/22 09:22

While working on benchmarking Paxos improvements in OSS Cassandra, the results for paxos variant V2 with 100 partitions is much lower than the V1. Results were shown to [~benedict] and he found the incorrect formula for waitNanos in Clock.java.

Also, tests were added to verify behaviour of ContentionStrategy backoff methods
Test behaviour fixed in CASSANDRA-18086 plus another one-line bug fix in Clock.waitUntil().
Bug: Server should be started in @Before rather than @BeforeClass in CQLTester
Status: Open
Priority: Normal
Created: 22/Nov/22 14:32

I can see that there is some misunderstanding in the project of {{@BeforeClass}} annotated methods. 

Given we have a static method {{@BeforeClass static void initA()}} in {{CQLTester}} and {{@BeforeClass static void initB()}} in {{SomeTestClass extends CQLTester}}, it will run like the following:

1. static initializer of {{CQLTester}}
2. {{initA()}}
3. static initializer of {{SomeTestClass}}
4. {{initB()}}

Since the server is started in (2), there is essentially no way to consistently set startup properties or configuration of {{DatabaseDescriptor}} in {{SomeTestClass}} so that the starting server can pick it consistently. That is, if something works, it is just because some server class has not been used yet, but this actually cannot be guaranteed.

When I look in the code, it seems like the understanding of JUnit was that if {{initB()}} is defined, then {{initA()}} is not called, but this is not true.

Therefore, I propose to move the server initialization from {{initA()}} (call to {{SchemaLoader.prepareServer()}} to a method annotated with {{@Before}} in {{CQLTester}} so that the server is started with the first test case. Then, we will be sure that call to {{initB()}} will happen before the server is initialized and we will be able to set configuration consistently.