av9ash/GitBugs huggingface dataset:
Bug: Upgrade ASM, ByteBuddy and Mockito on trunk
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Aug/22 20:28
Resolved: 24/Aug/22 16:17

In preparation to support Java 17 I would like to update ASM, ByteBuddy and Mockito on trunk.
 * ASM - from 9.1 to 9.3; From the [release notes|https://asm.ow2.io/versions.html] I don't see anything disturbing for us; 
 * ByteBuddy - 1.10.10 to 1.12.13; [Release notes|https://github.com/raphw/byte-buddy/blob/master/release-notes.md]; they support Java 17 since 1.12.0 but I did not see an issue updating to the latest version; 
 * Mockito - 3.2.4 to 4.7.0; [Release notes|https://github.com/mockito/mockito/releases/]. The bump in the major version was due to removing outdated deprecated APIs which it doesn't seem that we use so this is not a breaking change for us.
Bug: When bootstrap fails, CassandraRoleManager may attempt to do read queries that fail with "Cannot read from a bootstrapping node", and increments unavailables counters
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 14/Jul/22 22:58
Resolved: 25/Jul/22 21:35

If bootstrap fails we stay alive but avoid joining the ring.  CassandraRoleManager attempts to schedule its setup every 10 seconds, but this will fail due to Bootstrap being pending; the side effect to this is that readMetrics.unavailables gets incremented.

We should defer attempting to setup only after we know its safe to perform such as read
Bug: Include estimated active compaction remaining write size when starting a new compaction
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Sep/22 19:07
Resolved: 10/Oct/22 17:27

There are a few things we can augment in how we handle compactions locally to improve upon the very simple "keep X Mebibytes free on disk" we currently have:
 # Allow specification of free space available in percent rather than raw size
 # Do this on a per-disk basis based on what's going on with compactions
 # Include an estimate of the disk space of active compactions in flight when we do the above calculation check so we don't blow past our limits due to concurrent runs
Bug: Capital P gets confused in the parser for a Duration in places where IDENT are needed
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 23/Sep/22 18:05
Resolved: 27/Apr/23 08:17

This was found while adding Accord Transaction syntax into CQL and fuzz testing to validate all possible cases… in doing this the following was found

{code}
String query = "BEGIN TRANSACTION\n" +
                           "  LET P = (SELECT v FROM " + keyspace + ".tbl WHERE k=? AND c=?);\n" +
                           "  LET row2 = (SELECT v FROM " + keyspace + ".tbl WHERE k=? AND c=?);\n" +
                           "  SELECT v FROM " + keyspace + ".tbl WHERE k=? AND c=?;\n" +
                           "  IF P IS NULL AND row2.v = ? THEN\n" +
                           "    INSERT INTO " + keyspace + ".tbl (k, c, v) VALUES (?, ?, ?);\n" +
                           "  END IF\n" +
                           "COMMIT TRANSACTION";
{code}

Fails with

{code}
SyntaxException: line 2:6 mismatched input 'P' expecting IDENT (BEGIN TRANSACTION  LET [P]...)
{code}

The new LET syntax found this, but was able to reproduce in other cases

{code}
cqlsh:ks> CREATE TABLE P (k INT PRIMARY KEY);
SyntaxException: line 1:13 no viable alternative at input 'P' (CREATE TABLE [P]...)
cqlsh:ks>
cqlsh:ks> CREATE TABLE p (k INT PRIMARY KEY);
cqlsh:ks>
{code}
Bug: Mixed mode support for internode authentication during TLS upgrades
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/Sep/22 21:55
Resolved: 07/Oct/22 22:22

During upgrades from "non-ssl -> ssl" or "ssl-mTLS" the cluster should be able to function in mixed mode with some nodes supporting "non-ssl" authentication and the new nodes supporting "mTLS" authentication. Currently we do not have this supported and because of which upgrades are not possible for upgrading internode authentication strategies.

If a node is configured in optional mode for internode connections, retry with other SSL strategies If the node is not able to connect to other nodes due to authentication problems.
Bug: Raise test timeouts
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 05/Jul/22 04:34
Resolved: 14/Jul/22 09:45

We have seen for some time now junits timeout frequently on jenkins. This is probably down to it being a very loaded env. On circle we don't observe that behavior probably bc it's not so loaded.

The question is whether it is time to raise timeouts as they might be hiding legit failures. As en experiment I raised timeouts in a branch and ran jenkins against it. What I see is that the last 4.1 run had 14 failures out of which 12 were timeouts. Increasing timeouts reveals what looks to be 9 legit failures where 2 are timeouts that probably need to be investigated.
Bug: [jamm] Test failure: o.a.c.repair.RepairJobTest.testNoTreesRetainedAfterDifference
Status: Open
Priority: Normal
Created: 12/Sep/22 12:27

The unit test {{org.apache.cassandra.repair.RepairJobTest#testNoTreesRetainedAfterDifference}} seems to be slightly flaky at least in 4.0. We haven't seen it failing on Butler, but after [a failure on a patch branch|https://app.circleci.com/pipelines/github/jacek-lewandowski/cassandra/281/workflows/e6094c00-b611-4772-9772-205f4c76ecba/jobs/2126/tests], this repeated run shows a 0.28% flakiness:

* https://app.circleci.com/pipelines/github/adelapena/cassandra/2076/workflows/8adbfe99-afb5-43af-84ad-43df2a2a86e2/jobs/20816/tests
* https://app.circleci.com/pipelines/github/adelapena/cassandra/2076/workflows/e550d8c2-7c35-4326-bd95-6909978d344b/jobs/20817/tests

{code}
java.lang.reflect.InaccessibleObjectException: Unable to make field private jdk.internal.platform.cgroupv1.SubSystem$MemorySubSystem jdk.internal.platform.cgroupv1.Metrics.memory accessible: module java.base does not "opens jdk.internal.platform.cgroupv1" to unnamed module @157853da
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:340)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:280)
	at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:176)
	at java.base/java.lang.reflect.Field.setAccessible(Field.java:170)
	at org.github.jamm.MemoryMeter.addFieldChildren(MemoryMeter.java:330)
	at org.github.jamm.MemoryMeter.measureDeep(MemoryMeter.java:269)
	at org.apache.cassandra.utils.ObjectSizes.measureDeep(ObjectSizes.java:216)
	at org.apache.cassandra.repair.RepairJobTest.testNoTreesRetainedAfterDifference(RepairJobTest.java:271)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
{code}
Flakiness is really low but the failure is clearly reproducible in j11.
Bug: WEBSITE - September 2022 blog "Apache Cassandra Changelog #19"
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 09/Sep/22 06:01
Resolved: 27/Jul/23 08:46

This ticket is to capture the work associated with publishing the September 2022 blog "Apache Cassandra Changelog #19"

If this blog cannot be published by the September 9, 2022 publish date, please contact me, suggest changes, or correct the date when possible in the pull request for the appropriate time that the blog will go live (on both the blog.adoc and the blog post's file).
Bug: WEBSITE - September 2022 blog "Cassandra Days for London and Amsterdam announced"
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Sep/22 23:39
Resolved: 27/Jul/23 08:45

This ticket is to capture the work associated with publishing the September 2022 blog "Cassandra Days for London and Amsterdam announced"

If this blog cannot be published by the *September 29, 2022 publish date*, please contact me, suggest changes, or correct the date when possible in the pull request for the appropriate time that the blog will go live (on both the blog.adoc and the blog post's file).
Bug: WEBSITE - September 2022 blog "Cassandra Summit Returns in 2023"
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Sep/22 20:36
Resolved: 27/Jul/23 08:45

This ticket is to capture the work associated with publishing the September 2022 blog "Cassandra Summit Returns in 2023"

If this blog cannot be published by the *September 29, 2022 publish date*, please contact me, suggest changes, or correct the date when possible in the pull request for the appropriate time that the blog will go live (on both the blog.adoc and the blog post's file).
Bug: Incorrect cassandra.logdir on Debian systems
Status: Review In Progress
Priority: Normal
Created: 25/Jul/22 15:23

The Debian packaging patches bin/cassandra to use /var/log/cassandra for logs, it does so conditionally however, only if CASSANDRA_LOG_DIR is unset. This occurs _after_ cassandra-env.sh is sourced though, which also sets CASSANDRA_LOG_DIR if unset (to $CASSANDRA_HOME/logs).  The result is that -Dcassandra.lodir is set to /usr/share/cassandra/logs on Debian systems.
Bug: Settings Virtual Table should display the values assigned to a property in the DatabaseDescriptor on startup and not null (as per the yaml)
Status: Open
Priority: Normal
Created: 06/Jul/22 15:06

There are a few properties that after startup do not show their assigned values as per the DatabaseDescriptor assignment but the cassandra.yaml value.

They will not be also updated in the virtual table down the road in case they are updated through JMX, nodetool etc.

*EDIT: This ticket should serve to check the properties that are not type Duration, Data Storage and Data Rate; also that are not new to 4.1.* I will post a list of who are those later today for convenience. We target all those in Config class (some advanced properties are not broadly advertised in cassandra.yaml intentionally).

There is [Settings Virtual Table |https://cassandra.apache.org/doc/trunk/cassandra/new/virtualtables.html#settings-virtual-table] which is supposed to show the values for our config parameters at any time. Especially useful if any property was changed after startup through JMX/nodetool and it doesn't match anymore the value in cassandra.yaml. For this to be possible, we need to ensure that the parameters are always updated in the Config class. It was observed that some are not always updating in Config class, but after startup delegating to other internal variables. This is a bug and this task should review and address any new findings. 

Classes of interest - [SettingsTable|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/virtual/SettingsTable.java] where you can see how config parameters are listed; [Config|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/config/Config.java] class where our configuration parameters are defined.

We need patches 4.0 and above. I suggest you start looking into 4.0 branch and then merge into higher branches. As you won't be checking the data storage, data rate and duration type parameters, there shouldn't be many conflicts on merge. 

We have a lot of parameters and I suggest you split the list into batches to check and produce patches where/if needed to make the work more incremental and easier to work on and review it.
Bug: Ensure CircleCI and ASF Jenkins CI are aligned
Status: In Progress
Priority: Normal
Created: 27/Sep/22 16:57

As discussed in this [thread|https://lists.apache.org/list.html?dev@cassandra.apache.org] the Cassandra community wants to see CircleCI and ASF CI being aligned - running the same tests, configurations, all tests.

Exceptions: packaging.

A few examples of discrepancies we already noticed:
 * utests_system_keyspace_directory run only in CircleCI - CASSANDRA-17145
 * dtest-large run only in Jenkins
 * simulator tests run only in CircleCI
 * In a quick skim I think I didn't see [these|https://github.com/apache/cassandra-builds/blob/trunk/build-scripts/cassandra-dtest-pytest.sh#L84-L89] runs too in CircleCI - dtest-offheap, dtest-large, dtest-large-novnode
 * packaging is also only tested in CircleCI as far as I recall but that is deterministic and we will rely on Jenkins about that. 

And these are only a few examples on top of my mind. I am sure we will find more. We also need to verify the way we call those tests is correct and matches in both CIs. (I was looking to solve similar discrepancy in CASSANDRA-17912)

Some info on our tests suites here - [https://cassandra.apache.org/_/development/testing.html,]

 [cassandra-builds repo|https://github.com/apache/cassandra-builds] where our test images reside and the Jenkins build scripts, which I already referred to. 

CircleCI info can be found in the readme which resides in the in-tree folder dedicated to configuration and scripts for Circle CI - [https://github.com/apache/cassandra/tree/trunk/.circleci]

EDIT: More findings:
 *  burn tests missing in CircleIC
 * cqlshlib in 3.11.
 * 4.0+ we also need in-jvm JDK8 with 11, cqlshlib JDK8 with 11.

EDIT: Java distributed tests running with vnodes should be added to Jenkins
Bug: Node will not get schema updates if it replaces a removed node with the same address
Status: Open
Priority: Normal
Created: 12/Sep/22 09:25

If a node is removed and then a node with the same address is added, it is ignored by migration coordinator - that is, when that node fails to push the schema change, the other nodes will not be able pull it later during periodic update.

Such situation will last until the other nodes gets restarted.

This is caused by permanently adding the address of the node being removed to the ignore list in migration coordinator.
Bug: URL for KEYS in Debian installation returns a redirect
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Jul/22 01:56
Resolved: 12/Apr/23 15:33

On the page [https://cassandra.apache.org/doc/latest/cassandra/getting_started/installing.html] the Debian package instructions has a line:

`curl https://www.apache.org/dist/cassandra/KEYS | sudo apt-key add -`

that should be updated to

`curl https://downloads.apache.org/cassandra/KEYS | sudo apt-key add -`

 

[https://www.apache.org/dist/cassandra/KEYS] currently returns a redirect.
Bug: DOC - Fix typo in CQL types page
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Aug/22 09:25
Resolved: 12/Apr/23 13:13

Typo @ https://cassandra.apache.org/doc/trunk/cassandra/cql/types.html:

bq. the {color:#DE350B}while{color} collection has to be read

should be:

bq. the {color:#00875A}whole{color} collection has to be read
Bug: Test Failure: dtest-novnode.transient_replication_test.TestTransientReplicationRepairLegacyStreaming.test_optimized_primary_range_repair_with_lcs 
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 14/Sep/22 15:53
Resolved: 26/Sep/22 20:00

Link to failure: https://ci-cassandra.apache.org/job/Cassandra-4.1/162/testReport/dtest-novnode.transient_replication_test/TestTransientReplicationRepairLegacyStreaming/test_optimized_primary_range_repair_with_lcs/

Error Message
{code}
subprocess.CalledProcessError: Command '('/usr/lib/jvm/java-8-openjdk-amd64/bin/java', '-cp', '/usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar:/home/cassandra/cassandra/cassandra-dtest/tools/../lib/jolokia-jvm-1.7.1-agent.jar', 'org.jolokia.jvmagent.client.AgentLauncher', '--host', '127.0.0.1', '--port', '8778', 'start', '7601')' returned non-zero exit status 1.
Stacktrace
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '('/usr/lib/jvm/java-8-openjdk-amd64/bin/java', '-cp', '/usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar:/home/cassandra/cassandra/cassandra-dtest/tools/../lib/jolokia-jvm-1.7.1-agent.jar', 'org.jolokia.jvmagent.client.AgentLauncher', '--host', '127.0.0.1', '--port', '8778', 'start', '7601')' returned non-zero exit status 1.
{code}
Bug: Fix Settings Virtual Table for permissions_update_interval_in_ms, roles_update_interval_in_ms and credentials_update_interval_in_ms
Status: Open
Priority: Normal
Created: 28/Jul/22 21:33

permissions_update_interval_in_ms, roles_update_interval_in_ms and credentials_update_interval_in_ms should take other 3 properties value if they are -1 (4.0) or null (4.1+)

This is done in the DatabaseDescriptor but not handled with the Settings Virtual Table.

We need to ensure we update the Settings Virtual Table. In my opinion these 3 are tricky in the sense it is good to know how we are going to approach the Update access to the Settings Virtual Table before fixing them so we don't end up rewriting solutions a few times or miss it later
Bug: Validate that JMX and the Settings Virtual Table are aligned
Status: Open
Priority: Normal
Created: 06/Jul/22 14:07

I discovered JMX methods that are not updating the Config parameters but directly the usage. (see index_summary_resize_interval) This is a bug in 4.0+ as users will see wrong values in the Settings virtual table which rely on the Config parameters (if some value was changed after startup). It will be explicitly important to ensure there is no disconnect between those when we add update option for the Virtual Table.

This ticket is meant to serve as an umbrella for subtasks to check the properties getters&setters&JMX methods and for tickets to fix any discovered bugs.

 

CC [~b.lerer@gmail.com] and [~dcapwell]
Bug: Validate that JMX updates any old pre-4.1 properties (which are not moved to the new Config classes; non-duration, non-data rate, non-data storage)
Status: Open
Priority: Normal
Created: 06/Jul/22 15:29

Check that any properties which are added pre-4.1 (non-Duration, non-Data Storage, non-Data Rate) are always updated by JMX and there are no inconsistent validations that might cover bugs. Validate proper update in Settings Virtual Table
Bug: Allow reducing scope for L0 to L1 compactions when space is limited
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 14/Sep/22 19:12
Resolved: 16/Sep/22 19:01

When using LCS we currently only allow reducing scope when doing STCS-in-L0, reason being that removing an sstable from a compaction targeting higher level could cause overlap in the level.

It is safe to remove sstables from L0 when doing L0->L1 compactions though, and this can allow us to continue compacting when we have a few gigantic sstables in L0.
Bug: Allow node to serve traffic only once compactions settle
Status: Open
Priority: Normal
Created: 21/Jul/22 06:49

Today when nodes are joined to the cluster, once data streaming is completed, node starts serving traffic, but it's possible there are a lot of pending compactions, so having reads accessing sstables during that time is going to make these reads slower and load the server. In some cases performance is so bad it can bring the application down. 

Today I overcome this by stopping native transport to a node after join finishes, and enabling it after compactions pending reach a certain threshold, it would be nice to have that as part of the join process, only consider join completed once there are not that many compactions pending.
Bug: Failed replacements due to "Cannot replace token xxx which does not exist"
Status: Resolved
Priority: Normal
Resolution: Won't Fix
Created: 14/Sep/22 18:59
Resolved: 22/Sep/22 17:32

Host replacements require host ID and token, however reporting of this info is typically blocked when the “status” field is missing. We should expose a system env (-D) param to advanced operators to have the ability to specify the replace_addresses_token to be used during host replacement in cases where Gossip gets into a bad state.

Given this is inherently an unsafe / high risk manual intervention, we want to avoid promoting this to being a config param however Gossip quite often contorts into problematic split-brain states like this and having the ability to manually override things helps unstick host replacements that go straddled. Should update NEWS.txt at the very least and consider documentation as well

This is an interim solution that'll be obviated by CEP-21, but is still useful on existing clusters.
Bug: Typo in https://cassandra.apache.org/doc/trunk/cassandra/cql/ddl.html
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 21/Aug/22 09:30
Resolved: 23/Jan/23 09:43

It is common for a cluster to define only one keyspace for an actie application

 

should be:

 

It is common for a cluster to define only one keyspace for an active application
Bug: DOC - Fix typo in CQL Data Definition page
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 08/Jul/22 13:02
Resolved: 23/Jan/23 09:42

Fix typo in the first paragraph of the CQL [Data Definition|https://cassandra.apache.org/doc/latest/cassandra/cql/ddl.html] page:

bq. ... It is common for a cluster to define only one keyspace for an {color:#DE350B}actie{color} application.

I created this Pull Request to highlight the error

https://github.com/apache/cassandra/pull/1722
Bug: upgrade_tests/upgrade_udtfix_test.py failing in Circle CI for Cassandra 3.0 and 3.11
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 18/Jul/22 14:48
Resolved: 30/Sep/22 19:22

The tests are skipped in Jenkins and failing in CircleCI:

[3.0|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/1783/workflows/03577cf2-2be9-4b14-8339-d9ae4f4b4a22/jobs/13034]

[3.11|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/1779/workflows/ec3fe1cf-649c-46d1-aaff-eb3e71cec57e/jobs/13030]
Bug: Add solution for CASSANDRA-17581 to older branches for tests
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 12/Sep/22 14:54
Resolved: 19/Oct/22 15:09

Some of our tests use old branches for the purposes of testing upgrades and behavior in mixed versions, but those lacking CASSANDRA-17581 will fail on nodetool with a modern JDK.  We can port this fix to those branches and adjust the tests to use the newest version of them to solve this going forward.
Bug: upsert with adder support is not consistent with numbers and strings in LWT
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Aug/22 21:18
Resolved: 07/Sep/22 01:09

In 4.1 UPDATE name = name + offset support got extended to numbers and strings (not just counters) but the behavior isn’t consistent

Number: throws a error due to Int32Type.add(null, 42) failing
String: returns the non-null updated values

What Postgres does is keeps the value null

{code}
postgres=# create table x (pk int PRIMARY KEY, a int, b text);
CREATE TABLE
postgres=# insert into x (pk, a, b) values(0, null, null);
INSERT 0 1
postgres=# UPDATE x SET a = a + 1 WHERE pk = 0;
UPDATE 1
postgres=# select * from x;
 pk | a | b
----+---+---
  0 |   |
(1 row)
postgres=# UPDATE x SET b = b + 'one' WHERE pk = 0;
ERROR:  operator does not exist: text + unknown
LINE 1: UPDATE x SET b = b + 'one' WHERE pk = 0;
                           ^
HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
postgres=# UPDATE x SET b = b + CAST('one' as text) WHERE pk = 0;
ERROR:  operator does not exist: text + text
LINE 1: UPDATE x SET b = b + CAST('one' as text) WHERE pk = 0;
                           ^
HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
{code}
Bug: additionalWritePolicy in TableParams is not added into equals, hashcode and toString methods
Status: Open
Priority: Normal
Created: 26/Aug/22 14:32

This commit (1) introduces additionalWritePolicy into TableParams but equals, hashcode and toString are not updated and they do not contain it. Unless this was done on purpose (which I doubt), I think we should add it there.

(1) https://github.com/apache/cassandra/commit/4ae229f5cd270c2b43475b3f752a7b228de260ea#diff-1179d180c9b57e04b1088a10e5e168dcb924ace760acccc71fb28daf654453b1