av9ash/GitBugs huggingface dataset:
Bug: Fix error message handling when trying to use CLUSTERING ORDER with non-clustering column
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 16/Aug/22 00:44
Resolved: 14/Jun/23 16:41

Imagine ck1, ck2, v columns. For "CLUSTERING ORDER ck1 ASC, v DESC" error msg will suggest that information for ck2 is missing. But if you add it it will still be wrong as "v" cannot be used. So the problem here is really about using non-clustering column rather than about not providing information about some clustering column.

The following is example from 3.11, but the code is the same in 4.0, 4.1, trunk:
{code:java}
cqlsh:k_test> CREATE TABLE test2 (pk int, ck1 int, ck2 int, v int, PRIMARY KEY ((pk),ck1, ck2)) WITH CLUSTERING ORDER BY (v ASC);
InvalidRequest: Error from server: code=2200 [Invalid query] message="Missing CLUSTERING ORDER for column ck1"

cqlsh:k_test> CREATE TABLE test2 (pk int, ck1 int, ck2 int, v int, PRIMARY KEY ((pk),ck1, ck2)) WITH CLUSTERING ORDER BY (ck1 ASC, v ASC);
InvalidRequest: Error from server: code=2200 [Invalid query] message="Missing CLUSTERING ORDER for column ck2"

cqlsh:k_test> CREATE TABLE test2 (pk int, ck1 int, ck2 int, v int, PRIMARY KEY ((pk),ck1, ck2)) WITH CLUSTERING ORDER BY (ck1 ASC, ck2 DESC, v ASC);
InvalidRequest: Error from server: code=2200 [Invalid query] message="Only clustering key columns can be defined in CLUSTERING ORDER directive"{code}
We need to be sure that we return to the user the same correct error message in all three cases and it should be "Only clustering key columns can be defined in CLUSTERING ORDER directive"

+Additional information for newcomers+
 * [This|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/cql3/statements/schema/CreateTableStatement.java#L251-L252] is where we handle the issue incorrectly as proved by the example. The easiest way to handle this issue would be to  check the key set content of {_}clusteringOrder{_}.
 * It would be good also to add more unit tests in [CreateTableValidationTest|https://github.com/apache/cassandra/blob/trunk/test/unit/org/apache/cassandra/schema/CreateTableValidationTest.java] to cover different cases. 
 * I suggest we create patch first for 3.11 and then we can propagate it up to the next versions.
Bug: Improve documentation around query methods in CQLTester and GuardrailTester
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 22/Jul/22 17:23
Resolved: 25/Jul/22 15:23

Anything that relies on CQLTester.executeFormattedQuery (the assertInvalidThrowMessage methods for instance) will use internal client state rather than the client state specified for the query, thus nullifying any guardrail or systems keyspace permission checks as the ClientState.isInternal flag overrides all such permission checking. Reference: [link|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/ClientState.java#L385-L388]
 
See chain of CQLTester -> ClientState.isInternal here if interested:
[CQLTester|https://github.com/apache/cassandra/blob/trunk/test/unit/org/apache/cassandra/cql3/CQLTester.java#L1325]
[QueryProcessor|https://github.com/apache/cassandra/blob/8451acc8d8dcfee20d692d1e70ae11b60d2f004e/src/java/org/apache/cassandra/cql3/QueryProcessor.java#L447]
 
This is pretty easy to stumble across when testing guardrails as GuardrailTester adds a variety of assertFails and assertThrows signatures that _do_ respect the client state and thus apply guardrails ([example|https://github.com/apache/cassandra/blob/trunk/test/unit/org/apache/cassandra/db/guardrails/GuardrailTester.java#L271])
 
We should add documentation to the method calls in CQLTester and GuardrailTester to reflect this discrepancy as it can easily trip someone up writing tests for guardrails.
Bug: Allow disabling hotness persistence, or tuning of rate limiter
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/Aug/22 16:06
Resolved: 19/Sep/22 17:11

The persisting of the sstables hotness when there are 10s of thousands of sstables can have issues keeping up, and the rate limiter is hard coded. Another option may be nice to just completely disable the feature.

When sstables super backed up (from repairs) the hotness tracking tends to cause the STCS in L0 to make poor decisions, always grabbing the large sstables and skipping the tiny sstables that would benefit more.
Bug: Restore streaming_keep_alive_period functionality on the netty control streaming channel
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 22/Jul/22 00:08
Resolved: 09/Nov/22 11:06

While working on another ticket I noticed that after [CASSANDRA-16927] CEP-10 Phase 1: Refactor Streaming 

streaming_keep_alive_period is not used anymore, except to print it in error message [here|https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/streaming/StreamSession.java#L689]

If the property should not be used anymore, we need to deprecate it and fix the error message as it is misleading.

[~benedict] , [~samt] , [~aleksey] , can you, please, check and take care of this one as authors of that patch?

Thank you in advance!
Bug: Add --resolve-ip option on 'nodetool gossipinfo'
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Sep/22 17:55
Resolved: 27/Oct/22 13:04

Give nodetool gossipinfo the option of either displaying IPs or hostnames for the nodes in a ring.

Note: this option is already present for "nodetool status" and "nodetool ring"
Bug: Allow Release Version to be null
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/Aug/22 15:59
Resolved: 12/Sep/22 16:50

During node replacement, schema pulls in the {{MigrationCoordinator}} can have null values on the following:
{code}
final String releaseVersion = state.getApplicationState(ApplicationState.RELEASE_VERSION).value;
{code}
We can NPE out right now in certain edge cases.
Bug: Test failure: org.apache.cassandra.tools.StandaloneUpgraderOnSStablesTest.testUpgradeSnapshot
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 23/Aug/22 15:37
Resolved: 23/Aug/22 23:52

The test {{org.apache.cassandra.tools.StandaloneUpgraderOnSStablesTest.testUpgradeSnapshot}} is flaky on trunk, [as reported by Butler|https://butler.cassandra.apache.org/#/ci/upstream/workflow/Cassandra-trunk/failure/org.apache.cassandra.tools/StandaloneUpgraderOnSStablesTest/testUpgradeSnapshot]:
* https://ci-cassandra.apache.org/job/Cassandra-trunk/1287/testReport/org.apache.cassandra.tools/StandaloneUpgraderOnSStablesTest/testUpgradeSnapshot/
* https://ci-cassandra.apache.org/job/Cassandra-trunk/1286/testReport/org.apache.cassandra.tools/StandaloneUpgraderOnSStablesTest/testUpgradeSnapshot/
* https://ci-cassandra.apache.org/job/Cassandra-trunk/1285/testReport/org.apache.cassandra.tools/StandaloneUpgraderOnSStablesTest/testUpgradeSnapshot/

{code}
Failed 1 times in the last 27 runs. Flakiness: 3%, Stability: 96%
Error Message
Timeout occurred. Please note the time in the report does not reflect the time until the timeout.
Stacktrace
junit.framework.AssertionFailedError: Timeout occurred. Please note the time in the report does not reflect the time until the timeout.
	at java.util.Vector.forEach(Vector.java:1277)
	at java.util.Vector.forEach(Vector.java:1277)
	at java.util.Vector.forEach(Vector.java:1277)
	at org.apache.cassandra.anttasks.TestHelper.execute(TestHelper.java:53)
	at java.util.Vector.forEach(Vector.java:1277)
{code}
It seems that the test gets stuck until JUnit timeouts.
Bug: Read/Write/Truncate throw RequestFailure in a race condition with callback timeouts, should return Timeout instead
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 17/Aug/22 18:46
Resolved: 20/Aug/22 01:18

There is an edge case with write timeout where the condition gets signaled on the timeouts and this happens before await times out, this triggers us to send a Failure rather than a Timeout
Bug: When doing a host replacement, -Dcassandra.broadcast_interval_ms is used to know when to check the ring but checks that the ring wasn't changed in -Dcassandra.ring_delay_ms, changes to ring delay should not depend on when we publish load stats
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 25/Jul/22 22:57
Resolved: 26/Jul/22 20:49

In some cases changing the ring delay to be higher is desired (such as when network is slower), but in order to do a host replacement operators need to know that there is a relationship with -Dcassandra.broadcast_interval_ms which is used to publish LOAD via gossip. This relationship should not be relied on and instead we should sleep based off ring delay rather than delay when we publish loads.
Bug: Users of NativeLibrary should handle lack of JNA appropriately when running DBD in client mode
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/Aug/22 18:10
Resolved: 05/Aug/22 17:50

There are cases where users that are using cassandra as a library and initializing {{DatabaseDescriptor}} specifically in client mode without JNA should be able to safely continue w/out JNA but are currently getting killed w/FSReadErrors. We should loosen that up a bit.
Bug: Opcodes.ASM7 should be used in UDFByteCodeVerifier to support JDK11
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 31/Aug/22 20:29
Resolved: 20/Oct/22 18:22

In CASSANDRA-15108 ASM was updated and UDFByteCodeVerifier was updated to provide Opcodes.ASM7 but only in 1 of 3 places. 

We need to update that for 4.0, 4.1 and trunk where we support JDK11.

Also, I think it will be good to add one place where we update that when we add new JDK support as now I see we will have to update it at 4 places at least already (also for the simulator where currently it is Opcodes.ASM7 there, correctly added). Also, we can add a note in build.xml for people updating ASM. I think many people practice updating Opcodes.ASM* with the update of ASM version but in our case with update of JDK, from what I see. We will need to switch to ASM9 when we add the JDK 17 support. One stop place for that and adding a note for maintainers sounds like the right way to move forward at least on trunk. 

But for 4.0 and 4.1 we need at least to switch to ASM7 everywhere as far as I can tell.
Bug: Speculative execution threshold unit mismatch
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/Sep/22 14:49
Resolved: 06/Sep/22 16:10

CASSANDRA-16760 changed the ColumnFamilyStore.coordinatorRead/WriteLatency to be in microseconds rather than nanoseconds but did not change the calculation for ColumnFamilyStore.sampleReadLatencyNanos/additionalWriteLatency which is used as the threshold for speculative read execution / issuing additional writes.

The impact of this bug is that the threshold is 1000x smaller than intended when using percentile based speculation (tables default to 99p), so effectively speculation happens for ALL reads configured to speculate on a percentile, including min hybrid policies that use percentiles.
Bug: Clean up ScheduledExecutors, CommitLog, and MessagingService shutdown for in-JVM dtests
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 05/Jul/22 20:01
Resolved: 14/Jul/22 16:56

There appear to be two problems w/ the way we shut down {{ScheduledExecutors}} in Instance in 4.0+:

1.) We do it twice, Ince as part of a larger batch of shutdown activity, and then again in its own {{parallelRun()}} block.
2.) It happens before {{MessagingService}} shuts down, but some messaging-related threads (see {{StreamSession#closeSession()}}) can submit tasks to {{nonPeriodicTasks}}.

We should do it once, and do it after the {{MessagingService}} has properly shut down.
Bug: Fix potential test flake in MixedModeMessageForwardTest
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/Aug/22 16:02
Resolved: 12/Sep/22 17:09

This test will occasionally fail because there are no guarantees that the messaging service will be connected before the test tries to send the mutation request.

Given that this test is checking that {{FWD_FRM}} is being deserialized correctly, we can just wait until a {{CL.ALL}} request succeeds once to make sure messaging connections are established and then run the insert test.
Bug: Remove usage of deprecated javax certificate class
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/Aug/22 16:05
Resolved: 14/Sep/22 18:47

When initially implemented, we used the older {{javax.security.cert.X509Certificate}} when initializing an {{IAuthenticator}} with certs - as nobody is yet using this interface, and there is a reasonable way to fix it to use the correct, non-deprecated cert classes, we should switch to the modern {{java.security.cert.Certificate}} class.

The broader reasoning here is that consumers of this interface would have to jump through serialization/deserialization hoops on each authentication check to use the newer X509Certificate implementation, which has data on which we depend for our mTLS implementation, and in this case we can just cast them to the appropriate class.

See https://docs.oracle.com/javase/9/docs/api/javax/security/cert/X509Certificate.html for further details
Bug: fix restarting of services on gossipping-only members
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/Jul/22 11:47
Resolved: 27/Jul/22 13:32

When a node is started with -Dcassandra.join_ring=false, it is possible to talk to it via CQL. If we disable it via "nodetool disablebinary", it is not possible to enable it back with "nodetool enablebinary".

The reason why is that enablebinary command eventually calls "StorageService#checkServiceAllowedToStart" and it throws when:

1) the node is draining (which does not happen)
2) it is shutting down (which does not happen)
3) is not in NORMAL state - which happens.

The state of the gossipping-only member is STARTING for ever.

The check should make an exception here, like if it is in STARTING but -Dcassandra.join_ring=false, start it anyway.
Bug: Read Unavailables due to GossipStage backlog with high number of pending tasks
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 17/Aug/22 20:04
Resolved: 18/Aug/22 16:14

In CASSANDRA-16759, like we changed from iterating over liveMembers + unreachableMembers to all keys in endpointStateMap [here|https://github.com/apache/cassandra/commit/2fba5c80ce7bf71d04c62043ffa1088b9e832d83#diff-99267a2170b04fd7dd24d6c6bf2ba1fc26d6dc896cd74f8c5bd56c476e2540e4R208].

We can have an endpoint in quarantine (in justRemovedEndpoints) which has no release version information. In this case we don't memoize the value and instead recalculate the min version for every endpoint for every incoming gossip message... which can be bad.
Bug: Add the ability for operators to allow intentional loosening of definition of "empty" in Gossip for specific edge case failure scenarios
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 19/Aug/22 17:19
Resolved: 23/Aug/22 16:25

Right now {{empty}} is very specific to a single edge case (i.e. in {{isEmptyWithoutStatus()}} our usage of hbState() + applicationState), but there are other failure cases which block host replacements and require intrusive workarounds and human intervention to recover from when you have something in hbState() you don't expect.

If we allow opt-in to a more risky (i.e. we don’t know how we got there) definition of empty, then host replacements can make progress even when Gossip's gotten into a bad state. Which it does. All too often.

This parameter will obviously need some NEWS.txt and other documentation around it to explain the context for end users.

Now that I think of it, general "how to troubleshoot Gossip problems" might be worth writing up and including this as part of it for operators and users, specifically on our [Troubleshooting|https://cassandra.apache.org/doc/latest/cassandra/troubleshooting/index.html] page. Probably create that as another ticket and defer that update to there and rely on news.txt and the param documentation for this one just to get the functionality into the system for operators who need it.

A touch more context:
{code}
// In the very specific case where hbState.isEmpty and STATUS is missing, this is known to be safe to "fake"
// the data, as this happens when the gossip state isn't coming from the node but instead from a peer who
// restarted and is missing the node's state
//
// When hbState is *not* empty, then the node gossiped an empty STATUS, this happens during bootstrap and it's not
// possible to tell if this is ok or not (we can't really tell if the node is dead or having networking issues);
// for these cases we need to allow an external actor to verify and inform Cassandra that it is safe; this is done by
// updating the LOOSE_DEF_OF_EMPTY_ENABLED field.
{code}
Bug: Flaky test_rolling_upgrade
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 09/Aug/22 06:31
Resolved: 10/Aug/22 05:15

The fix on CASSANDRA-17140 needs to be extended into other places as it seems it now fails only one in a billion but still we can fix that one.

{noformat}
Regression

dtest-upgrade.upgrade_tests.upgrade_through_versions_test.TestProtoV3Upgrade_AllVersions_RandomPartitioner_EndsAt_3_11_X_HEAD.test_rolling_upgrade (from Cassandra dtests)
Failing for the past 1 build (Since
#115 )
Took 10 min.
Failed 1 times in the last 9 runs. Flakiness: 12%, Stability: 88%
Error Message

RuntimeError: A subprocess has terminated early. Subprocess statuses: Process-1 (is_alive: True), Process-2 (is_alive: False), attempting to terminate remaining subprocesses now.

Stacktrace

self = <upgrade_tests.upgrade_through_versions_test.TestProtoV3Upgrade_AllVersions_RandomPartitioner_EndsAt_3_11_X_HEAD object at 0x7f4d313e4e50>

    @pytest.mark.timeout(3000)
    def test_rolling_upgrade(self):
        """
            Test rolling upgrade of the cluster, so we have mixed versions part way through.
            """
>       self.upgrade_scenario(rolling=True)

upgrade_tests/upgrade_through_versions_test.py:340: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
upgrade_tests/upgrade_through_versions_test.py:417: in upgrade_scenario
    self._check_on_subprocs(self.fixture_dtest_setup.subprocs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <upgrade_tests.upgrade_through_versions_test.TestProtoV3Upgrade_AllVersions_RandomPartitioner_EndsAt_3_11_X_HEAD object at 0x7f4d313e4e50>
subprocs = [<Process name='Process-1' pid=10867 parent=389 stopped exitcode=-SIGKILL daemon>, <Process name='Process-2' pid=10881 parent=389 stopped exitcode=1 daemon>]

    def _check_on_subprocs(self, subprocs):
        """
            Check on given subprocesses.
    
            If any are not alive, we'll go ahead and terminate any remaining alive subprocesses since this test is going to fail.
            """
        subproc_statuses = [s.is_alive() for s in subprocs]
        if not all(subproc_statuses):
            message = "A subprocess has terminated early. Subprocess statuses: "
            for s in subprocs:
                message += "{name} (is_alive: {aliveness}), ".format(name=s.name, aliveness=s.is_alive())
            message += "attempting to terminate remaining subprocesses now."
            self._terminate_subprocs()
>           raise RuntimeError(message)
E           RuntimeError: A subprocess has terminated early. Subprocess statuses: Process-1 (is_alive: True), Process-2 (is_alive: False), attempting to terminate remaining subprocesses now.

upgrade_tests/upgrade_through_versions_test.py:475: RuntimeError
{noformat}
Bug: CircleCI: Automatically detect and repeat new or modified JUnit tests
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 30/Sep/22 10:46
Resolved: 18/Oct/22 16:02

The purpose of this ticket is adding a new CircleCI job that automatically detects new or modified test classes and runs them repeatedly. That way we wouldn't need to manually specify those tests with {{.circleci/generate.sh}}.
Bug: A user should not be able to manually remove ephemeral snapshots
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Jul/22 19:43
Resolved: 30/Aug/22 18:39

in CASSANDRA-16911 we introduced the "-e" flag to nodetool listsnaphots which are returning ephemerals as well. An operator might try to remove these snapshots by hand. This should not be possible as these snapshots are there for repair to work on and manual removal breaks it. To be complete, these snapshots are removed as part of repair mechanism automatically, or they are removed on the next reboot upon node' start. They should never be removed by a human.
Bug: Clean SyncUtil from dead code and update it for new JDK versions
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Sep/22 03:02
Resolved: 25/Jul/23 19:45

During code inspection I noticed [dead code|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/utils/SyncUtil.java#L77-L87] (JDK 7) in SyncUtil.

From a very quick skim _I think_ the [Java 8 section|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/utils/SyncUtil.java#L65-L75] is applicable in JDK 11, not sure for JDK 17 but it seems it should stay at least until we have JDK11. To be revisited.
Bug: Altering / creating of a keyspace on insufficient number of replicas should filter out gosspping only members
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Jul/22 08:23
Resolved: 24/Jul/22 18:00

When there is a CQL CREATE / ALTER KEYSPACE query executed on a gossipping-only member of a cluster (-Dcassandra.join_ring=false) where the replication factor is bigger than the number of the nodes, there is currently a warning emitted, which is ok, but this number also includes the gossipping node itself. This is incorrect as such a node is not part of a ring hence it does not hold any data.

This is not happening on "data" nodes (members of the ring) as from data nodes perspective, gossipping-only members are not visible.

We should filter gossipping-only members out of the computation.

EDIT:

For the sake of the completeness, I leave here the original description of this ticket:

The original issue for which we refused to do any action:

Imagine there is a 5-node cluster where two nodes are gossipping-only members (-Dcassandra.join_ring=false) - or in other words, 3 data nodes and 2 "coordinator" nodes.

Coordinator nodes are capable to speak CQL as well so requests can be executed against them. If we create a keyspace against such node, like "create keyspace ks1 with replication =

{class = "NTS", "dc1": 5}

, this query succeeds but if we set CONSISTENCY to ALL in cqlsh and we try to insert some data into a table of such keyspace, it will fail - because it does not have enough replicas. It has only 3.

If this query is executed on data node (a proper member of a ring), this should fail too. I think there is a mechanism how to do this, like by Guardrails but there is no check which would include gossipping-only members into consideration.

Ideally, we might introduce a check which would check that the replication factor is at most as big as the number of members - irrelevant of their current status, they just have to be members of the ring.
Bug: Jolokia agent fails to connect though port is available
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/Sep/22 19:49
Resolved: 24/Oct/22 13:40

In CASSANDRA-17872 we added protection around failures similar to this, caused by the port being in use, which is no longer the case:

{code}
subprocess.CalledProcessError: Command '('/usr/lib/jvm/java-8-openjdk-amd64/bin/java', '{-}cp', '/usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar:/home/cassandra/cassandra/cassandra-dtest/tools/../lib/jolokia-jvm-1.7.1-agent.jar', 'org.jolokia.jvmagent.client.AgentLauncher', '{-}{-}host', '127.0.0.2', '{-}-port', '8778', 'start', '1123')' returned non-zero exit status 1.
{code}

[https://ci-cassandra.apache.org/job/Cassandra-4.1/169/testReport/junit/dtest-novnode.auth_test/TestNetworkAuth/test_revoked_dc_access/]