av9ash/GitBugs huggingface dataset:
Bug: Add environment variables to CircleCI config generation script
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 23/Sep/21 17:58
Resolved: 28/Sep/21 16:57

The purpose of this ticket is adding arguments to the CircleCI config generation script allowing to set the values specific environment variables such as {{DTEST_REPO}} or {{DTEST_BRANCH}} in the generated {{.circleci/config.yml}} file. For example, we could generate a CircleCI config file with MIDRES specifying a dtest repo and branch by running:
{code}
generate.sh -m \
  -e DTEST_REPO=git://github.com/adelapena/cassandra-dtest.git \
  -e DTEST_BRANCH=CASSANDRA-8272 
{code}
Or we could set the test multiplexer for repeating a specific test with HIGHRES:
{code}
generate.sh -h \
  -e REPEATED_UTEST_TARGET=testsome \
  -e REPEATED_UTEST_CLASS=org.apache.cassandra.cql3.ViewTest \
  -e REPEATED_UTEST_METHODS=testCompoundPartitionKey,testStaticTable \
  -e REPEATED_UTEST_COUNT=100
{code}
This can be useful on its own so we don't have to manually edit the {{config-2_1.yml}}/{{config.yml}}, and it can also be useful for automation scripts manipulating these files.
Bug: repair prepare message would produce a wrong error message if network timeout happened rather than reply wait timeout
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Sep/21 14:42
Resolved: 19/Nov/21 22:33

Showed up on CASSANDRA-12988 test runs; can't reproduce locally.

 {quote}
junit.framework.AssertionFailedError: nodetool command [repair, distributed_test_keyspace, preparerpctimeout_full_parallel_false, --full] Error message 'Got negative replies from endpoints [/127.0.0.2:7012]' does not contain any of [Did not get replies from all endpoints.]
stdout:
[2021-09-22 17:37:02,187] After waiting for poll interval of 1 seconds queried for parent session status and discovered repair failed.
[2021-09-22 17:37:02,187] Got negative replies from endpoints [/127.0.0.2:7012]
[2021-09-22 17:37:02,187] Repair command #4 finished with error

stderr:
error: Got negative replies from endpoints [/127.0.0.2:7012]
-- StackTrace --
java.io.IOException: Got negative replies from endpoints [/127.0.0.2:7012]
	at org.apache.cassandra.tools.RepairRunner.queryForCompletedRepair(RepairRunner.java:167)
	at org.apache.cassandra.tools.RepairRunner.run(RepairRunner.java:72)
	at org.apache.cassandra.tools.NodeProbe.repairAsync(NodeProbe.java:461)
	at org.apache.cassandra.tools.nodetool.Repair.execute(Repair.java:171)
	at org.apache.cassandra.tools.NodeTool$NodeToolCmd.runInternal(NodeTool.java:363)
	at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:348)
	at org.apache.cassandra.tools.NodeTool.execute(NodeTool.java:251)
	at org.apache.cassandra.distributed.impl.Instance$DTestNodeTool.execute(Instance.java:895)
	at org.apache.cassandra.distributed.impl.Instance.lambda$nodetoolResult$39(Instance.java:805)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)
{quote}
Bug: Add soft/hard limits to local reads to protect against reading too much data in a single query
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Aug/21 23:42
Resolved: 28/Sep/21 17:53

Add soft/hard limits to local reads to protect against reading too much data in a single query.

This is an extension of the existing work to add warnings/aborts to large partitions (CASSANDRA-16850), with the core difference is that this applies locally rather than at the coordinator.
Bug: Missed wait latencies in the output of `nodetool tpstats -F`
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 09/Sep/21 14:53
Resolved: 10/Sep/21 12:34

The output of {{nodetool tpstats -F json}} always prints an empty map for {{WaitLatencies}}:
{code:java}
$ nodetool tpstats -F json
(...)"WaitLatencies":{},(...)
{code}
The same happens with yaml output:
{code:java}
$ nodetool tpstats -F yaml
(...)
WaitLatencies: {}
(...)
{code}
This happens because [this cast|https://github.com/apache/cassandra/blob/cassandra-4.0.1/src/java/org/apache/cassandra/tools/nodetool/stats/TpStatsHolder.java#L61-L63] silently fails inside a try-catch with an empty catch block:
{code:java}
String[] strValues = (String[]) Arrays.stream(probe.metricPercentilesAsArray(probe.getMessagingQueueWaitMetrics(entry.getKey())))
                                      .map(D -> D.toString())
                                      .toArray();
{code}
When we would need something like:
{code:java}
String[] strValues = Arrays.stream(probe.metricPercentilesAsArray(probe.getMessagingQueueWaitMetrics(entry.getKey())))
                           .map(Object::toString)
                           .toArray(String[]::new);
{code}
This conversion from {{Double[]}} to {{String[]}} was introduced during CASSANDRA-16230. My guess is that the conversion was done trying to work around a malformed JSON output detected in [the new tests|https://github.com/apache/cassandra/blob/cassandra-4.0.1/test/unit/org/apache/cassandra/tools/NodeToolTPStatsTest.java#L158] added by that ticket. Without the conversion the output would be something like:
{code:java}
$ nodetool tpstats -F json
(...)"WaitLatencies":{"READ_RSP":[Ljava.lang.Double;@398dada8,(...)
{code}
That's because {{json-simple}} doesn't handle well arrays. I think that instead of converting the array of doubles to an array of strings we can simply use {{jackson-mapper-asl}} to print the proper array.
Bug: Exclude Jackson 1.x dependency that leaks via old hadoop-core dependency
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/Aug/21 20:14
Resolved: 30/Aug/21 19:43

build.xml has a dependency for an old hadoop-core version (1.0.3). This is likely needed for some Hadoop compatibility code under `src/java/org/apache/cassandra/hadoop`. Since 1.0.3 was released in 2012, its dependencies are very old; in particular it depends on "jackson-mapper-asl" 1.0.1 (from 2009!).
An earlier issue CASSANDRA-15867 referenced this dependency as well (but did not actually remove it for some reason, which marked as resolved).

Although `hadoop-core` dependency is marked as "provided" (and should then not be included in distributed version) it seems best to avoid downloading it during build to "build/lib/jars". This can be done by adding 2 exclusions for "hadoop-core" and "hadoop-minicluster" dependencies in build.xml.
I will provide a patch.

I also tried updating "hadoop-core" to the latest public version (1.2.1), which would have upgraded jackson-mapper-asl to 1.8.8, but that breaks the build due to some other version incompatibility (asm, possibly). If anyone wants to tackle that issue it could be a good follow-up task; "hadoop-core" itself has been moved to "hadoop-client" it seems (and there's "hadoop-commons" too... confusing).
Bug: Gossipinfo should be available through a Virtual Table
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Sep/21 17:13
Resolved: 16/Mar/22 00:43

The {{nodetool}} utility provides a utility to print out the {{gossipinfo}} for the cluster. The output is then printed out as text to stdout and consumed by operators and tools built around {{nodetool gossipinfo}}.

With the introduction of virtual tables in Cassandra, the {{gossipinfo}} virtual table can help operators and the tools built around {{gossipinfo}} to consume the gossip information more easily by providing a tabular output of the data, which can be easily queried by different clients.
Bug: SSTableReaderTest#testGetPositionsKeyCacheStats failing sporadically on unexpected key cache hit for non-cached key
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/Aug/21 21:08
Resolved: 09/Sep/21 19:14

This has started to pop up on 4.0.x and trunk builds in both CircleCI and Apache CI:

[https://app.circleci.com/pipelines/github/josh-mckenzie/cassandra/60/workflows/15e09ea4-4d35-4035-9afc-ff0d1089041e/jobs/382]

[https://app.circleci.com/pipelines/github/maedhroz/cassandra/332/workflows/3bd588d9-717f-4877-8e78-7a127180dfee/jobs/2093/tests#failed-test-0]


https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/1067/testReport/junit/org.apache.cassandra.io.sstable/SSTableReaderTest/testGetPositionsKeyCacheStats_cdc/
 
It passes consistently in local runs, although I have seen one failure in about 10 runs.
Bug: Harden PrunableArrayQueue in the face of failures
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Aug/21 20:35
Resolved: 27/Aug/21 19:13

We've seen AssertionErrors in internode messaging deserialization lead to a property violation w/PrunableArrayQueue having null objects remaining present in the queue. We should harden the PAQ against cases like this and improve test coverage for exception handling as applicable.
Bug: Deploy multi-architecture (amd+arm) docker testing images
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 09/Aug/21 12:01
Resolved: 09/Aug/21 20:17

Docker's [buildx|https://github.com/docker/buildx/] supports building multi-architecture images.

This can be used to deploy our testing images: [apache/cassandra-testing-ubuntu2004-java11|https://hub.docker.com/layers/apache/cassandra-testing-ubuntu2004-java11] and [apache/cassandra-testing-ubuntu2004-java11-w-dependencies|https://hub.docker.com/layers/apache/cassandra-testing-ubuntu2004-java11-w-dependencies]; for both {{linux/amd64}} and {{linux/arm64}} architectures. 

(Up until now ci-cassandra.a.o is having to manually local build these images during each test run.)
Bug: offline_tools_test.py::TestOfflineTools::test_sstableverify fails on 3.11
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/Sep/21 13:27
Resolved: 27/Sep/21 21:52

As noted by [~stefan.miklosovic]:

===Flaky Test Report===

test_sstableverify failed and was not selected for rerun.
        <class 'AssertionError'>
        assert None
 +  where None = <function search at 0x7fef074b5550>(('WARNING: Corrupted SSTable : ' + '/tmp/dtest-z6njep37/test/node1/data2/keyspace1/standard1-9b45a1f0149411ecbc69f72e6826361e/me-12-big-Data.db'), "Subprocess sstableverify on keyspace1 : standard1 with options: ['-v'] exited with non-zero status; exit status: 1; \...p/dtest-z6njep37/test/node1/cdc_raw; setting cdc_total_space_in_mb to 3831.  You can override this in cassandra.yaml\n")
 +    where <function search at 0x7fef074b5550> = re.search
        [<TracebackEntry /home/ubuntu/cassandra-dtest/offline_tools_test.py:303>]
Bug: Save CircleCI resources with optional test jobs
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 25/Aug/21 12:21
Resolved: 14/Oct/21 11:42

This ticket implements the addition of approval steps in the CircleCI workflows as it was proposed in [this email|https://lists.apache.org/thread.html/r57bab800d037c087af01b3779fd266d83b538cdd29c120f74a5dbe63%40%3Cdev.cassandra.apache.org%3E] sent to the dev list:

The current CircleCI configuration automatically runs the unit tests, JVM dtests and cqhshlib tests. This is done by default for every commit or, with some configuration, for every push.

Along the lifecycle of a ticket it is quite frequent to have multiple commits and pushes, all running these test jobs. I'd say that frequently it is not necessary to run the tests for some of those intermediate commits and pushes. For example, one can show proofs of concept, or have multiple rounds of review before actually running the tests. Running the tests for every change can produce an unnecessary expense of CircleCI resources.

I think we could make running those tests optional, as well as clearly specifying in the documentation what are the tests runs that are mandatory before actually committing. We could do this in different ways:
 # Make the entire CircleCI workflow optional, so the build job requires
 manual approval. Once the build is approved the mandatory test jobs would
 be run without any further approval, exactly as it's currently done.
 # Make all the test jobs optional, so every test job requires manual approval, and the documentation specifies which tests are mandatory in the final steps of a ticket.
 # Make all the mandatory test jobs depend on a single optional job, so we have a single button to optionally run all the mandatory tests.

I think any of these changes, or a combination of them, would significantly
 reduce the usage of resources without making things less tested. The only
 downside I can think of is that we would need some additional clicks on the
 CircleCI GUI.
Bug: Tolerate missing DNS entry when completing a host replacement
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/Aug/21 16:57
Resolved: 27/Aug/21 19:55

In one of our deployments, after a host replacement a subset of nodes still saw the nodes as JOINING despite the rest of the cluster seeing it as NORMAL with a failure to gossip. This was traced to a DNS lookup failure on the nodes during an interim state leading to an exception being thrown and gossip state never transitioning.

Rather than implicitly requiring operators to bounce the node by throwing an exception, we should instead suppress the exception when checking if a node is replacing the same host address and ID if we get an UnknownHostException.
Bug: CQLSH shell script will fail on Python 3.10
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 23/Sep/21 10:46
Resolved: 27/Sep/21 19:20

The `is_supported_version()` function performs a lexicographical comparison between the Python version number in the format of `MAJOR.MINOR` with `3.6`. This works as expected if the Python version number is between `3.0` and `3.9` (inclusive), but will produce an unexpected result if the Python version number is `3.10` or above.

Recommended fix is to split the version number into two integers, `$version_major` and `$version_minor` and then compare them to `3` and `6` respectively.
Bug: CircleCI folder in dtest is confusing
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/Aug/21 12:10
Resolved: 07/Sep/21 09:19

I made a patch to dtest without affecting Cassandra. cassandra-dtest repo contains folder with CircleCI configuration file. So I tried to run a [build from CircleCI|https://app.circleci.com/pipelines/github/k-rus/cassandra-dtest/3/workflows/6ea8058d-37b0-40e8-89be-d9b4afdf279c/jobs/4] and it failed with an error related to configuration:
{code:java}
ERROR: flake8 3.9.2 has requirement pycodestyle<2.8.0,>=2.7.0, but you'll have pycodestyle 2.3.1 which is incompatible.
{code}
The CircleCI config contains
{code:java}
sudo pip install pycodestyle==2.3.1 flake8
{code}
, so it should lead to the error.

From the discussion in community I learned that dtest are run as part of Cassandra build and not on cassandra-dtest directly. Is the config in dtest maintained? If not, it might be good to remove. Or may be add some comment about it.
Bug: Fix flaky test DatabaseDescriptorRefTest
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 17/Aug/21 22:07
Resolved: 20/Sep/21 21:18

While working on another ticket I found out that DatabaseDescriptorRefTest is failing consistently locally for me and one more community member on 3.11, 4.0 and trunk.


{code:java}
 java.lang.AssertionError: thread started in clientInitialization 
Expected :5
Actual   :8
<Click to see difference>


	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:645)
	at org.apache.cassandra.config.DatabaseDescriptorRefTest.testDatabaseDescriptorRef(DatabaseDescriptorRefTest.java:285)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:221)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)
{code}
Bug: Fix usage instructions about sstabledump -k and -x options
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 28/Jul/21 16:10
Resolved: 30/Jul/21 17:18

The options {{-k}} and {{-x}} of {{sstabledump}} admit multiple arguments, so users can include or exclude multiple partitions. The intended usage is, for example:
{code:java}
$ sstablepartitions <sstable_path> -k 1
$ sstablepartitions <sstable_path> -k 1 2 3
{code}
However, the following command will fail:
{code:java}
$ sstablepartitions -k 1 <sstable_path>
You must supply exactly one sstable
usage: sstabledump <sstable file path> <options>

Dump contents of given SSTable to standard output in JSON format.
 -d         CQL row per line internal representation
 -e         enumerate partition keys only
 -k <arg>   Partition key
 -t         Print raw timestamps instead of iso8601 date strings
 -x <arg>   Excluded partition key
{code}
This command fails because the sstable path is considered a fourth partition key, and the mandatory argument for the sstable path is missed. While I think this behaviour is correct, it can be a bit confusing for users, especially when the information about usage describes both {{-k}} and {{-x}} as single-argument.

I think that at least we should fix the description of the options to indicate that there could be multiple included/excluded keys, and probably improve the message about the missing sstable path when those options are used.

Alternatively we could modify the options to have a single argument and allow to repeat them, so we could accept things like:
{code:java}
$ sstablepartitions -k 1 <sstable_path>
$ sstablepartitions -k 1 -k 2 -k 3 <sstable_path>
$ sstablepartitions <sstable_path> -k 1 -k 2 -k 3
{code}
The main downside of the latter approach is that the change in the syntax of the command might cause compatibility issues. Also we would need to upgrade {{commons-cli}} to at least 1.2 due to CLI-137.
Bug: Rename cassandra-topology.properties to cassandra-topology.properties.example
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/Sep/21 19:41
Resolved: 16/Nov/21 16:11

Far too often people leave this file in place when using GPFS, when it is only intended to be using during migration away from PFS.  If we simply rename this file, then the GPFS camp won't trip over it (without going to effort to do so at least) and the PFS camp can simple rename the file before/after editing it.
Bug: nodetool setstreamthroughput accepts invalid arguments that are not immediately applied
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Sep/21 14:27
Resolved: 01/Oct/21 11:03

Both {{nodetool setstreamthroughput}} and {{nodetool setinterdcstreamthroughput}} accept a negative throughput. The throughput value is not immediately applied to the corresponding rate limiters. Instead, the value is [set in the {{Config}}|https://github.com/apache/cassandra/blob/09c89e5f5f8604301c233130dfb6e82a36ae30f3/src/java/org/apache/cassandra/service/StorageService.java#L1488] and it's only applied to the singleton rate limiter when new sstable stream writer are created (see [here|https://github.com/apache/cassandra/blob/09c89e5f5f8604301c233130dfb6e82a36ae30f3/src/java/org/apache/cassandra/streaming/StreamManager.java#L66-L76]). This could happen much later than the definition of the new throughput, and by then the setting of the new rate in the rate limiter will fail with an {{IllegalArgumentException}} due to the negative value.

I think we should either immediately reject negative throughputs or consider them unlimited, as we do with zero. Also we should probably apply the new throughput to the rate limiter immediately, since I don't see why we should wait to start using the new throughput.
Bug: Fix ERROR message which prints data information in the logs
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 28/Jul/21 12:20
Resolved: 29/Jul/21 11:18

{{StorageProxy.mutateMV}} might log [an error message|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageProxy.java#L880] that prints user data in the logs beyond the row key, for example:
{code}
ERROR [MutationStage-2] 2021-07-28 13:08:52,609 StorageProxy.java:1002 - Error applying local view update to keyspace k: Mutation(keyspace='k', key='00000001', modifications=[
  [k.mv] key=1 partition_deletion=deletedAt=-9223372036854775808, localDeletion=2147483647 columns=[[] | []]
    Row[info=[ts=1627474132606719] ]: k=0, v=MY CONFIDENTIAL DATA |
])
{code}
We should probably change that log message so it doesn't print the entire mutation but only the keyspace, tables and partition key of the mutation.
Bug: Latest docker testing image (20210809) does not default support TLSv1 and TLSv1.1
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 12/Aug/21 11:51
Resolved: 13/Aug/21 21:46

Since deployment of the latest docker testing image (20210809), ci-cassandra.a.o is failing a few [tests|https://ci-cassandra.apache.org/job/Cassandra-trunk-jvm-dtest/833], e.g.:

* SSTableLoaderEncryptionOptionsTest.bulkLoaderCannotAgreeOnClientTLSProtocol
* NativeTransportEncryptionOptionsTest.negotiatedProtocolMustBeAcceptedProtocolTest
* InternodeEncryptionOptionsTest.negotiatedProtocolMustBeAcceptedProtocolTest


Looking at the [logs|https://ci-cassandra.apache.org/job/Cassandra-trunk-jvm-dtest/lastSuccessfulBuild/jdk=jdk_1.8_latest,label=cassandra,split=8/testReport/junit/org.apache.cassandra.distributed.test/SSTableLoaderEncryptionOptionsTest/bulkLoaderCannotAgreeOnClientTLSProtocol/], the in-jvm node is starting up with
{noformat}
JdkSslContext.java:97 - Default protocols (JDK): [TLSv1.2] 
{noformat}

Running the test locally works, and the matching line is
{noformat}
JdkSslContext.java:97 - Default protocols (JDK): [TLSv1.2, TLSv1.1, TLSv1]
{noformat}

The docker image is using {{openjdk version "1.8.0_282"}}.

Upgrading and using {{1.8.0_302}} locally reproduces the failure.