av9ash/GitBugs huggingface dataset:
Bug: Add a CircleCI job to run some tests repeatedly
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Apr/21 16:31
Resolved: 12/May/21 11:24

I think it could be useful to have an optional CircleCI job to run some specific tests n times. That way, tickets could attach CircleCI runs showing that the changes don't make a certain ticket flaky or, conversely, that they fix a flaky test. Doing this systematically should mitigate the risk of introducing new flaky tests, and I guess it would be more convenient and easy to share than running the tests locally or on a private CI system.

It would also be nice to have something similar in Jenkins, but I'm focusing this ticket on CircleCI because it's available also for non-committers, so assignees can run their tests before setting the tickets as ready for review.
Bug: LICENSE Text addition to Layout.html breaks Cassandra Website rendering
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 09/Jun/21 17:47
Resolved: 25/Jun/21 11:20

The newest docs contain a license agreement comment found in the layout.html of the theme.  This occurs before front matter indicator...
 [https://github.com/apache/cassandra/blob/cassandra-3.11/doc/source/_theme/cassandra_theme/layout.html]

Jekyll assumes that the front matter while be at the top "—".  This causes Jekyll to break when serving the page on the website.

There are a few other docs that seem to have this problem, and serving the latest docs of the website breaks the rendering.
Bug: Add Versions.Major.v4X to Java DTests (after cassandra-4.0 branch was created)
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/May/21 15:50
Resolved: 12/Jul/21 20:58

- https://github.com/apache/cassandra-in-jvm-dtest-api/compare/trunk...thelastpickle:mck/16649
- https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/16649/trunk
Bug: Flaky InsertUpdateIfConditionTest
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 19/May/21 06:10
Resolved: 19/May/21 06:43

Flaky [InsertUpdateIfConditionTest|https://ci-cassandra.apache.org/job/Cassandra-4.0/46/testReport/junit/org.apache.cassandra.cql3.validation.operations/InsertUpdateIfConditionTest/testListItem_2__clusterMinVersion_4_0_0_rc2_SNAPSHOT_/]
Bug: Bring the latest in-tree docs updates to the web site
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/Jun/21 15:16
Resolved: 06/Jun/21 11:04

Below two doc changes have to be published on the web site: 
 * [https://github.com/apache/cassandra/commit/55b4b8dd3bae0004d3128a7f9fe8474526f608c7]
 * CASSANDRA-16682
Bug: Flaky MessageFiltersTest
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Jun/21 06:27
Resolved: 30/Jun/21 07:43

Flaky [MessageFiltersTest|https://ci-cassandra.apache.org/job/Cassandra-4.0.0/39/testReport/junit/org.apache.cassandra.distributed.test/MessageFiltersTest/testFilters/]

{noformat}
Error Message

Operation failed - received 1 responses and 1 failures: TIMEOUT from /127.0.0.2:7012

Stacktrace

org.apache.cassandra.exceptions.ReadFailureException: Operation failed - received 1 responses and 1 failures: TIMEOUT from /127.0.0.2:7012
	at org.apache.cassandra.service.reads.ReadCallback.awaitResults(ReadCallback.java:127)
	at org.apache.cassandra.service.reads.range.SingleRangeResponse.waitForResponse(SingleRangeResponse.java:58)
	at org.apache.cassandra.service.reads.range.SingleRangeResponse.computeNext(SingleRangeResponse.java:65)
	at org.apache.cassandra.service.reads.range.SingleRangeResponse.computeNext(SingleRangeResponse.java:31)
	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
	at org.apache.cassandra.service.StorageProxy$6.hasNext(StorageProxy.java:1883)
	at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:93)
	at org.apache.cassandra.service.reads.range.RangeCommandIterator.computeNext(RangeCommandIterator.java:101)
	at org.apache.cassandra.service.reads.range.RangeCommandIterator.computeNext(RangeCommandIterator.java:54)
	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
	at org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:777)
	at org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:425)
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:296)
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:246)
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:88)
	at org.apache.cassandra.distributed.impl.Coordinator.executeInternal(Coordinator.java:103)
	at org.apache.cassandra.distributed.impl.Coordinator.lambda$executeWithResult$0(Coordinator.java:65)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

{noformat}
Bug: Data Points for the CommitLog's WaitingOnCommit Metric Should Describe Single Mutations
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/May/21 16:54
Resolved: 23/Jun/21 19:30

The metrics we have around the {{CommitLog}} aren’t as useful as they could be in the context of investigating the performance of local writes.

1.) We have no way to know how long the actual flush to disk takes in isolation, i.e. separate from the signaling apparatus between mutation threads and the sync thread. We should add a metric for this.

2.) The WaitingOnCommit metric can have multiple data points recorded for a single mutation, which is a little awkward when we’re trying to break down the latency of a local write (total time for CL add + Memtable put, etc.). More specifically, a thread waits for the sync thread to catch up to the position of its mutation, but it can wake up for a sync operation that hasn’t arrived there yet, which triggers another wait. A new data point is recorded for the metric each time this happens. We should move the scope of metric recording up a level so that there is a 1-1 relationship between it and WriteLatency in TableMetrics (which covers row cache updates and the Memtable put).

{noformat}
void waitForSync(int position, Timer waitingOnCommit)
{
    while (lastSyncedOffset < position)
    {
        WaitQueue.Signal signal = waitingOnCommit != null ?
                                  syncComplete.register(waitingOnCommit.time()) :
                                  syncComplete.register();
        if (lastSyncedOffset < position)
            signal.awaitUninterruptibly();
        else
            signal.cancel();
    }
}
{noformat}
Bug: Flaky NativeAllocatorTest.testBookKeeping
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/May/21 07:47
Resolved: 08/Jun/21 16:46

Flaky [here|https://ci-cassandra.apache.org/job/Cassandra-4.0/52/testReport/junit/org.apache.cassandra.utils.memory/NativeAllocatorTest/testBookKeeping_cdc/]

{noformat}
Error Message

java.lang.AssertionError: expected:<0> but was:<1>

Stacktrace

java.util.concurrent.ExecutionException: java.lang.AssertionError: expected:<0> but was:<1>
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at org.apache.cassandra.utils.memory.NativeAllocatorTest.testBookKeeping(NativeAllocatorTest.java:154)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Caused by: java.lang.AssertionError: expected:<0> but was:<1>
	at org.apache.cassandra.utils.memory.NativeAllocatorTest.lambda$testBookKeeping$2(NativeAllocatorTest.java:131)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
{noformat}
Bug: Reduce the log level on "expected" repair exceptions
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 30/Jun/21 19:23
Resolved: 21/Jul/21 21:27

Many of the repair errors we typically see in the logs are redundant. Say for example that one node has an unreadable SSTable...we should log that fact at ERROR, but then the failing repairs due to that unreadable SSTable should be at WARN, making it easier to find the actual problem.
Bug: Fix org.apache.cassandra.utils.SlidingTimeRateTest.testConcurrentUpdateAndGet
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/Jun/21 21:01
Resolved: 04/Jun/21 22:49

Example failure:

[https://ci-cassandra.apache.org/job/Cassandra-4.0/67/testReport/junit/org.apache.cassandra.utils/SlidingTimeRateTest/testConcurrentUpdateAndGet_cdc/]

History of failing 3-4 times lately like that in Jenkins:

https://jenkins-cm4.apache.org/job/Cassandra-4.0/76/testReport/junit/org.apache.cassandra.utils/SlidingTimeRateTest/testConcurrentUpdateAndGet_cdc/history/
Bug: Fix imports; run tests with packaged dependencies
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 28/May/21 16:03
Resolved: 12/Jun/21 10:05

Tests are currently run with a classpath containing _all_ downloaded jars. The tests would be more reflective of the behavior of a runtime environment if the test classpath only contained jars that are bundled with the binary release, together with explicit test dependencies. Ideally we'd use the build/lib/ jars for the classpath since that's what gets packaged, but since these aren't available at test compile time and should be identical to lib/ anyway, I've used the later.

Doing so exposed a couple of references in src/java to "org.apache.commons.lang", which is not available at runtime (should be "org.apache.commons.lang*3*").

Attached patch modifies the test classpath, fixes various imports in both test/ and src/ classes, and makes some simple substitutions in the tests such as using AbstractMap.SimpleEntry in place of org.apache.commons.collections.keyvalue.AbstractMapEntry.
Bug: Split ViewComplexTest to avoid timeouts after being moved away from 'long' junits
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/Jun/21 04:31
Resolved: 09/Jun/21 06:51

Under CASSANDRA-16670 we decided to move some tickets from the 'long' junit section to the std one. This required splitting them to avoid timing out but ViewComplexTest needs further splitting.
Bug: JmxHistogram#getRecentValues() is not thread-safe
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/Jun/21 18:24
Resolved: 03/Jun/21 18:41

Published {{JmxHistogram}} instances do not protect their “last” field. This isn’t a hot path, and there is a multi-part calculation based on it in {{getRecentValues()}}, so we should probably just synchronize that method.
Bug: Fix flaky test testHasVersion3Nodes - org.apache.cassandra.gms.GossiperTest
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/May/21 00:56
Resolved: 04/May/21 03:52

https://app.circleci.com/pipelines/github/dcapwell/cassandra/893/workflows/3737a7f7-e636-4aac-ac52-a215ec9b008f/jobs/5374

{code}
junit.framework.AssertionFailedError: expected null, but was:<4.0.0-SNAPSHOT>
	at org.apache.cassandra.gms.GossiperTest.testHasVersion3Nodes(GossiperTest.java:148)
{code}
Bug: StandaloneVerifier does not fail when unable to verify SSTables, it only fails if Corruption is thrown
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/May/21 00:45
Resolved: 21/May/21 23:58

offline_tools_test.py::TestOfflineTools::test_sstableverify has the following check

{code}
       try:
           (out, error, rc) = node1.run_sstableverify("keyspace1", "standard1", options=['-v'])
       except ToolError as e:
           # Process sstableverify output to normalize paths in string to Python casing as above
           error = re.sub("(?<=Corrupted: ).*", lambda match: os.path.normcase(match.group(0)), str(e))

           assert re.search("Corrupted: " + sstable1, error)
           assert e.exit_status == 1, str(e.exit_status)
{code}

This checks if the corrupt log is present IFF ToolError is thrown, but does not validate that the error is actually thrown.  I tried calling the same logic before the try to validate and see that it does not fail.  If we fix the test to check for error we also see that the log that is returned to the user does not match 2.2’s behavior but instead returns different logic as digest validation throws IOException, which we do not convert to a CorruptSSTableException (which is the message the test checks for).

This also shows another big issue, that when the digest fails verify passes
Bug: TimeWindowCompactionStrategyTest flaky
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 19/May/21 14:07
Resolved: 01/Jun/21 15:36

Seen in Jenkins:

https://ci-cassandra.apache.org/job/Cassandra-devbranch/785/

Failed two times with the multiplexer 

[https://app.circleci.com/pipelines/github/adelapena/cassandra/461/workflows/7a837b82-c0d1-4e10-8932-c5908d2585de/jobs/4114]
Bug: Implement nodetool getauditlog command
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 08/Jun/21 18:57
Resolved: 19/Aug/21 09:37

There is getfullquerylog already, there is not any reason why getauditlog should not be there too. A user can not retrieve runtime configuration of Audit log, it might be only enabled and disabled via jmx but its state can not be queried in runtime.
Bug: Junit RepeatableRunner flaky tests helper
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 05/May/21 09:06
Resolved: 18/May/21 04:47

Some flakies only fail when the full suite is ran. If it's a quick test you can do thousands of runs if a few seconds. Looping at the sh level is not an option as every loop takes a few seconds.

As I have found this very useful I am proposing committing it
Bug: Flaky TestTransientReplicationRing.test_move_forwards_and_cleanup
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/May/21 05:22
Resolved: 13/May/21 05:24

Flaky TestTransientReplicationRing.[test_move_forwards_and_cleanup|https://ci-cassandra.apache.org/job/Cassandra-4.0/38/testReport/junit/dtest-novnode.transient_replication_ring_test/TestTransientReplicationRing/test_move_forwards_and_cleanup/]
Bug: Parallelise docker container runs for tests in ci-cassandra.a.o
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Apr/21 06:41
Resolved: 06/May/21 10:00

This was raised on the dev ML, where the consensus was to remove it: https://lists.apache.org/thread.html/r1ca3c72b90fa6c57c1cb7dcd02a44221dcca991fe7392abd8c29fe95%40%3Cdev.cassandra.apache.org%3E

The idea is to then replace ant test parallelism with docker container parallelism.

PoC patch: 
https://github.com/apache/cassandra-builds/compare/trunk...thelastpickle:mck/16587-2/trunk

This is just a quick PoC, aimed at the ci-cassandra agents that have
4 cores and 16gb ram available to each executor, but I imagine instead
something that spawns a number of containers based on system
resources, like we currently do with get-cores and get-mem. 

Also worth noting the overhead here, compared with the ant parallelism approach, docker
builds everything in each container from scratch, but this too can be
improved easily enough.

Cleaning up any remnant `-Dtest.runners=` options is also part of this ticket.
Bug: Cassandra 4.0 RC1 ... Audit Logs going to wrong location
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/May/21 10:29
Resolved: 21/May/21 19:01

Version : Cassandra 4.0 RC1

Settings in my : <CASSANDRA_HOME>/conf/cassandra.yaml 
{code:yaml}
audit_logging_options: 
    enabled: true
     logger: 
      - class_name: FileAuditLogger
     excluded_categories: QUERY, DML
     audit_logs_dir: "/apps/opt/cassandra/logs/audit"
     roll_cycle: DAILY
     block: false
{code}
 After restart of Cassandra ... I can see the Audit Logs. But they are going into system.log and debug.log. They are NOT going into 
{noformat}
"/apps/opt/cassandra/logs/audit/audit.log"{noformat}
 

Also noticed that : it does not ignore the "excluded_categories" and logs everything ...
Bug: Update how_to_commit documentation after cassandra-4.0 branch
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/May/21 15:42
Resolved: 04/May/21 16:24


- 3.11: https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/16642/3.11 (also backports CASSANDRA-16626)
- cassandra-4.0: https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/16642/4.0
- trunk: https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/16642/trunk
Bug: Support upgrade tests in CircleCI multiplexer
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/May/21 11:17
Resolved: 27/May/21 15:44

The CircleCI jobs for running a test repeatedly added by CASSANDRA-16625 don't support upgrade tests. In the case of Python upgrade dtests we only need the {{--execute-upgrade-tests}} when calling {{pytest}}. For Java upgrade tests we need to build the dtest jars before running the repeated tests.
Bug: Password obfuscation for DCL audit log statements
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/May/21 01:44
Resolved: 15/Jun/21 12:46

The goal of this JIRA is to obfuscate passwords or any sensitive information from DCL audit log statements.

Currently, (Cassandra version 4.0-rc1) logs query statements for any DCL ([ROLE|https://cassandra.apache.org/doc/latest/cql/security.html#database-roles] and [USER|https://cassandra.apache.org/doc/latest/cql/security.html#users] ) queries with passwords in plaintext format in audit log files.

The current workaround to avoid plain text passwords from being logged in audit log files is either by [excluding|https://cassandra.apache.org/doc/latest/operating/audit_logging.html#options] DCL statements from auditing or by excluding the user who is creating these roles from auditing.

It would be ideal for Cassandra to provide an option or default to obfuscate passwords or any sensitive information from DCL audit log statements.

Sample audit logs with DCL queries
{code:sh}
Type: audit
LogMessage: user:cassandra|host:localhost/127.0.0.1:7000|source:/127.0.0.1|port:51908|timestamp:1620190499676|type:CREATE_ROLE|category:DCL|operation:CREATE ROLE new_role;
Type: audit
LogMessage: user:cassandra|host:localhost/127.0.0.1:7000|source:/127.0.0.1|port:51908|timestamp:1620190505313|type:CREATE_ROLE|category:DCL|operation:CREATE ROLE alice WITH PASSWORD = 'password_a' AND LOGIN = true;
Type: audit
LogMessage: user:cassandra|host:localhost/127.0.0.1:7000|source:/127.0.0.1|port:51908|timestamp:1620190519521|type:REQUEST_FAILURE|category:ERROR|operation:ALTER ROLE bob WITH PASSWORD = 'PASSWORD_B' AND SUPERUSER = false;; bob doesn't exist
Type: audit
LogMessage: user:cassandra|host:localhost/127.0.0.1:7000|source:/127.0.0.1|port:51908|timestamp:1620190525376|type:CREATE_ROLE|category:DCL|operation:CREATE ROLE bob WITH PASSWORD = 'password_b' AND LOGIN = true AND SUPERUSER = true;
Type: audit
LogMessage: user:cassandra|host:localhost/127.0.0.1:7000|source:/127.0.0.1|port:51908|timestamp:1620190532462|type:ALTER_ROLE|category:DCL|operation:ALTER ROLE bob WITH PASSWORD = 'PASSWORD_B' AND SUPERUSER = false;
{code}

It is also ideal to document this workaround or assumption in Cassandra audit log documentation until we close this JIRA
Bug: Python driver dependency missing
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/May/21 15:11
Resolved: 26/May/21 19:00

Cqlsh depends on the python driver, which is not included in the downloaded dependencies.
{noformat}
$ bin/cqlsh

Python Cassandra driver not installed, or not on PYTHONPATH.
You might try "pip install cassandra-driver".

Python: /usr/local/bin/python3
Module load path: ['/usr/local/share/java/cassandra/bin/../lib/geomet-0.1.0.zip', '/usr/local/share/java/cassandra/bin/../lib/six-1.12.0-py2.py3-none-any.zip', '/usr/local/share/java/cassandra/bin/../lib/futures-2.1.6-py2.py3-none-any.zip', '/usr/local/share/java/cassandra/bin', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload', '/usr/local/lib/python3.7/site-packages']

Error: No module named 'cassandra'

{noformat}
Bug: Update Cassandra build CI script for new website
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/Jun/21 06:03
Resolved: 27/Apr/22 07:46

Update the apache/cassandra-build repository has [.groovy job script|https://github.com/apache/cassandra-builds/blob/trunk/jenkins-dsl/cassandra_job_dsl_seed.groovy] to use the new tooling to build the website.

The apache/cassandra-build repository has [.groovy job script|https://github.com/apache/cassandra-builds/blob/trunk/jenkins-dsl/cassandra_job_dsl_seed.groovy]. It [builds and deploys the website|https://github.com/apache/cassandra-builds/blob/trunk/jenkins-dsl/cassandra_job_dsl_seed.groovy#L1255-L1268] to cassandra-website staging when the cassandra-website is updated.

We need to change the commands in the .groovy script to at least rebuild the site when a change happens in the cassandra-website repository.

It would be also good to build the docs when a change happens to doc/source directory in the cassandra repository.
Bug: Loss of commit log data possible after sstable ingest
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/Apr/21 10:17
Resolved: 17/May/21 08:47

SSTable metadata contains commit log positions of the sstable. These positions are used to filter out mutations from the commit log on restart and only make sense for the node on which the data was flushed.

If an SSTable is moved between nodes they may cover regions that the receiving node has not yet flushed, and result in valid data being lost should these sections of the commit log need to be replayed.

Solution:
The chosen solution introduces a new sstable metadata (StatsMetadata) - originatingHostId (UUID), which is the local host id of the node on which the sstable was created, or null if not known. Commit log intervals from an sstable are taken into account during Commit Log replay only when the originatingHostId of the sstable matches the local node's hostId.

For new sstables the originatingHostId is set according to StorageService's local hostId.
For compacted sstables the originatingHostId set according to StorageService's local hostId, and only commit log intervals from local sstables is preserved in the resulting sstable.

discovered by [~jakubzytka]
Bug: DOC - Latency in Client Request Metrics tables incorrectly labelled
Status: In Progress
Priority: Normal
Created: 14/Apr/21 02:13

[~sebastian.estevez@datastax.com] reported that the {{Latency}} metric in the the *Client Request Metrics* tables are incorrectly labelled.

For example, the metric in the {{CASRead}} table should have {{Latency}} as the Name and {{Histogram}} as the type:

 !client_request_metrics-CASRead.png|width=300!