av9ash/GitBugs huggingface dataset:
Bug: Gossip STATUS can be either missing during upgrade or stale after upgrade
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 16/Mar/21 22:56
Resolved: 19/Mar/21 15:57

In 4.0, new application states are added in Gossip and the corresponding old ones are deprecated, e.g. {{STATUS}} and the successor {{STATUS_WITH_PORT}}.

There are 2 issues discovered by the jvm (upgrade) dtest. First, the {{STATUS}} field of a peer in the lower version (e.g. 3.0) node can be missing. Second, it is possible the {{STATUS}} coexist with the new state {{STATUS_WITH_PORT}} in the 4.0 nodes after cluster is fully upgraded and the {{STATUS}} field can becomes stale as the 4.0 node filters out when applying new state.

The first issue can happen in this scenario. During upgrade, node1 and node2 are in v4, and node3 is still in v3. If node3 only gets the gossip info regarding node2 from node1, the {{STATUS}} field of node2 will be missing in node3's local state, which is unexpected. There could be many reasons that node3 does not exchange gossip with node2 directly, e.g. network issue between node2 and node3, or node2 simply does not select node3 when initiating the gossip round. Gossip should be resilient to it. I have a [jvm upgrade dtest|https://github.com/yifan-c/cassandra/blob/CASSANDRA-16525/trunk/test/distributed/org/apache/cassandra/distributed/upgrade/MixedModeGossipTest.java#L81] to demonstrate the unexpected behavior.

The cause of the second issue is more subtle. Heartbeat update happens as part of the Gossip task and outside of the GossipStage. When node2 just update its local application state and received a {{SYN}} from node1, node 2 just replies its gossip state without updating the heartbeat version. When node1 receives it, it first filters out the legacy {{STATUS}} field, and only saves the new one. So far so good. However, node2 soon updates its heart beat, and node1 realizes that its local version is less than the remove (node2) version in the next gossip round. So node2 sends {{STATUS}} along to node1. Because it does not come together with the new field, node1 does not filter it out when receiving. Boo! Node1 now has the {{STATUS}} field from node2. Such field can become stale and diverge with its successor in a live cluster. The jvm upgrade test [testStatusFieldShouldExistInOldVersionNodes|https://github.com/yifan-c/cassandra/blob/CASSANDRA-16525/trunk/test/distributed/org/apache/cassandra/distributed/upgrade/MixedModeGossipTest.java#L47] can fairly easy to reproduce it when the entire cluster is upgraded. And there is another jvm dtest (with source changes to help make deterministic result, see the attached Demonstrate-a-scenario-that-a-node-may-hold-the-stale-status.patch) that demonstrates the {{STATUS}} can be replicated to the peer and become stale.

The fix is to 
 1) retain the legacy fields if the cluster is still in mixed mode
 2) remove the legacy field when cluster is fully upgraded
Bug: Missing validation in AbstractType.writeValue():
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 08/Mar/21 11:01
Resolved: 25/Mar/21 19:59

Some validation checks present in {{AbstractType.writeValue()}} in {{cassandra-3.11}} are not there in {{trunk}}.
In 3.11 the checks used assertion. It would make sense to use {{IOExceptions}} instead as used in   {{AbstractType.readValue()}}.
Bug: CQL schema generated by DESCRIBE KEYSPACE fails to import with custom indexes
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Mar/21 18:17
Resolved: 04/Mar/21 16:59

The CQL schema generated by a DESCRIBE KEYSPACE on a keyspace with a table that contains a custom index cannot be reimported.

The reimport of the schema fails with:
{noformat}
InvalidRequest: Error from server: code=2200 [Invalid query] message="Cannot specify class_name as a CUSTOM option"
{noformat}
The reason for this is that the custom index definition is created with {{class_name}} and {{target}} options added to it in it's {{WITH OPTIONS}} section. This is then picked up by the {{IndexAttributes.validate}} method that specifically rejects them.

Note: This only seems to happen if the custom index was created with options in the first place. So if the {{CREATE CUSTOM INDEX}} statement didn't originally have a {{WITH OPTIONS}} section then it won't have the {{class_name}} and {{target}} options added to it.

h3. Steps to reproduce
{noformat}
CREATE KEYSPACE test WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'};
CREATE TABLE test.test (id int PRIMARY KEY, value text);
CREATE CUSTOM INDEX ON test.test(value) USING 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {'is_literal': 'false'};
DESCRIBE KEYSPACE test;
{noformat}
Trying to import the resultant schema will fail.
Bug: Hint messages are incorrectly re-serialized for filtering in in-jvm dtests
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Feb/21 10:18
Resolved: 25/Feb/21 13:12

Hint messages for dropped tables can still be dispatched, but they’re ignored on the receiving side all usual code paths. Since we’re attempting to re-serialize hint message for dropped table in in-jvm tests, we exercise path that is impossible in regular code, and for which there is no protocol specification. 


Stack trace: 

{code}
INFO  [AsyncAppender-Worker-ASYNC] 2021-02-17 18:50:13,759 SubstituteLogger.java:169 - ERROR [MutationStage-2] 2021-02-17 18:50:13,726 AbstractLocalAwareExecutorService.java:166 - Uncaught exception on thread Thread[MutationStage-2,5,node4]
java.lang.NullPointerException: null
	at org.apache.cassandra.hints.Hint$Serializer.serializedSize(Hint.java:150)
	at org.apache.cassandra.hints.HintMessage$Serializer.serializedSize(HintMessage.java:86)
	at org.apache.cassandra.hints.HintMessage$Serializer.serializedSize(HintMessage.java:77)
	at org.apache.cassandra.net.Message$Serializer.payloadSize(Message.java:1289)
	at org.apache.cassandra.net.Message$Serializer.access$1200(Message.java:607)
	at org.apache.cassandra.net.Message.payloadSize(Message.java:1341)
	at org.apache.cassandra.net.Message.access$900(Message.java:66)
	at org.apache.cassandra.net.Message$Serializer.serializePost40(Message.java:759)
	at org.apache.cassandra.net.Message$Serializer.serialize(Message.java:618)
	at org.apache.cassandra.distributed.impl.Instance.serializeMessage(Instance.java:322)
	at org.apache.cassandra.distributed.impl.Instance.lambda$registerInboundFilter$4(Instance.java:273)
	at org.apache.cassandra.net.InboundSink$Filtered.accept(InboundSink.java:62)
	at org.apache.cassandra.net.InboundSink$Filtered.accept(InboundSink.java:49)
	at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:93)
	at org.apache.cassandra.distributed.impl.Instance.lambda$null$6(Instance.java:365)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:119)
	at relocated.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)
{code}
Bug: Fix incorrect encoding for strings can be UTF8
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 06/Feb/21 13:00
Resolved: 23/Feb/21 03:00

Tables created with Japanese character name columns are working well in C* 3.11.10 when doing a SELECT * in cqlsh but will show as garbled (shown as "?") in 4.0-beta4. DESCRIBE shows the column names correctly in both cases.

Run the attached jptest.cql script in both envs with cqlsh -f. They will yield different results.

My test env (MacOS 10.15.7):

C* 3.11.10 with
 - OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_252-b09)
 - Python 2.7.16

C* 4.0-beta4
 - OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.9.1+1)
 - Python 3.8.2
Bug: Document the side-effect of Zero Copy Streaming when in a multiple data directories setup
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Feb/21 21:41
Resolved: 26/Feb/21 17:45

During the compaction test task (CASSANDRA-16345), we discovered that when zero copy streaming (ZCS) is enabled, it can lead to the streamed files contain tokens that exceed the disk boundaries (only when using multiple data directories) on the target host. Since ZCS streams the entire SSTable as is. 
We should add a proper note to the [https://cassandra.apache.org/doc/latest/new/streaming.html#zero-copy-streaming] section to inform people when using the feature.
Bug: MigrationCoordinatorTest#testWeKeepSendingRequests() Failing Sporadically
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Feb/21 20:10
Resolved: 16/Feb/21 17:36

This test has now failed at least on trunk and 3.11. Here are the data points we have:

*trunk*

https://ci-cassandra.apache.org/job/Cassandra-trunk/273/testReport/org.apache.cassandra.schema/MigrationCoordinatorTest/testWeKeepSendingRequests/

https://app.circleci.com/pipelines/github/adelapena/cassandra/181/workflows/84fa978e-6df6-455a-b9d0-8e1024e2657f/jobs/1390


*3.11*

https://app.circleci.com/pipelines/github/dcapwell/cassandra/801/workflows/34d9dd52-444a-465d-88ff-074db4a03132/jobs/4597

The only previous mention of this issue was [here|https://issues.apache.org/jira/browse/CASSANDRA-15158?focusedCommentId=17165619&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17165619]. In any case, that issue seems not to have resolved the problem.
Bug: UpgradeTest sporadically failing on schema updates
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Jan/21 22:57
Resolved: 15/Feb/21 13:48

We’ve observed {{UpdateTest}} failing during what appears to be a schema change:

https://app.circleci.com/pipelines/github/maedhroz/cassandra/192/workflows/ed5305e6-e4f9-420e-9f0a-6153333746dc/jobs/1068

It almost looks like the Gossiper can’t find its own endpoint state in the endpoint state map, and the failure is not consistent, which might suggest a race.
Bug: Add access and datacenters to unreserved keywords
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 22/Jan/21 01:27
Resolved: 03/Feb/21 19:31

The terms, "ACCESS TO ALL DATACENTERS" (for role control,) are introduced in CASSANDRA-13985. 
They are not marked as unreserved keywords. So in the non-role related DDL statements, for example, create table statement, the parser fails.
 
cqlsh> CREATE TABLE test.test_access ( foo text PRIMARY KEY, access text, bar text);
SyntaxException: line 1:54 mismatched input 'access' expecting ')' (... foo text PRIMARY KEY, [access]...)
 
cqlsh> CREATE TABLE test.test_datacenters ( foo text PRIMARY KEY, datacenters text, bar text);
SyntaxException: line 1:59 mismatched input 'datacenters' expecting ')' (... foo text PRIMARY KEY, [datacenters]...)
 
Since ACCESS and DATACENTERS are only applicable to the role defining statements. We should mark them as unreserved keywords in order to allow to use them in the DDL statement.
Bug: test case AuditLoggerTest  fail on aarch64 platform
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 14/Jan/21 03:19
Resolved: 22/Jan/21 09:09

When run ant; ant test on aarch64 platform, AuditLoggerTest failed，the output as follows:

 
{code:java}
[junit-timeout] Testsuite: org.apache.cassandra.audit.AuditLoggerTest
[junit-timeout] #
[junit-timeout] # A fatal error has been detected by the Java Runtime Environment:
[junit-timeout] #
[junit-timeout] #  SIGBUS (0x7) at pc=0x0000ffff81add020, pid=27652, tid=0x0000ffff80e1f1f0
[junit-timeout] #
[junit-timeout] # JRE version: OpenJDK Runtime Environment (8.0_222-b10) (build 1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10)
[junit-timeout] # Java VM: OpenJDK 64-Bit Server VM (25.222-b10 mixed mode linux-aarch64 compressed oops)
[junit-timeout] # Problematic frame:
[junit-timeout] # V  [libjvm.so+0xa4d020]
[junit-timeout] #
[junit-timeout] # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
[junit-timeout] #
[junit-timeout] # An error report file with more information is saved as:
[junit-timeout] # /home/cassandra/cassandra/hs_err_pid27652.log
[junit-timeout] #
[junit-timeout] # If you would like to submit a bug report, please visit:
[junit-timeout] #   http://bugreport.java.com/bugreport/crash.jsp
[junit-timeout] #
[junit-timeout] Testsuite: org.apache.cassandra.audit.AuditLoggerTest
[junit-timeout] Testsuite: org.apache.cassandra.audit.AuditLoggerTest Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 sec
[junit-timeout]
[junit-timeout] Testcase: org.apache.cassandra.audit.AuditLoggerTest:testConflictingPaths:      Caused an ERROR
[junit-timeout] Forked Java VM exited abnormally. Please note the time in the report does not reflect the time until the VM exit.
[junit-timeout] junit.framework.AssertionFailedError: Forked Java VM exited abnormally. Please note the time in the report does not reflect the time until the VM exit.
[junit-timeout]         at java.util.Vector.forEach(Vector.java:1275)
[junit-timeout]         at java.util.Vector.forEach(Vector.java:1275)
[junit-timeout]         at java.lang.Thread.run(Thread.java:748)
[junit-timeout]
[junit-timeout]
[junit-timeout] Test org.apache.cassandra.audit.AuditLoggerTest FAILED (crashed)
   [delete] Deleting directory /home/cassandra/cassandra/build/test/cassandra/commitlog:2
   [delete] Deleting directory /home/cassandra/cassandra/build/test/cassandra/data:2
   [delete] Deleting directory /home/cassandra/cassandra/build/test/cassandra/saved_caches:2
   [delete] Deleting directory /home/cassandra/cassandra/build/test/cassandra/hints:2

{code}
attached file hs_err_pid27652.log
Bug: Node restart during joining sets protocol version to V3
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Mar/21 02:18
Resolved: 08/Mar/22 22:44

While joining nodes to a cluster, an old node crashed. The old node was recovered however clients (datastax java) refused to connect to it.

The driver error:
{noformat}
Detected added or restarted Cassandra host /<ip>:<port> but ignoring it since it does not support the version V4 of the native protocol which is currently in use.{noformat}

In the recovered node cassandra logs:
{noformat}
INFO  o.a.c.transport.ConfiguredLimit Detected peers which do not fully support protocol V4. Capping max negotiable version to V3{noformat}

I confirmed that ALL the nodes in the cluster, joining or otherwise, were apache-cassandra-3.11.6 so that error message was rather confusing.

 Eventually after digging through the code we got to the bottom of the issue:

https://issues.apache.org/jira/browse/CASSANDRA-15193 adds a check for node version, which reverts the protocol version to V3 if any peer fails the version check. Joining nodes have NULL for their version in the peers table, which fails the version check.
Bug: Column rpc_address on system_local ignores broadcast_rpc_address yaml setting
Status: Open
Priority: Normal
Created: 14/Jan/21 19:04

While troubleshooting a driver issue I ran into a potential C* issue regarding the {{rpc_address}} and {{broadcast_rpc_address}} yaml settings. On {{system.peers}} I see that the {{rpc_address}} column is populated with the {{broadcast_rpc_address}} setting when {{rpc_address}} is {{0.0.0.0}} but this is not the same behavior for the {{system.local}} table.

On {{system.local}}, {{rpc_address}} is returned as {{0.0.0.0}} even if {{broadcast_rpc_address}} is set. In my opinion the {{rpc_address}} on {{system.local}} should also be populated with the {{broadcast_rpc_address}} setting if {{rpc_address}} is {{0.0.0.0}}.

I tested this with C* 3.11.8, 4.0-beta2 and 4.0-beta4 so it looks like this behavior has been like this for a while now.
Bug: Parent repair sessions leak may lead to node long pauses
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Feb/21 05:57
Resolved: 24/Feb/21 18:56

{{ActiveRepairService}} keeps  a map `parentRepairSessions`. If these sessions leak, that map can grow to a size when a node restarts {{ActiveRepairService.onRestart()}} triggers a cleanup of sessions that can pause nodes in a cluster for a long time.

The proposed solution is for repairs to cleanup these sessions on all nodes on completion by sending a CLEANUP message to involved nodes. Tests rely on a new {{parentRepairSessionsCount()}} method on the parent repair sessions MBean to keep track of these.
Bug: Apply CASSANDRA-16515 for other view tests
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Mar/21 10:48
Resolved: 16/Mar/21 13:07

CASSANDRA-16515 unfortunately fixes only on view test class, but there are also other classes which may suffer from the same issue:
- ViewComplextTest,
- ViewLongTest,
- ViewSchemaTest
- ViewTest
Bug: Don't fail all ViewFilteringTest cases because one case timed out
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 12/Mar/21 09:21
Resolved: 12/Mar/21 20:00

When {{ViewFilteringTest}} creates a view, it is added to the list of created views. Those views are removed after each test cases. If creating a view fail, it is not added to the list of views to be dropped because the assumption is that when the command fails, the view should not be created. This is ok in general however it is not a valid assumption when we get operation timed out because we cannot say anything about its real result. This way, it is possible that the view gets created, the driver throws time out and we do not remove the created view. All other test cases which tries to create that view will fail because the view already exists. 

While it seems very improbable, we actually hit this a couple of times on our infrastructure

 !screenshot-1.png! 

 !screenshot-2.png! 

 !screenshot-3.png!
Bug: Digest mismatches during upgrade
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/Feb/21 05:12
Resolved: 01/Mar/21 12:57

The test has been failing and can always be reproduced in the recent CI. 

Stack trace: 
{code:java}
junit.framework.AssertionFailedError: Found Digest Mismatch
 at org.apache.cassandra.distributed.upgrade.MixedModeReadTest.checkTraceForDigestMismatch(MixedModeReadTest.java:89)
 at org.apache.cassandra.distributed.upgrade.MixedModeReadTest.lambda$mixedModeReadColumnSubsetDigestCheck$0(MixedModeReadTest.java:63)
 at org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:171)
 at org.apache.cassandra.distributed.upgrade.MixedModeReadTest.mixedModeReadColumnSubsetDigestCheck(MixedModeReadTest.java:76) {code}
The initial investigation shows that 
 * The test only fails in the setup phase of mixedModeReadColumnSubsetDigestCheck. The cluster version is *Versions.Major.v3X*
 * The test failure is likely a consequence of CASSANDRA-15962. After dropping the commit in branch cassandra-3.11 and rebuild the dtest jar, the upgrade test can pass. Meanwhile, dropping the other commits does not help.
Bug: Running tests in parallel can sometimes result in port collision
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 11/Mar/21 14:20
Resolved: 12/Mar/21 20:42

When running unit tests with parallel execution we may sometimes notice the port collision. In this mode, test classes from the list get consecutive offsets which are added to native transport port, storage port and ssl storage port ( {{OffsetAwareConfigurationLoader}} ). However we add to the ports configured in {{test/conf/cassandra.yaml}} we as we can see we have the following default config:
{code:java}
storage_port: 7012
ssl_storage_port: 7011 {code}
This results in the situation that thread n has the same storage port as thread n+1 has ssl storage port. There are not much such tests, but we can see {{ConnectionTests}} failing frequently because of that. 

The other problem is that {{OffsetAwareConfigurationLoader}} does not adjust native transport ssl port.

 !screenshot-1.png! 
 

Patch at https://github.com/apache/cassandra/pull/925
Bug: Increase driver timeouts for CQL unit tests
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 12/Mar/21 09:06
Resolved: 14/Mar/21 17:51

Currently we use default timeouts in CQL driver (used in CQLTester) which is 5 seconds for connection and 12 seconds for read. This is ok in most cases. However when running tests using infrastructure which is not very stable, it may happen those timeouts are insufficient in rare cases and the builds result in flakiness.

I'd like to either increase those timeouts or make them configurable

 !screenshot-1.png!
Bug: ClientRequestSizeMetricsTest is flaky
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Mar/21 12:33
Resolved: 16/Mar/21 20:49

I don't know if it was reported but we can see it fail sometimes. 

The reason is that it checks the metrics before issuing and query and after. It expects that the executed operation between subsequent metrics checks is the only operation during that time. However the driver sometimes may refresh the schema in the middle and pollute the results. It is perfectly enough to slow down the execution a bit and the test starts to fail consistently rather than sporadically because we always hit schema refresh.

Patch at https://github.com/apache/cassandra/pull/930
Bug: Fix flaky testSkipScrubCorruptedCounterRowWithTool
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 23/Mar/21 08:23
Resolved: 06/Apr/21 13:03

Fix flaky [testSkipScrubCorruptedCounterRowWithTool|https://ci-cassandra.apache.org/job/Cassandra-trunk/365/testReport/junit/org.apache.cassandra.db/ScrubTest/testSkipScrubCorruptedCounterRowWithTool_compression/]
Bug: Add tool to display or export the contents of a virtual table
Status: Open
Priority: Normal
Created: 12/Mar/21 01:49

Several virtual tables were recently added, but they're currently only accessible via cqlsh or programmatically. While this is valuable for many use cases, operators are accustomed with the convenience of querying system metrics with a simple nodetool command.

In addition to that, a relatively common request is to provide nodetool output in different formats (JSON, YAML and even XML) (CASSANDRA-5977, CASSANDRA-12035, CASSANDRA-12486, CASSANDRA-12698, CASSANDRA-12503). However this requires lots of manual labor as each nodetool subcommand needs to be adapted to support new output formats.

I propose adding a new CLI tool that will consistently print to the standard output the contents of a virtual table. By default the command will print the output in a tabular format similar to cqlsh, but a "--format" parameter can be specified to modify the output to some other format like JSON or YAML.

It should be possible to add a limit to the amount of rows displayed and filter to display only rows from with specific keys (ie. keyspace or table). The command should be flexible and provide simple hooks for registration and customization of new virtual tables.

My vision is that this is a path towards deprecating JMX and toward CQL for management, as we move information currently available through JMX to virtual tables (as CASSANDRA-14457 did with compactionstats) and easily expose them in this new tool as more virtual tables are added. Eventually we can also add setters when we start supporting writeable virtual tables.

I propose calling this tool admintool (naming bikeshedding welcome), for example:
{noformat}
admintool help

admintool <subcommand> <entity>

Available subcommands and entities are:

subcommands:
 - show
 - set (future)

entities:
 - caches
 - internode_inbound
 - internode_outbound
 - settings
 - sstable_tasks
 - system_properties
 - thread_pools

nodetool show clients --format yaml
...
nodetool show internode_outboud --format json
...
nodetool show sstabletasks --filter keyspace=my_ks --filter table=my_table
...
{noformat}