Bug: Test Failure: MixedModeAvailabilityV30AllOneTest.testAvailabilityCoordinatorUpgraded
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/Jun/23 16:06
Resolved: 11/Sep/23 15:54

The JVM upgrade dtest {{MixedModeAvailabilityV3XAllOneTest.testAvailabilityCoordinatorUpgraded}} seems to be flaky at least in {{trunk}}:
{code}
junit.framework.AssertionFailedError: Error in test '4.0.11 -> [5.0]' while upgrading to '5.0'; successful upgrades []
	at org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:348)
	at org.apache.cassandra.distributed.upgrade.MixedModeAvailabilityTestBase.testAvailability(MixedModeAvailabilityTestBase.java:154)
	at org.apache.cassandra.distributed.upgrade.MixedModeAvailabilityTestBase.testAvailabilityCoordinatorUpgraded(MixedModeAvailabilityTestBase.java:74)
Caused by: java.lang.AssertionError: Unexpected error while reading in case write-read consistency ALL-ONE with upgraded coordinator and 2 nodes down: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
	at org.apache.cassandra.distributed.upgrade.MixedModeAvailabilityTestBase.lambda$testAvailability$6(MixedModeAvailabilityTestBase.java:145)
	at org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:339)
Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
	at org.apache.cassandra.service.reads.ReadCallback.awaitResults(ReadCallback.java:162)
	at org.apache.cassandra.service.reads.AbstractReadExecutor.awaitResponses(AbstractReadExecutor.java:387)
	at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:2124)
	at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1995)
	at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1873)
	at org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1286)
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:364)
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:293)
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:105)
	at org.apache.cassandra.distributed.impl.Coordinator.unsafeExecuteInternal(Coordinator.java:122)
	at org.apache.cassandra.distributed.impl.Coordinator.unsafeExecuteInternal(Coordinator.java:103)
	at org.apache.cassandra.distributed.impl.Coordinator.lambda$executeWithResult$0(Coordinator.java:66)
	at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
	at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:750)
{code}
This has failed 143 times in 500 iterations of this CircleCI run:
* https://app.circleci.com/pipelines/github/adelapena/cassandra/2927/workflows/fcd1cd60-826b-484a-8e81-d3ba640f7de9/jobs/47659/tests

The failure has also recently appeared on Jenkins too:
* https://ci-cassandra.apache.org/job/Cassandra-trunk/1585/testReport/org.apache.cassandra.distributed.upgrade/MixedModeAvailabilityV3XAllOneTest/testAvailabilityCoordinatorUpgraded__jdk11/

Given that the failure has just appeared on Jenkins and it fails relatively easily on CircleCI, it's likely that it has been broken by a very recent change.