Bug: Test failure: org.apache.cassandra.distributed.upgrade.DropCompactStorageTest.testDropCompactStorage
Status: Open
Priority: Normal
Created: 31/May/22 17:10

The Java upgrade dtest {{org.apache.cassandra.distributed.upgrade.DropCompactStorageTest.testDropCompactStorage}} is ~68% flaky on 4.0 and ~2% flaky on trunk, at least in CircleCI:
 * 4.0: [https://app.circleci.com/pipelines/github/adelapena/cassandra/1622/workflows/0086c3b1-a552-4c7a-8278-2f759cee5bdf/jobs/17288]
 * trunk: [https://app.circleci.com/pipelines/github/adelapena/cassandra/1624/workflows/c4ce2b95-998f-459b-830e-8e3fa6637e15/jobs/17293]

The error for 4.0 is:
{code:java}
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 218] did not terminate on time
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
	at org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:227)
	at org.apache.cassandra.distributed.upgrade.DropCompactStorageTest.testDropCompactStorage(DropCompactStorageTest.java:49)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 218] did not terminate on time
	at org.apache.cassandra.distributed.impl.IsolatedExecutor$ThrowingRunnable.lambda$toRunnable$0(IsolatedExecutor.java:235)
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:83)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 218] did not terminate on time
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.apache.cassandra.distributed.impl.IsolatedExecutor$ThrowingRunnable.lambda$toRunnable$0(IsolatedExecutor.java:231)
Caused by: java.lang.RuntimeException: java.util.concurrent.TimeoutException: org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 218] did not terminate on time
	at org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:62)
	at org.apache.cassandra.distributed.impl.Instance.lambda$shutdown$28(Instance.java:810)
	at org.apache.cassandra.distributed.impl.IsolatedExecutor.lambda$null$8(IsolatedExecutor.java:114)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Caused by: java.util.concurrent.TimeoutException: org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 218] did not terminate on time
	at org.apache.cassandra.utils.ExecutorUtils.awaitTerminationUntil(ExecutorUtils.java:107)
	at org.apache.cassandra.utils.ExecutorUtils.awaitTermination(ExecutorUtils.java:96)
	at org.apache.cassandra.utils.ExecutorUtils.shutdownNowAndWait(ExecutorUtils.java:139)
	at org.apache.cassandra.concurrent.StageManager.shutdownAndWait(StageManager.java:128)
	at org.apache.cassandra.distributed.impl.Instance.lambda$null$26(Instance.java:803)
	at org.apache.cassandra.distributed.impl.Instance.lambda$parallelRun$32(Instance.java:968)
{code}
Whereas the error for trunk is:
{code:java}
org.apache.cassandra.distributed.shared.ShutdownException: Uncaught exceptions were thrown during test
	at org.apache.cassandra.distributed.impl.AbstractCluster.checkAndResetUncaughtExceptions(AbstractCluster.java:1056)
	at org.apache.cassandra.distributed.impl.AbstractCluster.close(AbstractCluster.java:1042)
	at org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:244)
	at org.apache.cassandra.distributed.upgrade.DropCompactStorageTest.testDropCompactStorage(DropCompactStorageTest.java:49)
	Suppressed: java.lang.NullPointerException
		at org.apache.cassandra.gms.Gossiper.getHostId(Gossiper.java:992)
		at org.apache.cassandra.gms.Gossiper.getHostId(Gossiper.java:987)
		at org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:2512)
		at org.apache.cassandra.service.StorageService.onChange(StorageService.java:2086)
		at org.apache.cassandra.gms.Gossiper.markAsShutdown(Gossiper.java:479)
		at org.apache.cassandra.gms.GossipShutdownVerbHandler.doVerb(GossipShutdownVerbHandler.java:37)
		at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:69)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
		at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
		at java.lang.Thread.run(Thread.java:748)
{code}
I haven't managed to reproduce it on 4.1, although more iterations could do it.

This was discovered during CASSANDRA-17213.