Bug: Cassandra Dtest Cluster is not fully up after upgrading and may fail on queries
Status: Triage Needed
Priority: Normal
Created: 02/Jul/24 06:29

This may not be a bug, but the Cassandra Dtest framework can definitely be improved.
h2. What happened

In DTest framework, the cluster node may not fully up and become fully operational when the test logic in {{runAfterNodeUpgrade()}} is executed. This may cause expected behavior and even test flakiness.
h2. How to reproduce

Use the following upgrade tests as example, Put the following test under cassandra/test/distributed/org/apache/cassandra/distributed/upgrade/, and build dtest jars.
{code:java}
package org.apache.cassandra.distributed.upgrade;public class demoUpgradeTest extends UpgradeTestBase {
    @Test
    public void demoTest() throws Throwable {
    {
        new TestCase()
        .nodes(2)
        .nodesToUpgrade(1)
        .withConfig(c -> c.with(GOSSIP, NATIVE_PROTOCOL).set("drop_compact_storage_enabled", true))
        .upgradesToCurrentFrom(v3X)
        .setup((cluster) -> {
            cluster.schemaChange("CREATE KEYSPACE k WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1}");
            cluster.schemaChange("CREATE TABLE k.t ( k int, c int, total counter, PRIMARY KEY (k, c))");        })
        .runAfterNodeUpgrade((cluster, node) -> {
            ConsistencyLevel cl = ConsistencyLevel.ONE;
            String select = "SELECT total FROM k.t WHERE k = 1 AND c = ?";            for (int i = 1; i <= cluster.size(); i++)
            {
                ICoordinator coordinator = cluster.coordinator(i);                coordinator.execute("UPDATE k.t SET total = total + 1 WHERE k = 1 AND c = ?", cl, i);
                assertRows(coordinator.execute(select, cl, i), row(1L));                coordinator.execute("UPDATE k.t SET total = total - 4 WHERE k = 1 AND c = ?", cl, i);
                assertRows(coordinator.execute(select, cl, i), row(-3L));
            }
        }).run();
    }
} {code}
Run the test with:
{code:java}
$ ant test-jvm-dtest-some-Duse.jdk11=true -Dtest.name=org.apache.cassandra.distributed.upgrade.demoUpgradeTest {code}
You will see the following failure:
{code:java}
[junit-timeout] Testcase: demoTest(org.apache.cassandra.distributed.upgrade.demoUpgradeTest)-_jdk11:    FAILED
[junit-timeout] Error in test '4.0.13 -> [4.1-alpha1]' while upgrading to '4.1-alpha1'; successful upgrades []
[junit-timeout] junit.framework.AssertionFailedError: Error in test '4.0.13 -> [4.1-alpha1]' while upgrading to '4.1-alpha1'; successful upgrades []
[junit-timeout]     at org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:442)
[junit-timeout]     at org.apache.cassandra.distributed.upgrade.demoUpgradeTest.demoTest(demoUpgradeTest.java:62)
[junit-timeout]     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit-timeout]     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit-timeout]     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit-timeout] Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level ONE
[junit-timeout]     at org.apache.cassandra.exceptions.UnavailableException.create(UnavailableException.java:37)
[junit-timeout]     at org.apache.cassandra.exceptions.UnavailableException.create(UnavailableException.java:31)
[junit-timeout]     at org.apache.cassandra.service.StorageProxy.findSuitableReplica(StorageProxy.java:1617)
[junit-timeout]     at org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:1565)
[junit-timeout]     at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:809)
[junit-timeout]     at org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:1054)
[junit-timeout]     at org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:476)
[junit-timeout]     at org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:454)
[junit-timeout]     at org.apache.cassandra.distributed.impl.Coordinator.executeInternal(Coordinator.java:103)
[junit-timeout]     at org.apache.cassandra.distributed.impl.Coordinator.lambda$executeWithResult$0(Coordinator.java:65)
[junit-timeout]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[junit-timeout]     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[junit-timeout]     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[junit-timeout]     at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[junit-timeout]     at java.base/java.lang.Thread.run(Thread.java:829) {code}
This is actually due to the fact that the upgraded node is not fully operational after a restart and directly executes the UPDATE statement. This can be manually fixed by adding a sleep in the beginning of runAfterNodeUpgrade like below:
{code:java}
        ...
        .runAfterNodeUpgrade((cluster, node) -> {
            // Wait for the node to be fully operational
            cluster.get(node).nodetool("status");            // Adding a small delay to ensure the node is fully integrated
            Thread.sleep(10000);
            ...
        }).run();
    }
} {code}
However, by design, the Dtest framework should wait for the node to be fully operational before executing the runAfterNodeUpgrade(). It would be good to add some waiting logic for this purpose to prevent such unexpected behavior from happening.