Bug: Schema change failed in Cassandra Dtest after node restart and upgrade but success in setup 
Status: Triage Needed
Priority: Normal
Created: 02/Jul/24 19:52

h2. What happened

Cassandra dtest {{schemaChange}} behaves differently in {{setup}} and {{{}runAfterNodeUpgrade{}}}.
h2. How to reproduce:

Put the following test under {{{}cassandra/test/distributed/org/apache/cassandra/distributed/upgrade/{}}}, and build dtest jars.

 
{code:java}
package org.apache.cassandra.distributed.upgrade;
public class demoUpgradeTest extends UpgradeTestBase {
    @Test
    public void firstDemoTest() throws Throwable {
        new TestCase()
        .nodes(2)
        .nodesToUpgrade(1)
        .withConfig((cfg) -> cfg.with(Feature.NETWORK, Feature.GOSSIP))
        .upgradesToCurrentFrom(v3X)
        .setup((cluster) -> {
            cluster.schemaChange(withKeyspace("CREATE TABLE %s.tbl (pk int, ck int, v1 int, PRIMARY KEY (pk, ck))"));
        })
        .runAfterNodeUpgrade((cluster, node) -> {
         // let's do nothing here.
        }).run();
    }
    @Test
    public void secondDemoTest() throws Throwable {
        new TestCase()
        .nodes(2)
        .nodesToUpgrade(1)
        .withConfig((cfg) -> cfg.with(Feature.NETWORK, Feature.GOSSIP))
        .upgradesToCurrentFrom(v3X)
        .setup((cluster) -> {
             // let's do nothing here.
        })
        .runAfterNodeUpgrade((cluster, node) -> {
            cluster.schemaChange(withKeyspace("CREATE TABLE %s.tbl (pk int, ck int, v1 int, PRIMARY KEY (pk, ck))"));
        }).run();
    }
} {code}
Run the test with:
{code:java}
$ ant test-jvm-dtest-some-Duse.jdk11=true -Dtest.name=org.apache.cassandra.distributed.upgrade.demoUpgradeTest {code}
firstDemoTest passes, but secondDemoTest fails with the following output:

 
{code:java}
java.lang.AssertionError: Error in test '5.0-beta2 -> [5.1]' while upgrading to '5.1'; successful upgrades []
    at org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:396)
 at org.apache.cassandra.distributed.upgrade.demoUpgradeTest.secondDemoTest(demoUpgradeTest.java:56)
Caused by: java.lang.IllegalStateException: Can't commit transformations when running in gossip mode. Enable the ClusterMetadataService with `nodetool addtocms`.
 at org.apache.cassandra.tcm.migration.GossipProcessor.commit(GossipProcessor.java:34)
 at org.apache.cassandra.tcm.ClusterMetadataService$SwitchableProcessor.commit(ClusterMetadataService.java:836)
 at org.apache.cassandra.tcm.Processor.commit(Processor.java:45)
 at org.apache.cassandra.tcm.ClusterMetadataService.commit(ClusterMetadataService.java:511)
 at org.apache.cassandra.schema.Schema.submit(Schema.java:292) {code}
As a previous issue suggested, {{schemaChange}} would fail to execute when gossip mode is disabled. However, even with gossip mode enabled, {{schemaChange}} behaves differently in {{setup}} and {{{}runAfterNodeUpgrade{}}}. Executing {{schemaChange}} in {{runAfterNodeUpgrade}} would call the {{commit}} function in [{{org.apache.cassandra.tcm.migration.GossipProcessor.java}}|https://github.com/apache/cassandra/blob/cbe07fd57e3d94a1f2512fced3f38e69ad4b3eb2/src/java/org/apache/cassandra/tcm/migration/GossipProcessor.java#L32]:

 
{code:java}
public Commit.Result commit(Entry.Id entryId, Transformation transform, Epoch lastKnown, Retry.Deadline retryPolicy)
{
    throw new IllegalStateException("Can't commit transformations when running in gossip mode. Enable the ClusterMetadataService with `nodetool addtocms`.");
} {code}
which directly throws an exception.

The expected behavior should be that {{schemaChange}} consistently in {{setup}} and {{{}runAfterNodeUpgrade{}}}.