Bug: Cannot correctly create keyspace statement with replication during schemaChange
Status: Resolved
Priority: Normal
Resolution: Not A Problem
Created: 28/Jun/24 21:36
Resolved: 02/Jul/24 07:09

h3. What happened

A specific schema change for creating keyspace with replications failed during Cassandra upgrade testing, but can pass under Cassandra distributed testing (non-upgrade).
h3. How to reproduce:

Put the following test under {{{}cassandra/test/distributed/org/apache/cassandra/distributed/upgrade/{}}}, and build dtest jars for any versions within [4.1.3, 5.0-alpha2].
{code:java}
package org.apache.cassandra.distributed.upgrade;
public class demoUpgradeTest extends UpgradeTestBase
    @Test
    public void demoTest() throws Throwable {
        new TestCase()
                .nodes(1)
                .nodesToUpgrade(1)
                .withConfig(config -> config.with(NETWORK, GOSSIP, NATIVE_PROTOCOL))
                .upgradesToCurrentFrom(v41)
                .setup((cluster) -> {
                    cluster.schemaChange(withKeyspace("CREATE KEYSPACE %s WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 2}"));
                }).runAfterNodeUpgrade((cluster, node) -> {
                    // let's do nothing here.
                }).run();
    }
} {code}
Run the test with
{code:java}
$ ant test-jvm-dtest-some-Duse.jdk11=true -Dtest.name=org.apache.cassandra.distributed.upgrade.demoUpgradeTest {code}
You will see the following failure:
{code:java}
[junit-timeout] Testcase: demoTest(org.apache.cassandra.distributed.upgrade.demoUpgradeTest)-_jdk11:    Caused an ERROR
[junit-timeout] Cannot add existing keyspace "distributed_test_keyspace"
[junit-timeout] org.apache.cassandra.exceptions.AlreadyExistsException: Cannot add existing keyspace "distributed_test_keyspace"
[junit-timeout]     at org.apache.cassandra.cql3.statements.schema.CreateKeyspaceStatement.apply(CreateKeyspaceStatement.java:78)
[junit-timeout]     at org.apache.cassandra.schema.DefaultSchemaUpdateHandler.apply(DefaultSchemaUpdateHandler.java:230)
[junit-timeout]     at org.apache.cassandra.schema.Schema.transform(Schema.java:597)
[junit-timeout]     at org.apache.cassandra.cql3.statements.schema.AlterSchemaStatement.execute(AlterSchemaStatement.java:114)
[junit-timeout]     at org.apache.cassandra.cql3.statements.schema.AlterSchemaStatement.execute(AlterSchemaStatement.java:60)
[junit-timeout]     at org.apache.cassandra.distributed.impl.Coordinator.unsafeExecuteInternal(Coordinator.java:122)
[junit-timeout]     at org.apache.cassandra.distributed.impl.Coordinator.unsafeExecuteInternal(Coordinator.java:103)
[junit-timeout]     at org.apache.cassandra.distributed.impl.Coordinator.lambda$executeWithResult$0(Coordinator.java:66)
[junit-timeout]     at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
[junit-timeout]     at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
[junit-timeout]     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[junit-timeout]     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[junit-timeout]     at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[junit-timeout]     at java.base/java.lang.Thread.run(Thread.java:829) {code}
I have tested version pairs 4.1.3_4.1.4, 4.1.4_4.1.5, 4.1.5_5.0-alpha1, and 5.0-alpha1_5.0-alpha2. All of them have the same issue.

I wrote a very similar test with Cassandra distributed test framework (non-upgrade test) as below:
{code:java}
package org.apache.cassandra.distributed.test.streaming;public class LCSStreamingKeepLevelTest extends TestBaseImpl
{
    @Test
    public void demoTest() throws IOException
    {
        try (Cluster cluster = builder().withNodes(1)
                .withConfig(config -> config.with(NETWORK, GOSSIP, NATIVE_PROTOCOL))
                .start())
        {
            cluster.schemaChange(withKeyspace("CREATE KEYSPACE %s WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 2}"));
        }
    }
} {code}
This distributed test would pass successfully without any issues.

 

The expected behavior should be that the upgrade test above can also perform cluster.schemaChange() successfully.