Bug: Unknown column cdc during deserialization ERROR when upgrading from 3.0.28 to 3.11.14
Status: Resolved
Priority: Normal
Resolution: Not A Problem
Created: 10/Dec/22 04:32
Resolved: 11/Dec/22 18:27

When we performed a rolling upgrade from 3.0.28 to 3.11.14, we encountered the following error message during the upgrade process. It doesn't prevent the upgrade process, but it's in ERROR log level and causes concerns for the upgrade process.
{code:java}
ERROR [MessagingService-Incoming-/192.168.107.2] 2022-12-10 04:01:12,449 CassandraDaemon.java:265 - Exception in thread Thread[MessagingService-Incoming-/192.168.107.2,5,main]
java.lang.RuntimeException: Unknown column cdc during deserialization
	at org.apache.cassandra.db.Columns$Serializer.deserialize(Columns.java:445)
	at org.apache.cassandra.db.SerializationHeader$Serializer.deserializeForMessaging(SerializationHeader.java:446)
	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.deserializeHeader(UnfilteredRowIteratorSerializer.java:190)
	at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:734)
	at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:722)
	at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:337)
	at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:346)
	at org.apache.cassandra.service.MigrationManager$MigrationsSerializer.deserialize(MigrationManager.java:574)
	at org.apache.cassandra.service.MigrationManager$MigrationsSerializer.deserialize(MigrationManager.java:557)
	at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98)
	at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:216)
	at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:193)
	at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:96)
INFO  [MigrationStage:1] 2022-12-10 04:01:13,394 MigrationCoordinator.java:523 - Sending schema pull request to /192.168.107.2 at 1670644873394 with timeout 10000
ERROR [MessagingService-Incoming-/192.168.107.2] 2022-12-10 04:01:13,427 CassandraDaemon.java:265 - Exception in thread Thread[MessagingService-Incoming-/192.168.107.2,5,main]
java.lang.RuntimeException: Unknown column cdc during deserialization
	at org.apache.cassandra.db.Columns$Serializer.deserialize(Columns.java:445)
	at org.apache.cassandra.db.SerializationHeader$Serializer.deserializeForMessaging(SerializationHeader.java:446)
	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.deserializeHeader(UnfilteredRowIteratorSerializer.java:190)
	at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:734)
	at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:722)
	at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:337)
	at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:346)
	at org.apache.cassandra.service.MigrationManager$MigrationsSerializer.deserialize(MigrationManager.java:574)
	at org.apache.cassandra.service.MigrationManager$MigrationsSerializer.deserialize(MigrationManager.java:557)
	at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98)
	at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:216)
	at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:193)
	at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:96){code}
h1. Steps to reproduce

Configure a 3-node Cassandra cluster (N0, N1 and N2) in 3.0.28 using the default configuration setting. Configure N0 as the seed node.

For the new version configuration (3.11.14), set *cdc_enabled: true* in cassandra.yaml. All the other configurations use default values.

 Perform a rolling upgrade.
 # Start up the 3-node cluster in 3.0.28.
 # Drain, shutdown and upgrade N0 to 3.11.14.
 # Drain, shutdown and upgrade N1 to 3.11.14.
 # Drain, shutdown and upgrade N2 to 3.11.14.

The +Unknown column cdc ERROR message+ occurs in *N2* once both N0 and N1 are upgraded. 

I attached the system.log for our three-node cluster.
 * N0 (Seed): 192.168.86.2
 * N1: 192.168.86.3
 * N2: 192.168.86.4