Bug: Streamline the serialized format for index status gossip messages
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 05/Nov/24 19:45
Resolved: 21/Nov/24 21:20

I spoke with someone today hitting an error on a cluster recently upgraded from 4.0.5 to 5.0.2, when creating or dropping tables.  Immediately after creating a table, nodes are reported down and they all immediately start throwing the below exception repeatedly in the logs.  The cluster has to be restarted to come back online.

Unfortunately I don't have access to the cluster, this was sent to me on a phone call.

Here's the exception:

 
{noformat}
ERROR [GossipStage:1] 2024-11-03 09:48:04,187 JVMStabilityInspector.java:70 - 
Exception in thread Thread[GossipStage:1,5,GossipStage]
java.lang.RuntimeException: java.lang.AssertionError
        at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:108)
        at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:45)
        at org.apache.cassandra.net.InboundMessageHandler$ProcessMessage.run(InboundMessageHandler.java:430)
        at org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.AssertionError: null
        at org.apache.cassandra.db.TypeSizes.sizeof(TypeSizes.java:44)
        at org.apache.cassandra.gms.VersionedValue$VersionedValueSerializer.serializedSize(VersionedValue.java:365)
        at org.apache.cassandra.gms.VersionedValue$VersionedValueSerializer.serializedSize(VersionedValue.java:343)
        at org.apache.cassandra.gms.EndpointStateSerializer.serializedSize(EndpointState.java:344)
        at org.apache.cassandra.gms.EndpointStateSerializer.serializedSize(EndpointState.java:300)
        at org.apache.cassandra.gms.GossipDigestAckSerializer.serializedSize(GossipDigestAck.java:96)
        at org.apache.cassandra.gms.GossipDigestAckSerializer.serializedSize(GossipDigestAck.java:61)
        at org.apache.cassandra.net.Message$Serializer.payloadSize(Message.java:1088)
        at org.apache.cassandra.net.Message.payloadSize(Message.java:1131)
        at org.apache.cassandra.net.Message$Serializer.serializedSize(Message.java:769)
        at org.apache.cassandra.net.Message.serializedSize(Message.java:1111)
        at org.apache.cassandra.net.OutboundConnections.connectionTypeFor(OutboundConnections.java:215)
        at org.apache.cassandra.net.OutboundConnections.connectionFor(OutboundConnections.java:207)
        at org.apache.cassandra.net.OutboundConnections.enqueue(OutboundConnections.java:96)
        at org.apache.cassandra.net.MessagingService.doSend(MessagingService.java:473)
        at org.apache.cassandra.net.OutboundSink.accept(OutboundSink.java:70)
        at org.apache.cassandra.net.MessagingService.send(MessagingService.java:462)
        at org.apache.cassandra.net.MessagingService.send(MessagingService.java:437)
        at org.apache.cassandra.gms.GossipDigestSynVerbHandler.doVerb(GossipDigestSynVerbHandler.java:110)
        at org.apache.cassandra.net.InboundSink.lambda$new$0(InboundSink.java:78)
        at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:97)
        ... 7 common frames omitted{noformat}