Bug: Running read on a mixed cluster with C* version 3.11.6 and version 4.0-alpha1 failed with multiple exceptions
Status: Triage Needed
Priority: Normal
Created: 10/Nov/20 01:11

Steps to reproduce: 
 # Set up a  cluster with 2 nodes (1 seed using 3.11.6, the other node using 4.0-alpha1). 
 # Run stress testing on the seed node running C* 3.11.6. 

Error messages on each node: 

Seed (3.11.6):

 
{code:java}
ERROR [RequestResponseStage-1] 2020-06-25 01:58:56,745 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[RequestResponseStage-1,5,main]
java.lang.IllegalArgumentException: Unknown request failure reason error code: 3
  at org.apache.cassandra.exceptions.RequestFailureReason.fromCode(RequestFailureReason.java:49) ~[main/:na]
  at org.apache.cassandra.net.MessageIn.getFailureReason(MessageIn.java:189) ~[main/:na]
  at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:47) ~[main/:na]
  at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) ~[main/:na]
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_222]
  at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:165) ~[main/:na]
  at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:137) [main/:na]
  at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:113) [main/:na]
  at java.lang.Thread.run(Thread.java:748) [na:1.8.0_222]{code}
 

 

Looks like this error message might be because the enum class RequestFailureReason is incompatible between 3.11.6 and 4.0-alpha1: In 3.11.6 it contains only 2 possible values ([https://github.com/apache/cassandra/blob/cassandra-3.11.6/src/java/org/apache/cassandra/exceptions/RequestFailureReason.java#L21]), while in 4.0-alpha1 it contains 4 possible values ([https://github.com/apache/cassandra/blob/cassandra-4.0-alpha1/src/java/org/apache/cassandra/exceptions/RequestFailureReason.java#L33]). 

 

Non-seed node (4.0-alpha1):

 
{code:java}
INFO  [Messaging-EventLoop-3-5] 2020-06-25 01:58:56,743 NoSpamLogger.java:91 - 250.16.238.1:7000->250.16.238.2:7000-LEGACY_MESSAGES-0e4d7ae2 incompatible schema encountered while deserializing a message
org.apache.cassandra.exceptions.UnknownTableException: Couldn't find table with id 5bc52802-de25-35ed-aeab-188eecebb090. If a table was just created, this is likely due to the schemanot being fully propagated.  Please wait for schema agreement on table creation.
  at org.apache.cassandra.schema.Schema.getExistingTableMetadata(Schema.java:456)
  at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:614)
  at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:344)
  at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:362)
  at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:326)
  at org.apache.cassandra.net.Message$Serializer.deserializePayloadPre40(Message.java:949)
  at org.apache.cassandra.net.Message$Serializer.deserializePre40(Message.java:930)
  at org.apache.cassandra.net.Message$Serializer.deserializePre40(Message.java:918)
  at org.apache.cassandra.net.Message$Serializer.deserialize(Message.java:618)
  at org.apache.cassandra.net.InboundMessageHandler.processSmallMessage(InboundMessageHandler.java:321)
  at org.apache.cassandra.net.InboundMessageHandler.processOneContainedMessage(InboundMessageHandler.java:304)
  at org.apache.cassandra.net.InboundMessageHandler.processFrameOfContainedMessages(InboundMessageHandler.java:271)
  at org.apache.cassandra.net.InboundMessageHandler.processIntactFrame(InboundMessageHandler.java:256)
  at org.apache.cassandra.net.InboundMessageHandler.process(InboundMessageHandler.java:247)
  at org.apache.cassandra.net.FrameDecoder.deliver(FrameDecoder.java:320)
  at org.apache.cassandra.net.FrameDecoder.channelRead(FrameDecoder.java:284)
  at org.apache.cassandra.net.FrameDecoder.channelRead(FrameDecoder.java:268)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360
  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
  at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
  at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
  at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424)
  at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326)
  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
  at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
  at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
  at java.lang.Thread.run(Thread.java:748){code}
 

Not exactly sure why how is this UnknownTableException triggered.  

Client:

 
{code:java}
WARN  01:59:41,160 No schema agreement from live replicas after 10 s. The schema may not be up to date on some nodes.
com.datastax.driver.core.exceptions.InvalidQueryException: Keyspace 'stresscql' does not exist
  at com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:50)
  at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
  at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
  at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
  at org.apache.cassandra.stress.util.JavaDriverClient.execute(JavaDriverClient.java:189)
  at org.apache.cassandra.stress.StressProfile.maybeCreateSchema(StressProfile.java:238)
  at org.apache.cassandra.stress.StressProfile.newGenerator(StressProfile.java:681)
  at org.apache.cassandra.stress.StressProfile.printSettings(StressProfile.java:129)
  at org.apache.cassandra.stress.settings.StressSettings.printSettings(StressSettings.java:383)
  at org.apache.cassandra.stress.Stress.run(Stress.java:95)
  at org.apache.cassandra.stress.Stress.main(Stress.java:62)
Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Keyspace 'stresscql' does not exist
  at com.datastax.driver.core.Responses$Error.asException(Responses.java:136)
  at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:179)
  at com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:177)
  at com.datastax.driver.core.RequestHandler.access$2500(RequestHandler.java:43)
  at com.datastax.driver.core.RequestHandler$SpeculativeExecution.setFinalResult(RequestHandler.java:792)
  at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:611)
  at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1013)
  at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:936)
  at com.datastax.shaded.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
  at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
  at com.datastax.shaded.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
  at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
  at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
  at com.datastax.shaded.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
  at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
  at com.datastax.shaded.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:276)
  at com.datastax.shaded.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:263)
  at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
  at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
  at com.datastax.shaded.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
  at com.datastax.shaded.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
  at com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  at com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  at com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  at com.datastax.shaded.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  at com.datastax.shaded.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
  at java.lang.Thread.run(Thread.java:748)
{code}