Bug: BufferOverflowException when writing during ALTER
Status: Triage Needed
Priority: Normal
Created: 23/Sep/21 12:36

This is somewhat tricky to reproduce, but Harry tests throw BufferOverflowException from time to time. It looks like it happens on the node that ALTER is happening on, about the time ALTER is happening. A race condition arises from the fact that in 3.0 we can compute the buffer size with one set of columns, but serialise a different one, resulting into a buffer overflow. 4.0 is not susceptible to this problem because of immutable metadata.

{code}
java.lang.RuntimeException: java.nio.BufferOverflowException
   at org.apache.cassandra.distributed.impl.IsolatedExecutor.waitOn(IsolatedExecutor.java)
   at org.apache.cassandra.distributed.impl.IsolatedExecutor.lambda$sync$5(IsolatedExecutor.java)
   at org.apache.cassandra.distributed.impl.Instance.executeInternalWithResult(Instance.java)
   at org.apache.cassandra.distributed.api.IInstance.executeInternal(IInstance.java)
   at org.apache.cassandra.distributed.impl.DelegatingInvokableInstance.executeInternal(DelegatingInvokableInstance.java)
   at org.apache.cassandra.distributed.fuzz.InJvmSutBase.execute(InJvmSutBase.java)
   at org.apache.cassandra.distributed.fuzz.InJvmSutBase.execute(InJvmSutBase.java)
   at org.apache.cassandra.distributed.fuzz.InJvmSutBase.lambda$executeAsync$0(InJvmSutBase.java)
   at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java)
   at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.BufferOverflowException
   at org.apache.cassandra.io.util.DataOutputBufferFixed.doFlush(DataOutputBufferFixed.java)
   at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java)
   at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java)
   at org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java)
   at org.apache.cassandra.db.marshal.CollectionType$CollectionPathSerializer.serialize(CollectionType.java)
   at org.apache.cassandra.db.rows.BufferCell$Serializer.serialize(BufferCell.java)
   at org.apache.cassandra.db.rows.UnfilteredSerializer.writeComplexColumn(UnfilteredSerializer.java)
   at org.apache.cassandra.db.rows.UnfilteredSerializer.lambda$serializeRowBody$0(UnfilteredSerializer.java)
   at org.apache.cassandra.utils.btree.BTree.applyLeaf(BTree.java)
   at org.apache.cassandra.utils.btree.BTree.apply(BTree.java)
   at org.apache.cassandra.utils.btree.BTree.apply(BTree.java)
   at org.apache.cassandra.utils.btree.BTree.apply(BTree.java)
   at org.apache.cassandra.db.rows.BTreeRow.apply(BTreeRow.java)
   at org.apache.cassandra.db.rows.UnfilteredSerializer.serializeRowBody(UnfilteredSerializer.java)
   at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java)
   at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java)
   at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java)
   at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java)
   at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java)
   at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.serialize(PartitionUpdate.java)
   at org.apache.cassandra.db.Mutation$MutationSerializer.serialize(Mutation.java)
   at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java)
   at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java)
   at org.apache.cassandra.db.Keyspace.apply(Keyspace.java)
   at org.apache.cassandra.db.Mutation.apply(Mutation.java)
   at org.apache.cassandra.db.Mutation.apply(Mutation.java)
   at org.apache.cassandra.db.Mutation.apply(Mutation.java)
   at org.apache.cassandra.cql3.statements.BatchStatement.executeInternalWithoutCondition(BatchStatement.java)
   at org.apache.cassandra.cql3.statements.BatchStatement.executeInternal(BatchStatement.java)
   at org.apache.cassandra.distributed.impl.Instance.lambda$executeInternalWithResult$1(Instance.java)
   at java.util.concurrent.FutureTask.run(FutureTask.java)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java)
   at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java8)
   ... 1 more
{code}