Bug: ChannelProxy read might not read complete buffer causing "Checksum didn't match" error for sstableloader
Status: Triage Needed
Priority: Normal
Created: 24/Oct/24 00:38

CassandraCompressedStreamWriter expects the outbuffer to be filled completely while issuing the read call for ChannelProxy. Please refer the code snippet below from CassandraCompressedStreamWriter.
{code:java}
  out.writeToChannel(bufferSupplier -> {
      ByteBuffer outBuffer = bufferSupplier.get(toTransfer);
      long read = fc.read(outBuffer, position);
      assert read == toTransfer : String.format("could not read required number of bytes from file to be streamed: read %d bytes, wanted %d bytes", read, toTransfer);
      outBuffer.flip();
  }, limiter);  {code}
This is problematic since the underlying FileChannel implementation for ChannelProxy can have reads less than the buffer size causing the file upload operation to fail. 

Also, to catch the above problem an assertion is used instead of using an Exception. Assertions have to enabled explicitly by passing -ea flag to the java runtime. 

If assertions are disabled, the above partial read operation can cause an invalid buffer to get transferred to the cassandra peer causing "Checksum didn't match" error on the peer as shown below.
{code:java}
org.apache.cassandra.streaming.StreamReceiveException: java.io.IOException: Checksum didn't match (expected: 1967481976, actual: 2124622197)
        at org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:60)
        at org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:38)
        at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53)
        at org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:172)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: Checksum didn't match (expected: 1967481976, actual: 2124622197)
        at org.apache.cassandra.db.streaming.CompressedInputStream.maybeValidateChecksum(CompressedInputStream.java:206)
        at org.apache.cassandra.db.streaming.CompressedInputStream.loadNextChunk(CompressedInputStream.java:157)
        at org.apache.cassandra.db.streaming.CompressedInputStream.reBuffer(CompressedInputStream.java:121)
        at org.apache.cassandra.io.util.RebufferingInputStream.read(RebufferingInputStream.java:90)
        at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
        at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:62)
        at org.apache.cassandra.io.util.TrackedDataInputPlus.readFully(TrackedDataInputPlus.java:93)
        at org.apache.cassandra.db.marshal.ByteArrayAccessor.read(ByteArrayAccessor.java:101)
        at org.apache.cassandra.db.marshal.ByteArrayAccessor.read(ByteArrayAccessor.java:38)
        at org.apache.cassandra.db.marshal.AbstractType.read(AbstractType.java:500)
        at org.apache.cassandra.db.marshal.AbstractType.readArray(AbstractType.java:480)
        at org.apache.cassandra.db.ClusteringPrefix$Serializer.deserializeValuesWithoutSize(ClusteringPrefix.java:441)
        at org.apache.cassandra.db.Clustering$Serializer.deserialize(Clustering.java:164)
        at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeOne(UnfilteredSerializer.java:477)
        at org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:434)
        at org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:84)
        at org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:62)
        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
        at org.apache.cassandra.db.streaming.CassandraStreamReader$StreamDeserializer.hasNext(CassandraStreamReader.java:249)
        at org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:67)
        at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:205)
        at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48)
        at org.apache.cassandra.io.sstable.format.RangeAwareSSTableWriter.append(RangeAwareSSTableWriter.java:107)
        at org.apache.cassandra.db.streaming.CassandraStreamReader.writePartition(CassandraStreamReader.java:175)
        at org.apache.cassandra.db.streaming.CassandraCompressedStreamReader.read(CassandraCompressedStreamReader.java:98)
        at org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:84)
        at org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:53)
        ... 5 common frames omitted {code}