Bug: Read fail with NullPointerException after migrating data from 2.2.19 to 3.0.30
Status: Triage Needed
Priority: Normal
Created: 16/May/24 22:25

After migrating data from 2.2.19 to 3.0.30, reading the legacy data in 3.0.30 would fail with the following exception.
{code:java}
ERROR [SharedPool-Worker-8] 2024-05-16 22:01:55,742 AbstractLocalAwareExecutorService.java:166 - Uncaught exception on thread Thread[SharedPool-Worker-8,10,main]
java.lang.RuntimeException: java.lang.NullPointerException
        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2656)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
        at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.NullPointerException: null
        at org.apache.cassandra.db.Slice$Bound.exclusiveStartOf(Slice.java:386)
        at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.readRow(UnfilteredDeserializer.java:570)
        at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:503)
        at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:329)
        at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.handlePreSliceData(SSTableIterator.java:105)
        at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:164)
        at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:336)
        at org.apache.cassandra.db.Slices$ArrayBackedSlices$1.prepareNext(Slices.java:506)
        at org.apache.cassandra.db.Slices$ArrayBackedSlices$1.hasNext(Slices.java:474)
        at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129)
        at org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:71)
        at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:799)
        at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:654)
        at org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:489)
        at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:417)
        at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1912)
        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2652)
        ... 5 common frames omitted {code}
h1. Reproduce

1. Start up cassandra-2.2.19, single node. Set column_index_size_in_kb in cassandra.yaml to 1.
{code:java}
// cassandra.yaml
column_index_size_in_kb: 1 {code}
2. Execute the following commands
{code:java}
CREATE KEYSPACE ks WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };
CREATE TABLE IF NOT EXISTS ks.tb (c0 TEXT,W_Col set<INT>,c2 INT,c3 INT,c4 INT,c5 set<TEXT>, PRIMARY KEY (c3, c2, c0, c4));
DELETE FROM ks.tb WHERE c3 = 2 AND c2 = 2;
INSERT INTO ks.tb (c5, W_Col, c0, c4, c3, c2) VALUES ({'nAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA','AAAAAAAAAAAAAAAAAAAAA','AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA','AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA','AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'},{0,1,2,3,4},'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA',1320973978,2,2);
DELETE FROM ks.tb WHERE c3 = 2 AND c2 = 1584505374 AND c0 = 'z';
ALTER TABLE ks.tb DROP W_Col ; {code}
3. Drain 2.2.19.
{code:java}
bin/nodetool -h ::FFFF:127.0.0.1 drain
bin/nodetool -h ::FFFF:127.0.0.1 stopdaemon;  {code}
4. Start up 3.0.30 (with default config is enough) and execute the following read command
{code:java}
cqlsh> SELECT *  FROM ks.tb WHERE c3 = 2 AND c2 = 100;
ReadFailure: Error from server: code=1300 [Replica(s) failed to execute read] message="Operation failed - received 0 responses and 1 failures" info={'failures': 1, 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}{code}
NPE Error in the log file
{code:java}
ERROR [SharedPool-Worker-8] 2024-05-16 22:01:55,742 AbstractLocalAwareExecutorService.java:166 - Uncaught exception on thread Thread[SharedPool-Worker-8,10,main]
java.lang.RuntimeException: java.lang.NullPointerException
    at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2656)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
    at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
    at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
    at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.NullPointerException: null
    at org.apache.cassandra.db.Slice$Bound.exclusiveStartOf(Slice.java:386)
    at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.readRow(UnfilteredDeserializer.java:570)
    at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:503)
    at org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:329)
    at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.handlePreSliceData(SSTableIterator.java:105)
    at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:164)
    at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:336)
    at org.apache.cassandra.db.Slices$ArrayBackedSlices$1.prepareNext(Slices.java:506)
    at org.apache.cassandra.db.Slices$ArrayBackedSlices$1.hasNext(Slices.java:474)
    at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129)
    at org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:71)
    at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:799)
    at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:654)
    at org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:489)
    at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:417)
    at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1912)
    at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2652)
    ... 5 common frames omitted {code}
 

I attached (1) the log file and (2) a data file (start up 3.0.30 with this folder and execute the read command can directly reproduce it).