Bug: IllegalStateException when query on table having static columns during the Cassandra cluster upgrade from 3.11.4 to 4.0.11
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 05/Jul/24 03:57
Resolved: 10/Jul/24 00:06

We are upgrading Cassandra cluster from 3.11.4 to 4.0.11. This cluster has SSL enabled.
While performing upgrade on 1st DC, we observed below WARN/ERROR messages on C* 3 and C* 4 nodes.

+C*3 nodes:+


{noformat}
WARN  [ReadStage-1] 2024-06-11 08:04:09,088 AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread Thread[ReadStage-1,5,main]: {}
java.lang.IllegalStateException: [last_metadata_updt_ts, price_metadata] is not a subset of [price_metadata]

WARN  [ReadStage-1] 2024-06-19 05:10:31,226 AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread Thread[ReadStage-1,5,main]: {}
java.lang.IllegalStateException: [default_price_json, last_metadata_updt_ts, price_metadata] is not a subset of [price_metadata]
{noformat}


+C*4 nodes:+

{noformat}
ERROR [ReadStage-1] 2024-06-19 05:48:47,388 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[ReadStage-1,5,main]
java.lang.IllegalStateException: [last_metadata_updt_ts, price_metadata] is not a subset of [price_metadata]
{noformat}


Table definition for which above columns are associated is as below:


{noformat}
CREATE TABLE omni_price_ks_v2.location_price_mstr (
    tcin text,
    location_id bigint,
    price_change_id text,
    default_price_json text static,
    end_ts bigint,
    last_metadata_updt_ts bigint static,
    last_update_ts bigint,
    price_json text,
    price_metadata text static,
    price_type text,
    start_ts bigint,
    status text,
    version text,
    PRIMARY KEY (tcin, location_id, price_change_id)
) WITH CLUSTERING ORDER BY (location_id ASC, price_change_id ASC)
    AND bloom_filter_fp_chance = 0.1
    AND caching = {'keys': 'ALL', 'rows_per_partition': '100'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';
{noformat}

App team also observed below error in their application logs when try to read from this table.

{noformat}
{ "code": "ERR_GETPRICE_0034", "message": "Cassandra failure during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded, 1 failed)" }
{noformat}

Because of this error, the application is getting impacted during the upgrade.
Once the upgrade on all DCs is completed, this error stops.

I found below bug which matches our case.
https://issues.apache.org/jira/browse/CASSANDRA-17601

It seems like we are hitting some bug and hence raising this Jira.

Can you please have a look if this is still a bug and what would be the fix?

Let me know if you need any more details.