Bug: High Local read latency for few tables
Status: Triage Needed
Priority: Normal
Created: 20/May/20 16:06

Hi Team,

I am seeing high Local read latency for 3 tables in node(its 5 node cluster) and keyspace has total 16 sstables and hitting 10 sstables for read from that table, can you please suggest any path forward to fix read latency. Appreciate your help.. Thanks

Cassandra version : 3.11.3

SSTable Hitratio:

==============

k2view_usp/service_network_element_relation histograms
Percentile SSTables Write Latency Read Latency Partition Size Cell Count
 (micros) (micros) (bytes)
50% 3.00 0.00 219.34 179 10
75% 7.00 0.00 315.85 179 10
95% 10.00 0.00 454.83 179 10
98% 10.00 0.00 545.79 215 10
99% 10.00 0.00 545.79 310 20
Min 0.00 0.00 51.01 43 0
Max 10.00 0.00 545.79 89970660 8409007

 

TABLE STATS:

==========

Table: service_network_element_relation_mirTable: service_network_element_relation_mir SSTable count: 3 Space used (live): 283698097 Space used (total): 283698097 Space used by snapshots (total): 0 Off heap memory used (total): 5335824 SSTable Compression Ratio: 0.39563345719027554 Number of partitions (estimate): 2194136 Memtable cell count: 0 Memtable data size: 0 Memtable off heap memory used: 0 Memtable switch count: 0 Local read count: 0 Local read latency: NaN ms Local write count: 0 Local write latency: NaN ms Pending flushes: 0 Percent repaired: 100.0 Bloom filter false positives: 0 Bloom filter false ratio: 0.00000 Bloom filter space used: 4567016 Bloom filter off heap memory used: 4566992 Index summary off heap memory used: 705208 Compression metadata off heap memory used: 63624 Compacted partition minimum bytes: 104 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 154 Average live cells per slice (last five minutes): NaN Maximum live cells per slice (last five minutes): 0 Average tombstones per slice (last five minutes): NaN Maximum tombstones per slice (last five minutes): 0 Dropped Mutations: 0

 

 

Table: service_network_element_relationTable: service_network_element_relation SSTable count: 11 Space used (live): 8067239427 Space used (total): 8067239427 Space used by snapshots (total): 0 Off heap memory used (total): 143032693 SSTable Compression Ratio: 0.21558247949161227 Number of partitions (estimate): 29357598 Memtable cell count: 2714 Memtable data size: 691617 Memtable off heap memory used: 0 Memtable switch count: 9 Local read count: 6369399 Local read latency: 0.311 ms Local write count: 161229 Local write latency: NaN ms Pending flushes: 0 Percent repaired: 99.91 Bloom filter false positives: 1508 Bloom filter false ratio: 0.00012 Bloom filter space used: 113071680 Bloom filter off heap memory used: 113071592 Index summary off heap memory used: 27244541 Compression metadata off heap memory used: 2716560 Compacted partition minimum bytes: 43 Compacted partition maximum bytes: 89970660 Compacted partition mean bytes: 265 Average live cells per slice (last five minutes): 1.1779891304347827 Maximum live cells per slice (last five minutes): 103 Average tombstones per slice (last five minutes): 1.0 Maximum tombstones per slice (last five minutes): 1 Dropped Mutations: 0

 

Table: service_relationTable: service_relation SSTable count: 7 Space used (live): 281354042 Space used (total): 281354042 Space used by snapshots (total): 35695068 Off heap memory used (total): 6423276 SSTable Compression Ratio: 0.17685515178431085 Number of partitions (estimate): 1719400 Memtable cell count: 1150 Memtable data size: 67482 Memtable off heap memory used: 0 Memtable switch count: 3 Local read count: 5506327 Local read latency: 0.182 ms Local write count: 5237 Local write latency: 0.084 ms Pending flushes: 0 Percent repaired: 55.48 Bloom filter false positives: 17 Bloom filter false ratio: 0.00000 Bloom filter space used: 5549664 Bloom filter off heap memory used: 5549608 Index summary off heap memory used: 737348 Compression metadata off heap memory used: 136320 Compacted partition minimum bytes: 87 Compacted partition maximum bytes: 4055269 Compacted partition mean bytes: 351 Average live cells per slice (last five minutes): 1.5555555555555556 Maximum live cells per slice (last five minutes): 3 Average tombstones per slice (last five minutes): 1.0 Maximum tombstones per slice (last five minutes): 1 Dropped Mutations: 0