tjake/Solandra — Issue #164: TL and TI  huge rows
Author: shubhammmt
Created: 2012-02-26 11:03:02+00:00
State: open

I have a Solandra setup with around 300K docs and 78 indexed fields. After a initial full import I saw that both TL and TI row sizes are pretty large specifically TL.Saw through the code(IndexWriter : addDoc) that all the terms are written inside a single row in TL which will make it pretty large anyways over time. Would like to get an understanding and reasoning of this data model.

The problem I am facing with this is by any delta or full import causes the nodes a continuous GC activity dropping any read / writes etc leaving it unusable.this may be because of lot wide rows getting in and out of memory while read and writes are happening on the same nodes.
I am using solr:3.3 with cassandra:0.8.6.

Below is the cfstat for the same:

Keyspace: system
Read Count: 19
Read Latency: 8.130157894736843 ms.
Write Count: 31
Write Latency: 0.03029032258064516 ms.
Pending Tasks: 0
Column Family: NodeIdInfo
SSTable count: 1
Space used (live): 4516
Space used (total): 4516
Number of Keys (estimate): 128
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 0
Read Latency: NaN ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Key cache capacity: 1
Key cache size: 0
Key cache hit rate: NaN
Row cache: disabled
Compacted row minimum size: 87
Compacted row maximum size: 103
Compacted row mean size: 103

Column Family: HintsColumnFamily
SSTable count: 2
Space used (live): 9880
Space used (total): 34550
Number of Keys (estimate): 256
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 5
Read Count: 10
Read Latency: NaN ms.
Write Count: 28
Write Latency: NaN ms.
Pending Tasks: 0
Key cache capacity: 2
Key cache size: 2
Key cache hit rate: NaN
Row cache: disabled
Compacted row minimum size: 125
Compacted row maximum size: 149
Compacted row mean size: 149

Column Family: Schema
SSTable count: 1
Space used (live): 192414
Space used (total): 192414
Number of Keys (estimate): 128
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 3
Read Latency: NaN ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Key cache capacity: 1
Key cache size: 1
Key cache hit rate: NaN
Row cache: disabled
Compacted row minimum size: 104
Compacted row maximum size: 8239
Compacted row mean size: 7794

Column Family: Migrations
SSTable count: 1
Space used (live): 300650
Space used (total): 300650
Number of Keys (estimate): 128
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 0
Read Latency: NaN ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Key cache capacity: 1
Key cache size: 0
Key cache hit rate: NaN
Row cache: disabled
Compacted row minimum size: 263211
Compacted row maximum size: 315852
Compacted row mean size: 315852

Column Family: IndexInfo
SSTable count: 0
Space used (live): 0
Space used (total): 0
Number of Keys (estimate): 0
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 0
Read Latency: NaN ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Key cache capacity: 1
Key cache size: 0
Key cache hit rate: NaN
Row cache: disabled
Compacted row minimum size: 0
Compacted row maximum size: 0
Compacted row mean size: 0

Column Family: LocationInfo
SSTable count: 2
Space used (live): 11496
Space used (total): 11496
Number of Keys (estimate): 256
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 2
Read Count: 6
Read Latency: NaN ms.
Write Count: 3
Write Latency: NaN ms.
Pending Tasks: 0
Key cache capacity: 2
Key cache size: 2
Key cache hit rate: NaN
Row cache: disabled
Compacted row minimum size: 73
Compacted row maximum size: 310
Compacted row mean size: 146

----------------
Keyspace: L
Read Count: 343746
Read Latency: 5.11407672525644 ms.
Write Count: 541480
Write Latency: 0.023806109182241266 ms.
Pending Tasks: 0
Column Family: FC
SSTable count: 4
Space used (live): 353742759
Space used (total): 370460012
Number of Keys (estimate): 512
Memtable Columns Count: 64604
Memtable Data Size: 65810913
Memtable Switch Count: 4
Read Count: 93
Read Latency: NaN ms.
Write Count: 138640
Write Latency: NaN ms.
Pending Tasks: 0
Key cache: disabled
Row cache: disabled
Compacted row minimum size: 104
Compacted row maximum size: 10090808
Compacted row mean size: 2650773

Column Family: Docs
SSTable count: 9
Space used (live): 1645341386
Space used (total): 1726113393
Number of Keys (estimate): 356608
Memtable Columns Count: 65268
Memtable Data Size: 27459963
Memtable Switch Count: 4
Read Count: 74566
Read Latency: 1.995 ms.
Write Count: 4688
Write Latency: NaN ms.
Pending Tasks: 0
Key cache: disabled
Row cache: disabled
Compacted row minimum size: 87
Compacted row maximum size: 51012
Compacted row mean size: 5557

Column Family: SI
SSTable count: 4
Space used (live): 88904425
Space used (total): 105431969
Number of Keys (estimate): 288512
Memtable Columns Count: 1141
Memtable Data Size: 767070
Memtable Switch Count: 4
Read Count: 8473
Read Latency: 0.615 ms.
Write Count: 2447
Write Latency: NaN ms.
Pending Tasks: 0
Key cache: disabled
Row cache: disabled
Compacted row minimum size: 104
Compacted row maximum size: 25109160
Compacted row mean size: 4694

Column Family: TL
SSTable count: 5
Space used (live): 334547547
Space used (total): 348394559
Number of Keys (estimate): 640
Memtable Columns Count: 90495
Memtable Data Size: 8602591
Memtable Switch Count: 3
Read Count: 5236
Read Latency: 30.199 ms.
Write Count: 2335
Write Latency: NaN ms.
Pending Tasks: 0
Key cache: disabled
Row cache: disabled
Compacted row minimum size: 454827
Compacted row maximum size: 186563160
Compacted row mean size: 77287206

Column Family: TI
SSTable count: 10
Space used (live): 3056043044
Space used (total): 3084009723
Number of Keys (estimate): 8494720
Memtable Columns Count: 181003
Memtable Data Size: 212200530
Memtable Switch Count: 4
Read Count: 255378
Read Latency: 0.962 ms.
Write Count: 393370
Write Latency: NaN ms.
Pending Tasks: 0
Key cache: disabled
Row cache: disabled
Compacted row minimum size: 104
Compacted row maximum size: 7007506
Compacted row mean size: 633