Bug: full repair getting failed for 1 large table version 3.11.2
Status: Resolved
Priority: Urgent
Resolution: Not A Bug
Created: 28/Feb/20 13:01
Resolved: 29/Feb/20 21:19

nodetool repair -pr test table1
[2020-02-28 17:56:00,192] Starting repair command #1 (7a634f80-5a25-11ea-8b5a-3b5edc400697), repairing keyspace test with repair options (parallelism: parallel, primary range: true, incremental: true, job threads: 1, ColumnFamilies: [table1], dataCenters: [], hosts: [], # of ranges: 256, pull repair: false)
[2020-02-28 17:56:17,173] Repair session 7ae020a1-5a25-11ea-8b5a-3b5edc400697 for range [(-1398494450650640175,-1398484603620354999]] finished (progress: 1%)
[2020-02-28 17:56:22,633] Repair session 7ad3c491-5a25-11ea-8b5a-3b5edc400697 for range [(-2055080187802596770,-2055069591393124910]] finished (progress: 2%)
[2020-02-28 17:56:23,888] Repair session 7ace4651-5a25-11ea-8b5a-3b5edc400697 for range [(-5916076129714268351,-5916054112636649354]] finished (progress: 2%)
[2020-02-28 17:56:27,675] Repair session 7ad83161-5a25-11ea-8b5a-3b5edc400697 for range [(3262419392837600448,3262453314317655259]] finished (progress: 3%)
[2020-02-28 17:56:31,674] Repair session 7ad32851-5a25-11ea-8b5a-3b5edc400697 for range [(2146167582342729236,2146207858477993492]] finished (progress: 3%)
[2020-02-28 17:56:52,964] Repair session 7af15eb0-5a25-11ea-8b5a-3b5edc400697 for range [(-7088663015651548617,-7088601571353808807]] finished (progress: 3%)
[2020-02-28 17:57:03,237] Repair session 7acf30b2-5a25-11ea-8b5a-3b5edc400697 for range [(-5392084277480362928,-5391992322619247178]] finished (progress: 4%)
[2020-02-28 17:57:08,937] Repair session 7ada5443-5a25-11ea-8b5a-3b5edc400697 for range [(-2247846360993532740,-2247745485112718002]] finished (progress: 4%)
[2020-02-28 17:57:14,102] Repair session 7ae1a740-5a25-11ea-8b5a-3b5edc400697 for range [(4804514170425290662,4804611019437533765]] finished (progress: 5%)
[2020-02-28 17:57:18,190] Repair session 7ae6fe71-5a25-11ea-8b5a-3b5edc400697 for range [(4989724097640378549,4989803009764075856]] finished (progress: 5%)
[2020-02-28 17:57:27,549] Repair session 7acfccf1-5a25-11ea-8b5a-3b5edc400697 for range [(4512226173347132723,4512312572318114003]] finished (progress: 5%)
[2020-02-28 17:57:37,534] Repair session 7ae1ce50-5a25-11ea-8b5a-3b5edc400697 for range [(7146381406763943813,7146516394221309657]] finished (progress: 6%)
Feb 28, 2020 5:59:00 PM ClientCommunicatorAdmin Checker-run
WARNING: Failed to check the connection: java.net.SocketTimeoutException: Read timed out
[2020-02-28 17:59:36,419] Repair session 7aeacf01-5a25-11ea-8b5a-3b5edc400697 for range [(4717535938324360043,4717683452072792471]] finished (progress: 6%)

Current setup Details:

Allocated Memory: 128gb

No of nodes:12 DC1 and 12 DC2
|*Sr No*|*Parameter Name*|*Old value*|*New Value*|*File Name*|
|1|Min Heap Size -Xms|-Xms32G|-Xms64G|jvm.options|
|2|Max Heap Size -Xmx|-Xms32G|-Xms64G|jvm.options|
|3|Young generation size -Xmn|#-Xmn800M|-Xmn3072M|jvm.options|
|4|GC SETTINGS|CMS|G1GC|jvm.options|
|5|Disk Read_ahead_kb|4096|64|/sys/class/block/sdd/queue/read_ahead_kb|
|6|Disk Rotational|1|1|/sys/class/block/sdd/queue/rotational|
|7|block devices attributes|8192|128|blockdev --setra 128 /dev/sdd (To set it)|
|8|commitlog_segment_size_in_mb|32|64|cassandra.yaml|
|9|memtable_heap_space_in_mb|1/4 the size of the heap|8192|cassandra.yaml|
|10|memtable_offheap_space_in_mb|1/4 the size of the heap|8192|cassandra.yaml|
|11|phi_convict_threshold|#8|12|cassandra.yaml|