Bug: Add FastPath feature to HBase RWQueueRpcExecutor
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Dec/21 06:53

In ticket [HBASE-17808|https://issues.apache.org/jira/browse/HBASE-17808], the author introduced a fastpath implementation for RWQueueRpcExecutor. It aggregated 3 different independent RpcExecutor to implement the mechanism. This redundancy costed more memory and from its own performance test, it cannot outperform the original implementation. This time, I directly extended RWQueueRpcExecutor to implement the fast path mechanism. From my test result, it has a better queue time performance than before.

YCSB Test:

Constant Configurations:
hbase.regionserver.handler.count: 1000
hbase.ipc.server.callqueue.read.ratio: 0.5
hbase.ipc.server.callqueue.handler.factor: 0.2

Test Workload:

YCSB: 50% Write, 25% Get, 25% Scan. Max Scan length: 1000.
Client Threads: 100


||FastPathRWQueueRpcExecutor||RWQueueRpcExecutor||
|[OVERALL], RunTime(ms), 909365
[OVERALL], Throughput(ops/sec), 5498.3422498116815
[TOTAL_GCS_PS_Scavenge], Count, 1208
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 8006
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.8803945610398465
[TOTAL_GCS_PS_MarkSweep], Count, 2
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 96
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.010556817119638429
[TOTAL_GCs], Count, 1210
[TOTAL_GC_TIME], Time(ms), 8102
[TOTAL_GC_TIME_%], Time(%), 0.8909513781594849
[READ], Operations, 1248885
[READ], AverageLatency(us), 14080.154160711354
[READ], MinLatency(us), 269
[READ], MaxLatency(us), 180735
[READ], 95thPercentileLatency(us), 29775
[READ], 99thPercentileLatency(us), 39391
[READ], Return=OK, 1248885
[CLEANUP], Operations, 200
[CLEANUP], AverageLatency(us), 311.78
[CLEANUP], MinLatency(us), 1
[CLEANUP], MaxLatency(us), 59647
[CLEANUP], 95thPercentileLatency(us), 26
[CLEANUP], 99thPercentileLatency(us), 173
[INSERT], Operations, 1251067
[INSERT], AverageLatency(us), 14235.898240461942
[INSERT], MinLatency(us), 393
[INSERT], MaxLatency(us), 204159
[INSERT], 95thPercentileLatency(us), 29919
[INSERT], 99thPercentileLatency(us), 39647
[INSERT], Return=OK, 1251067
[UPDATE], Operations, 1249582
[UPDATE], AverageLatency(us), 14166.923049467741
[UPDATE], MinLatency(us), 321
[UPDATE], MaxLatency(us), 203647
[UPDATE], 95thPercentileLatency(us), 29855
[UPDATE], 99thPercentileLatency(us), 39551
[UPDATE], Return=OK, 1249582
[SCAN], Operations, 1250466
[SCAN], AverageLatency(us), 30056.68854251135
[SCAN], MinLatency(us), 787
[SCAN], MaxLatency(us), 509183
[SCAN], 95thPercentileLatency(us), 57823
[SCAN], 99thPercentileLatency(us), 74751
[SCAN], Return=OK, 1250466|[OVERALL], RunTime(ms), 958763
[OVERALL], Throughput(ops/sec), 5215.053146606617
[TOTAL_GCS_PS_Scavenge], Count, 1264
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 8680
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.9053332262509086
[TOTAL_GCS_PS_MarkSweep], Count, 1
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 38
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.00396344039142103
[TOTAL_GCs], Count, 1265
[TOTAL_GC_TIME], Time(ms), 8718
[TOTAL_GC_TIME_%], Time(%), 0.9092966666423298
[READ], Operations, 1250961
[READ], AverageLatency(us), 14663.084518222391
[READ], MinLatency(us), 320
[READ], MaxLatency(us), 204415
[READ], 95thPercentileLatency(us), 30815
[READ], 99thPercentileLatency(us), 43071
[READ], Return=OK, 1250961
[CLEANUP], Operations, 200
[CLEANUP], AverageLatency(us), 366.845
[CLEANUP], MinLatency(us), 1
[CLEANUP], MaxLatency(us), 70719
[CLEANUP], 95thPercentileLatency(us), 36
[CLEANUP], 99thPercentileLatency(us), 80
[INSERT], Operations, 1248183
[INSERT], AverageLatency(us), 14334.938754974231
[INSERT], MinLatency(us), 390
[INSERT], MaxLatency(us), 2828287
[INSERT], 95thPercentileLatency(us), 30271
[INSERT], 99thPercentileLatency(us), 41919
[INSERT], Return=OK, 1248183
[UPDATE], Operations, 1250212
[UPDATE], AverageLatency(us), 14283.836318960304
[UPDATE], MinLatency(us), 337
[UPDATE], MaxLatency(us), 2828287
[UPDATE], 95thPercentileLatency(us), 30255
[UPDATE], 99thPercentileLatency(us), 41855
[UPDATE], Return=OK, 1250212
[SCAN], Operations, 1250644
[SCAN], AverageLatency(us), 33153.01709839091
[SCAN], MinLatency(us), 742
[SCAN], MaxLatency(us), 645119
[SCAN], 95thPercentileLatency(us), 62879
[SCAN], 99thPercentileLatency(us), 80447
[SCAN], Return=OK, 1250644|

In the metrics screenshot, the first interval is the performance of RWQueueRpcExecutor and the second interval is FastPathRWQueueRpcExecutor.