Bug: rpc read handler can get stuck on LruBlockCache#getBlock
Status: Open | Priority: Major | Resolution: 
Created: 10/Aug/23 07:42
Affects Version/s: 3.0.0-alpha-4

In our production, we found lots of read handlers got stuck on LruBlockCache#getBlock. This may be caused by a bug in jdk8 ConcurrentHashMap. To make common fast, I think we'd better get and check it before call ConcurrentHashMap#computeIfPresent.

 

 

"RpcServer.priority.RWQ.Fifo.scan.handler=190,queue=57,port=60020" #1807 daemon prio=5 os_prio=0 cpu=9703.28ms elapsed=88160.93s tid=0x00007f38d338a800 nid=0x8f4 waiting for monitor entry [0x00007f0af4baa000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.util.concurrent.ConcurrentHashMap.computeIfPresent(ConcurrentHashMap.java:1760)
        - waiting to lock <0x00007f2fc6495fe0> (a java.util.concurrent.ConcurrentHashMap$Node)
        at org.apache.hadoop.hbase.io.hfile.LruBlockCache.getBlock(LruBlockCache.java:538)
        at org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.getBlock(CombinedBlockCache.java:88)
        at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.getCachedBlock(HFileReaderImpl.java:1124)
        at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1300)
        at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$CellBasedKeyBlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:331)
        at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:679)
        at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:631)
        at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:315)
        at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:216)
        at org.apache.hadoop.hbase.regionserver.StoreFileScanner.backwardSeek(StoreFileScanner.java:561)
        at org.apache.hadoop.hbase.regionserver.ReversedKeyValueHeap.backwardSeek(ReversedKeyValueHeap.java:117)
        at org.apache.hadoop.hbase.regionserver.ReversedStoreScanner.backwardSeek(ReversedStoreScanner.java:134)
        at org.apache.hadoop.hbase.regionserver.ReversedStoreScanner.seekAsDirection(ReversedStoreScanner.java:94)
        at org.apache.hadoop.hbase.regionserver.StoreScanner.seekOrSkipToNextColumn(StoreScanner.java:821)
        at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:727)
        at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:155)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7515)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7683)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7447)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3403)
        - locked <0x00007f2ff1fc8f40> (a org.apache.hadoop.hbase.regionserver.ReversedRegionScannerImpl)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3662)
        at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45253)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:447)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:136)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)

   Locked ownable synchronizers:
        - None