Bug: BootstrapNodeManager "failed to request region server" warnings
Status: Open | Priority: Major | Resolution: 
Created: 07/Apr/22 23:16
Affects Version/s: 2.6.0

Warnings from HBASE-26220 (Use P2P communicate between region servers to sync the list for bootstrap node) in normal operation with some write load applied:

{noformat}
[01]2022-04-07T23:14:21,365 WARN  [BootstrapNodeManager] regionserver.BootstrapNodeManager: failed to request region server ip-172-31-51-4.us-west-2.compute.internal,8120,1649373044119
[01]org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Call to address=ip-172-31-51-4.us-west-2.compute.internal:8120 failed on local exception: org.apache.hadoop.hbase.exceptions.Connectio
nClosedException: Connection closed
[01]    at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:215) ~[hbase-client-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
[01]    at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:387) ~[hbase-client-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
[01]    at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:93) ~[hbase-client-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
[01]    at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:424) ~[hbase-client-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
[01]    at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:419) ~[hbase-client-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
[01]    at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:119) ~[hbase-client-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
[01]    at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:134) ~[hbase-client-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
[01]    at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.cleanupCalls(NettyRpcDuplexHandler.java:207) ~[hbase-client-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
[01]    at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelInactive(NettyRpcDuplexHandler.java:215) ~[hbase-client-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
[01]    at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[hbase-shaded-netty-4.1.0.jar:?]
[01]    at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[hbase-shaded-netty-4.1.0.jar:?]
[01]    at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[hbase-shaded-netty-4.1.0.jar:?]
[01]    at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:392) ~[hbase-shaded-netty-4.1.0.jar:?]
[01]    at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:357) ~[hbase-shaded-netty-4.1.0.jar:?]
[01]    at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[hbase-shaded-netty-4.1.0.jar:?]
[01]    at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[hbase-shaded-netty-4.1.0.jar:?]
[01]    at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[hbase-shaded-netty-4.1.0.jar:?]
[01]    at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) ~[hbase-shaded-netty-4.1.0.jar:?]
[01]    at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277) ~[hbase-shaded-netty-4.1.0.jar:?]
[01]    at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[hbase-shaded-netty-4.1.0.jar:?]
[01]    at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[hbase-shaded-netty-4.1.0.jar:?]
[01]    at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[hbase-shaded-netty-4.1.0.jar:?]
[01]    at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) ~[hbase-shaded-netty-4.1.0.jar:?]
[01]    at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[hbase-shaded-netty-4.1.0.jar:?]
[01]    at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[hbase-shaded-netty-4.1.0.jar:?]
[01]    at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) ~[hbase-shaded-netty-4.1.0.jar:?]
[01]    at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813) ~[hbase-shaded-netty-4.1.0.jar:?]
[01]    at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[hbase-shaded-netty-4.1.0.jar:?]
[01]    at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[hbase-shaded-netty-4.1.0.jar:?]
[01]    at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) ~[hbase-shaded-netty-4.1.0.jar:?]
[01]    at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[hbase-shaded-netty-4.1.0.jar:?]
[01]    at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[hbase-shaded-netty-4.1.0.jar:?]
[01]    at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[hbase-shaded-netty-4.1.0.jar:?]
[01]    at java.lang.Thread.run(Thread.java:829) ~[?:?]
[01]Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Connection closed
 {noformat}