av9ash/GitBugs huggingface dataset:
Bug: StoreFileInfo should set its initialPath in a consistent way
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/May/22 16:21
Affects Version/s: 2.5.0

Currently, StoreFileInfo provides overloaded public constructors where the related file path can be passed as either a Path or FileStatus instance. This can lead to the StoreFileInfo instances related to the same file entry to have different representations of the file path, which could create problems for functions relying on equality for comparing store files. One example I could find is the StoreEngine.refreshStoreFiles method, which list some files from the SFT, then compares against a list of files from the SFM to decide how it should update SFM internal cache. Here's a sample output from the TestHStore.testRefreshStoreFiles:

-------

2022-05-10T15:06:42,831 INFO [Time-limited test] regionserver.StoreEngine(399): Refreshing store files for org.apache.hadoop.hbase.regionserver.DefaultStoreEngine@69d58ac1 files to add: [file:/hbase/hbase-server/target/test-data/e3eac5ce-9bdf-8624-bcec-09c89790d682/TestStoretestRefreshStoreFiles/data/default/table/da6a3cf38941b37cd16438d554b13bbc/family/6e92c2f5cf1f40f7b8c6b6b34a176fa5, file:/hbase/hbase-server/target/test-data/e3eac5ce-9bdf-8624-bcec-09c89790d682/TestStoretestRefreshStoreFiles/data/default/table/da6a3cf38941b37cd16438d554b13bbc/family/{*}fa4d5909da644d94873cbfdc6b5a07da{*}] files to remove: [/hbase/hbase-server/target/test-data/e3eac5ce-9bdf-8624-bcec-09c89790d682/TestStoretestRefreshStoreFiles/data/default/table/da6a3cf38941b37cd16438d554b13bbc/family/{*}fa4d5909da644d94873cbfdc6b5a07da{*}]

-------

The above will wrongly add it to SFM's list of compacted files, making a valid file potentially eligible for deletion and data loss.

I think we can avoid that by always converting Path instances passed in StoreFileInfo constructors to a FileStatus, for consistently build the internal StoreFileInfo path.
Bug: HBaseInterClusterReplicationEndpoint should honor replication adaptive timeout
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jun/22 15:25

HBASE-23293 introduced replication.source.shipedits.timeout which is adaptive, ReplicationSourceShipper#shipEdits() set the adaptive timeout based on retries. But on CallTimeoutException in HBaseInterClusterReplicationEndpoint#replicate(), it keep retrying the replication after sleep with same timeout value.
Bug: Hbase SecureBulkload permission regression
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/May/22 13:09
Affects Version/s: 2.5.0

HBASE-26707 has introduced a bug, where setting the permission of the bulk loaded HFile to 777 is made conditional.

However, as discussed in HBASE-15790, that permission is essential for HBase's correct operation.
Bug: two phase bulkload is broken when SFT is in use.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/May/22 04:13
Affects Version/s: 2.5.0

In HBASE-26707 for the SFT case, we are writing files directly to the region location. For that we are using HRegion.regionDir as the staging directory. The problem is that in reality, this dir is pointing to the WAL dir, so for S3 deployments that would be pointing to the hdfs. As the result during the execution of LoadIncrementalHFiles the process failed with the exception:
{noformat}
2022-05-24 03:31:23,656 ERROR org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager: Failed to complete bulk load
java.lang.IllegalArgumentException: Wrong FS hdfs://ns1//hbase-wals/data/default/employees/4f367b303da4fed7667fff07fd4c6066/department/acd971097924463da6d6e3a15f9527da -expected s3a://hbase
        at org.apache.hadoop.fs.s3native.S3xLoginHelper.checkPath(S3xLoginHelper.java:224)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.checkPath(S3AFileSystem.java:1375)
        at org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:647)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.makeQualified(S3AFileSystem.java:1337)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.qualify(S3AFileSystem.java:1363)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3521)
        at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:511)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:397)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:387)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
        at org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$SecureBulkLoadListener.prepareBulkLoad(SecureBulkLoadManager.java:397)
        at org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:6994)
        at org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$1.run(SecureBulkLoadManager.java:291)
        at org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$1.run(SecureBulkLoadManager.java:266)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:360)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1879)
        at org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager.secureBulkLoadHFiles(SecureBulkLoadManager.java:266)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.bulkLoadHFile(RSRpcServices.java:2453)
        at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45821)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:392)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:140)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:359)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:339)

 {noformat}
Bug: Under compaction mark may leak
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jun/22 05:05
Affects Version/s: 2.4.12

HBASE-26249 introduced an under compaction mark to reduce repeatedly compactions for the same stores for a short period of time after bulk loading files.

Since the mark adding and removing are in difference threads,
{code:java}
pool.execute(
  new CompactionRunner(store, region, compaction, tracker, completeTracker, pool, user));
if (LOG.isDebugEnabled()) {
  LOG.debug(
    "Add compact mark for store {}, priority={}, current under compaction "
      + "store size is {}",
    getStoreNameForUnderCompaction(store), priority, underCompactionStores.size());
}
underCompactionStores.add(getStoreNameForUnderCompaction(store)); {code}
 it happens that the concurrently of thread-1 using underCompactionStores.add() and thread-2 which running a CompactionRunner that using underCompactionStores.remove(). If the removing happens before adding, the mark will leak.
Bug: Non-interactive missing for the hbase-shell
Status: Patch Available | Priority: Major | Resolution: 
Created: 14/Jun/22 01:31
Affects Version/s: 2.4.12

During test 2.4.12, we found a lot of our integration testing failure for the non-interactive mode in hbase-shell, looks like in 2.4.12, we removed the "shell.eval_io" for the non-interactive shell, this caused the input/output handle behavior for shell is the same for interactive and non-interactive which caused unexpected error, for example:

 

```

echo -e "disable_all '.*'\ny" | hbase shell -n


=====Old version 2.4.4=====

t1

Disable the above 1 tables (y/n)?

1 tables successfully disabled
Took 0.7811 seconds



=====New version 2.4.12=====


hbase:001:0> disable_all '.*'t1
test_tableDisable the above 2 tables (y/n)?
Took 0.5270 seconds  

NoMethodError: undefined method `chomp' for nil:NilClass
                     command at uri:classloader:/shell/commands/disable_all.rb:41
                command_safe at uri:classloader:/shell/commands.rb:49
  translate_hbase_exceptions at uri:classloader:/shell/commands.rb:122
                command_safe at uri:classloader:/shell/commands.rb:49
....
                         run at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:427
                      <main> at classpath:/jar-bootstrap.rb:223}}

```
Bug: Add a tool command list_unknownservers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jun/22 14:59
Affects Version/s: 3.0.0-alpha-3

Sometimes, RegionServer will become an unknown server. It is necessary to add a tool command to list unknown servers.
!image-2022-06-10-15-49-33-178.png!
Bug: Add a config that allows us to configure region-level storage policies
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jun/22 10:29

<property>
    <name>hbase.hregion.block.storage.policy</name>
    <value>HOT|ALL_SSD|...</value>
  </property>
With this config, we can set region-level storage policies.

 

We have this config about CF storage policy:
{code:java}
 <property>
    <name>hbase.hstore.block.storage.policy</name>
    <value>ALL_SSD</value>
  </property>
{code}
But in addition to CF, we also have some other path under region path, such like .splits, 

recovered.edits, .tmp .

So i want to add a region-level config which could cover the whole path, and if you have any other requirements for the cf directory, you can continue to use the previous hbase.hstore.block.storage.policy to specify them separately.
Bug: Log archiving fail cause regionserver abort
Status: In Progress | Priority: Major | Resolution: 
Created: 18/Jun/22 08:49
Affects Version/s: 2.3.4

RegionServer log

java.io.IOException: Unable to rename hdfs://clusterA/hbase/WALs/hostmame1,16020,1647853003113/hostmame1%2C16020%2C1647853003113.default.1655296917964 to hdfs://clusterA/hbase/oldWALs/hostmame1,16020,1647853003113/hostmame1%2C16020%2C1647853003113.default.1655296917964

NameNode log

[2022-06-15T20:51:49.277+08:00] [INFO] [IPC Server handler 0 on 8021] : Starting log segment at 25945020756
[2022-06-15T20:51:58.050+08:00] [WARN] [IPC Server handler 180 on 8020] : DIR* FSDirectory.unprotectedRenameTo: failed to rename /h base/WALs/hostmame1,16020,1647853003113/hostmame1%2C16020%2C16478530031 13.default.1655296917964 to /hbase/oldWALs/hostmame1,16020,1647853003113/HTYD-clusterA-11-13-112-165.hado op.jd.local%2C16020%2C1647853003113.default.1655296917964 because destination's parent does not exist
Bug: Eliminate MOB renames when SFT is enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Apr/22 15:10
Affects Version/s: 2.5.0

MOB file compaction and flush still relies on renames even when SFT is enabled.

My proposed changes are:
 * when requireWritingToTmpDirFirst is false during mob flush/compact instead of using the temp writer we should create a different writer using a {color:#000000}StoreFileWriterCreationTracker that writes directly to the mob store folder{color}
 * {color:#000000}these StoreFileWriterCreationTracker should be stored in the MobStore. This would requires us to extend MobStore with a createWriter and a finalizeWriter method to handle this{color}
 * {color:#000000}refactor {color}MobFileCleanerChore to run on the RS instead on Master to allow access to the {color:#000000}StoreFileWriterCreationTracker{color}s to make sure the currently written files are not cleaned up
Bug: [HBCK2] extraRegionsInMeta does not work If RegionInfo is null
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jun/22 00:07
Affects Version/s: hbase-operator-tools-1.2.0

extraRegionsInMeta will not clean/fix meta if info:regioninfo columns is missing.

 

Somehow, the customer has the following empty row in meta as a stale. 
'I1xxxxxx,16332508xxxxx.xxxxf53609cc1ae366b43205dxxxxxxx', 'info:state', 163333223xxxx
 

And no corresponding table "I1xxxxxx" exist. 

 

We use extraRegionsInMeta but it didn't clean. Also, we created same table again and used extraRegionsInMeta after removing HDFS data but the stale row never cleaned. It looks extraRegionsInMeta works only when "info:regioninfo" is present. 

 

We need to handle the scenario for other columns I.e info:state, info:server, etc
Bug: TestQuotaThrottle times out
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jun/22 22:25
Affects Version/s: 2.5.0

With branch-2.5, TestQuotaThrottle times out. Need to investigate.

 
h3. Error Message

Failed after attempts=7, exceptions: 2022-06-03T11:26:33.418Z, RpcRetryingCaller\{globalStartTime=2022-06-03T11:26:33.418Z, pause=250, maxAttempts=7}, org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/master 2022-06-03T11:26:33.418Z, RpcRetryingCaller\{globalStartTime=2022-06-03T11:26:33.418Z, pause=250, maxAttempts=7}, org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/master 2022-06-03T11:26:33.418Z, RpcRetryingCaller\{globalStartTime=2022-06-03T11:26:33.418Z, pause=250, maxAttempts=7}, org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/master 2022-06-03T11:26:33.418Z, RpcRetryingCaller\{globalStartTime=2022-06-03T11:26:33.418Z, pause=250, maxAttempts=7}, org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/master 2022-06-03T11:26:33.418Z, RpcRetryingCaller\{globalStartTime=2022-06-03T11:26:33.418Z, pause=250, maxAttempts=7}, org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/master 2022-06-03T11:26:33.418Z, RpcRetryingCaller\{globalStartTime=2022-06-03T11:26:33.418Z, pause=250, maxAttempts=7}, org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/master 2022-06-03T11:26:33.418Z, RpcRetryingCaller\{globalStartTime=2022-06-03T11:26:33.418Z, pause=250, maxAttempts=7}, org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/master
Bug: The filenum in AbstractFSWAL should be monotone increasing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/May/22 04:49

This is the current code

{code}
  /**
   * retrieve the next path to use for writing. Increments the internal filenum.
   */
  private Path getNewPath() throws IOException {
    this.filenum.set(EnvironmentEdgeManager.currentTime());
    Path newPath = getCurrentFileName();
    while (fs.exists(newPath)) {
      this.filenum.incrementAndGet();
      newPath = getCurrentFileName();
    }
    return newPath;
  }
{code}

In some tests, we inject our own EnvironmentEdge, it may return the same ts always or even go backwards, the logic here is not rnough to keep the filenum monotone increasing, as we may have already archive the old file.
Bug: Phoenix PartialIndexRebuilderIT regression on 2.4.11
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 29/Apr/22 14:38
Affects Version/s: 2.4.11

We have noticed that using HBase 2.4.11 reliably breaks
_PartialIndexRebuilderIT.testConcurrentUpsertsWithRebuild_
and makes _ConcurrentMutationsExtendedIT.testConcurrentUpserts_ very flakey.

The same tests run file with earlier versions, including HBase 2.4.10.

At best this is a behaviour change in an HBase minor version, at worst this is a plain regression.
Bug: Server side scanner time limit should account for time in queue
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/May/22 16:16

When a scan request comes in with a timeout specified and heartbeats/partials allowed, we calculate a time limit for running the scan to be half of that timeout. The idea is to return before the timeout expires.

The calculation of that time limit is "now + timeout / 2", where now is the point at which the scan is starting to run. What's missed here is the scan may have spent upwards of a few seconds in the IPC queue before being serviced. In this case, the time limit may extend beyond the timeout of the request and the server will not return in time.

We should calculate the time limit from ServerCall.getReceiveTime instead to avoid these timeouts.
Bug: TestReplicationValueCompressedWAL is flaky
Status: Open | Priority: Major | Resolution: 
Created: 27/Jun/22 02:41

https://nightlies.apache.org/hbase/HBase-Flaky-Tests/master/3752/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.replication.regionserver.TestReplicationValueCompressedWAL-output.txt

{noformat}
2022-06-26T08:41:27,359 DEBUG [RS_REFRESH_PEER-regionserver/jenkins-hbase19:0-0.replicationSource,2.replicationSource.wal-reader.jenkins-hbase19.apache.org%2C44659%2C1656232862372,2] wal.ProtobufLogReader(448): Encountered a malformed edit, seeking back to last good position in file, from 65537 to 65536
java.io.EOFException: Partial PB while reading WAL, probably an unexpected EOF, ignoring. current offset=65537
	at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:390) ~[classes/:?]
	at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:104) ~[classes/:?]
	at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:92) ~[classes/:?]
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:264) ~[classes/:?]
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:178) ~[classes/:?]
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:103) ~[classes/:?]
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:212) ~[classes/:?]
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:148) ~[classes/:?]
{noformat}

Actually, we should still have a lot of edits after this point. This could  be a critical problem as we will hang replication forever...
Bug: Scanner timeout should take precedence over rpc timeout
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 20/Jun/22 19:23
Affects Version/s: 2.4.12

In the new async client (see AsyncScanSingleRegionRpcRetryingCaller.call()), we ignore hbase.rpc.timeout for scanner next calls. The reason is:
{quote}// As we have a call sequence for scan, it is useless to have a different rpc timeout which is
// less than the scan timeout. If the server does not respond in time(usually this will not
// happen as we have heartbeat now), we will get an OutOfOrderScannerNextException when
// resending the next request and the only way to fix this is to close the scanner and open a
// new one.
{quote}
In branch-2's blocking client (i.e. ClientScanner/ScannerCallable) we make use of RpcRetryingCallerImpl, which calculates the timeout for each call based on the smaller of the two: remaining operation timeout or configured rpc timeout. For scans, the operation timeout is dictated by hbase.client.scanner.timeout.period, which should take precedence over the configured rpc timeout per the above comment.

We can fix this by using RpcRetryingCallerFactory.newCaller(int) in ClientScanner, passing the scanner timeout period in as the argument.
Bug: Allow configuring a separate timeout for meta scans
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/May/22 16:35

There is a {{hbase.client.meta.operation.timeout}} but it does not apply to meta scans, which are the primary use-case for clients (i.e. through RegionLocator). 

Many user-facing clients may want to have low rpc and scan timeouts. However, in periods of meta hotspotting, those timeouts can be way too low for the meta scans. The problem with low timeouts for meta scans is that without a populated MetaCache, user requests cannot succeed. In fact, user requests will continually try to re-scan meta until the MetaCache is populated. So having a lower rpc timeout will cause a situation where meta scans cannot succeed, and thus user requests cannot succeed. In this case I think it'd be preferable to relax the rpc timeout for meta requests so that a few long requests can unblock many faster requests.

My suggestion would be to add an {{hbase.client.meta.rpc.timeout}} and ensure that it applies to meta scans. I also think it would be less confusing to have {{hbase.client.meta.operation.timeout}} apply as the scanner timeout period for meta scans.
Bug: TestSeparateClientZKCluster is flaky
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jun/22 15:03

https://nightlies.apache.org/hbase/HBase-Flaky-Tests/master/3773/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestSeparateClientZKCluster-output.txt

{noformat}
org.apache.hadoop.hbase.exceptions.MasterStoppedException: null
	at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:3177) ~[classes/:?]
	at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1954) ~[classes/:?]
	at org.apache.hadoop.hbase.master.MasterRpcServices.balance(MasterRpcServices.java:743) ~[classes/:?]
	at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) ~[hbase-protocol-shaded-3.0.0-alpha-4-SNAPSHOT.jar:3.0.0-alpha-4-SNAPSHOT]
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:385) ~[classes/:?]
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) ~[classes/:?]
	at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:104) ~[classes/:?]
	at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:84) ~[classes/:?]
{noformat}

I think the problem is that, MasterStoppedException is a sub class of DoNotRetryIOException, so when hitting this issue, we will fail immediately.

And the client zk syncer is asynchoronous, so it is possible that when we call admin.balance, we haven't synced the new location yet, and it will throw the MasterStoppedException out soon and fail the UT.

Let me see how to fix it.
Bug: disable_peer stuck
Status: Open | Priority: Major | Resolution: 
Created: 29/Jun/22 12:29

!image-2022-06-29-20-30-53-545.png!

 

hbase(main):001:0> list_procedures
 PID Name State Submitted Last_Update Parameters
 201155 org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure SUCCESS 2022-06-29 17:38:26 +0800 2022-06-29 17:38:29 +0800 [\{"peerId"=>"circe_to_phoebe", "type"=>"DISABLE_PEER", "targetServer"=>{"hostName"=>"htyd-circe-11-13-109-241.hadoop.jd.local", "port"=>16020, "startCode"=>"1650537882474"}}]
 201154 org.apache.hadoop.hbase.master.replication.DisablePeerProcedure WAITING 2022-06-29 17:38:26 +0800 2022-06-29 17:38:26 +0800 [\{"state"=>[1, 2, 3, 8]}, \{"peerId"=>"circe_to_phoebe"}, {}]
 201159 org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure RUNNABLE 2022-06-29 17:38:26 +0800 2022-06-29 17:38:26 +0800 [\{"peerId"=>"circe_to_phoebe", "type"=>"DISABLE_PEER", "targetServer"=>{"hostName"=>"hostname1", "port"=>16020, "startCode"=>"1650537882461"}}]
 201158 org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure RUNNABLE 2022-06-29 17:38:26 +0800 2022-06-29 17:38:26 +0800 [\{"peerId"=>"circe_to_phoebe", "type"=>"DISABLE_PEER", "targetServer"=>{"hostName"=>"hostname2", "port"=>16020, "startCode"=>"1650537882485"}}]
 201157 org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure SUCCESS 2022-06-29 17:38:26 +0800 2022-06-29 17:38:30 +0800 [\{"peerId"=>"circe_to_phoebe", "type"=>"DISABLE_PEER", "targetServer"=>{"hostName"=>"hostname3", "port"=>16020, "startCode"=>"1650537882484"}}]
 201156 org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure SUCCESS 2022-06-29 17:38:26 +0800 2022-06-29 17:38:29 +0800 [\{"peerId"=>"circe_to_phoebe", "type"=>"DISABLE_PEER", "targetServer"=>{"hostName"=>"hostname4", "port"=>16020, "startCode"=>"1650537882482"}}]

RS jstack :  

!image-2022-06-29-20-53-32-300.png!

!image-2022-06-29-20-54-37-919.png!