av9ash/GitBugs huggingface dataset:
Bug: Introduce an AdaptiveFastPathRWRpcExecutor to make the W/R/S separations fit various workloads 
Status: Open | Priority: Major | Resolution: 
Created: 05/Jul/23 09:18

We currently use the FastPathRWQueueRpcExecutor. But the numbers of read/write handlers are fixed, which make the RegionServer performance not so good in our prod env.

The logic of it is described below:
 * The basic architecture is same as FastPathRWRpcExecutor.
 * Introduced a float shared_ratio (0, 1.0), to indicate the ratio of the number of shared handlers. (for example, when we set the ratio to 0.2, if we have 100 handlers, 50 for write, 25 for get, 25 for scan, there will be 10 + 5 + 5 shared handlers and 40 isolated handlers for write, 20 for get and 20 for scan).
 * Shared handler could run all the three kinds of requests.
 * Shared handler will be shared only when it is idle.
 * Shared handler is also bounded to a kind of RPCQueue, it will process the requests in that queue first.

This improvement will improve the resource utility under various workloads and guarantee a level of R/W/S isolation for requests processing at the same time.
Bug: CompactionServer cause the loss of HFile references in snapshot
Status: Open | Priority: Major | Resolution: 
Created: 14/Jul/23 10:27

We encountered an issue with the loss of HFile references in the snapshot after enabling the CompactionServer feature in our cluster.

The relevant log fragment is approximately as follows:
{code:java}
File does not exist: /.../.tmp/data/default/......../b72cab4efb074defb1bd9acd9087891f
File does not exist: /....../archive/data/default/........../b72cab4efb074defb1bd9acd9087891f
File does not exist: /....../data/default/......./b72cab4efb074defb1bd9acd9087891f {code}
From the displayed HDFS logs, we observed that this HFile 'b72cab4efb074defb1bd9acd9087891f' was renamed by the CompactionServer and RegionServer, and eventually deleted by the HMaster.
{code:java}
2022-07-13,00:50:01,727 INFO FSNamesystem.audit:  
cmd=rename   operator=CompactionServer
src=/...../data/default/...../b72cab4efb074defb1bd9acd9087891f dst=/....../archive/data/default/....../b72cab4efb074defb1bd9acd9087891f 

2022-07-13,00:51:23,802 INFO FSNamesystem.audit:  
cmd=rename 
operator=RegionServer
src=/....../archive/data/default/....../b72cab4efb074defb1bd9acd9087891f
dst=/....../archive/data/default/....../b72cab4efb074defb1bd9acd9087891f.1657644683801 

2022-07-13,01:51:57,823 INFO FSNamesystem.audit:        
cmd=delete     
operator=HMaster
src=/....../archive/data/default/....../b72cab4efb074defb1bd9acd9087891f.1657644683801
 {code}
 

Based on HBASE-26722 and HBASE-22163, we understand that if a region A on RS1 is not closed, and another RS2 (in this case, CompactionServer) opens the same region A, it may trigger an "archived" state. Consequently, when RS1 closes this region, it will be archived again, resulting in the deletion of the HFile from the archived directory. As a result, the snapshot will lose its reference to the HFile.
Bug: MetaRWQueueRpcExecutor should extend FastPathRWQueueRpcExecutor
Status: Open | Priority: Major | Resolution: 
Created: 11/Jul/23 19:23

Introduced in HBASE-22280, MetaRWQueueRpcExecutor protects the cluster from a case where heavy read volume to meta causes a delay in writes to meta. Delay in writes to meta can affect RIT time, among other things. 

At the end of the comments in HBASE-22280, some concerns were raised about a perf regression in throughput for meta. Reverting to FastPathRWQueueRpcExecutor fixed the regression, and we've also noticed in other testing that fast path is a somewhat universally good optimization. We should change MetaRWQueueRpcExecutor to extend it so that one can take advantage of the isolation it provides and the speed of fast path.
Bug: Replication stuck when switch to new reader
Status: Open | Priority: Major | Resolution: 
Created: 06/Jul/23 03:08
Affects Version/s: 2.4.17

After creating new reader for next WAL, it immediately seek() to the  currentPositionOfEntry, but this position may be spill over the length of current WAL.
{code:java}
WARN  [RpcServer.default.FPRWQ.Fifo.read.handler=101,queue=1,port=16020.replicationSource.wal-reader.XXXXXXX] regionserver.ReplicationSourceWALReader: Failed to read stream of replication entries
java.io.EOFException: Cannot seek after EOF
        at org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1488)
        at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:62)
        at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.seekOnFs(ProtobufLogReader.java:495)
        at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.seek(ReaderBase.java:138)
        at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.seek(WALEntryStream.java:399)
        at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.openReader(WALEntryStream.java:341)
        at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.handleFileNotFound(WALEntryStream.java:328)
        at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.openReader(WALEntryStream.java:347)
        at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.openNextLog(WALEntryStream.java:310)
        at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.checkReader(WALEntryStream.java:300)
        at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:176)
        at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:102)
        at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.tryAdvanceStreamAndCreateWALBatch(ReplicationSourceWALReader.java:260)
        at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:142) {code}
Bug: Fix ChaosMonkey documentation code block rendering
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Aug/23 10:53

The code blocks in document for ChaosMonkey isnot rendered correctly. Fix them and also add few more example. See 
 * [https://github.com/apache/hbase/blob/master/src/main/asciidoc/_chapters/developer.adoc#1054-chaosmonkey-without-ssh] or
 * [https://hbase.apache.org/book.html#_chaosmonkey_without_ssh]
Bug: RSProcedureDispatcher to log the request details during retries
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Aug/23 03:43

As of today, RSProcedureDispatcher only logs the exception details for the given RPC request, however it does not log any other details. We should log:
 * whether the request is for region open/close
 * proc id, proc class names
 * region name

 

Sample log without any of the above mentioned details:
{code:java}
2023-08-25 02:21:02,821 WARN [ispatcher-pool-40777] procedure.RSProcedureDispatcher - request to rs1,61020,1692930044498 failed due to java.io.IOException: Call to address=rs1:61020 failed on local exception: java.io.IOException: org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException: org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed, try=0, retrying... {code}
Bug: Synchronous replication should check if the file system supports truncate API
Status: Open | Priority: Minor | Resolution: 
Created: 18/Jul/23 21:03

Ok. I missed this but I was just told that the synchronous replication leverages the truncate() FS API.

https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/replication/SyncReplicationReplayWALManager.java#L282

Ozone does not implement truncate so calling this method on the WAL FS will result in an exception. It would be a better user experience to alert user that this is not supported from the start.
Bug: Change the label from InterfaceAudience.Private to InterfaceAudience.LimitedPrivate for org.apache.hadoop.hbase.Server.java 
Status: Open | Priority: Minor | Resolution: 
Created: 07/Jul/23 04:05
Affects Version/s: 2.4.4

Currently the class org.apache.hadoop.hbase.Server.java is marked InterfaceAudience.Private.

{{{}This prevents in getting shared zookeeper watcher (ZKWatcher) instances from org.apache.hadoop.hbase.regionserver.RegionServerServices.java (extends org.apache.hadoop.hbase.Server.java{}}}{{{}) using the method getZooKeeper().{}}}

 

{{This creates a problem in writing custom Coprocessors because now we don't have a shared ZKWatcher instance.}}

 

{{The proposed solution is to use the InterfaceAudience.LimitedPrivate. }}
Bug: Add hbase.normalizer.merge.merge_request_max_number_of_regions property to limit max number of regions in a merge request for merge normalization
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Sep/23 13:46
Affects Version/s: 2.4.0

In our production environment, while investigating an issue, we observed that the Noramlizer had scheduled one single merge procedure to an RS providing 27K+ empty regions of a table (this was a result of a failed copy table job that left 27K+ empty regions of the table) to merge.

This action led the procedure to go to stuck state and eventually the procedure framework bailed out after ~40mins. This was happening with each normalizer run until we deleted the table manually.


Logs

Normalizer triggers a merge procedure

normalizer.RegionNormalizerWorker - NormalizationTarget[regionInfo=\{ENCODED => 6e8606335a62f6bafceb017dc7edfdf5, NAME => 'TEST.TEST_TABLE,XXXX.', STARTKEY => 'XXXX', ENDKEY => 'YYYY'},{*}regionSizeMb=0{*}], NormalizationTarget[regionInfo=\{ENCODED => 79607df308d7618e632abe8a12c1bf6b, NAME => 'TEST.TEST_TABLE,XXXX', STARTKEY => 'XXYY', ENDKEY => 'YYZZ'},{*}regionSizeMb=0]{*}]] resulting in *pid 21968356*

procedure immediately gets stuck

procedure2.ProcedureExecutor - Worker *stuck* PEWorker-56(pid=21968356), run time 12.4850 sec

Finally fails after ~40 mins

procedure2.ProcedureExecutor - Worker *stuck* PEWorker-56(pid=21968356), run time *40 mins, 58.055 sec*


Bails out with RuntimeException

procedure2.ProcedureExecutor - force=false
java.lang.UnsupportedOperationException: pid=21968356, state=FAILED:MERGE_TABLE_REGIONS_UPDATE_META, locked=true, exception=java.lang.{*}RuntimeException via CODE-BUG: Uncaught runtime exception{*}: pid=21968356, state=RUNNABLE:MERGE_TABLE_REGIONS_UPDATE_META, locked=true; MergeTableRegionsProcedure table=TEST.TEST_TABLEXXXX, {*}regions={*}{*}[269a1b168af497cce9ba6d3d581568f2{*}
.
.
.
.
*27K+ regions printed here]*
Bug: Move snapshot storage from filesystem to a separated HBase table
Status: Open | Priority: Minor | Resolution: 
Created: 27/Sep/23 13:04

As we know, rename and list are very expensive operations on object storage. Currently, the snapshot in hbase relies on these two operations. For example, when taking snapshot, we first write snapshot description and data manifest file to a temporary directory ,then commit it by a rename operation. When list all snapshots, we will scan the snapshot directory to find all completed snapshots.

So maybe we can try to introduce a new snapshot storage, using hbase table to store it.
Here are a few points from which maybe we can gain benefits:
1. make hbase easier to deploy on object storage, like s3
2. will make snapshots faster and more lightweight. In the current filesystem-based snapshot implementation, when consolidating snapshot manifest, we will first list all region manifests with a thread pool, read content and then delete them. When the number of regions is large, this process may take a lot of time. In comparison, the read and write operations of hbase tables are more lightweight than the read and write operations of hdfs files.
3. more likely to reduce hdfs small files
Bug: NPE on initialization error in RecoveredReplicationSourceShipper
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Sep/23 13:14
Affects Version/s: 2.4.17

When we run into problems starting RecoveredReplicationSourceShipper, we try to stop the reader thread which we haven't initialized yet, resulting in an NPE.
{noformat}
ERROR org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Unexpected exception in redacted currentPath=hdfs://redacted
java.lang.NullPointerException
        at org.apache.hadoop.hbase.replication.regionserver.RecoveredReplicationSourceShipper.terminate(RecoveredReplicationSourceShipper.java:100)
        at org.apache.hadoop.hbase.replication.regionserver.RecoveredReplicationSourceShipper.getRecoveredQueueStartPos(RecoveredReplicationSourceShipper.java:87)
        at org.apache.hadoop.hbase.replication.regionserver.RecoveredReplicationSourceShipper.getStartPosition(RecoveredReplicationSourceShipper.java:62)
        at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.lambda$tryStartNewShipper$3(ReplicationSource.java:349)
        at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
        at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.tryStartNewShipper(ReplicationSource.java:341)
        at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.initialize(ReplicationSource.java:601)
        at java.lang.Thread.run(Thread.java:750)
{noformat}
A simple null check should fix this.
Bug: Drop duplicate test class TestShellRSGroups.java
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Sep/23 18:47

Just noticed that {{TestShellRSGroups.java}} is at {{hbase-shell/src/test/rsgroup/org/apache/hadoop/hbase/client/rsgroup/TestShellRSGroups.java,}} but ideally it should be at {{hbase-shell/src/test/java/org/apache/hadoop/hbase/client/rsgroup/TestShellRSGroups.java}} instead.
Also because of same misplacement spotless skipped this file. Also need to run spotless for the same.
Bug: Update tech details of backup
Status: Open | Priority: Minor | Resolution: 
Created: 12/Sep/23 16:01

The [technical details|https://hbase.apache.org/book.html#br.technical.details] of HBase backup mention that the backup works by copying WALs which are converted to HFiles during the restore process.

However, the code seems to imply that WALs are already converted to HFiles during the backup process. (See IncrementalTableBackupClient#convertWALsToHFiles.) I suspect the docs still describe the old behavior, which (I suspect) was changed in HBase-14141.

Knowing this also confirms that bulk-imported HFiles are also backed-up using the same mechanism (see IncrementalTableBackupClient#handleBulkLoad) - something I managed to test with experimentation, but wasn't explained in the backup docs.

In short, this section in the docs requires an update.
Bug: Record the replication start timestamp for tables/peers
Status: Open | Priority: Minor | Resolution: 
Created: 22/Aug/23 01:06
Affects Version/s: 2.5.5

Currently, the peer info showed on the UI miss the start timestamp for the replications, we need the start timestamp (e.g. the create time of the peer, the time when tables added to peers) to distinguish if the replication is continuous or discontinuous.
Bug: some syntax errors detected by IDEA in pom.xml
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Jul/23 03:54
Affects Version/s: 2.2.7

When I open the project by IDEA,  there are some syntax error detected in pom.xml, which can confuse developers. Has anyone else meet this? 

It seems does not affect compilation, but I don't know why the red highlight comes here, annoying...

!image-2023-07-18-11-54-47-997.png|width=840,height=171!

 

!image-2023-07-18-11-55-15-582.png|width=445,height=110!
Bug: The logStats about LruBlockCache is not accurate
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Aug/23 14:15
Affects Version/s: 2.4.13

LruBlockCache.logStats would print info, as follow:
{code:java}
//代码占位符
INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=2.42 MB, freeSize=3.20 GB, max=3.20 GB, blockCount=14, accesses=31200, hits=31164, hitRatio=99.88%, , cachingAccesses=31179, cachingHits=31156, cachingHitsRatio=99.93%, evictions=426355, evicted=0, evictedPerRun=0.0 {code}
I think the description about *totalSize=2.42 MB* is not accurate, It actually represents the used size of BlockCache.
Bug: change the random scope of random row for PerformanceEvaluation
Status: Open | Priority: Minor | Resolution: 
Created: 01/Jul/23 10:28
Affects Version/s: 2.3.2

*question description:*

when we use PerformanceEvaluation tool to run randomWrite test, we find out that when one region happened regiontoobusy, the requests in the ui will dramatically decreased in a short time from several million to 0. 

the mechanism of regiontoobusy is really nice to maximum the throughput of hbase cluster, it will only influence the current region which happened regiontoobusy. but when i look into the whole procedure of the random write(include client and server), i found there may be some problem in PerformanceEvaluation tool(client) which cause the current question.

 

*cause of the issue:*

before trying to illustrate the issue, here are some preconditions we need to know first:

1、one request generated by TestClient thread will contain the data of all regions(we will accumulate 2M(default client buffer size) request, the request will include many mutate operations. each mutate operation is generated in random from a whole table)

2、when one 2M's request doesn't finish, the TestClient won't generate a new request(due to the PE's mechanism)

let's try to illustrate the cause of the issue:

1、when one region happened regiontoobusy, the 2M's request will not be finished util the region unblocked. so do the other 2M's request generated by other TestClients.

2、both the client(PerformanceEvaluation) and server(regionserver) will block and we will see the request in the ui and cpu util decreased in several seconds util the unblock of the region.

*probable solution：*

the issue is not due to regionserver(regiontoobusy is a good mechanism from the regionserver's side), but the client. 

if we try to change the scope of random, we will find a way to solve this problem.

the origin scope is the whole table, if every TestClient generate random row in it's own scope we will solve this problem, let's take a example:

assume that we have 5 TestClient, each TestClient charge for 1000 requests.

TestClient1 will generate random rows from 0–999;

TestClient2 will generate random rows from 1000–1999;

TestClient3 will generate random rows from 2000–2999;

TestClient4 will generate random rows from 3000--3999;

TestClient5 will generate random rows from 4000–4999;

 

*some other words to say:*

i raise a question encountered in my work. i hope to have a further discussion with the experts from the community and other fields in order to seek a better solution. if my solution is acceptable, i will push a patch to solve the issue.
Bug: add durability setting on PerformanceEvaluation
Status: Open | Priority: Minor | Resolution: 
Created: 01/Jul/23 07:50
Affects Version/s: 2.3.2

I wish we add a durability setting on PerformanceEvaluation. Although we can use writeToWAL setting to implement SKIP_WAL and SYNC_WAL, but sometimes it's not enough. we will hope to set other durabilities, such as ASYNC_WAL、FSYNC_WAL. i suggest we could add a separate durability setting.
Bug: Bump exec-maven-plugin to 3.1.0
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 27/Jul/23 07:21

I frequently see IOException in hbase-shaded-with-hadoop-check-invariants make-sure-validation-files-are-in-sync phase. I'm not sure what is the root cause but it worth to bump the plugin version to the latest.
{noformat}
[INFO] --- exec-maven-plugin:1.6.0:exec (make-sure-validation-files-are-in-sync) @ hbase-shaded-with-hadoop-check-invariants ---
[ERROR] Command execution failed.
java.io.IOException: Stream closed
    at java.lang.ProcessBuilder$NullOutputStream.write (ProcessBuilder.java:433)
    at java.io.OutputStream.write (OutputStream.java:116)
    at java.io.BufferedOutputStream.flushBuffer (BufferedOutputStream.java:82)
    at java.io.BufferedOutputStream.flush (BufferedOutputStream.java:140)
    at java.io.FilterOutputStream.close (FilterOutputStream.java:158)
    at org.apache.commons.exec.DefaultExecutor.closeProcessStreams (DefaultExecutor.java:306)
    at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:387)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:166)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:804)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:751)
    at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:313)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:190)
    at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:186)
    at java.util.concurrent.FutureTask.run (FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511)
    at java.util.concurrent.FutureTask.run (FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
    at java.lang.Thread.run (Thread.java:748)
{noformat}
Bug: correct span name in AbstractRpcBasedConnectionRegistry#getActiveMaster
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 13/Sep/23 02:22
Affects Version/s: 3.0.0-alpha-4

It looks like that the span name does not correspond to what is actually done.
 
public CompletableFuture<ServerName> getActiveMaster() {
return tracedFuture(
() -> this
.<GetActiveMasterResponse> call(
(c, s, d) -> s.getActiveMaster(c, GetActiveMasterRequest.getDefaultInstance(), d),
GetActiveMasterResponse::hasServerName, "getActiveMaster()")
.thenApply(resp -> ProtobufUtil.toServerName(resp.getServerName())),
getClass().getSimpleName() + ".getClusterId");
}
Bug: The javadoc about RegionProcedureStore.delete is incorrect
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 30/Aug/23 13:59
Affects Version/s: 2.4.13

As shown in the following figure.
!image-2023-08-30-21-54-59-999.png!
 
Actually, we would fill the {color:#ff0000}*proc:d*{color} column with an empty byte array when calling RegionProcedureStore.delete().

!image-2023-08-30-21-57-32-393.png!
Bug: NPE in AddPeerProcedure recovery
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 23/Jul/23 13:26
Affects Version/s: 4.0.0-alpha-1

AddPeerProcedure will restore syncReplicationPeerLock when replayed in master recovery, however the replicationPeerManager has not been initialized when replay procedure, which will cause a nullPointerException and master to abort.
{code:java}
@Override
protected void afterReplay(MasterProcedureEnv env) {
  // ......
  if (peerConfig.isSyncReplication()) {
    if (!env.getReplicationPeerManager().tryAcquireSyncReplicationPeerLock()) {
      throw new IllegalStateException(
        "Can not acquire sync replication peer lock for peer " + peerId);
    }
  }
}  {code}