av9ash/GitBugs huggingface dataset:
Bug: Do not need to maintain rollback step when root procedure does not support rollback
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Nov/23 04:01

In HBASE-28210, we need to synchronized on RootProcedureState when updating procedure store, which will decrease the performance a lot when there are lots of sub procedures for a procedure, such as SCP, so it will impact MTTR.
But actually, SCP does not support rollback, so I do not think we need to maintain this information. If so, I think we do not need the synchronization added in HBASE-28210, at least for SCP then. This could get the performance back.
Bug: The flush table operation encountered an NPE and failed.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Nov/23 03:49
Affects Version/s: 3.0.0-beta-1

Executing a merge region before the flush table is completed will result in an NPE error for the flush table and cause it to fail

Triggering logic
 # Execute the flush table command.
 # Merge regions before the flush is completed.
 # After the merge is completed, the two parent regions will be cleaned up.
 # An NPE will be reported when the flush operation is performed on that parent region.
Bug: Leak in ExportSnapshot during verifySnapshot on S3A
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Nov/23 23:15

Each S3AFileSystem creates an S3AInstrumentation and various metrics sources, with no real way to disable that. In HADOOP-18526, a bug was fixed so that these are not leaked. But in order to use that, you must call S3AFileSystem.close() when done.

In ExportSnapshot, ever since HBASE-12819 we set fs.impl.disable.cache to true. It looks like that was added in order to prevent conflicting calls to close() between mapper and main thread when running in a single JVM.

When verifySnapshot is enabled, SnapshotReferenceUtil.verifySnapshot iterates all storefiles (could be many thousands) and calls SnapshotReferenceUtil.verifyStoreFile on them. verifyStoreFile makes a number of static calls which end up in CommonFSUtils.getRootDir, which does Path.getFileSystem().

Since the FS cache is disabled, every single call to Path.getFileSystem() creates a new FileSystem instance. That FS is short lived, and gets GC'd. But in the case of S3AFileSystem, this leaks all of the metrics stuff.

We have two easy possible fixes:
 # Only set fs.impl.disable.cache when running hadoop in local mode, since that was the original problem.
 # When calling verifySnapshot, create a new Configuration which does not include the fs.impl.disable.cache setting.

I tested out #2 in my environment and it fixed the leak.
Bug: Graceful shutdown of active hmaster
Status: Open | Priority: Major | Resolution: 
Created: 14/Nov/23 17:19

We recently had an operational incident due to bad interplay between an ongoing maintenance of regionservers, and a new maintenance of hmasters. We'd been running a rolling restart of regionservers. Someone unknowingly started a rolling restart of hmasters. This caused long RITs on a few clusters, what happened was:
 * regionserver restart executed a region move
 * hmaster saw it and started TSRP, sending request to RS to close region
 * immediately after, the hmaster stopped due to that rolling restart
 * the regionserver saw the close request, closed the region, and tried to report state back to the hmaster.
 * This spammed tons of failures for the next 30+ secs while the new hmaster became active.
 * Finally the new hmaster started up and recovered the state and the RIT finished.

Long RITs are really painful and effectively downtime for hbase. I think we should have a shutdown hook on the hmaster which:
 * Sets state so new move requests are rejected
 * Waits for any existing move TSRP to finish
 * Shuts down
Bug: Add a check for getQueueStorage().hasData() in the getDeletableFiles method of ReplicationLogCleaner
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Nov/23 01:32
Affects Version/s: 3.0.0

There are a large number of oldwal files in the oldwal directory, and the default value of hbase.master.logcleaner.ttl is 10 minutes. Upon observation, it has been noticed that no oldwal files have been cleared.
Through analyzing the source code, it has been discovered that the following logic contains a problem. When LogCleaner executes the checkAndDeleteFiles method, it calls the getDeletableFiles method of ReplicationLogCleaner. One of the logics in this method is that if canFilter is false, it will directly return Collections.emptyList(), which means that the wal data filtered out by TimeToLiveLogCleaner will be filtered out. This in turn leads to the files under oldwal not being able to be cleared.
Bug: Use x86_64 protoc 2.5.0 on Aarch64 Mac
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 16/Nov/23 08:09
Affects Version/s: 2.4.17

Hbase as a profile for building on Aarch64 Linux, but it doesn't work on Mac.

We have solved the same problem on Phoenix by using x86_64 in this case via emulation.
Bug: Add Endpoint  and Method Name to COPROC_EXEC Spans
Status: Open | Priority: Major | Resolution: 
Created: 14/Nov/23 06:29

If we assume parentBased=on, then it's enough to add this information on the client side.
However, we may want to also add this on the server side for stochastic tracing.

We could call these:

_db.hbase.endpoint.name_
_db.hbase.endpoint.method_

or 

_db.hbase.coprocessor.name_
_db.hbase.coprocessor.method_
Bug: Enrich trace span information
Status: Open | Priority: Major | Resolution: 
Created: 14/Nov/23 06:13
Affects Version/s: 2.5.6

As I'm working on adding usable Trace data to Phoenix, I find that HBase trace spans are missing crucial information.

This an umbrella ticket to track those issues, and add crucial information that is needed for interpreting trace data.

The goal is to add the most useful data while keeping the Span data as compact as possible.
Bug: Meta browser can happen NPE when the server or target server of region is null
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Nov/23 14:10
Affects Version/s: 2.4.13

Reproduction Steps:

1. We create a new table with many regions , for example: 
{code:java}
//代码占位符
create 't01' , 'info' , SPLITS => Array(1..999).map{|e|e.to_s}{code}
2. At the same time, we scan meta entries of this table on meta browser by hbase UI

3. We would get NPE if those regions is initializing and the server or target server is null
Bug: RpcLogDetails' Message can become corrupt before log is consumed
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Oct/23 19:20

The RpcLogDetails class represents a slow (or large) log event which will later be consumed by the SlowLogQueueService.

Right now the RpcLogDetails' param field points to the slow call's Message, and this Message is backed by a CodedInputStream which may be overwritten before the given log is consumed. This overwriting of the Message may result in slow query payloads for which the metadata derived post-consumption is inaccurate.

To solve this bug I think we need to copy the Message in the RpcLogDetails constructor. I have this bug reproduced in a QA environment and will test out this idea and open a PR shortly if the test results are promising.
Bug: Narrow the netty3 dependency scope
Status: Open | Priority: Major | Resolution: 
Created: 06/Nov/23 03:47

Netty 3 has a bunch of CVEs and will never be fixed.

In HBase, we poll in netty 3 dependency through hadoop, and till hadop 3.3.6, the dependency is still there.

The only place for hadoop 3.1.x where we depend on netty 3 is in MR's ShuffleHandler.

https://issues.apache.org/jira/browse/HADOOP-15327

So I think at least wecould narrow the dependency scope for netty 3 to test scope, as it is only used in tests we start a MiniMRCluster.
Bug: Complete release hbase-connectors 1.0.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Oct/23 23:53

# Release the artifacts on repository.apache.org
 # Move the binaries from dist-dev to dist-release
 # Add xml to download page
 # Push tag 1.0.1RC as tag rel/1.0.1
 # Release 1.0.1 on JIRA https://issues.apache.org/jira/projects/HBASE/versions/12345436
 # Add release data on [https://reporter.apache.org/addrelease.html?hbase]
 # Send announcement email
Bug: Upgrade ZooKeeper for CVE-2023-44981
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 25/Oct/23 23:59

CVE-2023-44981 is a high scoring (9.1/10) authentication bypass vulnerability in ZooKeeper related to SASL quorum authentication. The bug is fixed in versions 3.7.2, 3.8.3, and 3.9.1. 
Upgrade ZK versions on all active branches. At least 3.7.2.
Bug: Remove ServerManager's rsAdmins map
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Oct/23 13:02
Affects Version/s: 2.5.5

On 2.x [the ServerManager registers admins in a HashMap|https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java]. This can result in thread safety issues — we recently observed an exception which caused a region to be indefinitely stuck in transition until we could manually intervene. We saw the following exception in the HMaster logs:
{code:java}
2023-10-11 02:20:05.213 [RSProcedureDispatcher-pool-325] ERROR org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher: Unexpected error caught, this may cause the procedure to hang forever
    java.lang.ClassCastException: class java.util.HashMap$Node cannot be cast to class java.util.HashMap$TreeNode (java.util.HashMap$Node and java.util.HashMap$TreeNode are in module java.base of loader 'bootstrap')
        at java.util.HashMap$TreeNode.moveRootToFront(HashMap.java:1900) ~[?:?]
        at java.util.HashMap$TreeNode.treeify(HashMap.java:2016) ~[?:?]
        at java.util.HashMap.treeifyBin(HashMap.java:768) ~[?:?]
        at java.util.HashMap.putVal(HashMap.java:640) ~[?:?]
        at java.util.HashMap.put(HashMap.java:608) ~[?:?]
        at org.apache.hadoop.hbase.master.ServerManager.getRsAdmin(ServerManager.java:723){code}
Bug: Make use of assertThrows in TestShadeSaslAuthenticationProvider
Status: Open | Priority: Major | Resolution: 
Created: 21/Oct/23 18:34

This JIRA isa to address review comments as given in https://github.com/apache/hbase/pull/5434 and make use of assertThrows in TestShadeSaslAuthenticationProvider instead of fail.

CC: [~zhangduo]
Bug: Fix hbase-connectors for release scripts 
Status: Open | Priority: Major | Resolution: 
Created: 18/Oct/23 04:55
Affects Version/s: hbase-connectors-1.0.1

During the release for hbase-connectors 1.0.1 HBASE-28161, I found several issues 

1. CHANGELOG.md was being used but our release script is using CHANGES.md to create new changes, see https://github.com/apache/hbase/blob/master/dev-support/create-release/release-util.sh#L578-L584
2. the do-release.sh script isn't working because of due 1/, I hacked it to make it work, but this would bring a backward compatible issues because no matter what the archive https://archive.apache.org/dist/hbase/hbase-connectors-1.0.0/ does not have the CHANGES.md for hbase-connectors.  

Proposing to fix it after the releases,
Bug: Intra-process client connections cause netty EventLoop deadlock
Status: Open | Priority: Major | Resolution: 
Created: 16/Oct/23 19:44

We've had a few operational incidents over the past few months where our HMaster stops accepting new connections, but can continue processing requests from existing ones. Finally I was able to get heap and thread dumps to confirm what's happening.

The core trigger is HBASE-24687, where the MobFileCleanerChore is not using ClusterConnection. I've prodded the linked PR to get that resolved and will take it over if I don't hear soon.

In this case, the chore is using the NettyRpcClient to make a local rpc call to the same NettyRpcServer in the process. Due to [NettyEventLoopGroupConfig|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/util/NettyEventLoopGroupConfig.java#L98], we use the same EventLoopGroup for both the RPC Client and the RPC Server.

What happens rarely is that the local client for MobFileCleanerChore gets assigned to RS-EventLoopGroup-1-1. Since we share the EventLoopGroupConfig, and [we don't specify a separate parent group|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java#L155], that group is also the group which processes new connections.

What we see in this case is that RS-EventLoopGroup-1-1 gets hung in Socket.accept. Since the client side is on the same EventLoop, it's tasks get stuck in a queue waiting for the executor. So the client can't send the request that the server Socket is waiting for.

Further, the client/chore gets stuck waiting on BlockingRpcCallback.get(). We use an HWT TimerTask to cancel overdue requests, but it only gets scheduled [once NettyRpcConnection.sendRequest0 is executed|https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcConnection.java#L371]. But sendRequest0 [executes on the EventLoop|https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcConnection.java#L393], and thus gets similarly stuck. So we never schedule a timeout and the chore gets stuck forever.

While fixing HBASE-24687 will fix this case, I think we should improve our netty configuration here so we can avoid problems like this if we ever do intra-process RPC calls again (there may already be others, not sure).
Bug: Replace scala.util.parsing.json with org.json4s.jackson which used in Spark too
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Oct/23 17:48
Affects Version/s: connector-1.0.0

In https://issues.apache.org/jira/browse/HBASE-28137 to support Spark 3.4 the scala-parser-combinators was added as direct dependency to HBase Spark Connector. 

This was needed as Spark 3.4 is not using scala-parser-combinators and it is not inherited as transitive dependency.  

But this solution has a disadvantage. As the HBase Spark Connector assembly jar does not include any 3rd party libraries the scala-parser-combinators must be added to the spark classpath for HBase Spark Connector to work.

A much better solution is to replace  scala.util.parsing.json with org.json4s.jackson which is used by Spark core, see https://github.com/apache/spark/blob/branch-3.4/core/pom.xml#L279-L280.
Bug: AbstractWALProvider may miss the WAL which is under creation in getWALs method
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Oct/23 11:24

Found this problem when analysis HBASE-28114.

In the AbstractFSWALProvider implementation, if wal field is null, we will acquire the creation lock so we can get the WAL instance which is under creation, so we will not miss the WAL instance, and also will not get an uninitialized WAL instance.

We should try to do the same for remote WAL, otherwise the logic in WALEntryStream, where we rely on the return value of getWALs to test whether a WAL file is currently being written, will be broken.
Bug: Specify -Xms for tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Oct/23 07:04

The default -Xms value is JVM dependent, but the host memory size is usually included in the calculation.

-Xms in turn is used to calculate some GC parameters, for example NewSize and OldSize, which affect the behaviour of tests.

As the memory consumption on the tests is not dependent on the host VM size, we could set -Xms for the tests explictly, and enjoy more consistent test results.
Bug: TestSyncTimeRangeTracker fails with OOM with small -Xms values
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Oct/23 15:53
Affects Version/s: 2.4.17

Edit2: It's not the OS, it's the -Xms value determined from the host memory size.

Edit: It's related to the OS and it's default java 8 , not to the processor architecture.

This test seems to be cutting real close to the heap size.
On ARM, it consistently fails on my RHEL8.8 Aarch64 VM with Java 8.
{noformat}
mvn test -P runDevTests -Dtest.build.data.basedirectory=/ram2G -Dhadoop.profile=3.0 -fn -B -Dtest=TestSyncTimeRangeTracker* -pl hbase-server
...
[ERROR] org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker.testConcurrentIncludeTimestampCorrectness  Time elapsed: 1.969 s  <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
{noformat}
It seems that Java on ARM has some higher memory overhead than x86_64.

Simply bumping -Xmx from the default 2200m to 2300m allows it to pass. 
{noformat}
mvn test -P runDevTests -Dtest.build.data.basedirectory=/ram2G -Dhadoop.profile=3.0 -fn -B -Dtest=TestSyncTimeRangeTracker* -pl hbase-server -Dsurefire.Xmx=2300m
...
[INFO] Running org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.395 s - in org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker
{noformat}
However, the real solution should be reducing the memory usage for this test.
Bug: Canary publish read failure fails with NPE if region location is null
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Oct/23 22:26
Affects Version/s: 2.5.5

Region with null server name causes canary failures while publishing read failure i.e. while updating perServerFailuresCount map:
{code:java}
2023-10-09 15:24:11 [CanaryMonitor-1696864805801] ERROR tool.Canary(1480): Sniff region failed!
java.util.concurrent.ExecutionException: java.lang.NullPointerException
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.apache.hadoop.hbase.tool.CanaryTool$RegionMonitor.run(CanaryTool.java:1478)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.NullPointerException
	at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1837)
	at org.apache.hadoop.hbase.tool.CanaryTool$RegionStdOutSink.incFailuresCountDetails(CanaryTool.java:327)
	at org.apache.hadoop.hbase.tool.CanaryTool$RegionStdOutSink.publishReadFailure(CanaryTool.java:353)
	at org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.readColumnFamily(CanaryTool.java:548)
	at org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.read(CanaryTool.java:587)
	at org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.call(CanaryTool.java:502)
	at org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.call(CanaryTool.java:470)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more {code}
Bug: Disable table may hang when SCP create TRSP but DTP didn't set tableState to disabling
Status: Open | Priority: Major | Resolution: 
Created: 07/Oct/23 09:25
Affects Version/s: 2.4.17

Consider the following case：

1. RegionServer stop or abort, ServerCrashProcedure start running;

2. Now, DisableTableProcedure start running but not set tableState to disabling;

3. ServerCrashProcedure create TransitStateProcedure with success, but can not running because of xLock had hold by DisableTableProcedure ;

4. DisableTable Procedure UnassignProcedure RPC fails because server is down.
Bug: Is there plan or timeline that HBase java client will fully support JDK 17?
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 09/Oct/23 02:39

We are using JDK 17 as our application programing platform,  but we found HBase document said java client only well supported {*}JDK 8{*}, may I ask whether there is any plan or timeline *JDK 17* can be supported by HBase java client?

[https://hbase.apache.org/book.html#basic.prerequisites]

Thanks.
Bug: Do not retry refreshSources when region server is already stopping
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Oct/23 12:17

Found this when debugging HBASE-28114, it is designed to retry forever but when aborting a region server, we will stop the rpc client, and it will retry a lot of times and output a lot of logs, which even cause the Jenkins job OOM…

I think we should throw the exception out directly and give up retry when aborting the region server.

==== Updated ====

It turns out that the problem is the retry when initialize new shipper failed. We should stop retrying when region server is already stopped.
Bug: Reject requests at RPC layer when RegionServer is aborting
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Oct/23 14:15

We recently had an operational incident where the RegionServer got aborted, but failed to exit within a reasonable timeframe. We're going to tune hbase.regionserver.abort.timeout much lower than the 20m default, but even with that it makes little sense to accept requests when the server is aborting.

In our case, the server was impaired and not processing requests. The call queue was full, so NettyRpcServer kept trying and failing to add requests to the queue. This results in CallQueueTooBigException, which is not a meta cache clearing exception. It continued throwing these exceptions for multiple minutes until we finally manually killed the server.

I'd like to add a check in ServerRpcConnection.processRequest, where we check if regionServer.isAborted() and throw a RegionServerAbortedException rather than attempt to enqueue the request.
Bug: TestTableShell times out when run in background
Status: Open | Priority: Major | Resolution: 
Created: 05/Oct/23 11:59
Affects Version/s: 2.4.17

When starting maven in the backround, TestTableShell hangs.

*mvn test -P runDevTests -Dtest.build.data.basedirectory=/ram2G -fn -pl hbase-shell -B:*
pass

*nohup mvn test -P runDevTests -Dtest.build.data.basedirectory=/ram2G -fn -pl hbase-shell -B*
pass

*nohup mvn test -P runDevTests -Dtest.build.data.basedirectory=/ram2G -fn -pl hbase-shell -B &*
hangs
 
Tests run via maven shouldn't behave differently depending on whether maven runs in the foreground or background.

Technically, the test doesn't hang forver, it times out after 10000 seconds and fails.
Bug: CatalogReplicaMode.LOAD_BALANCE introduces a client dependency on the active master
Status: Open | Priority: Major | Resolution: 
Created: 05/Oct/23 00:38
Affects Version/s: 2.4.17

This only a 2.x releases, and only usages of MetaTableAccessor. It's hard to trace all of the overloads of the scanMeta method in there, but it is at the very least exposed to clients via RegionLocator.getAllRegionLocations().

The code in AsyncTable and normal ConnectionImplementation meta scans (i.e. looking up a single location) both use a CatalogReplicaLoadBalanceSelector which fetches the meta locations from the ConnectionRegistry. Those do not have a dependency on master unless you define the RpcConnectionRegistry to hit master.

The way MetaTableAccessor works is to call getMetaTable(conn) to get an HTable, then table.getDescriptor() and parses the getRegionReplication() method. The call to getDescriptor() hits the active HMaster.

We should fix MetaTableAccessor to use the same connection registry based lookup of meta replicas.