av9ash/GitBugs huggingface dataset:
Bug: After modifying the 'zookeeper.znode.parent' parameter and restarting HBase, HBase is unable to complete its initialization.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Oct/24 06:57
Affects Version/s: 2.4.17

In my tests, after modifying the 'zookeeper.znode.parent' parameter and restarting HBase, HBase is unable to complete its initialization with versions 2.3.* and 2.4.* , but this issue does not occur with version 2.2.* .

If the Zookeeper cluster crashes and the data stored in zk cannot be recovered, and only a new ZK cluster can be set up, the HBase cluster will never be able to return to a normal state. This problem is particularly severe under certain failure scenarios.

 

The following describes the situation after I modified the znode and restarted HBase. The data in the hbase:meta table was lost, the number of regions displayed on the Master UI remained at 1, and the meta table could not come online. Moreover, there were no obvious error messages found in the logs of the Master or Region Servers.

 

Maybe it's related to the issue fixed by HBASE-26193?
Bug: Limit the number of retries in FanOutOneBlockAsyncDFSOutputHelper.completeFile
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Nov/24 07:00

After HBASE-28955, when shutting down MiniDFSCluster, we will close all the output streams. 

And in some WAL related tests, we want to keep the WAL file as open so we set namenode to safe mode before shutting down, like TestWALFactory.testAppendClose.

And in FanOutOneBlockAsyncDFSOutputHelper.completeFile, we will only give up when hitting LeaseExpireException, so in this scenario, we will block there forever...

In general, if there is an error while complete file, we could go with the recoverLease logic so it is OK to throw exception when completeFile. We should change the implementation so it does not block there forever.
Bug: Update nightlies to run with HADOOP2_VERSION = 2.10.2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Oct/24 10:21

We are running nighlies with older hadoop2 and hence expected failures are not seen:
 * I see hadoop distributed 2.10.0 installed for last run at [https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/lastCompletedBuild/execution/node/121/ws/hadoop-2/share/hadoop/hdfs/]
 ** Ref: [https://github.com/apache/hbase/blob/master/dev-support/Jenkinsfile#L133]

IMO they are wrong should be 2.10.2 for all branches.

CC: [~stoty]
Bug: Backport "HBASE-27598 Upgrade mockito to 4.x" to branch-2.5
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Oct/24 11:36
Affects Version/s: 2.5.10

Mockito 2.28.2 is release May 29, 2019, which is very old now. Let's upgrade to the latest 4.x version, as mockito 5.0.0 requires java 11, which is not suitable for us.

Need to backport this to branch-2.5 as we face https://issues.apache.org/jira/browse/HBASE-28944 w/o this.
Bug: FSHLog can not roll the WAL log properly
Status: Open | Priority: Major | Resolution: 
Created: 07/Nov/24 21:03
Affects Version/s: 2.4.18

For the FSHLog, when we try to roll the writer, we will
 # initiate the zigzagLatch, and wait for the safe point
 # After the safe point obtained, continue to close the writer

For above process, looks like we have [assumption|https://github.com/apache/hbase/blob/branch-2.6/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java#L388] that the highestSyncedTxid will must be bigger than highestUnsyncedTxid, I do not think it must be true, because for [attainSafePoint|https://github.com/apache/hbase/blob/branch-2.6/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java#L1119] we did not limit the save point much be sync workload, if its stop at append workload, the highestUnsyncedTxid will always be bigger highestSyncedTxid, right?

In our environment, we can reproduce the issue which will causing the wal log pilling up very quickly if a lot of writing, if we want to to make sure the the existing logic is working, we need to add a check and make attainSafePoint get correct safe point, or make sure the doReplaceWriter handle the WAL close correctly when highestUnsyncedTxid > highestSyncedTxid
Bug: Add Clearwater Analytics to powered by hbase
Status: In Progress | Priority: Major | Resolution: 
Created: 16/Oct/24 22:08

According to [https://hbase.apache.org/poweredbyhbase.html] is this the right ticket to add my organization name? If so, could you please add the content below? LMK if you have any questions. Thank you!

==========

Clearwater Analytics ([https://clearwateranalytics.com/]) has an HBase cluster with 9 nodes in production. We have a handful of tables hosting billions of rows (and growing) in a time series manner. Some tables are pretty wide, with several hundreds of columns, while others are very tall and narrow.

==========

Thanks!
Sanjeev Dhiman
Principal Engineer
Clearwater Analytics
Bug: Avoid bundling hbase-webapps folder in default jars
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Oct/24 12:23

We are bundling all webapp resources in hbase-server, hbase-thrift and transitively to hbase-shaded-mapreduce jar. This can be an issue as if any of the js projects used by hbase are vulnerable, security scan tools like sonatype start flagging the jars too as vulnerable since they contain vulnerable code.

With this JIRA, we want to avoid bundling static webapp resources in our jars.

For example: Bootstrap 3.4.1 which is used by hbase, has multiple medium CVEs reported recently. See [https://security.snyk.io/package/npm/bootstrap/3.4.1] for details.

And since we are bundling all webapp resources in hbase-server, hbase-thrift and transitively to hbase-shaded-mapreduce jar. And hence sonatype reports all such jars also as vulnerable:
|3|CVE-2024-6484|2.3|bootstrap 3.4.1|
|3|CVE-2024-6484|2.3|org.apache.hbase : hbase-server : 2.6.0|
|3|CVE-2024-6484|2.3|org.apache.hbase : hbase-shaded-mapreduce : 2.6.0|
|3|CVE-2024-6484|2.3|org.apache.hbase : hbase-thrift : 2.6.0|

It is wise to remove such files from our jars to avoid any bigger hiccups in future.
Bug: Remove all jackson 1.x dependencies for hadoop-3 profile, since all jackson 1.x versions have vulnerabilities
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Oct/24 10:40
Affects Version/s: 2.5.10

Building hbase with hadoop-3 profile on branch-2, still requires jackson 1.x jars, which has vulnerabilities. Ideally these should not be needed as with HADOOP-13332 hadoop has already "Remove jackson 1.9.13 and switch all jackson code to 2.x code line" for branch-3.

Also in HBASE-27148, where we worked on "Move minimum hadoop 3 support version to 3.2.3" we had did a similar cleanup for branch-3 but somehow we missed to port the relevant changes to the branch-2 backport of same jira. This task is to take care of this so that we donot need jackson 1.x to build/run hbase with hadoop-3 profile on branch-2.x.

 

We have following in our dependency tree:
{code:java}
[INFO] ----------< org.apache.hbase:hbase-shaded-client-byo-hadoop >-----------
[INFO] Building Apache HBase - Shaded - Client 2.7.0-SNAPSHOT           [33/53]
[INFO]   from hbase-shaded/hbase-shaded-client-byo-hadoop/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] +- org.codehaus.jackson:jackson-jaxrs:jar:1.9.13:provided
[INFO] +- org.codehaus.jackson:jackson-xc:jar:1.9.13:provided
.
.
[INFO] --------------< org.apache.hbase:hbase-shaded-mapreduce >---------------
[INFO] Building Apache HBase - Shaded - MapReduce 2.7.0-SNAPSHOT        [34/53]
[INFO]   from hbase-shaded/hbase-shaded-mapreduce/pom.xml
[INFO] --------------------------------[ jar ]--------------------------------- 
[INFO] 
[INFO] +- org.codehaus.jackson:jackson-jaxrs:jar:1.9.13:provided
[INFO] +- org.codehaus.jackson:jackson-xc:jar:1.9.13:provided
.
.
[INFO] -------------< org.apache.hbase:hbase-shaded-testing-util >-------------
[INFO] Building Apache HBase - Shaded - Testing Util 2.7.0-SNAPSHOT     [46/53]
[INFO]   from hbase-shaded/hbase-shaded-testing-util/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] +- org.codehaus.jackson:jackson-jaxrs:jar:1.9.13:compile
[INFO] |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  \- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  +- org.codehaus.jackson:jackson-jaxrs:jar:1.9.13:test
.
.
[INFO] ---------< org.apache.hbase:hbase-shaded-testing-util-tester >----------
[INFO] Building Apache HBase - Shaded - Testing Util Tester 2.7.0-SNAPSHOT [47/53]
[INFO]   from hbase-shaded/hbase-shaded-testing-util-tester/pom.xml
[INFO] --------------------------------[ jar ]--------------------------------- 
[INFO] 
[INFO] +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:test
[INFO] |  \- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:test {code}
Bug: [hbase-thirdparty] add jersey-media-json-jackson in  hbase-shaded-jersey
Status: Open | Priority: Major | Resolution: 
Created: 07/Nov/24 09:54
Affects Version/s: thirdparty-4.1.9

In a REST service built using hbase-shaded-jersey.jar, I'm unable to convert a Response object into a JSON string.
The reason I cannot convert a Response object to a JSON string in my REST service built with hbase-shaded-jersey.jar is because it lacks the necessary dependencies from jersey-media-json-jackson.
Bug: Clean up explicit depency versions in poms
Status: Open | Priority: Major | Resolution: 
Created: 21/Oct/24 18:45

Unless there is some specific reason, all dependency versions should be set in DependencyManage sections.

Right now we at least some Hadoop dependencies where the versions are redundantly set directly in the Dependency entries.

We should also consistently set hadoop.version from the hadoop3 profiles in the modules that have them. (I think that those aren't relevant in branch-3+, but we should either set them consistently, or just remove them outside branch-2.x)
Bug: Apply workaround for HADOOP-19164 for Hadoop 3.4.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Nov/24 10:45

HADOOP-19164 is also present in 3.4.1, not just 3.4.0.
We need to update the scripts to apply the workaround for both 3.4.0 and 3.4.1
Bug: TestMemStoreLAB often fails with OOM
Status: Open | Priority: Major | Resolution: 
Created: 06/Nov/24 07:27

I see this quite often in the failed nightly tests:
IIRC I've always seen this on the JDK8 tests, but the sample size is very small.

[https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/619/]
Bug:  RegionMover tool fails when table is deleted
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Oct/24 13:20

The issue occurs when the RegionMover tool encounters a TableNotFoundException while moving regions. This happens if the table being processed is deleted while the RegionMover is running.
{noformat}
2024-10-30 11:58:20,500 INFO org.apache.hadoop.hbase.util.RegionMover: Moving 10 regions to server0.example.org,16020,1730289478970 using 30 threads.Ack mode:false
2024-10-30 11:58:20,564 ERROR org.apache.hadoop.hbase.util.RegionMover: Error while loading regions to server0.example.org org.apache.hadoop.hbase.TableNotFoundException: cluster_test
        at org.apache.hadoop.hbase.client.HBaseAdmin$46.rpcCall(HBaseAdmin.java:2120)
        at org.apache.hadoop.hbase.client.HBaseAdmin$46.rpcCall(HBaseAdmin.java:2116)
        at org.apache.hadoop.hbase.client.RpcRetryingCallable.call(RpcRetryingCallable.java:57)
        at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104)
        at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3088)
        at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3080)
        at org.apache.hadoop.hbase.client.HBaseAdmin.checkTableExists(HBaseAdmin.java:2116)
        at org.apache.hadoop.hbase.client.HBaseAdmin.isTableEnabled(HBaseAdmin.java:978)
        at org.apache.hadoop.hbase.util.MoveWithAck.getServerNameForRegion(MoveWithAck.java:140)
        at org.apache.hadoop.hbase.util.RegionMover.loadRegions(RegionMover.java:358)
        at org.apache.hadoop.hbase.util.RegionMover.lambda$getRegionsMovePlan$2(RegionMover.java:333)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
{noformat}
Bug: Purge all netty 3 dependencies by default
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Oct/24 07:02

After bumping default hadoop version to 3.4.1, we should be able to remove all netty 3 dependencies.

Now the problem is in hbase-external-blockcache, where we use jmemcached for testing, it is an old library which still depends on netty 3.

We should find a way to deal with this.
Bug: Prioritize "orphan" blocks for eviction inside BucketCache.freespace
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Oct/24 15:23
Affects Version/s: 2.6.1

The Persistent Cache feature brought the ability to recover the cache in the event of a restart or crash. Under certain conditions, the cache recovery can lead to _orphan_ blocks hanging on the cache, causing unnecessary extra cache usage.

For example, when a region server crashes or restarts, the original regions on this region server would be immediately reassigned on the remaining servers. Once the crashed/restarted server comes back online, persistent cache will recover the cache state prior to the crash/restart, which would contain the blocks from the regions it was holding prior to the incident. This can lead to the _orphan_ blocks situation described above in the following conditions:
 * If balancer is off, or cache aware balancer fails to move back the very same regions from prior the crash;
 * If compaction completes for the regions on the temporary servers;

Also, with the default evictsOnClose is set to false, any region move would leave "orphans" blocks behind. 

This proposes additional logic for identifying blocks not belonging to any files from current online regions inside BucketCache.freeSpace method.

This would need to modify both BlockCacheFactory and BucketCache to pass along the map of online regions kept by HRegionServer.

Inside BucketCache.freeSpace method, when iterating through the backingMap and before separating the entries between the different eviction priority groups, we can check if the given entry belongs to a block from any of the online regions files, using BucketCache.evictBucketEntryIfNoRpcReferenced method to remove it if its file is not found on any of the online regions.

An additional configurable grace period should be added, to consider only blocks cached before this grace period as potentially orphans. This is to avoid evicting blocks from currently being written files by flushes/compactions, when “caching on write/caching on compaction” is enabled.
Bug: Should check if compaction are needed after flushing
Status: Open | Priority: Major | Resolution: 
Created: 24/Oct/24 10:59
Affects Version/s: 2.4.13

Regardless of whether we have enabled flsuh procedure or not, even if the threshold is reached, hbase would not execute comapct after flushing operation.
I think we should check the HRegion.FlushResult, and send compact request if it is needed.
Bug: Update nightlies to run with HADOOP3_VERSION = 3.2.4
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 30/Oct/24 10:11
Affects Version/s: 2.5.10

We are running nighlies with older hadoop3 on branch-2.5 and hence expected failures are not seen:
 * Also I see hadoop 3.1.1 for hadoop3 at [https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/lastCompletedBuild/execution/node/121/ws/hadoop-3/share/hadoop/client/] 
 ** Ref: [https://github.com/apache/hbase/blob/branch-2.5/dev-support/Jenkinsfile#L157]

IMO is is wrong should 3.2.4 for branch-2.5

CC: [~stoty]
Bug: Handle NPE in Split/Merge table when getMasterQuotaManager returns null
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Oct/24 09:51
Affects Version/s: 2.5.8

Currently when doing splits or merge we notify quota manager about the same. But if quota instance is null on below call
{code:java}
// code placeholder
env.getMasterServices().getMasterQuotaManager() {code}
So the below two lines will throw unexpected exception which ideally can be handled. 
{code:java}
// SplitTableRegionProcedure.java
env.getMasterServices().getMasterQuotaManager().onRegionSplit(this.getParentRegion());
 {code}
{code:java}
// MergeTableRegionProcedure.java
 env.getMasterServices().getMasterQuotaManager().onRegionMerged(this.mergedRegion);
 {code}
Bug: Fix spotbugs issues introduced by refactoring to hbase-diagnostics with HBASE-28432
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Oct/24 09:31

Seeing a lot of spotbugs issues across PRs lately. These are coming due to newly added main code refactored as part of https://issues.apache.org/jira/browse/HBASE-28432 I was not aware spotbugs issues will be on complete code and not just PR code change. Also wanted to avoid changing code as we were just 'refactoring'.

As now code sits inside main and not test, we should try to fix all the spotbugs issues.

CC: [~ndimiduk], [~stoty]
Bug: The HFile Reader creation should not be blocked due to waits for cache initialisation.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Oct/24 12:28
Affects Version/s: 2.7.0

When persistent bucket cache is enabled, bucket cache needs to read and load the persistent cache file contents during region server starting time. This cache initialisation is currently asynchronous. The server restart completes after spawning a thread to initialise the cache.

When the subsequent HFile readers are created, the current implementation is such that the constructor of the hfile (HFilePreadReader) waits until the cache is initialised. Due to this wait, all client requests to these regions would fail during the whole cache initialisation period, as these regions are not yet online.

The correct way to handle this is that, the constructors of HFile readers should not wait for cache initialisation. Only the prefetch threads should wait for the cache initialisation.

Subsequently, any client requests should be served by accessing the underlying file system if the cache is not yet initialised, and a cache miss should be computed in the cache stats.

Thanks,

Janardhan
Bug: Use reflection to access recoverLease(), setSafeMode() APIs.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Oct/24 03:04

HBASE-27769 used the new Hadoop API (available since Hadoop 3.3.6/3.4.0) to access recoverLease() and setSafeMode() APIs, and committed the change in a feature branch.

However, until we move beyond Hadoop 3.3.6, we can not use them directly.

I'd like to propose to use reflection to access these APIs in the interim so HBase can use Ozone sooner.
Bug: Automatic SSL keystore reloading for HttpServer
Status: Open | Priority: Major | Resolution: 
Created: 13/Oct/24 11:05

Since https://issues.apache.org/jira/browse/HADOOP-16524, Hadoop can reload SSL keystore without restarting.
It can be adopted to HBase similarly.

It is painful to rolling-restart HBase master and all the HBase regionservers on large cluster in order to refresh ssl keystore every year.
Bug: Add support for multi region splits in Catalog Janitor
Status: Open | Priority: Major | Resolution: 
Created: 24/Oct/24 10:39

This is to add the API to support multi region splitting - HBASE-28438

Catalog janitor checks ongoing region splits  and compactions on the daughter regions completed or not by looking at the reference files present in the daughter region directories to delete the parent region from meta.


All the APIs used by catalog janitor check only two daughter regions and reference files of those two daughter regions which need to be changed to support multi region split feature - HBASE-28438.
Bug: Set hadoop-three.version in Hadoop 3 backwards compatibility tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Oct/24 11:52

Found by the new 3.4.0 nighty tests:


{noformat}
[ERROR] Errors: 
[ERROR] org.apache.hadoop.hbase.backup.TestBackupRestoreOnEmptyEnvironment.null
[ERROR]   Run 1: TestBackupRestoreOnEmptyEnvironment.testRestoreToCorrectTable:126->backup:223 » TestTimedOut test timed out after 1560 seconds
[ERROR]   Run 2: TestBackupRestoreOnEmptyEnvironment »  Appears to be stuck in thread MiniHBaseClusterRegionServer-EventLoopGroup-3-1
[INFO] 
[ERROR]   TestBackupRestoreOnEmptyEnvironment.testRestoreCorrectTableForIncremental:156->backup:223 » NoClassDefFound org/apache/hadoop/util/Sets
[ERROR] org.apache.hadoop.hbase.backup.TestIncrementalBackupMergeWithBulkLoad.null
[ERROR]   Run 1: TestIncrementalBackupMergeWithBulkLoad.testMergeContainingBulkloadedHfiles:127->backup:219 » TestTimedOut test timed out after 1560 seconds
[ERROR]   Run 2: TestIncrementalBackupMergeWithBulkLoad »  Appears to be stuck in thread MiniHBaseClusterRegionServer-EventLoopGroup-3-1
[INFO] 
[ERROR]   TestIncrementalBackupMergeWithBulkLoad.testMergeContainingBulkloadedHfiles:134->backup:219 » NoClassDefFound org/apache/hadoop/util/Sets
[ERROR]   TestIncrementalBackupMergeWithBulkLoad.testMergeContainingBulkloadedHfiles:127->backup:219 » InterruptedIO
[INFO] {noformat}
Bug: CopyTable to accept custom cf/qualifier for timestamp range filtering
Status: In Progress | Priority: Major | Resolution: 
Created: 22/Oct/24 23:15
Affects Version/s: 2.4.18

Currently CopyTable starttime/endtime param filters on HBase's built-in timestamp.

Add the following options to support filtering based on the value of custom timestamp column.
{noformat}
 ts.cf        column family for time range checking (use HBase built-in timestamp if not specified)
 ts.qualifier qualifier for time range checking (use HBase built-in timestamp if not specified)
{noformat}
Bug: Supports enabling storage policy in the data copying scenario of bulkload
Status: Open | Priority: Major | Resolution: 
Created: 05/Oct/24 09:39

The Bulkload process consists of two steps:

1. generate hfiles using MR/SPARK and write them to an HDFS cluster.
2. execute 'hbase completebulkload [OPTIONS] </PATH/TO/HFILEOUTPUTFORMAT-OUTPUT> <TABLENAME> ' or  invoke the BulkLoadHFilesTool API.

[HBASE-1721|https://issues.apache.org/jira/browse/HBASE-15172] implements tiered storage capabilities for bulkload, but it is only applicable to scenarios where hfiles generated by MR/SPARK are directly written to the HDFS cluster used by HBase (tiered storage is configured). However, in most bulkload scenarios, hfiles generated by MR/SPARK are first written to an offline HDFS cluster (non-HBase HDFS Cluster, and tiered storage is not configured). Subsequently, the 'hbase completebulkload' command is used to copy these hfiles from the offline HDFS cluster to the HDFS cluster used by HBase, and rename them to the appropriate table/region/columnfamily directory. This scenario is not supported by [HBASE-1721|https://issues.apache.org/jira/browse/HBASE-15172], this issue will support tiered storage for this more general bulkload scenario.
Bug: Refactor chaos monkey related code to a new module hbase-chaos
Status: Open | Priority: Major | Resolution: 
Created: 16/Oct/24 09:11
Affects Version/s: 3.0.0-beta-2

As part of this Jira we will create a new module named {{hbase-chaos}} and optionally include it in the assembly. This change is necessary because we will be removing all test jars from our assembly (with HBASE-28433), resulting in the removal of chaos monkey and all its code, which may need to be manually copied by operators. Currently, all the code resides in {{{}hbase-it{}}}.

This fix will provide an option to build hbase such that we can bundle hbase-chaos in assembly, albeit we wont do this by default.

CC: [~stoty], [~ndimiduk]