av9ash/GitBugs huggingface dataset:
Bug: WALs are not cleaned even after all entries are flushed
Status: Open | Priority: Minor | Resolution: 
Created: 21/Aug/24 05:51
Affects Version/s: 2.5.6

In our production fleet we have observed that WAL files are not cleaned up even when all the entries have been flushed. I have fixed the WAL  close issue when there is issue with the wal closures as part of [HBASE-28665|https://issues.apache.org/jira/projects/HBASE/issues/HBASE-28665]. There is a case in case of unflushed entried that can lead for the wal not be cleaned even after all the entries have been flushed
[FSHLog.java |https://github.com/apache/hbase/blob/branch-2.6/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java#L388]

{code:java}
 if (isUnflushedEntries() || closeErrorCount.get() >= this.closeErrorsTolerated) {
          try {
            closeWriter(this.writer, oldPath, true);
          } finally {
            inflightWALClosures.remove(oldPath.getName());
            if (!isUnflushedEntries()) {
              markClosedAndClean(oldPath);
            }
          }
{code}
If there are unflushed entries then wal will never be marked close and won't be cleaned further

{code:java}
private synchronized void cleanOldLogs() {
    List<Pair<Path, Long>> logsToArchive = null;
    // For each log file, look at its Map of regions to the highest sequence id; if all sequence ids
    // are older than what is currently in memory, the WAL can be GC'd.
    for (Map.Entry<Path, WALProps> e : this.walFile2Props.entrySet()) {
      if (!e.getValue().closed) {
        LOG.debug("{} is not closed yet, will try archiving it next time", e.getKey());
        continue;
      }
      Path log = e.getKey();
      Map<byte[], Long> sequenceNums = e.getValue().encodedName2HighestSequenceId;
      if (this.sequenceIdAccounting.areAllLower(sequenceNums)) {
        if (logsToArchive == null) {
          logsToArchive = new ArrayList<>();
        }
        logsToArchive.add(Pair.newPair(log, e.getValue().logSize));
        if (LOG.isTraceEnabled()) {
          LOG.trace("WAL file ready for archiving " + log);
        }
      }
    }
{code}
Bug: Procedure queues & PE pool metrics
Status: Open | Priority: Minor | Resolution: 
Created: 13/Sep/24 02:32

While investigating a production incident, we observed that some procedures are getting created but never getting executed until a HMaster failover.
 - master-2 was active & rs-1 holding meta
 - 18:40, bunch of RSs (~80) reported crashed & SCPs were created & being executed
 - 19:51, balancer decided to move Meta region to another RS -> TRSP created -> Meta region went offline
 - 19:52, RS carrying meta crashed -> SCP created
 - 19:52 - Both TRSP & SCP seemed stuck/not executing  - No more logs related to these procedures
 - 19:55 - RPC queue size slightly increased to ~700
 - 21:09 - Master failed over from master-2 to master-3
 - Procs were loaded from store & attached.
 - 21:17 -  When the TRSP for meta had completed, meta came back online.

I will post the logs in some time.

We have a theory that the TRSP & SCP related to meta were submitted but never got executed possibly due to sitting in procedure queue. HMaster thread dumps would have been helpful but unfortunately, one was not availble. We do have RPC queue metrics at high level but having Procedure queue metrics would holistically indicate what could have happened to the procedures.
Bug: compactionsQueuedCount metric is not cleared on 'clear_compaction_queues'
Status: Open | Priority: Minor | Resolution: 
Created: 11/Sep/24 03:14

h2. Problem Description

Region-level {{compactionsQueuedCount}} metric is not reset when the compaction queue is cleared by {{clear_compaction_queues}} command on HBase shell.

Also, the compaction state of the regions are not reset, and table.jsp page reports incorrect compaction state.

h2. To reproduce

Trigger compaction and clear the compaction queue while the compaction is in progress.

h2. Cause

The counters of HRegion are not updated accordingly when the queue is cleared.

 * {{compactionsQueued}}
 * {{majorInProgress}}
 * {{minorInProgress}}
Bug: HBase should support scan limit 0
Status: Open | Priority: Minor | Resolution: 
Created: 09/Sep/24 10:53
Affects Version/s: 2.4.14

When do scan use a sannner, but i set limit=0(some code logic cause...), but the scanner return result....

alought hbase shell convert limit 0 to limit -1, but it not tell user...

but we also could set limit 0 when use client api

so should support limit 0? when I execute SELECT ... LIMIT 0 on MySQL...it work fine, not resturn any result...
Bug: [hbase-operator-tools] Enable infra automation: autolink to Jira and 'pull-request-available' label
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Aug/24 17:42

Patterns like HBASE-28786 will be clickable, so it's easier to go to the Jira page.

{{.asf.yaml}} provides an option to do this via config: See [https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-AutolinksforJira]

Even without manual configuring this already works for hbase somehow. See [https://github.com/apache/hbase/commits/master/]
Maybe configured at repo level in INFRA?

But same does not work for hbase-connectors and other repos. Will use this config to fix the issue.

Also will enable label feature for 'pull-request-available', same as hbase main repo: HBASE-28325
Bug: [hbase-connectors] Enable infra automation: autolink to Jira and 'pull-request-available' label
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Aug/24 12:13

Patterns like HBASE-28786 will be clickable, so it's easier to go to the Jira page.

{{.asf.yaml}} provides an option to do this via config: See [https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-AutolinksforJira]

Even without manual configuring this already works for hbase somehow. See [https://github.com/apache/hbase/commits/master/]
Maybe configured at repo level in INFRA?

But same does not work for hbase-connectors and other repos. Will use this config to fix the issue.

Also will enable label feature for 'pull-request-available', same as hbase main repo: HBASE-28325
Bug: Fix classname for command: copyreppeers in bin/hbase
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Aug/24 08:22

Stumbled upon this. Dug deeper seems during review we missed to rename the classname in bin/hbase when the actual class was renamed from ReplicationPeerMigrationTool -> CopyReplicationPeers

 

See https://github.com/apache/hbase/compare/69603351b3f2817c74d869d32da0596bab3c409e..1d11ce96c44277df6ccdd16ae2c9d8a1c419f3da

[hbase@hostname~]$ hbase copyreppeers
Error: Could not find or load main class org.apache.hadoop.hbase.replication.ReplicationPeerMigrationTool
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.replication.ReplicationPeerMigrationTool 

 

FYI [~zhangduo]
Bug: Expose HTTP context in REST Client
Status: Open | Priority: Minor | Resolution: 
Created: 31/Jul/24 06:51

We already expose the Apache HTTP Client object in the REST client, but we specify the context for each call separately, so it is not possible to retrieve it.

Add a getter and setter for the stickyContext object.

The use case for this is copying session cookies between clients to avoid re-authentication by each client object, but this may also be useful for debugging purposes.
Bug: Canary should not create RefreshCredentials chore if not in daemon mode
Status: Open | Priority: Minor | Resolution: 
Created: 14/Aug/24 17:10

For kerberos enabled clusters, on startup Canary.runMonitor [will create a chore service running a RefreshCredentials chore regardless of the canary mode|https://github.com/apache/hbase/blob/97de2912396d303b78a0d85552953e3e9955a145/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L1057-L1067]. The [documentation in the reference|https://hbase.apache.org/book.html#_running_canary_in_a_kerberos_enabled_cluster] for running a canary against kerberos enabled clusters specifies that the RefreshCredentials chore will only be created in daemon mode, which makes sense as the canary is intended to be long lived if run in daemon mode. If the canary is being run one off (not in daemon mode) it should be relatively short lived, the default canary timeout we have set is 10min, and should not require credential refresh if one has a sane kerberos ticket lifetime. 

If we do see a need to keep that behavior then this can turn into a ticket to correct what's documented in the reference.
Bug: SLF4J: Class path contains multiple SLF4J bindings.
Status: Resolved | Priority: Minor | Resolution: Not A Bug
Created: 29/Jul/24 02:30
Affects Version/s: 2.5.10

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data/app/hadoop-3.3.6/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data/app/hbase-2.5.10/lib/client-facing-thirdparty/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an explanation.
 
The above log dependency conflict causes the regionserver to be unable to output logs after it is started.

By default, in the hbase script file in the bin directory, the value of HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP is true, which will append the hadoop lib to the classpath. In this way, after the hbase process is started, the hadoop jar will be loaded, which may cause dependency conflicts.

Is it possible to set the variable HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP in the hbase-env.sh file, set the default value to true, and only modify this value to false when necessary?
Bug: Users of QuotaRetriever should pass an existing connection
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Jul/24 13:36

Every call to {{HBaseAdmin#getQuota()}} opens a new {{Connection}}, and then closes it. As far as I can tell, this is pointless, since it could use the existing {{Connection}} object held by the {{HBaseAdmin}}. If you have a custom RpcClient (like we do at Hubspot), new connections have an expense that we'd rather not pay unnecessarily. I'll submit a PR that addresses this situation and similar ones.
Bug: Remove the duplicate configurations named hbase.wal.batch.size
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Jul/24 10:14
Affects Version/s: 3.0.0-beta-1

The following code appears in two places: AsyncFSWAL and AbstractFSWAL
{code:java}
public static final String WAL_BATCH_SIZE = "hbase.wal.batch.size";
public static final long DEFAULT_WAL_BATCH_SIZE = 64L * 1024; {code}
Bug: Improve HBase shell snapshot command Doc with TTL option 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Jul/24 09:39

The current HBase shell snapshot command allows users to create a snapshot of a specific table. While this command is useful, it could be enhanced by adding a TTL (Time-to-Live) option. This would allow users to specify a time period after which the snapshot would automatically be deleted.

I propose we introduce a TTL option in the snapshot command doc as follows:

hbase> snapshot 'sourceTable', 'snapshotName', \{TTL => '7d'}

This would create a snapshot of 'sourceTable' called 'snapshotName' that would automatically be deleted after 7 days. The addition document of a TTL option would provide a better user experience and assist with efficient storage management.
Bug: Default Zookeeper ConnectionRegistry APIs timeout should be less
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Jul/24 19:13

HBASE-28428 introduces timeout for Zookeeper ConnectionRegistry APIs. However, the default timeout value we have set is 60s. Given that connection registry are metadata APIs, they should have much lesser timeout value, including default.

Let's set default timeout to 10s.
Bug: SteppingSplitPolicy may not work when table enables region replication
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Jul/24 05:18
Affects Version/s: 2.4.13

Reproduction:

1. Create a table with region replication, and ensure that the primary region and replica region are on the same RS (eg: the HBase cluster has only one RS)
create 't01', 'info', \{REGION_REPLICATION => 2}

2. The first region does not split when storefile size exceed  flushsize * 2, because that we get 2 regions about this table on this RS (1 primary region and 1 replica region)

 

I think we should ignore the replica reggion when getting the count of regions on this same regionserver.

Is my idea correct? maybe can discuss it .
Bug: Merge repairs always fail to restore the BackupSystemTable snapshot
Status: Open | Priority: Minor | Resolution: 
Created: 09/Jul/24 15:39
Affects Version/s: 2.6.0

In `BackupCommands.RepairCommand#execute`, the following code block is present:
{code:java}
repairFailedBackupDeletionIfAny(conn, sysTable);
repairFailedBackupMergeIfAny(conn, sysTable); {code}
Both of these methods can be sketched as:
{code:java}
if (there are no unresolved merges/deletes) {
    BackupSystemTable.deleteSnapshot(conn);
} else {
    BackupSystemTable.restoreFromSnapshot(conn);
    retry merge/delete
}{code}
In case of failed merges, there will be no unresolved deletes, hence the BackupSystemTable snapshot will be deleted. As a result, the restore for the merges will always fail.

Currently unsure what the risks of this are, haven't explored the merge code yet.

Update: it looks like the merge code doesn't create a BackupSystemTable snapshot. So the restore/delete snapshot for the merge method is pointless.
Bug: Ignored flags in Yetus nightly runs
Status: Open | Priority: Trivial | Resolution: 
Created: 27/Sep/24 07:18

{noformat}
22:08:23  |  -0  |         yetus  |   0m  6s   | Unprocessed flag(s): 
22:08:23  |      |                |            | --blanks-tabs-ignore-file
22:08:23  |      |                |            | --blanks-eol-ignore-file
22:08:23  |      |                |            | --author-ignore-list
{noformat}

The flags seem to be still present in the Yetus source code.
Maybe the modules are no longer getting activated ?
Bug: Possible memory leak in MemStoreLAB implementation
Status: Open | Priority: Critical | Resolution: 
Created: 19/Jun/23 15:35

We got this error message when running ITBLL against branch-3.

{noformat}
2023-06-09 14:44:15,386 ERROR [regionserver/core-1-2:16020-shortCompactions-0] util.ResourceLeakDetector: LEAK: RefCnt.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
Created at:
        org.apache.hadoop.hbase.nio.RefCnt.<init>(RefCnt.java:59)
        org.apache.hadoop.hbase.nio.RefCnt.create(RefCnt.java:54)
        org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.<init>(MemStoreLABImpl.java:108)
        sun.reflect.GeneratedConstructorAccessor13.newInstance(Unknown Source)
        sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        org.apache.hadoop.hbase.util.ReflectionUtils.instantiate(ReflectionUtils.java:55)
        org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:43)
        org.apache.hadoop.hbase.regionserver.MemStoreLAB.newInstance(MemStoreLAB.java:116)
        org.apache.hadoop.hbase.regionserver.SegmentFactory.createMutableSegment(SegmentFactory.java:81)
        org.apache.hadoop.hbase.regionserver.AbstractMemStore.resetActive(AbstractMemStore.java:93)
        org.apache.hadoop.hbase.regionserver.AbstractMemStore.<init>(AbstractMemStore.java:83)
        org.apache.hadoop.hbase.regionserver.DefaultMemStore.<init>(DefaultMemStore.java:79)
        sun.reflect.GeneratedConstructorAccessor12.newInstance(Unknown Source)
        sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        org.apache.hadoop.hbase.util.ReflectionUtils.instantiate(ReflectionUtils.java:55)
        org.apache.hadoop.hbase.util.ReflectionUtils.newInstance(ReflectionUtils.java:92)
        org.apache.hadoop.hbase.regionserver.HStore.getMemstore(HStore.java:377)
        org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:283)
        org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:6904)
        org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1173)
        org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1170)
        java.util.concurrent.FutureTask.run(FutureTask.java:266)
        java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        java.util.concurrent.FutureTask.run(FutureTask.java:266)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:750)
{noformat}

Need to dig more.
Bug: RegionServer OOM under load when TLS is enabled
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 22/Jun/23 18:45
Affects Version/s: 2.6.0

We are rolling out the server side TLS settings to all of our QA clusters. This has mostly gone fine, except on 1 cluster. Most clusters, including this one have a sampled {{nettyDirectMemory}} usage of about 30-100mb. This cluster tends to get bursts of traffic, in which case it would typically jump to 400-500mb. Again this is sampled, so it could have been higher than that. When we enabled SSL on this cluster, we started seeing bursts up to at least 4gb. This exceeded our {{{}-XX:MaxDirectMemorySize{}}}, which caused OOM's and general chaos on the cluster.
 
We've gotten it under control a little bit by setting {{-Dorg.apache.hbase.thirdparty.io.netty.maxDirectMemory}} and {{{}-Dorg.apache.hbase.thirdparty.io.netty.tryReflectionSetAccessible{}}}. We've set netty's maxDirectMemory to be approx equal to ({{{}-XX:MaxDirectMemorySize - BucketCacheSize - ReservoirSize{}}}). Now we are seeing netty's own OutOfDirectMemoryError, which is still causing pain for clients but at least insulates the other components of the regionserver.
 
We're still digging into exactly why this is happening. The cluster clearly has a bad access pattern, but it doesn't seem like SSL should increase the memory footprint by 5-10x like we're seeing.
Bug: AssertionError in AsyncRequestFutureImpl when timing out during location resolution
Status: Patch Available | Priority: Major | Resolution: 
Created: 06/Apr/23 01:08

In AsyncFutureRequestImpl we fail fast when operation timeout is exceeded during location resolution [here|https://github.com/apache/hbase/blob/branch-2.5/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java#L460-L462]. In that handling, we loop all actions and set them as failed. The problem is, some number of actions may already finished when we get to this spot. So the actionsInProgress would have been decremented for those already, and now we're going to decrement by all actions. This causes an assertion error since we go negative [here|https://github.com/apache/hbase/blob/branch-2.5/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java#L1197], causing the HBase client to throw an unchecked exception which can kill the caller thread that invoked the operation which should have timed out, as callers of the client should not be catching {{Error}} and its subclasses like {{AssertionError}}. 

We still want to fail all actions, because none will be executed. But we need special handling to avoid this case. Maybe don't bother decrementing the actionsInProgress at all, instead set to 0.
Bug: Define RPC API for cache cleaning
Status: Resolved | Priority: Major | Resolution: Abandoned
Created: 14/Apr/23 10:31

We should add an RPC API to allow for a "limited manual" cache cleaning. If hbase.rs.evictblocksonclose is set to false, blocks may hang in the cache upon regions move between RSes. 

The method at the RS level, should compare the files from its online regions against the files in the prefetch list file, evicting blocks from the files in the prefetch list file that are not in any of the online regions for the given RS.
Bug: Update hadoop.version from 3.3.5 to 3.3.6
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 14/Jun/23 18:04

HBase's default Hadoop3 version is 3.2.4 but HBase already supports Haddoop 3.3.x.

Hadoop 3.2 line has not been updated for over a year. It is perhaps the time to update the Hadoop dependency to the 3.3.x line. (I'll start a DISCUSS thread if the test goes well)

3.3.6 RC is out which fixed a bunch of CVEs and I'd like to test HBase against it. Additionally, Hadoop 3.3.6 will permit us to use non-HDFS as WAL storage.
Bug: Region split and merge time while offline is O(n) with respect to number of store files
Status: Open | Priority: Major | Resolution: 
Created: 03/May/23 16:07
Affects Version/s: 2.5.4

This is a significant availability issue when HFiles are on S3. =

HBASE-26079 ({_}Use StoreFileTracker when splitting and merging{_}) changed the split and merge table procedure implementations to indirect through the StoreFileTracker implementation when selecting HFiles to be merged or split, rather than directly listing those using file system APIs. It also changed the commit logic in HRegionFileSystem to add the link/ref files on resulting split or merged regions to the StoreFileTracker. However, the creation of a link file is still a filesystem operation and creating a “file” on S3 can take well over a second. If, for example there are 20 store files in a region, which is not uncommon, after the region is taken offline for a split (or merge) it may require more than 20 seconds to create the link files before the results can be brought back online, creating a severe availability problem. Splits and merges are supposed to be fast, completing in less than a second, certainly less than a few seconds. This has been true when HFiles are stored on HDFS only because file creation operations there are nearly instantaneous. 

There are two issues but both can be handled with modifications to the store file tracker interface and the file based store file tracker implementation. 

When the file based store file file tracker is enabled the HFile links should be virtual entities that only exist in the file manifest. We do not require physical files in the filesystem to serve as links now. That is the magic of the this file tracker, the manifest file replaces requirements to list the filesystem.

Then, when splitting or merging, the HFile links should be collected into a list and committed in one batch using a new FILE file tracker interface, requiring only one update of the manifest file in S3, bringing the time requirement for this operation to O(1) down from O[n].
Bug: Bump up jruby to 9.4.2.0 and related joni and jcodings to 2.1.48 and 1.0.58 respectively
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 07/Jun/23 19:22
Affects Version/s: 3.0.0-alpha-4

Current version of *jruby* (9.3.9.0) has snakeyaml dependency version 1.33.
{code:java}
> find ./jruby-complete-9.3.9.0/ -name "*snakeyaml*"
./jruby-complete-9.3.9.0/META-INF/jruby.home/lib/ruby/stdlib/org/yaml/snakeyaml
./jruby-complete-9.3.9.0/META-INF/jruby.home/lib/ruby/stdlib/org/yaml/snakeyaml/1.33/snakeyaml-1.33.jar {code}
*snakeyaml-1.33* has a critical CVE [https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-1471] 

{*}jruby-complete-9.4.2{*}{*}.0{*} uses {*}snakeyaml-engine-2.6{*}.