av9ash/GitBugs huggingface dataset:
Bug: Make sure that filterAllRemaining() is not called from filterRowKey()
Status: Open | Priority: Minor | Resolution: 
Created: 03/Jun/24 04:51

Many of the current filters call filterAllRemaining() from filterRowKey().

This should not be necessary, as in the normal (RS) code path, filterRowKey() is only called AFTER filterAllReamaining()  has returned false.

Well-written filters do cache their filterAllReamaining() status, so this is not very expensive, but we could still save a few cycles for each cell.

* Change the filter API definition to explicitly state this
* Fix the code where this is not true. At first glance, org.apache.hadoop.hbase.mapreduce.Import seems to be one place that does not confirm to this behaviour.
Bug: Reduce memory copying in Rest server when serializing CellModel to Protobuf
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Apr/24 07:56

The REST server does a lot of unneccessary coping, which could be avoided at least for protobuf encoding.

- -It uses ByteStringer to handle ByteBuffer backed Cells. However, it uses the client API, so it should never encounter ByteBuffer backed cells.-
- It clones everything from the cells (sometimes multiple times) before serializing to protbuf.

We could mimic the structure in Cell, with array, offset and length for each field, in CellModel and use the appropriate protobuf setters to avoid the extra copies.

There may or may not be a way to do the same for JSON and XML via jax-rs, I don't know the frameworks well enough to tell, but if not, we could just do the copying in the getters for them, which would not make things worse.
Bug: Use the latest Httpclient/Httpcore 5.x  in HBase
Status: Open | Priority: Minor | Resolution: 
Created: 08/Apr/24 07:09

HttpClient 4.x is not actively developed.

We use Httpclient directly in the REST client code, and in the tests for several modules.

Httpclient 4.5 is a transitive dependency at least from Hadoop and Thrift, but httpclient 5.x uses a separate java package, so 4.5 and 5.x  should be able to co-exist fine.

As of now, Httpclient 4.5 is in maintenance mode:
https://hc.apache.org/status.html
Bug: After one RegionServer restarts, another RegionServer leaks a connection to ZooKeeper
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Jun/24 08:07
Affects Version/s: 2.4.5

The peer "to_pd_A" has been removed, but there is an error log in RegionServer, error log：
{code:java}
2024-06-11 09:42:34.074 ERROR [ReplicationExecutor-0.replicationSource,to_pd_A-172.30.12.12,6002,1709612684705-SendThread(bjtx-hbase-onll-meta-01:2181)] client.StaticHostProvider: Unable to resolve address: bjtx-hbase-onll-meta-03:2181
java.net.UnknownHostException: bjtx-hbase-onll-meta-03
   at java.net.InetAddress$CachedAddresses.get(InetAddress.java:764)
   at java.net.InetAddress.getAllByName0(InetAddress.java:1291)
   at java.net.InetAddress.getAllByName(InetAddress.java:1144)
   at java.net.InetAddress.getAllByName(InetAddress.java:1065)
   at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
   at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
   at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
   at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137){code}
Here are the steps to reproduce:

I have 3 RegionServers. The following steps can reproduce the phenomenon of ZK connection leakage:
1. Enable replication
2. Create a peer
3. Shut down any two RegionServers for a few minutes and restart them
4. Print the thread stack on the RegionServer that is not shut down, search for the keyword <peerId>, and you can see that there are 4 more threads with ZooKeeper
5. By removing the peer, the extra 4 threads still exist

The following is the thread stack leak in one of my RegionServers：
{code:java}
"ReplicationExecutor-0.replicationSource,lizy_test_replication-10.0.16.29,6002,1718180442225-EventThread" #610 daemon prio=5 os_prio=0 cpu=0.27ms elapsed=466.94s tid=0x00007efc58179000 nid=0x5a051 waiting on condition [0x00007efc2cdef000]

"ReplicationExecutor-0.replicationSource,lizy_test_replication-10.0.16.29,6002,1718180442225-SendThread(10.0.16.100:2181)" #609 daemon prio=5 os_prio=0 cpu=3.02ms elapsed=466.94s tid=0x00007efc58178800 nid=0x5a050 runnable [0x00007efc2cef0000]

"ReplicationExecutor-0.replicationSource,lizy_test_replication-10.0.16.9,6002,1718180457260-EventThread" #505 daemon prio=5 os_prio=0 cpu=0.27ms elapsed=556.09s tid=0x00007efc50094800 nid=0x59c04 waiting on condition [0x00007efc2d7f7000]

"ReplicationExecutor-0.replicationSource,lizy_test_replication-10.0.16.9,6002,1718180457260-SendThread(10.0.16.100:2181)" #504 daemon prio=5 os_prio=0 cpu=3.72ms elapsed=556.09s tid=0x00007efc50093000 nid=0x59c03 runnable [0x00007efc2d8f8000] {code}
Bug: Support only remove expired files in the compact process
Status: In Progress | Priority: Minor | Resolution: 
Created: 11/Jun/24 13:26

As is known to all, the compact processes generate a certain amount of I/O. But in some cases, not being compact in time does not affect online services.

For example, in the scenario where the TTL time of a table is short, there are more writes than reads. To ensure write performance and reduce cluster I/O pressure, Only expired files are removed in the compact process is a good idea.

Another usage scenario: Tables have primary and backup, and the backup table can add this table attribute to reduce the compact IO of the backup cluster.

Optimization solution: Add a table attribute for the HBase table
Bug: WALs not marked closed when there are errors in closing WALs
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Jun/24 08:22
Affects Version/s: 2.5.8

In our production clusters we have observed that when WAL close fails It causes the the oldWAL files not marked as close and not letting them cleaned. When a WAL close fails in closeWriter it increments the error count. 

{code:java}
Span span = Span.current();
 try {
      span.addEvent("closing writer");
      writer.close();
      span.addEvent("writer closed");
    } catch (IOException ioe) {
      int errors = closeErrorCount.incrementAndGet();
      boolean hasUnflushedEntries = isUnflushedEntries();
      if (syncCloseCall && (hasUnflushedEntries || (errors > this.closeErrorsTolerated))) {
        LOG.error("Close of WAL " + path + " failed. Cause=\"" + ioe.getMessage() + "\", errors="
          + errors + ", hasUnflushedEntries=" + hasUnflushedEntries);
        throw ioe;
      }
      LOG.warn("Riding over failed WAL close of " + path
        + "; THIS FILE WAS NOT CLOSED BUT ALL EDITS SYNCED SO SHOULD BE OK", ioe);
    }
{code}

When there are errors in closing WAL only twice doReplaceWALWriter enters this code block

{code:java}
if (isUnflushedEntries() || closeErrorCount.get() >= this.closeErrorsTolerated) {
          try {
            closeWriter(this.writer, oldPath, true);
          } finally {
            inflightWALClosures.remove(oldPath.getName());
          }
        }
{code}
 as we don't mark them closed here like we do it here 
  
{code:java}
  Writer localWriter = this.writer;
          closeExecutor.execute(() -> {
            try {
              closeWriter(localWriter, oldPath, false);
            } catch (IOException e) {
              LOG.warn("close old writer failed", e);
            } finally {
              // call this even if the above close fails, as there is no other chance we can set
              // closed to true, it will not cause big problems.
             {color:red} markClosedAndClean(oldPath);{color}
              inflightWALClosures.remove(oldPath.getName());
            }
          });
{code}
Bug: [HBASE] The Archives folder in the HBASE root directory starts to grow uncontrollably under load
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Jun/24 08:12

If there are empty .snapshotinfo and data.manifest in .hbase-snapshot dir, we got a lot of this warns on master server
{code:java}
WARN  [snapshot-hfile-cleaner-cache-refresher] snapshot.SnapshotFileCache: Failed to refresh snapshot hfile cache!
org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Couldn't read snapshot info from:hdfs://host/apps/hbase/data/.hbase-snapshot/test/.snapshotinfo{code}
{color:#172b4d}how to reproduce:{color}
 * {color:#172b4d}create a folder (for example snap2) at the path /apps/base/data/.base-snapshot {color}
 * {color:#172b4d}create two empty files there: .snapshotinfo data.manifest {color}
 * {color:#172b4d}give a load, for example like this: hbase lt -write 1:10:100 -num_keys 1000000 -read 100:30 -num_tablets 20 -data_block_encoding NONE -tn load_test_tool {color}
 * {color:#172b4d}wait... {color}
 * {color:#172b4d}the /apps/base/data/archive folder will start to grow (approximately to the size of /apps/base/data/data) {color}
 * {color:#172b4d}if you delete the previously created snap2 folder, the size of /aps/base/data/archive will return to its original state{color}
Bug: Show the create time of the peer
Status: Open | Priority: Minor | Resolution: 
Created: 30/May/24 09:14

When the table data needs to be migrated due to the closure of the computer room or the business side needs to have a primary and standby table, we need to create a peer. 
By checking the creation time of the peer, we can understand when and what operations were performed, and the impact of these operations on the HBase system status.
When there is a problem with the HBase cluster, such as abnormal data synchronization, performance degradation, etc., displaying the creation time of the peer can help us quickly locate the possible source of the problem. For example, if a peer has a problem shortly after it is created, we can prioritize checking the configuration and status of the peer, optimize the related configuration of replication, etc.
Recording the creation time of the peer helps to audit and track system changes, ensuring that any changes to the replication configuration can be traced. This is very important for meeting data governance, security compliance, and business continuity requirements.
Bug: REST multiget endpoint returns 500 error if no rows are specified
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Jun/24 05:43

Should return 404 instead.

Need to check if there are any "row" params before trying to iterate over them

{noformat}
Caused by: java.lang.NullPointerException
	at org.apache.hadoop.hbase.rest.MultiRowResource.get(MultiRowResource.java:100)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hbase.thirdparty.org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.apache.hbase.thirdparty.org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:134)
	at org.apache.hbase.thirdparty.org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:177)
	at {noformat}
Bug: Dropping unclosed WALTailingReaders leads to leaked sockets
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Jun/24 15:34
Affects Version/s: 2.6.0

{{WALEntryStream#prepareReader()}} will, in some cases, reach [the line|https://github.com/apache/hbase/blob/ba15d67a350adb11ae1d4c44d214216406ae0b5a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALEntryStream.java#L258]
{code}
reader = WALFactory.createTailingReader(fs, nextPath, conf, currentPositionOfEntry > 0 ? currentPositionOfEntry : -1);
{code}
when {{reader}} is non-null. In this case, the old object pointed to by {{reader}} becomes un-referenced and is garbage-collected. However, that object was never closed.

At Hubspot we see the effects of this when doing tests that use inter-cluster replication. Machines in the source cluster experience a build-up of sockets. Eventually this causes the machine to run out of TCP kernel memory and start dropping packets. The only workaround currently is to restart the RegionServer process.

I have found that simply putting
{code}
closeReader();
{code}
immediately before the line quoted above appears to resolve the issue and causes no obvious problems. However, I'm still developing a proper test for this fix.
Bug: Log hfile name when unpack is failed
Status: Open | Priority: Minor | Resolution: 
Created: 14/Jun/24 22:09

When an HFile block is corrupted, identifying the corrupt HFile name is challenging because we only log a "Corrupted block detected" message. So, we should log the HFile name (fileContext.getHFileName()) whenever an unpack operation is unsuccessful.

 

Ref: https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java#L659

!Screenshot 2024-06-14 at 2.59.04 PM.png!
Bug: Cache Results in org.apache.hadoop.hbase.rest.client.RemoteHTable.Scanner
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Apr/24 07:52

The implementation of org.apache.hadoop.hbase.rest.client.RemoteHTable.Scanner
is very inefficient, as the standard next() methods makes separate a http request for each row.

Performance can be improved by not specifying the row count in the REST call and caching the returned Results.

Chunk size can still be influenced by scan.setBatch();
Bug: Introduce a field to display whether the snapshot is expired
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/May/24 10:18

HBase supports to create snapshot with TTL, and expired snapshots will be periodically deleted.

This period is 30 min by default, as follow.
{code:java}
private static final String SNAPSHOT_CLEANER_INTERVAL = "hbase.master.cleaner.snapshot.interval";
private static final int SNAPSHOT_CLEANER_DEFAULT_INTERVAL = 1800 * 1000; // Default 30 min {code}
 

Therefore, the following situation may occur: 
The expired snapshot would still exist for a period of time on hbase cluster, and would not be deleted until the next operation of the periodic thread.
So, Sometimes, we may use the expired snapshot because we donot know whether the snapshot is expired.

 

So, I think we can introduce a expired field for this situation in HBase UI.
And on hbase shell ,adding snapshot TTL info and displaying expired if the snaphost has already expired.

Or any better suggestions? Thanks a lot!
Bug: Fix the inaccurate message when snapshot doesn't exist
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/May/24 11:57
Affects Version/s: 2.4.13

We would get the as following message when restore a non-existing snapshot.
{code:java}
hbase:021:0> restore_snapshot  'non_existing_snap'

ERROR: Unable to find the table name for snapshot=non_existing_snap

For usage try 'help "restore_snapshot"'

Took 0.0170 seconds {code}
 
ERROR: {color:#FF0000}Unable to find the table{color} name for snapshot=non_existing_snap
This error message is inaccurate.
Bug: Client Does not Stop Retrying after DoNotRetryException
Status: Open | Priority: Minor | Resolution: 
Created: 11/May/24 15:03
Affects Version/s: 1.2.0

I recently discovered that the fix for HBase-14598 does not completely resolve the issue. Their fix addressed two aspects: first, when the Scan/Get RPC attempts to allocate a very large array that could potentially lead to an out-of-memory (OOM) error, it will check the size of the array before allocation and directly throw an exception to prevent the region server from crashing and avoid possible cascading failures. Second, the developer intends for the client to stop retrying after such a failure, as retrying will not resolve the issue.

However, their fix involved throwing a DoNotRetryException. After ByteBufferOutputStream.write throws the DoNotRetryException, in the call stack (ByteBufferOutputStream.write --> encoder.write --> encodeCellsTo --> his.cellBlockBuilder.buildCellBlockStream --> call.setResponse), the DoNotRetryException is ultimately caught in the CallRunner.run function, with only a log printed. Consequently, the DoNotRetryException is not sent back to the client side. Instead, the client receives a generic exception for the failed RPC request and continues retrying, which is not the desired behavior. I have reproduced this on the cluster.

In the code of CallRunner, it is obvious that the DoNotRetryException in call.setResponse will be swallowed in the error handler with just a LOG printed.
Bug: Closing ZooKeeper in ZKMainServer
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/May/24 00:37

Users can switch the Zookeeper client/server communication framework to Netty.
ZKMainServer process fails to terminate due to when users utilize Netty for ZooKeeper connections.
Netty threads identified as non-Daemon threads.
Enforce the calling of close() on ZooKeeper before ZKMainServer termination.
Bug: Always printing error log when snapshot table 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/May/24 15:18
Affects Version/s: 2.4.13

Reproduction.

1. 
Disable snapshot procedure if your hbase support snapshot procedure feature
Set hbase.snapshot.procedure.enabled to false to disable snapshot procedure.

2.
Executing snapshot against a table, this step is no problem
snapshot 't01', 'sn0001'

3. 
HBase outputs error logs, as follow.
2024-05-07T23:16:37,175 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/archlinux:16000-0] snapshot.TakeSnapshotHandler: Couldn't delete snapshot working directory:file:/opt/hbase/hbase-4.0.0-alpha-1-SNAPSHOT/tmp/hbase/.hbase-snapshot/.tmp/sn001



 

The Reason.
HBase would clean tmp of the snapshot after snapshot. 
The tmp would be empty if snapshot was executed successfully
We would get false when calling `Filesystem.delete()` to delete the tmp which does not exist, so hbase outputs error logs.
Bug: Fix `Disable Stripe Compaction` run error in  document
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 18/Apr/24 13:53
Affects Version/s: 2.5.6

*Disable Stripe Compaction* in document is
{code:java}
alter 'orders_table', CONFIGURATION =>
{'hbase.hstore.engine.class' => 'rg.apache.hadoop.hbase.regionserver.DefaultStoreEngine'}{code}
This should be 'org.apache.hadoop.hbase.regionserver.DefaultStoreEngine' 

This will cause all regions to be in the openning state.Finally, I went through the disable table and corrected it before enable it.
Bug: Visualize current state of region assignment
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 07/May/21 18:23

After several months of debugging and tuning the balancer and normalizer on a large production cluster, we found that working from visualizations of the current region state was very useful for understanding behaviors and quantifying improvements we made along the way. Specifically, we found that a chart of total assigned region count and total assigned region store files size per table per host was immensely useful for tuning the balancer. Histograms of store file size made understanding normalizer activity much more intuitive.

Our scripts would parse the output of the shell's {{status 'detailed'}} command, extract the desired metric, and produce charts. I'd like to build into the master UI the equivalent functionality, with data coming directly from the {{ClusterMetrics object}}, and data rendered into an interactive chart rendered in the browser.
Bug: [branch-1] add -U for maven build to force a check for updated releases and snapshots on remote repositories
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 14/May/21 07:31

{code}
[ERROR] Failed to execute goal on project hbase-assembly: Could not resolve dependencies for project org.apache.hbase:hbase-assembly:pom:1.7.0: Could not find artifact org.apache.hbase:hbase-thrift:jar:1.7.0 in apache release (https://repository.apache.org/content/repositories/releases/) -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hbase-assembly
{code}

{code}
[ERROR] Failed to execute goal on project hbase-assembly: Could not resolve dependencies for project org.apache.hbase:hbase-assembly:pom:1.7.0: Failure to find org.apache.hbase:hbase-thrift:jar:1.7.0 in https://repository.apache.org/content/repositories/releases/ was cached in the local repository, resolution will not be reattempted until the update interval of apache release has elapsed or updates are forced -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hbase-assembly
{code}
Bug: [branch-1] make hbase-thrift optional in hbase-assembly module
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 06/May/21 11:00

{code}
[ERROR] Failed to execute goal on project hbase-assembly: Could not resolve dependencies for project org.apache.hbase:hbase-assembly:pom:1.7.0: Failure to find org.apache.hbase:hbase-thrift:jar:1.7.0 in https://repository.apache.org/content/repositories/releases/ was cached in the local repository, resolution will not be reattempted until the update interval of apache release has elapsed or updates are forced -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
{code}
Bug: [branch-1] remove thrift examples out of hbase-examples module for bypassing the thrift version check
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 30/Apr/21 09:00

[ERROR] Failed to execute goal on project hbase-examples: Could not resolve dependencies for project org.apache.hbase:hbase-examples:jar:1.7.0: Could not find artifact org.apache.hbase:hbase-thrift:jar:1.7.0 in apache release (https://repository.apache.org/content/repositories/releases/) -> [Help 1]

This is the msg when I tried to run make_rc.sh, we need to remove thrift related codes from hbase-examples for making release successfully.
Bug: Master has completed initialization but hbase shell can't connect and throw a unhandledexception
Status: Open | Priority: Blocker | Resolution: 
Created: 13/May/21 08:08
Affects Version/s: 2.2.2

I modified some values in the hbase-site.xml file, and then ran hbase in stand-alone mode. Hbase can be initialized normally, but the connection from the client cannot be processed, and the client will throw an unhandled exception. 

Please see the attachment for the configuration file, log information and error information