av9ash/GitBugs huggingface dataset:
Bug: Improvements to backup of bulkloaded files
Status: Open | Priority: Major | Resolution: 
Created: 06/Mar/24 19:39

Backup/Restore has support for including bulkloaded files in incremental backups. There is a coprocessor hook which registers all bulkloads into a backup:system_bulk table. A cleaner plugin ensures that these files are not cleaned up from the archive until they are backed up. When the incremental backup occurs, the files are deleted from the system_bulk table and then cleaned up.

We have encountered two problems to be solved with this:
 # The deletion process only happens during incremental backups, not full backups. A full backup already includes all data in the table via a snapshot export. So we should clear any pending bulkloads upon full backup.
 # There is currently no linking of bulkload state to backupRoot. It's possible to have multiple backupRoots for tables. For example, you might backup to 2 destinations with different schedules. Currently whichever backupRoot does an incremental backup first will be the one to include bulkloads, then the system_bulk table. We need some sort of mapping of bulkload to backupRoot, and we should only delete the rows from system_bulk once the files have been included in all active backupRoots.
Bug: Fix race condition in TestCleanerChore.retriesIOExceptionInStatus
Status: Patch Available | Priority: Major | Resolution: 
Created: 01/Mar/24 06:45
Affects Version/s: 2.6.0

We occasionally get a test failure in TestCleanerChore.retriesIOExceptionInStatus. For example, from a [recent PR build|https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5725/1/testReport/org.apache.hadoop.hbase.master.cleaner/TestCleanerChore/precommit_checks___yetus_jdk11_hadoop3_checks___retriesIOExceptionInStatus/] on branch-2.6,

{noformat}
java.util.concurrent.ExecutionException: java.io.IOException: whomp whomp.
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
	at org.apache.hadoop.hbase.master.cleaner.TestCleanerChore.retriesIOExceptionInStatus(TestCleanerChore.java:163)
...
Caused by: java.io.IOException: whomp whomp.
	at org.apache.hadoop.hbase.master.cleaner.TestCleanerChore$1.listStatus(TestCleanerChore.java:134)
	at org.apache.hadoop.hbase.master.cleaner.CleanerChore.traverseAndDelete(CleanerChore.java:475)
	at org.apache.hadoop.hbase.master.cleaner.CleanerChore.lambda$chore$0(CleanerChore.java:258)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	... 1 more
{noformat}

This looks like a race condition where the chore manages an entire execution between when the flag is flipped and when the test thread gets back around to continuing execution. Make the test a little more pessimistic about its view of the world.
Bug: Improve debugging for failures in procedure tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Feb/24 11:51

We see unit test failures in Jenkins that look like this:

{noformat}
java.lang.IllegalArgumentException: run queue not empty
	at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:143)
	at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:332)
	at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:665)
	at org.apache.hadoop.hbase.procedure2.ProcedureTestingUtility.restart(ProcedureTestingUtility.java:132)
	at org.apache.hadoop.hbase.procedure2.ProcedureTestingUtility.restart(ProcedureTestingUtility.java:100)
	at org.apache.hadoop.hbase.master.procedure.MasterProcedureTestingUtility.restartMasterProcedureExecutor(MasterProcedureTestingUtility.java:85)
	at org.apache.hadoop.hbase.master.assignment.TestRollbackSCP.testFailAndRollback(TestRollbackSCP.java:180)
{noformat}

This isn't enough information to debug the situation. The test code in question looks reasonable enough -- it clears the object for re-use between tests. However, somewhere between stop/clear/start we miss something. Add some toString implementations and dump the objects in the preconditions.
Bug: RegionSizeCalculator throws NPE when regions are in transition
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Feb/24 13:44

When a region is in transition, it may briefly have a null ServerName in meta. The RegionSizeCalculator calls RegionLocator.getAllRegionLocations() and does not handle the possibility that a RegionLocation.getServerName() could be null. The ServerName is eventually passed into an Admin call, which results in an NPE.

This has come up in other contexts. For example, taking a look at getAllRegionLocations() impl, we have checks to ensure that we don't call null server names. We need to similarly handle the possibility of nulls in RegionSizeCalculator.
Bug: StorefileRefresherChore should not refresh readonly table
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jan/24 08:47
Affects Version/s: 2.4.17

According to semantics, when StoreFile Refresher is turned on, only the area with replicas should be refreshed. *Tables with read-only attributes will not be refreshed.*
There are the following scenarios: Assume that some tables in the database have replica enabled, while others do not. If a table that does not have replica enabled needs to perform migration tasks - first enable readonly and then export the snapshot, StorefileRefresherChore will always refresh the table that does not have replica enabled but performs migration tasks. This is obviously illogical.
Therefore, StorefileRefresherChore needs to determine whether the region has a read-only copy, and refresh it if it has a read-only copy. And tables with read-only properties are not refreshed.

patch:[HBASE-28313|https://github.com/apache/hbase/pull/5641]
Bug: When AbstractRpcClient calls getConnection, the status of conn should be checked
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 19/Jan/24 04:12
Affects Version/s: 2.5.7

h3. 1. Question

The connection will only be created when it does not exist. Since the conn in *connections* are cleaned asynchronously and regularly, it is very likely that we will get a closed connection.
{code:java}
/**
 * Get a connection from the pool, or create a new one and add it to the pool. Connections to a
 * given host/port are reused.
 */
private T getConnection(ConnectionId remoteId) throws IOException {
  if (failedServers.isFailedServer(remoteId.getAddress())) {
    if (LOG.isDebugEnabled()) {
      LOG.debug("Not trying to connect to " + remoteId.getAddress()
        + " this server is in the failed servers list");
    }
    throw new FailedServerException(
      "This server is in the failed servers list: " + remoteId.getAddress());
  }
  T conn;
  synchronized (connections) {
    if (!running) {
      throw new StoppedRpcClientException();
    }
    conn = connections.getOrCreate(remoteId, () -> createConnection(remoteId));
    conn.setLastTouched(EnvironmentEdgeManager.currentTime());
  }
  return conn;
} {code}
h3. 2. Solution
 * When getConnection, we should to check whether the conn is active
 * Optimize the impl of isActive and increase the judgment of channel status

{code:java}
@Override
public boolean isActive() {
  return channel != null && channel.isActive();
} {code}
I am new to HBase, so my understanding may be wrong. Please correct me.
Bug: Client ingegration tests fails for branch-2/branch-2.6
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Feb/24 13:12

In HBASE-28331, we fixed the problem that we should use the latest hadoop3 tarballs, for generating hdfs-site.xml.

But for branch-2.x, we still use hadoop2 when compiling hbase by default, so there is no org.apache.hadoop.ipc.ProtobufRpcEngine2 class. But the hadoop 3.3.x will generate hdfs-site.xml with this config, then leads to the failure of hbase start up.

Should we change to build with hadoop3 while running on top of hadoop3? Since for branch-2.x, we will publish artifacts for hadoop3.
Bug: Broken test TestHRegionWithInMemoryFlush
Status: Open | Priority: Major | Resolution: 
Created: 21/Feb/24 09:17
Affects Version/s: 4.0.0-alpha-1

{{TestHRegionWithInMemoryFlush}} is broken in Jenkins ([PR35693|https://github.com/apache/hbase/pull/5693]) and in local testing. It times out while waiting for HMaster to startup.
Bug: Remove the need for ADMIN permissions for listDecommissionedRegionServers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Feb/24 23:07
Affects Version/s: 2.4.17

Why we need {{ADMIN}} permissions for {{AccessController#preListDecommissionedRegionServers}} ?

From Phoenix, we are calling {{Admin#getRegionServers(true)}} where the argument {{excludeDecommissionedRS}} is set to true. Refer [here|https://github.com/apache/hbase/blob/branch-2.5/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java#L1721-L1730].
If {{excludeDecommissionedRS}}  is set to true and if we have {{AccessController}} co-proc attached, it requires ADMIN permissions to execute {{listDecommissionedRegionServers}} RPC. Refer [here|https://github.com/apache/hbase/blob/branch-2.5/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java#L1205-L1207].
 
{code:java}
  @Override
  public void preListDecommissionedRegionServers(ObserverContext<MasterCoprocessorEnvironment> ctx)
    throws IOException {
    requirePermission(ctx, "listDecommissionedRegionServers", Action.ADMIN);
  }
{code}
I understand that we need ADMIN permissions for _preDecommissionRegionServers_ and _preRecommissionRegionServer_ because it changes the membership of regionservers but I don’t see any need for ADMIN permissions for _listDecommissionedRegionServers_.  Do you think we can remove need for ADMIN permissions for  _listDecommissionedRegionServers_ RPC?
Bug: Cleanup / Resurrect hbase-table-reporter 
Status: Open | Priority: Major | Resolution: 
Created: 28/Feb/24 17:31

Seems [hbase-table-reporter|https://github.com/apache/hbase-operator-tools/tree/master/hbase-table-reporter] code is not maintained or maybe not even used? We have not had any change since a long time. 

With this JIRA, I plan on doing following:
 * Bump dependencies to be latest, remove redundant ones
 * Update docs as I am not sure why it references com.apple.hbase.Reporter
 * Do basic sanity based on docs and ensure it works
Bug: region size can be wrong from RegionSizeCalculator
Status: Open | Priority: Major | Resolution: 
Created: 23/Feb/24 09:17
Affects Version/s: 3.0.0-beta-1

The RegionSizeCalculator calculates region byte size using the following method
{code:java}
private static final long MEGABYTE = 1024L * 1024L;

long regionSizeBytes =
  ((long) regionLoad.getStoreFileSize().get(Size.Unit.MEGABYTE)) * MEGABYTE; {code}
However, this method will lose accuracy. For example, the result of 
{code:java}
((long) new Size(1, Size.Unit.BYTE).get(Size.Unit.MEGABYTE)) * MEGABYTE {code}
is 0. This will result in a TableInputSplit with a length of 0, but in fact this TableInputSplit has a small amount of data.

 

This TableInputSplit will be ignored if we enable spark.hadoopRDD.ignoreEmptySplits.
Bug: Move namespace to RS group can support multiple options to move tables under it
Status: Open | Priority: Major | Resolution: 
Created: 27/Feb/24 13:36

As discussed here
[https://github.com/apache/hbase/pull/5661#issuecomment-1949090826]

Move namespaces to rsgroup can have multiple options like:
 # *Move All Tables:* This option involves moving all tables within a namespace to the new RS group, regardless of the current RS group they belong to. This would effectively consolidate all tables of the namespace into the new RS group. This is the same as the current implementation so it can be default behavior.

 # {*}Move Tables present in current RS group{*}: With this option, only tables belonging to the current RS group of the namespace would be moved to the new RS group. This provides more granular control, allowing users to choose specific tables to move based on their needs. If the namespace does not belong to any RS group, the namespace tables in the default RS group would be moved to the new RS group.

 # *Move No Tables:* This option involves changing the RS group of the namespace without moving any tables. Existing tables would remain in their current RS group. This could be useful if there's a desire to separate the namespaces by RS group but without immediately moving the tables.

With this options we can give control to user choose proper option based on the requirements.
Bug: [hbase-thirdparty] bump deps for 4.1.6 release
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Feb/24 15:18

* protobuf 3.24.3 -> 3.25.2
* guava 32.1.2-jre -> 33.0.0-jre
* commons-cli 1.5.0 -> 1.6.0
* jetty 9.4.53.v20231009 -> 9.4.54.v20240208
* jersey 2.40 -> 2.41
* javassist 3.29.2-GA -> 3.30.2-GA
* jackson-jaxrs-json-provider 2.15.2 -> 2.16.1
Bug: Default user quotas are refreshing too frequently
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Feb/24 17:21

In [https://github.com/apache/hbase/pull/5666] we introduced default user quotas, but I accidentally called UserQuotaState's default constructor rather than passing in the current timestamp. The consequence is that we're constantly refreshing these default user quotas, and this can be a bottleneck for horizontal cluster scalability.

This should be a 1 line fix in QuotaUtil's buildDefaultUserQuotaState method.
Bug: WAL readers treat any exception as EOFException, which can lead to data loss
Status: Open | Priority: Major | Resolution: 
Created: 23/Feb/24 19:16

In HBASE-28390, I found a bug in our WAL compression which manifests as an IllegalArgumentException or ArrayIndexOutOfBoundException. Even worse is that ProtobufLogReader.readNext catches any Exception and rethrows it as an EOFException. EOFException gets handled in a variety of ways by the readers of WALs, and not all of them make sense for an exception that isn't really EOF.

For example, WALInputFormat catches EOFException and returns false for nextKeyValue(), effectively skipping the rest of the WAL file but not failing the job.

ReplicationSourceWALReader has some much more complicated handling of EOFException.

RegionServer replayRecoveredEdits stops trying to read the WAL, which might mean data loss.

We probably should better handle these exceptions which are not EOF exceptions.
Bug: Makeuse of jenkins libraries to reuse the stage definition code to run with multiple JDKs
Status: Open | Priority: Major | Resolution: 
Created: 22/Feb/24 17:51

As for the discussion at https://github.com/apache/hbase/pull/5689#pullrequestreview-1892363058 would be better to make use jenkins libraries to reuse most of the code in JenkinsFiles to support multiple JDKs.
Bug: Field sorting is broken in HBase Web UI
Status: Open | Priority: Major | Resolution: 
Created: 21/Feb/24 10:10
Affects Version/s: 2.6.0

Field sorting is broken in HBase Web UI for tables.jsp. As part of this Jira plan on auditing all such elements which support sorting and fixing them. Also we should avoid index based field sorting (which is prone to getting broken) as addition of new fields breaks them as the indexes may change.
Bug: HBase Operator Tools fails to compile with hbase 2.6.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Feb/24 16:10

HBase Operator Tools fails to compile with hbase 2.6.0.
{code:java}
[ERROR] /file_path/hbase-operator-tools/hbase-hbck2/src/main/java/org/apache/hbase/hbck1/ReplicationChecker.java:[59,49] method getReplicationPeerStorage in class org.apache.hadoop.hbase.replication.ReplicationStorageFactory cannot be applied to given types;
[ERROR]   required: org.apache.hadoop.fs.FileSystem,org.apache.hadoop.hbase.zookeeper.ZKWatcher,org.apache.hadoop.conf.Configuration
[ERROR]   found: org.apache.hadoop.hbase.zookeeper.ZKWatcher,org.apache.hadoop.conf.Configuration
[ERROR]   reason: actual and formal argument lists differ in length {code}
Seems there is a breaking change between [https://github.com/apache/hbase/blob/branch-2.5/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ReplicationStorageFactory.java] vs [https://github.com/apache/hbase/blob/branch-2.6/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ReplicationStorageFactory.java] where a public method has been dropped, which is used by operator tools and hence  the build will fail for it. See [https://github.com/apache/hbase-operator-tools/blob/master/hbase-hbck2/src/main/java/org/apache/hbase/hbck1/ReplicationChecker.java#L58] where the effected method is invoked.

Since ReplicationStorageFactory is @InterfaceAudience.Private so maybe it is fine.

Will try to fix and make changes in hbase-operator-tools to fall back to new method, in case if build with branch-2.6

CC: [~zhangduo]
Bug: ChaosMonkey batch suspend/resume action assume shell implementation
Status: Open | Priority: Major | Resolution: 
Created: 13/Feb/24 22:59

These two actions have code like this:
{code:java}
case SUSPEND:
  server = serversToBeSuspended.remove();
  try {
    suspendRs(server);
  } catch (Shell.ExitCodeException e) {
    LOG.warn("Problem suspending but presume successful; code={}", e.getExitCode(), e);
  }
  suspendedServers.add(server);
  break; {code}
This only catches that one Shell.ExitCodeException, but operators may have an implementation of ClusterManager which does not use shell. We should expand this to catch all exceptions.

The implication here is that the uncaught exception propagates, and we don't add the server to suspendedServers. If the suspension actually succeeded, this leaves some processes in a permanently suspended state until manual intervention occurs.
Bug: Noisy exception from FlushRegionProcedure when result is CANNOT_FLUSH
Status: Open | Priority: Major | Resolution: 
Created: 13/Feb/24 22:19

Running ITBLL with chaos monkey in HBASE-28233. I noticed lots of exceptions:
{code:java}
[RS_FLUSH_OPERATIONS-regionserver/test-host:60020-1 {event_type=RS_FLUSH_REGIONS, pid=741536}] ERROR org.apache.hadoop.hbase.regionserver.handler.RSProcedureHandler: pid=741536
java.io.IOException: Unable to complete flush {ENCODED => 371d2ba6875913542893642c94634226, NAME => 'IntegrationTestBigLinkedList,-\x82\xD8-\x82\xD8-\x80,1707761077516.371d2ba6875913542893642c94634226.', STARTKEY =
> '-\x82\xD8-\x82\xD8-\x80', ENDKEY => '33333330'}
        at org.apache.hadoop.hbase.regionserver.FlushRegionCallable.doCall(FlushRegionCallable.java:61) ~[hbase-server-2.6-hubspot-SNAPSHOT.jar:2.6-hubspot-SNAPSHOT]
        at org.apache.hadoop.hbase.procedure2.BaseRSProcedureCallable.call(BaseRSProcedureCallable.java:35) ~[hbase-server-2.6-hubspot-SNAPSHOT.jar:2.6-hubspot-SNAPSHOT]
        at org.apache.hadoop.hbase.procedure2.BaseRSProcedureCallable.call(BaseRSProcedureCallable.java:23) ~[hbase-server-2.6-hubspot-SNAPSHOT.jar:2.6-hubspot-SNAPSHOT]
        at org.apache.hadoop.hbase.regionserver.handler.RSProcedureHandler.process(RSProcedureHandler.java:51) ~[hbase-server-2.6-hubspot-SNAPSHOT.jar:2.6-hubspot-SNAPSHOT]
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) ~[hbase-server-2.6-hubspot-SNAPSHOT.jar:2.6-hubspot-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:840) ~[?:?] {code}
I took a look at the HRegion.flushcache code, and there are 3 reasons for CANNOT_FLUSH. All only print at debug log level and none look like actual errors.

I think we shouldn't throw an exception here, or at least should downgrade to debug. It looks like a problem, but isn't (i dont think).

cc [~frostruan]
Bug: NPE calling bootstrapNodeManager during RegionServer initialization
Status: Open | Priority: Major | Resolution: 
Created: 13/Feb/24 22:11

Shortly after starting up, if a RegionServer is getting requests from clients before it's ready (i.e. it restarts and they haven't cleared meta cache yet), it will throw an NPE. This is because netty may bind and start accepting requests before HRegionServer.preRegistrationInitialization finishes.

I think this is similar to https://issues.apache.org/jira/browse/HBASE-28088. It's not critical because the RS self-resolves within a few seconds, but it causes noise in the logs and probably errors for clients.
{code:java}
2024-02-13T18:24:02,537 [RpcServer.default.FPBQ.handler=6,queue=6,port=60020 {}] ERROR org.apache.hadoop.hbase.ipc.RpcServer: Unexpected throwable object
java.lang.NullPointerException: Cannot invoke "org.apache.hadoop.hbase.regionserver.BootstrapNodeManager.getBootstrapNodes()" because "this.bootstrapNodeManager" is null
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getBootstrapNodes(HRegionServer.java:4179) ~[hbase-server-2.6-hubspot-SNAPSHOT.jar:2.6-hubspot-SNAPSHOT]
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.getAllBootstrapNodes(RSRpcServices.java:4140) ~[hbase-server-2.6-hubspot-SNAPSHOT.jar:2.6-hubspot-SNAPSHOT]
        at org.apache.hadoop.hbase.shaded.protobuf.generated.BootstrapNodeProtos$BootstrapNodeService$2.callBlockingMethod(BootstrapNodeProtos.java:1259) ~[hbase-protocol-shaded-2.6-hubspot-SNAPSHOT.jar:2.6-hubspot-SNAPSHOT]
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:438) ~[hbase-server-2.6-hubspot-SNAPSHOT.jar:2.6-hubspot-SNAPSHOT]
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) ~[hbase-server-2.6-hubspot-SNAPSHOT.jar:2.6-hubspot-SNAPSHOT] {code}