av9ash/GitBugs huggingface dataset:
Bug: Backup tests are categorically flakey
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/May/21 20:50

Here's some logs from a PR build vs. master that suffered a significant number of failures in the backup tests. I suspect that a single improvement could fix all of these tests to be more robust.

{noformat}
Test Name
Duration
Age
 precommit checks / yetus jdk8 Hadoop3 checks / org.apache.hadoop.hbase.backup.TestBackupDeleteRestore.testBackupDeleteRestore	6 min 23 sec	1
 precommit checks / yetus jdk8 Hadoop3 checks / org.apache.hadoop.hbase.backup.TestBackupDeleteRestore.(?)	1 min 6 sec	1
 precommit checks / yetus jdk8 Hadoop3 checks / org.apache.hadoop.hbase.backup.TestBackupMerge.TestIncBackupMergeRestore	5 min 3 sec	1
 precommit checks / yetus jdk8 Hadoop3 checks / org.apache.hadoop.hbase.backup.TestBackupMerge.(?)	1 min 6 sec	1
 precommit checks / yetus jdk8 Hadoop3 checks / org.apache.hadoop.hbase.backup.TestFullBackupSet.testFullBackupSetExist	6 min 16 sec	1
 precommit checks / yetus jdk8 Hadoop3 checks / org.apache.hadoop.hbase.backup.TestFullBackupSet.(?)	1 min 6 sec	1
 precommit checks / yetus jdk8 Hadoop3 checks / org.apache.hadoop.hbase.backup.TestIncrementalBackupMergeWithFailures.TestIncBackupMergeRestore	5 min 55 sec	1
 precommit checks / yetus jdk8 Hadoop3 checks / org.apache.hadoop.hbase.backup.TestIncrementalBackupMergeWithFailures.(?)	1 min 6 sec	1
 precommit checks / yetus jdk8 Hadoop3 checks / org.apache.hadoop.hbase.backup.TestIncrementalBackupWithBulkLoad.TestIncBackupDeleteTable	5 min 56 sec	1
 precommit checks / yetus jdk8 Hadoop3 checks / org.apache.hadoop.hbase.backup.TestIncrementalBackupWithBulkLoad.(?)	1 min 6 sec	1
 precommit checks / yetus jdk8 Hadoop3 checks / org.apache.hadoop.hbase.backup.TestRestoreBoundaryTests.testFullRestoreSingleEmpty	6 min 5 sec	1
 precommit checks / yetus jdk8 Hadoop3 checks / org.apache.hadoop.hbase.backup.TestRestoreBoundaryTests.testFullRestoreMultipleEmpty	0.17 sec	1
 precommit checks / yetus jdk8 Hadoop3 checks / org.apache.hadoop.hbase.backup.TestRestoreBoundaryTests.(?)
{noformat}

https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3249/4/testReport/
Bug: Replication may lose data during regionserver restart when multiwal is in use
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/May/21 11:47
Affects Version/s: 2.1.4

When multiwal is used, restarting the region server may result in the loss of replication data.This may be due to a non thread safe update of the queues in the ReplicationSource#
enqueueLog.

.
Bug: 'False' should be 'True' in auditlog of listLabels
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/May/21 12:18

In method `listLabels` at line 932, it should be
{code:java}
logResult(true, "listLabels", "Listing labels allowed", null, null, regex);{code}
instead of 
{code:java}
logResult(false, "listLabels", "Listing labels allowed", null, null, regex);
{code}
as listLabels is executed normally.
Bug: Provide javadoc strings for all significant public symbols in o.a.h.h.client.*
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 13/May/21 23:34

While playing with the jshell via HBASE-25841, I noticed that several of our classes and methods in this package don't have javadoc. We should do this anyway, but with jshell's tab-completion to show javadoc, this became particularly noticeable. Specifically, the very first class one must use, {{ConnectionFactory}}, is without a class-level javadoc. I assume there are plenty more.
Bug: [hbase-operator-tools] Extra checks on RegionsMerger to avoid submitting merge requests for regions with merge qualifiers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/May/21 19:56
Affects Version/s: hbase-operator-tools-1.1.0

When iterating over existing region over cycles to perform merges, RegionsMerger may try to merge two regions that had been recently merged and would then have still have a "merge" qualifier to sign these still refers to parent regions files. That means the request is useless, the created SCPs would be aborted right away once it finds the so called merge qualifiers. RegionsMerger should avoid submitting these attempts, by checking for merge qualifiers and skipping those that still have merge qualifiers.
Bug: Add documentation for LoadBalancer about synchronization
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/May/21 13:08

In general, the balanceCluster call is not thread safe and we only allow one balanceCluster call at the same time, so we need to synchronize balanceCluster, onConfigurationChange, and setClusterMetrics(which will be called periodically, maybe we should rename it to updateClusterMetrics).

Most setters do not need to be synchronized as they will only be called once before initialization.

And anothe group of methods are XXXAssignment, these methods should be thread safe and should not be synchronized to improve the assignment performance.
Bug: Revisit the synchronization of balancer implementation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Apr/21 08:03

In StochasticLoadBalancer, some methods are synchronized, for example, setClusterMetrics, setConf, balanceTable, etc.

What I can see is that, we have ConfigurationObserver support, so we need to make sure that the configuration updating does not effect the balance operation, so we need synchronization here. But do we support two balance operations at the same time? And what about other balancer implementations?

I think we need to make the synchronization more clearer.
Bug: NPE while getting Balancer decisions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/May/21 19:22

With HBASE-25790 in place, we have new balancer queue: BalancerRejection in addition to BalancerDecision. However, now while retrieving records from ring buffer, only if exactly one of the ring buffers are enabled, we get NPE while retrieving the other one. Ideally it should return empty results.

Result of get_balancer_decision when only rejection queue is enabled but not the balancer decision one.

 
{code:java}
2021-05-02T21:22:31,830 WARN  [RPCClient-NioEventLoopGroup-1-2] client.AsyncRpcRetryingCaller: Call to master failed, tries = 6, maxAttempts = 7, timeout = 1200000 ms, time elapsed = 2360 ms2021-05-02T21:22:31,830 WARN  [RPCClient-NioEventLoopGroup-1-2] client.AsyncRpcRetryingCaller: Call to master failed, tries = 6, maxAttempts = 7, timeout = 1200000 ms, time elapsed = 2360 msjava.io.IOException: java.io.IOException at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:463) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:135) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.master.MasterRpcServices.getBalancerDecisions(MasterRpcServices.java:3432) at org.apache.hadoop.hbase.master.MasterRpcServices.getLogEntries(MasterRpcServices.java:3395) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:395) ... 3 more
{code}
Bug: [hbase-operator-tools]Add tool for identifying "unknown servers" from master logs, then submit SCPs for each of those.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/May/21 10:50
Affects Version/s: hbase-operator-tools-1.1.0

Mostly useful when reusing hbase data dir from a previously destroyed cluster to a new cluster definition (see HBASE-24286) with hbase versions prior to 2.2.7, 2.3.5 and 2.4.7, where there's no support for *scheduleSCPsForUnknownServers*. 

*HBCK2* has this *scheduleRecoveries* method, that submits SCPs for individual RS names passed as parameter, but it still requires operators to manually parse these unknown server names from master logs in order to build the command. This tool automates the parsing step, building the list of proper server names, then reusing *HBCK2.scheduleRecoveries* to trigger SCPs.
Bug: NamedQueue 'BalancerRejection' for recent history of balancer skipping
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Apr/21 10:34

[HBASE-24528|https://issues.apache.org/jira/browse/HBASE-24528] implements a NamedQueue that saved recent history of balancer decision details. Since we want to know exactly why the balancer is skipping to run, we need a new namedQueue called 'BalancerRejection' to hold the history of all details information why the balancer was rejected to run(skipping). LogEntry of 'BalancerRejection' should contain the information below:

1. Reason, to show why this time balancer was rejected

Maybe look like:

 
{code:java}
(cost1*multiplier1+cost2*multiplier2+...+costn*multipliern)/sumMultiplier=0.025 <= minCostNeedBalance(0.35)
{code}
2. Detail cost factors and multipliers

Maybe look like:
{code:java}
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer$RegionCountSkewCostFunction cost:0.0 multiplier:500.0

org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer$MoveCostFunction cost:0.0 multiplier:7.0

org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer$RackLocalityCostFunction cost:1.0 multiplier:15.0
{code}
Bug: Move all the intialization work of LoadBalancer implementation to initialize method
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/May/21 14:58

To make sure that we have all the necessary properties set before initialization.

And also, this could also help to make the synchronization more clear, as before initialization, we should not call balance or assignment methods yet, so we do not need to synchronize initialization method. We only need to synchronize when we want to change the configuration.
Bug: Fix spotbugs warnings on branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/May/21 08:12

{noformat}
Dodgy code Warnings
Code	Warning
UC	Useless condition: it's known that major == true at this point
Bug type UC_USELESS_CONDITION (click for details)
In class org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor
In method org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.performCompaction(Compactor$FileDetails, InternalScanner, CellSink, long, boolean, ThroughputController, boolean, int)
Value major == true
Unreachable code at DefaultMobStoreCompactor.java:[line 210]
At DefaultMobStoreCompactor.java:[line 210]
{noformat}

https://github.com/apache/hbase/blob/afa9836b87f9c909ac4fc18d904a5b5b0a3dfff6/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java#L211

{code}
      if (major) {
        try {
          delFileWriter = mobStore.createDelFileWriterInTmp(new Date(fd.latestPutTs),
            fd.maxKeyCount, major ? majorCompactionCompression : minorCompactionCompression,
            store.getRegionInfo().getStartKey());
        } catch (IOException e) {
          LOG.warn(
            "Failed to create del writer, "
            + "we will continue the compaction by writing delete markers directly in store files",
            e);
        }
      }
{code}

The 'major ? majorCompactionCompression : minorCompactionCompression' is useless as we have a 'if (major)' test above.
Bug: Remove redundant AM in-memory state changes in CatalogJanitor
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/May/21 16:50

In CatalogJanitor we schedule GCRegionProcedure to clean up both filesystem and in-memory state after a split, and GCMultipleMergedRegionsProcedure to do the same for merges. 

Both GCRegionProcedure and GCMultipleMergedRegionsProcedure clean up in-memory state, but CatalogJanitor also does this redundantly just after scheduling the procedures. 

The cleanup should be done in only one place. Presumably we are using the procedures to do it in a principled way.  This is least a nit, but probably a source of future bugs. Remove the redundancy in CatalogJanitor and fix any follow on issues, like test failures.
Bug: Make LoadBalancer not extend Configurable interface
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/May/21 12:58

On master, LoadBalancer has a setClusterInfoProvider method and on branch-2, LoadBalancer has a setMasterServices method. Both ClusterInfoProvider and MasterServices could give a Configuration out so I do not think we need to expose another setConf method.

When refactoring StochasticLoadBalancer, the initialization order is also a pain as after setConf, we will load a lot of configuration but at that time, the ClusterInfoProvider is not present, and easy to cause NPE...

And it could also help simplity locking.
Bug: Deadlock during peer cleanup with NoNodeException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Apr/21 01:35
Affects Version/s: 1.7.0

We have observed that replication source metrics for peer exists on some region servers even though peer has been removed.  This is because when we encounter the NoNodeException in ReplicationSource, it calls the `peerRemoved` workflow which should eventually terminate the source and removes the source from the source manager. Now, the problem is ReplicationSource thread terminates itself and thus the action to removePeer is not complete leaving the metrics there forever for source. This is the flow, replication source trying to clean wals [here|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java#L801] and on NoNodeException it calls the [peerRemoved|https://github.com/apache/hbase/blob/b231dd620f107b488b88599e16dc846eb856972c/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java#L244] and terminate the source (itself), leaving the terminated source there in sourcemanager and not clearing it's [metrics|https://github.com/apache/hbase/blob/b231dd620f107b488b88599e16dc846eb856972c/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java#L645].
Bug: Ignore duplicate split requests from regionserver reports
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/May/21 20:52
Affects Version/s: 2.4.3

A SplitTableRegionProcedure may already be running when a regionserver report is received that includes a split request. The outcome is multiple SplitTableRegionProcedure procedures scheduled for the split request, only one of which can succeed. The others error out. Perhaps it is only WARN level noise in the logs that will cause operator concern, but is clearly not correct. 

Do not create a split procedure in response to a region state change report if the region is not open or already splitting.
Bug: Use double instead of Double in StochasticLoadBalancer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/May/21 14:28

{code}
  // to save and report costs to JMX
  private Double curOverallCost = 0d;
  private Double[] tempFunctionCosts;
  private Double[] curFunctionCosts;
{code}

It seems we never do null check on these field or the elements, so I do not see any reason we should use Double here. Just use double instead.
Bug: RegionStates#getAssignmentsForBalancer should only care about OPEN or OPENING regions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/May/21 20:56

RegionStates#getAssignmentsForBalancer is used by the HMaster to collect all regions of interest to the balancer for the next chore iteration. We check if a table is in disabled state to exclude regions that will not be of interest (because disabled regions are or will be offline) or are in a state where they shouldn't be mutated (like SPLITTING ). The current checks do not get to the point so are not comprehensive. 

We should filter out all regions that are not in OPEN or OPENING state when building the set of interesting regions for the balancer to consider. Only regions open (or opening) on the cluster are of interest to balancing calculations. Regions in all other states can be expected to not be of interest -- either offline (OFFLINE, or FAILED_*), not subject to balancer decisions now (SPLITTING, SPLITTING_NEW, MERGING, MERGING_NEW), or will be offline shortly (CLOSING) -- until at least the next chore iteration.
Bug: [branch-1] Precommit fails to build docker due to python-dateutil
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/May/21 03:43

02:34:22  Downloading/unpacking python-dateutil
02:34:22    Cannot fetch index base URL https://pypi.python.org/simple/
02:34:22    Could not find any downloads that satisfy the requirement python-dateutil
02:34:22  Cleaning up...
02:34:22  No distributions at all found for python-dateutil
02:34:22  Storing debug log for failure in /root/.pip/pip.log
02:34:22  The command '/bin/sh -c pip install python-dateutil' returned a non-zero code: 1
02:34:22  ERROR: Docker failed to build yetus/hbase:edccfe439a.
Bug: TestCacheOnWrite sometimes times out
Status: Open | Priority: Major | Resolution: 
Created: 28/Apr/21 22:00
Affects Version/s: 3.0.0-alpha-1

From a failed Jenkins run, I see

{noformat}
-------------------------------------------------------------------------------
Test set: org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite
-------------------------------------------------------------------------------
Tests run: 107, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 780.027 s <<< FAILURE! - in org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite
org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite  Time elapsed: 7.651 s  <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 780 seconds
	at app//org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testCachingDataBlocksDuringCompactionInternals(TestCacheOnWrite.java:505)
	at app//org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testCachingDataBlocksDuringCompaction(TestCacheOnWrite.java:582)
{noformat}
Bug: Find a way to know whether logQueue goes empty when ReplicationSource is running on ReplicationServer
Status: Open | Priority: Major | Resolution: 
Created: 28/Apr/21 11:47

HBASE-25110 we choose to use ZK to notify ReplicationServer that a new wal was generated, this is asynchronous. And then we got a problem, the shipper thread and the wal reader thread may go terminated as logQueue goes empty before receiving the notification of new wal.

So we now need find a way to know whether logQueue is really empty after the last wal in logQueue is consumed.
Bug: Other protobuf services, other than ClientService, cannot be invoked with RpcClient
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 19/Apr/21 09:48
Affects Version/s: 2.4.2

I'd like to invoke a coprocessor with RpcClient, and I find the service_name in function RpcSerde::Header() has been hard coded, and when I change the service_name to another one, the call failed.


Before I change the service_name, I get the following error message:
{noformat}
# ./adamas-echo-test 
/usr/src/hbase/hbase-native-client/src/test/adamas-echo-test.cc:83: Failure
Failed
Failed because: N5hbase15RemoteExceptionE: /172.17.0.4:16000 is unable to read call parameter from client 172.17.0.4;
java.lang.UnsupportedOperationException: adamasEcho,
Shouldn't get here, exception is not expected for RPC adamasEcho.{noformat}

As I look into the hbase server, I find the service name is ClientService, which is not what I expected.

then I hard code the "ClientService" to my own service, i.e. the service name in my own .proto file, by changing the INTERFACE macro.
but when I connect to the hbase server, it reset the connection immediately, showing:
{noformat}
# ./adamas-echo-test 
/usr/src/hbase/hbase-native-client/src/test/adamas-echo-test.cc:83: Failure
Failed
Failed because: N5hbase11IOExceptionE: Connection closed to server:172.17.0.4:16000,
Shouldn't get here, exception is not expected for RPC adamasEcho.{noformat}
and I look into the wingle library, I find the errno is 104, which means 'Connection reset by peer'.

So what is the problem?
Thank you!
Bug: Http InfoServers should honor gzip encoding when requested
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Apr/21 23:15
Affects Version/s: 2.3.0

Working on another project, I noticed that our InfoServer does not honor the {{Accept-Encoding}} HTTP header when present. It appears to be a relatively trivial matter to add the GzipHandler provided by Jetty. I'm not sure if there are other encodings we should support, but this is the only one I found available in Jetty itself (the other one by browser sent was "deflate").
Bug: [branch-1] Backport improvements to FSHLog from branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Apr/21 19:00
Affects Version/s: 1.6.0

From c96b642f15d (HBASE-15265 Implement an asynchronous FSHLog)
    - Better formatting; use { } instead of dangling ifs, elses, etc. in RingBufferEventHandler.
    
    From b1269ec57ff (HBASE-19811 Fix findbugs and error-prone warnings in hbase-server (branch-2))
    - Change syncFuturesCount in RingBufferEventHandler from 'volatile int' to AtomicInteger.
      'volatile' is insufficient for multithreaded access.
    
    From afc1746757f (HBASE-24034 [Flakey Tests] A couple of fixes and cleanups)
    - Make a local copy of takeSyncFuture after we get it in SyncRunner#run.
      This looks like a workaround for a JIT/compiler bug in some Java versions. (SCARY)
Bug: [branch-1] TestAtomicOperation.testMultiRowMutationMultiThreads deadlock
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Apr/21 22:08

TestAtomicOperation.testMultiRowMutationMultiThreads deadlocks.

There is an easy fix for the test that synchronizes on the CHM instead of the object. We already have a Findbugs exceptions for synchronization on the CHM and get-then-set on it is what we synchronizing for anyway.

This is only relevant for branch-1 because we have to synchronize get-then-set due to Java 7 compatibility. For branch-2 and master we use CHM#computeIfPresent.
Bug: [branch-1] Upgrade com.lmax.disruptor to 3.4.2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Apr/21 00:33

branch-1 is on 3.3.0. 

Partial changelog from 3.3.0 to 3.4.2:
* Fix for race condition on restart of BatchEventProcessor with 3 or more threads (in 3.4.2)
* Fix race between run() and halt() on BatchEventProcessor (in 3.4.1)
* Ensure that if halt is call the BatchEventProcessor will always exit cleanly (in 3.4.1)
* Remove unsafe based padding from RingBuffer. (in 3.4.0)
* Fix bug were the cursor could be accidentally changed (in 3.3.7)
* Add StoreLoad barrier before reading the gating sequences (in 3.3.6)
* Resignal any waiting threads when trying to publish to a full ring buffer (in 3.3.5)
* NPE using time wait blocking strategy (in 3.3.5)

branch-2 and master are on 3.4.2. 

3.4.3 change is "Add Automatic-Module-Name to MANIFEST.MF" which is harmless. 

Compatibility for branch-1 tested with openjdk version "1.7.0_272 (OpenJDK Runtime Environment (Zulu 7.40.0.15-CA-macosx) (build 1.7.0_272-b10) OpenJDK 64-Bit Server VM (Zulu 7.40.0.15-CA-macosx) (build 24.272-b10, mixed mode))"
Bug: RegionMover tool reassigning the already opened regions
Status: Open | Priority: Major | Resolution: 
Created: 21/Apr/21 10:49

{color:#000000}{color:#000000}Steps:
1)Install cluster with 3 RS (RS1,RS2,RS3)
2)Create table with 1000 regions 
3)Perform RegionMover to unload from RS1
4)While unloading the regions stop RS1 

Observation:
{color}{color}
{color:#000000}Regions already asigned to RS2 or RS3(by SCP) getting reassigned by the tool{color}
{color:#000000}2021-03-17 15:08:15,484 INFO [pool-3-thread-1] util.RegionMover: Moved Region table1,user25612,1615963232549.b19f9a6dac88a1fa21191c9fd0d515aa. cost:2.485
2021-03-17 15:08:16,663 INFO [pool-3-thread-1] util.RegionMover: Moving region:b19f9a6dac88a1fa21191c9fd0d515aa from RS1,16040,1615810740162 to RS2,16040,1615810791637{color}