av9ash/GitBugs huggingface dataset:
Bug: [HBOSS] HBaseObjectStoreSemantics.close should call super.close to make sure its own instance always get removed from FileSystem.CACHE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Aug/20 10:15
Affects Version/s: hbase-filesystem-1.0.0-alpha1

This came up when running bulkloads on hbase deployments using HBOSS. The fixes introduced by HBASE-23679 use *_FileSystem.closeAllForUGI(ugi)_* to make sure _*FileSystem*_ instances get cleared for the specific running UGI. Problem is that _*FileSystem.closeAllForUGI*_ does not remove the instance from _*FileSystem.CACHE*_ explicitly, it rather calls _*FileSystem.close*_, which in turn removes itself from _*FileSystem.CACHE*_. In this case, though, our _*FileSystem*_ implementation is _*HBaseObjectStoreSemantics*_, so _*FileSystem.closeAllForUGI*_ closes it, but does not remove it from _*FileSystem.CACHE*_, leading to all attempts to _*FileSystem.get*_ by the same UGI retrieving a closed _*HBaseObjectStoreSemantics*_ instance, ultimately failing as below:

 
{noformat}
2020-08-26 12:43:57,528 ERROR org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager: Failed to complete bulk load
java.io.IOException: Exception while testing a lock
        at org.apache.hadoop.hbase.oss.sync.ZKTreeLockManager.isLocked(ZKTreeLockManager.java:312)
        at org.apache.hadoop.hbase.oss.sync.ZKTreeLockManager.writeLockAbove(ZKTreeLockManager.java:183)
        at org.apache.hadoop.hbase.oss.sync.TreeLockManager.treeReadLock(TreeLockManager.java:282)
        at org.apache.hadoop.hbase.oss.sync.TreeLockManager.lock(TreeLockManager.java:449)
        at org.apache.hadoop.hbase.oss.HBaseObjectStoreSemantics.exists(HBaseObjectStoreSemantics.java:498)
        at org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$1.run(SecureBulkLoadManager.java:281)
        at org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$1.run(SecureBulkLoadManager.java:266)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:360)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1856)
        at org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager.secureBulkLoadHFiles(SecureBulkLoadManager.java:266)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.bulkLoadHFile(RSRpcServices.java:2445)
        at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42280)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:418)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
Caused by: java.lang.IllegalStateException: Expected state [STARTED] was [STOPPED] {noformat}
Bug: [Flakey Test] TestStochasticLoadBalancer.testMoveCostMultiplier
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/20 21:57
Affects Version/s: 2.3.0

I've noticed this test has gotten flakey on a couple PRs. The only recent change to the class under test is HBASE-24709.

Failure looks like:

{noformat}
[ERROR] Failures: 
[ERROR] org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer.testMoveCostMultiplier
[ERROR]   Run 1: TestStochasticLoadBalancer.testMoveCostMultiplier:304 expected:<3.0> but was:<7.0>
[ERROR]   Run 2: TestStochasticLoadBalancer.testMoveCostMultiplier:304 expected:<3.0> but was:<7.0>
[ERROR]   Run 3: TestStochasticLoadBalancer.testMoveCostMultiplier:304 expected:<3.0> but was:<7.0>
{noformat}
Bug: Optimize FSTableDescriptors.get to not always go to fs when cache miss
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Aug/20 09:37

Since in the below getAll method, if fsvisited is true, we will just return what we have in cache.

We should also do this in the get method, if fsvisited is true, we do not need to go to fs any more.

Another benefit here is that, when executing procedures on master, we do not need to go to meta region any more, we could just check if we have a table descriptor in cache.
Bug: Create versatile hbase-shell table formatter
Status: Open | Priority: Major | Resolution: 
Created: 18/Aug/20 19:13
Affects Version/s: 3.0.0-alpha-1

As a user, I would like a simple interface for shell output that can be expressed as a table (ie. output with a fixed number of columns and potentially many rows). To be clear, this new formatter is not specifically for HBase "tables." Table is used in the broader sense here.
h2. Goals
 - Do not require more than one output cell loaded in memory at a time
 - Support many implementations like aligned human-friendly tables, unaligned delimited, and JSON

h2. Non-goals
 - Don't load all the headers into memory at once.
 ** This may seem like a goal with merit, but we are unlikely to find a use case for this formatter with many columns. For example: since HBase tables aren't relational, our scan output will not have an output column for every HBase column. Instead, each output row will correspond to an HBase cell.
 ** It's also really useful to have the headers ahead of time, because it allows us to do things like JSON object output (where each row is represented with key-value pairs).

h2. Implementation

This patch was implemented as a stateful output formatter for data with a fixed number of output columns. Tracking state inside the formatter is an important design feature so that we don't have to feed the formatter all the data at once.

This patch comes with three implementations of the table formatter: Aligned, Unaligned, and JSON.
h2. Formatter Usage Pattern

The verbose way to use the formatter to print a table is as follows:
 1. call start_table to reset the formatter's state and pass configuration options
 2. call start_row to start writing a row
 3. call cell to write a single cell
 4. call close_row
 5. call close_table

Sometimes, it will feel like this is a lot of method calls, but these calls act as "hooks"
 and give each of the formatter implementations a chance to fill out all the content necessary
 between cells. To cut down on boilerplate, there are shortcut methods like row and single_value_table.
Bug: The postAppend method in SimpleRegionObserver should return result instead of null
Status: Open | Priority: Major | Resolution: 
Created: 27/Aug/20 07:27
Affects Version/s: 1.4.13

I found this error when testing batch append with SimpleRegionObserver：

java.lang.RuntimeException: java.lang.IllegalStateException: actions.getActionCount=2, actionResult.getResultOrExceptionCount=0 for region type: REGION_NAME

 

The postAppend method in SimpleRegionObserver should return result instead of null.
Bug: Add a command to support archive the earliest log file manually
Status: Open | Priority: Major | Resolution: 
Created: 05/Aug/20 09:28

This new command named wal_archive, belong to the group of "tools".

It is a sync call, if the data related to earliest log file not all persisted, they will be flushed immediately. 

Usage:
{code:java}
wal_archive 'desktop-ud83s84,16020,1596595277350'{code}
Bug: balanceRSGroup should skip generating balance plan for disabled table and splitParent region
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Aug/20 14:48

now ,we generate balance plan for disabled tables, which is useless
{code:java}
2020-08-20,20:47:54,702 WARN [RpcServer.default.RWQ.Fifo.read.handler=310,queue=6,port=22500] org.apache.hadoop.hbase.master.HMaster: Failed balance plan: hri=aa325467924edc865ab2ef6d82f9e2a7, source=tj1-hadoop-staging-st02.kscn,22600,1572403947348, destination=, just skip it
org.apache.hadoop.hbase.client.DoNotRetryRegionException: Unexpected state for rit=CLOSED, location=tj1-hadoop-staging-st02.kscn,22600,1572403947348, table=galaxysds:sds_staging_258z, region=aa325467924edc865ab2ef6d82f9e2a7
        at org.apache.hadoop.hbase.master.assignment.AssignmentManager.preTransitCheck(AssignmentManager.java:580)
        at org.apache.hadoop.hbase.master.assignment.AssignmentManager.createMoveRegionProcedure(AssignmentManager.java:635)
        at org.apache.hadoop.hbase.master.assignment.AssignmentManager.moveAsync(AssignmentManager.java:652)
        at org.apache.hadoop.hbase.master.HMaster.executeRegionPlansWithThrottling(HMaster.java:1776)
        at org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.balanceRSGroup(RSGroupAdminServer.java:486)
        at org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl.balanceRSGroup(RSGroupAdminEndpoint.java:293)
        at org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService.callMethod(RSGroupAdminProtos.java:13890)
        at org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:908)
        at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:135)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)

{code}
Bug: Clarify patch upgrade compatibility guarantees
Status: Open | Priority: Major | Resolution: 
Created: 26/Aug/20 16:04
Affects Version/s: 1.7.0

Per the [compatibility|https://hbase.apache.org/book.html#hbase.versioning] guidelines (specifically section "Client-Server wire protocol compatibility
 ") which says "We could only allow upgrading the server first. I.e. the server would be backward compatible to an old client, that way new APIs are OK."

This gives an impression that it is fine to break API compatibility in patch upgrades and expect the users to upgrade server binaries first before upgrading clients. However, when considering a back-port of HBASE-24765, it was noted by [~zhangduo] and [~ndimiduk] that this compatibility shouldn't be broken. Seems like something that should be clarified in the docs.
Bug: Fix flaky TestMasterAbortAndRSGotKilled for branch-2.2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Aug/20 08:38

I meet this problem on branch-2.2 too. This case happened because the DelayCloseCP. The event execute order is:
 # Close regiong. But because the DelayCloseCP, it will close after 10 seconds.
 # Finish ut and shutdown cluster.
 # Shutdown master.
 # Shutdown RS. Call waitOnAllRegionsToClose method. But abortRequested is false now.
 # Close region and failed because master is down and report master error. Then abort RegionServer and set abortRequested to ture.
 # waitOnAllRegionsToClose hanged because the online regions cannot be empty.

 

waitOnAllRegionsToClose(final boolean abort) already consider the abort case but the problem is abortRequested is false when call this method. I thought the fix should be that keep to check the abortRequested in waitOnAllRegionsToClose method internal.
Bug: On Master restart all servers are assigned to default rsgroup.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jul/20 13:19
Affects Version/s: 2.2.3

Steps to reproduce:
# Install a HBase cluster with three RS(rs1,rs2 and rs3) and one Master
# Create two rsgroups r1 and r2 and move rs1 to r1 and rs2 to r2
{code:java}
add_rsgroup 'r1';add_rsgroup 'r2';move_servers_rsgroup 'r1',['host1:16020'];move_servers_rsgroup 'r2',['host2:16020']
{code}
# Restart Master
# Run list_rsgroups for hbase shell, all region servers are assigned to default regroup.
Bug: ReplicationSink should limit the batch rowcount for batch mutations based on hbase.rpc.rows.warning.threshold
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jul/20 14:59

At times there are quite a large no of WAL Edits to ship as part of Replication and sometimes replication queues accumulate huge list of Edits to process. ReplicationSink at the sink server usually goes through all Edits and creates map of table -> list of rows grouped by clusterIds, and performs batch mutation of all rows per table level. However, there is no limit to no of Rows that are sent as part of batch mutate call. If no of rows > limit threshold defined by hbase.rpc.rows.warning.threshold, we usually get warn "Large batch operation detected". If hbase.rpc.rows.size.threshold.reject is turned on, RS will reject the whole batch without processing.

We should let Replication Sink honour this threshold value and accordingly keep the size lower per batch mutation call.

Replication triggered batch mutations should always be consumed but keeping limit of mutation low enough will let the system function at the same pace and without triggering redundant warnings. This will also restrict exploitation of cpu cycles at the destination server.
Bug: Fix the connection leaks on getting hbase admin from unclosed connection
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jul/20 19:44
Affects Version/s: 1.6.0

Observed the significant increase in ZK connection on performance testing on map reduce jobs. Turns out the [TableOutputFormat.checkOutputSpecs()|https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java#L182]  is not closing the connection it uses to get the hbase admin. It closes the hbase admin but never close the connection to get the admin.
Bug: Remove the metrics assert in TestClusterRestartFailover
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Aug/20 00:29

MetricsMasterSource masterSource = UTIL.getHBaseCluster().getMaster().getMasterMetrics()
 .getMetricsSource();
metricsHelper.assertCounter(MetricsMasterSource.SERVER_CRASH_METRIC_PREFIX+"SubmittedCount",
 4, masterSource);

 

Introduced by HBASE-24199. But flaky now as this unit test will restart all clusters. Meanwhile, this metric already tested by TestMasterMetrics. I plan to remove this assert for branch-2.2.
Bug: MergeTableRegionsProcedure should not call clean merge region
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Aug/20 10:22

In MergeTableRegionsProcedure, we will call CatalogJanitor.cleanMergeQualifier to test whether the regions we want to merge still have merge qualifier. If so we will skip merge.

But the problem is that, in cleanMergeQualifier, we will call cleanMergeRegion, where we may schedule GCMultipleMergedRegionsProcedure to actually clean the merged regions.

I think the intention here is to speed up the cleaning of merged region. CatalogJanitor, we have a alreadyRunning guard to prevent multiple scan at the same time, to prevent scheduling duplicated GCMultipleMergedRegionsProcedure for the same region. But if we call cleanMergeQualifier directly from MergeTableRegionsProcedure, we may schedule duplicated GCMultipleMergedRegionsProcedure as there is no guard to prevent CatalogJanitor to run at the same time.

And there is also another problem that, even if you disable CatalogJanitor, which means you want to stop cleaning meta, a call from MergeTableRegionsProcedure could still clean the records in meta...

So I think here we should just check the qualifiers. If there are merge qualifiers we just skip merging, without scheduling any procedures.
Bug: Replication may loss data when refresh recovered replication sources
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Aug/20 06:52
Affects Version/s: 2.2.5

See code in [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java#L519]

 

The WAL enqueued to wrong replication source.
{code:java}
    for (String queueId : previousQueueIds) {
        ReplicationSourceInterface replicationSource = createSource(queueId, peer);
        this.oldsources.add(replicationSource);
        this.queueStorage.getWALsInQueue(this.server.getServerName(), queueId)
          .forEach(wal -> src.enqueueLog(new Path(wal)));
        toStartup.add(replicationSource);
      }
{code}
Bug: Enlarge the wait time in TestReplicationEndpoint/TestMetaWithReplicasBasic
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Aug/20 08:06

Failed many times. But the failed reason are different. The replicated entries number are different. So it means the replication is work and it need more time to replicate all 2500 entries.
h3. Error Message

Waiting timed out after [30,000] msec Failed to replicate all edits, expected = 2500 replicated = 2499

 
h3. Error Message

Waiting timed out after [30,000] msec Failed to replicate all edits, expected = 2500 replicated = 2481

 
h3. Error Message

Waiting timed out after [30,000] msec Failed to replicate all edits, expected = 2500 replicated = 2491
Bug: Speed up some unit tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Aug/20 07:41

See [https://ci-hadoop.apache.org/job/HBase/job/HBase-Flaky-Tests/job/branch-2.2/42/testReport/org.apache.hadoop.hbase.client/TestAsyncTableAdminApi/]

[https://ci-hadoop.apache.org/job/HBase/job/HBase-Flaky-Tests/job/branch-2.2/61/testReport/junit/org.apache.hadoop.hbase.client/TestSnapshotTemporaryDirectoryWithRegionReplicas//]

 

These ut are flaky because they take too much time which more than 780 seconds. 

 

Split TestAsyncTableAdminApi/TestAdminShell/TestLoadIncrementalHFiles
 
 Reduce region numbers in TestSnapshotTemporaryDirectoryWithRegionReplicas/TestRegionReplicaFailover/TestSCP*
Bug: Enlarge MemstoreFlusherChore/CompactionChecker period for unit test
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Aug/20 05:46

Too many debug logs when run unit test now.

 

2020-08-19 01:20:59,899 DEBUG [regionserver/asf909:0.Chore.1] hbase.ScheduledChore(192): CompactionChecker execution time: 0 ms.
2020-08-19 01:20:59,899 DEBUG [regionserver/asf909:0.Chore.1] hbase.ScheduledChore(192): MemstoreFlusherChore execution time: 0 ms.
2020-08-19 01:20:59,900 DEBUG [regionserver/asf909:0.Chore.1] hbase.ScheduledChore(192): CompactionChecker execution time: 0 ms.
2020-08-19 01:20:59,900 DEBUG [regionserver/asf909:0.Chore.1] hbase.ScheduledChore(192): MemstoreFlusherChore execution time: 0 ms.
2020-08-19 01:20:59,905 DEBUG [regionserver/asf909:0.Chore.1] hbase.ScheduledChore(192): MemstoreFlusherChore execution time: 0 ms.
2020-08-19 01:20:59,905 DEBUG [regionserver/asf909:0.Chore.1] hbase.ScheduledChore(192): CompactionChecker execution time: 0 ms.
2020-08-19 01:21:00,001 DEBUG [regionserver/asf909:0.Chore.1] hbase.ScheduledChore(192): CompactionChecker execution time: 0 ms.
Bug: A tool to rewrite corrupted HFiles
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 20/Aug/20 20:20

Typically I have been dealing with corrupted HFiles (due to loss of hdfs blocks) by just removing them. However, It always seemed wasteful to throw away the entire HFile (which can be hundreds of gigabytes), just because one hdfs block is missing (128MB).

I think there's a possibility for a tool that can rewrite an HFile by skipping corrupted blocks. 

There can be multiple types of issues with hdfs blocks but any of them can be treated as if the block doesn't exist:
1. All the replicas can be lost
2. The block can be corrupted due to some bug in hdfs (I've recently run into HDFS-15186 by experimenting with EC).

At the simplest the tool can be a local mapreduce job (mapper only) with a custom HFile reader input that can seek to next DATABLK to skip corrupted hdfs blocks.
Bug: Running InitMetaProcedures are missed if Hmaster startup happens after Meta assign
Status: Open | Priority: Major | Resolution: 
Created: 21/Aug/20 00:43
Affects Version/s: 3.0.0-alpha-1

In HMaster startup, if the InitMetaProcedure was stopped after the meta assign stage, it will have a RegionState assigned to it and will not fall into this if statement: https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java#L1044. 

This means if an InitMetaProcedure is running (in the INIT_META_CREATE_NAMESPACES state), we won't look for any instances of running InitMetaProcedures and won't correctly await until they are done: https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java#L1065

The check for running InitMetaProcedures should be moved outside of the if statement and only if it is null should it try to add a new InitMetaProcedure (if there is no RegionState for meta or if meta is offline). This way we will correctly wait for procedures that are running that have passed the INIT_META_ASSIGN_META state (which sets the RegionStates that HMaster is looking at).
Bug: InitMetaProcedure create Namespace is not Idempotent
Status: Open | Priority: Major | Resolution: 
Created: 21/Aug/20 00:31
Affects Version/s: 3.0.0-alpha-1

If InitMetaProcedure is retried after creating at least one namespace directory: https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/InitMetaProcedure.java#L115, it will not be able to recover as the procedure does not check for existence of the directories before attempting to create them.

We should check for existence of the directory first and only execute createDirectories if they do not exist. The rest of the procedure looks like it is idempotent. We can additionally optimize to not overwrite the column families.
Bug: Speed up TestFromClientSide3 by reduce the table regions number
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Aug/20 01:58

[https://ci-hadoop.apache.org/job/HBase/job/HBase-Flaky-Tests/job/branch-2.2/52/testReport/junit/org.apache.hadoop.hbase.client/TestFromClientSide3//]

 
|[testHTableExistsMethodMultipleRegionsMultipleGets|https://ci-hadoop.apache.org/job/HBase/job/HBase-Flaky-Tests/job/branch-2.2/52/testReport/junit/org.apache.hadoop.hbase.client/TestFromClientSide3//testHTableExistsMethodMultipleRegionsMultipleGets]|2 min 58 sec|Regression|
|[testHTableExistsMethodMultipleRegionsSingleGet|https://ci-hadoop.apache.org/job/HBase/job/HBase-Flaky-Tests/job/branch-2.2/52/testReport/junit/org.apache.hadoop.hbase.client/TestFromClientSide3//testHTableExistsMethodMultipleRegionsSingleGet]|4 min 20 sec|Passed|

 

It take too many time and timeout.
Bug: Fix hbase-shell access to ModifiableTableDescriptor methods
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Aug/20 20:18
Affects Version/s: 3.0.0-alpha-1

HBASE-20819 prepared us for HBase 3.x by removing usages of the deprecated HTableDescriptor and HColumnDescriptor classes from the shell. However, it did use two methods from the ModifiableTableDescriptor, which was only public for compatibility/migration and was marked with {{@InterfaceAudience.Private}}. When {{ModifiableTableDescriptor}} was made private last week by HBASE-24507 it broke two hbase-shell commands (*describe* and *alter* when used to set a coprocessor) that were using methods from {{ModifiableTableDescriptor}} (these methods are not present on the general {{TableDescriptor}} interface).

This story will remove the two references in hbase-shell to methods on the now-private {{ModifiableTableDescriptor}} class and will find appropriate replacements for the calls.
Bug: [hbase-operator-tools] Add missing ASF headers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Aug/20 12:50
Affects Version/s: hbase-operator-tools-1.1.0

There are 4 files in hbase-table-reporter module without ASF headers:

* hbase-table-reporter/pom.xml
* hbase-table-reporter/README.md
* hbase-table-reporter/src/main/gnuplot/freq.p
* hbase-table-reporter/src/main/gnuplot/histo.p