av9ash/GitBugs huggingface dataset:
Bug: TestStochasticLoadBalancerRegionReplicaLargeCluster.testRegionReplicasOnLargeCluster is flaky  
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/May/22 05:12
Affects Version/s: 2.5.0

TestStochasticLoadBalancerRegionReplicaLargeCluster.testRegionReplicasOnLargeCluster  . Looks like we can be off by one on either side of an expected value.

Any idea what is going on here [~dmanning]? 

{noformat}
org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaLargeCluster.testRegionReplicasOnLargeCluster  Time elapsed: 77.779 s  <<< FAILURE!
java.lang.AssertionError: All servers should have load no less than 60.
server=srv1351292323,46522,-3543799643652531264 , load=59
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.apache.hadoop.hbase.master.balancer.BalancerTestBase.assertClusterAsBalanced(BalancerTestBase.java:200)
	at org.apache.hadoop.hbase.master.balancer.BalancerTestBase.testWithCluster(BalancerTestBase.java:577)
	at org.apache.hadoop.hbase.master.balancer.BalancerTestBase.testWithCluster(BalancerTestBase.java:544)
	at org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaLargeCluster.testRegionReplicasOnLargeCluster(TestStochasticLoadBalancerRegionReplicaLargeCluster.java:41)
{noformat}

{noformat}
org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaLargeCluster.testRegionReplicasOnLargeCluster  Time elapsed: 77.781 s  <<< FAILURE!
java.lang.AssertionError: All servers should have load no more than 60. 
server=srv1402325691,7995,26308078476749652 , load=61
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.apache.hadoop.hbase.master.balancer.BalancerTestBase.assertClusterAsBalanced(BalancerTestBase.java:198)
	at org.apache.hadoop.hbase.master.balancer.BalancerTestBase.testWithCluster(BalancerTestBase.java:577)
	at org.apache.hadoop.hbase.master.balancer.BalancerTestBase.testWithCluster(BalancerTestBase.java:544)
	at org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaLargeCluster.testRegionReplicasOnLargeCluster(TestStochasticLoadBalancerRegionReplicaLargeCluster.java:41)

{noformat}
Bug: Let lock wait timeout to improve performance of SnapshotHFileCleaner
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/May/22 13:49
Affects Version/s: 2.4.12

Currently, hfile cleaner uses the dir scanning threads to get deletable files, by checking all the files under the scanned directories through the cleaner chain. And before scanning a directory, cleaner sorted the subdirectories by consumed spaces, but  we all know getContentSummary is a time consuming operation for HDFS.

SnapshotHFileCleaner filters all the unreferenced files of snapshots to delete, and it tries to get write lock of SnapshotManager#takingSnapshotLock before determining the deletable files. [https://github.com/apache/hbase/blob/ad64a9baae2ef8ee56aa3ed6b96cb3d51f5daf0a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java#L195]

But when there is any snapshot taking and the cleaner gets the lock failed, all the scanned files will be determined to be none-deletable, and all the dir scanning threads will scan and getContentSummary of other dirs(no files are deletable too) one by one until the snapshot taking lock is released.

This is a low efficiency behavior, we should let the clear wait the lock to determine if the files it currently hold are deletable, instead of the meaningless scanning and getContentSummary other/new directories while the lock is acquired by taking snapshot operations.

I deployed this optimization in our production environment, and the effect is very obvious.
Bug: Add a tool command list_liveservers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/May/22 00:31

To make it easier for us to query the living region Servers. We can add a command `list_liveservers`. There are already `list_deadServers` and `list_Decommissioned_regionServers`.

!image-2022-05-10-08-34-33-711.png|width=457,height=123!
Bug: Add DeleteSortReducer to HFileOutputFormat2
Status: Open | Priority: Major | Resolution: 
Created: 20/May/22 21:32

Currently if you want to bulk load Deletes you need to find some way to create KeyValues instead, because HFileOutputFormat2 doesn't support Delete as a map output value class. A savvy user will realize that you can first create your Delete then use {{delete.cellScanner()}} to gain access to the KeyValue's and write out those.

This feels a little buried next to the ability to bulk load Puts directly. Additionally, KeyValue is an IA.Private class and in order to make this work you need to at the very least do {{{}job.setMapOutputValueClass(KeyValue.class){}}}. It seems a bit wrong to make the user import an IA.Private class for this.

We can make HFileOutputFormat2 directly accept Deletes. It can do the work in a DeleteSortReducer of breaking up the Deletes into KeyValues, like we do with PutSortReducer.

We could also use this moment to add support for full row deletes. If DeleteSortReducer sees a Delete without any getFamilyMap() values, it can look up the current families for the table.
Bug: Trace a one-shot execution of a Master procedure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Apr/22 15:18
Affects Version/s: 2.5.0

A self-contained effort towards wrangling the stray traces identified in HBASE-26366. This one adds a tracing context to the Master's ProcedureV2 execution. This ties all the spans that result from a procedure execution back to a procedure by name.

Maybe in a follow-on issue, we can serialize span context along with procedures and display procedure start/stop/retry events and parent-child hierarchy in tracing visualization.
Bug: Flakey test detector generated garbage, breaks build automation
Status: Open | Priority: Major | Resolution: 
Created: 16/May/22 10:43
Affects Version/s: 2.6.0

Flakey test automation is injecting nonsense into the mvn invocation used by builds. The skip list includes: {{**/htWtApR.NTIeNsGt:H tAtnp Sielrlveegra.java}}, which causes the build to fail with

{noformat}
[ERROR] Unknown lifecycle phase "tAtnp". You must specify a valid lifecycle phase or a goal in the format <plugin-prefix>:<goal> or <plugin-group-id>:<plugin-artifact-id>[:<plugin-version>]:<goal>. Available lifecycle phases are: validate, initialize, generate-sources, process-sources, generate-resources, process-resources, compile, process-classes, generate-test-sources, process-test-sources, generate-test-resources, process-test-resources, test-compile, process-test-classes, test, prepare-package, package, pre-integration-test, integration-test, post-integration-test, verify, install, deploy, pre-clean, clean, post-clean, pre-site, site, post-site, site-deploy. -> [Help 1]
{noformat}
Bug: Backport "HBASE-27013 Introduce read all bytes when using pread for prefetch" to branch-2.4
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/May/22 20:44
Affects Version/s: 2.4.13

Backport HBASE-27013 to branch-2.4, it's required because it's not a clean backport.
Bug: failed to set file permission  when node crash
Status: Open | Priority: Major | Resolution: 
Created: 14/May/22 10:48

 in SecureBulkLoadManager#secureBulkLoadHFiles, we have code like that:
{code:java}
for(Pair<byte[], String> el: familyPaths) {
              Path stageFamily = new Path(bulkToken, Bytes.toString(el.getFirst()));
              if(!fs.exists(stageFamily)) {
                fs.mkdirs(stageFamily);
                fs.setPermission(stageFamily, PERM_ALL_ACCESS);
             }
} {code}
if process crashbefore setpermission， and reboot, we can't setpermission again.

 

we should make this code like SnapshotScannerHDFSAclHelper#setCommonDirectoryPermission

 
{code:java}
 for (Path path : paths) {
      createDirIfNotExist(path);
      fs.setPermission(path, new FsPermission(
          conf.get(COMMON_DIRECTORY_PERMISSION, COMMON_DIRECTORY_PERMISSION_DEFAULT)))
} {code}
Bug: MasterFileSystem create directory without permission check
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/May/22 07:10
Affects Version/s: 2.4.12

Method checkStagingDir and checkSubDir first check if directory is exist ,if not , create it with special permission. If exist then setPermission for this directory. BUT if not exist ,we still need set special permission for this directory
Bug: Another case for unnecessary replication suspending in RegionReplicationSink
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Apr/22 09:59
Affects Version/s: 3.0.0-alpha-2

Besides HBASE-26768, there is another case replication  in {{RegionReplicationSink}} would be suspend:
For {{RegionReplicationSink}}, when there is a replication error , {{RegionReplicationSink}} invokes {{MemStoreFlusher#requestFlush}} to request a flush, and after receiving the {{FlushAction#START_FLUSH}} or {{FlushAction#CANNOT_FLUSH}} flush marker, it would resume the replication. But when {{MemStoreFlusher}}  flushing, it invokes following method {{HRegion.flushcache}} with the {{writeFlushRequestWalMarker}} set to false:
{code:java}
  public FlushResultImpl flushcache(List<byte[]> families,
      boolean writeFlushRequestWalMarker, FlushLifeCycleTracker tracker) throws IOException {
 }
{code}
When  {{writeFlushRequestWalMarker}} is set to false, {{HRegion.flushcache}} does not write the {{FlushAction#CANNOT_FLUSH}} flush marker to {{WAL}} when the memstore is empty, just as following {{HRegion.writeFlushRequestMarkerToWAL}} illustrated:
{code:java}
private boolean writeFlushRequestMarkerToWAL(WAL wal, boolean writeFlushWalMarker) {
    if (writeFlushWalMarker && wal != null && !writestate.readOnly) {
      FlushDescriptor desc = ProtobufUtil.toFlushDescriptor(FlushAction.CANNOT_FLUSH,
        getRegionInfo(), -1, new TreeMap<>(Bytes.BYTES_COMPARATOR));
      try {
        WALUtil.writeFlushMarker(wal, this.getReplicationScope(), getRegionInfo(), desc, true, mvcc,
          regionReplicationSink.orElse(null));
        return true;
      } catch (IOException e) {
        LOG.warn(getRegionInfo().getEncodedName() + " : " +
          "Received exception while trying to write the flush request to wal", e);
      }
    }
    return false;
  }
{code}

so when there is a replication error when the memstore is empty(eg. replicating the {{FlushAction#START_FLUSH}}  or {{FlushAction#COMMIT_FLUSH}} ), the replication may suspend until next memstore flush,even though later there are user writes and it could replicate normally.

I simulate this problem in the PR , and for {{writeFlushRequestWalMarker}} paramter, it is introduced by HBASE-11580 and just only determines whether or not writing the {{FlushAction#CANNOT_FLUSH}} flush marker to WAL when the memstore is empty, so I think for simplicity, we could set it to true always for {{MemStoreFlusher}}.
Bug: TestRegionProcessRowsWithLocks#testProcessExceptionAndRollBack is broken in branch-1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/May/22 06:44
Affects Version/s: 1.7.1

As titled, I found that {{TestRegionProcessRowsWithLocks#testProcessExceptionAndRollBack}} is broken while reviewing the PR for HBASE-26977 ([pre-commit test log here|https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4377/7/artifact/out/patch-unit-hbase-server.txt]) with below exceptions:

{noformat}
[ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 10.166 s <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRegionProcessRowsWithLocks
[ERROR] testProcessExceptionAndRollBack(org.apache.hadoop.hbase.regionserver.TestRegionProcessRowsWithLocks)  Time elapsed: 5.849 s  <<< ERROR!
com.google.protobuf.ServiceException: Error calling method hbase.pb.RowProcessorService.Process
	at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callBlockingMethod(CoprocessorRpcChannel.java:75)
	at org.apache.hadoop.hbase.protobuf.generated.RowProcessorProtos$RowProcessorService$BlockingStub.process(RowProcessorProtos.java:1631)
	at org.apache.hadoop.hbase.regionserver.TestRegionProcessRowsWithLocks.incrementCounter(TestRegionProcessRowsWithLocks.java:193)
	at org.apache.hadoop.hbase.regionserver.TestRegionProcessRowsWithLocks.testProcessExceptionAndRollBack(TestRegionProcessRowsWithLocks.java:173)
{noformat}

Manually test the single case in my local environment also failed. Using {{git bisect}}, I found the issue was introduced by HBASE-26393 and have been existing for several months (since Oct 23, 2021).

The fix is straight forward and I will prepare a PR soon.
Bug: Auto renew scanner lease in TableRecordReader
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 03/May/22 19:21

A common problem with hadoop jobs is when the mapper takes too long to process individual inputs. This is especially problematic with TableInputFormat because if you don't process a scanner.next() batch within the scanner timeout period your job will fail with UnknownScannerException.

The fix here is usually to reduce Scan.setCaching, so that fewer rows are returned within each batch. This isn't always a great solution because maybe not all batches are uniform in their processing time, or maybe even processing a single row (the smallest caching size) might take a while.

We can improve this for users by providing a configurable period at which the TableRecordReader will automatically call scanner.renewLease() unless next() was recently called.
Bug: Do not add --threads when running 'mvn site'
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Apr/22 05:27

We will get this in the mvnsite task in our nightly and pre commit jobs
{noformat}
[WARNING] *****************************************************************
[WARNING] * Your build is requesting parallel execution, but project      *
[WARNING] * contains the following plugin(s) that have goals not marked   *
[WARNING] * as @threadSafe to support parallel building.                  *
[WARNING] * While this /may/ work fine, please look for plugin updates    *
[WARNING] * and/or request plugins be made thread-safe.                   *
[WARNING] * If reporting an issue, report it against the plugin in        *
[WARNING] * question, not against maven-core                              *
[WARNING] *****************************************************************
[WARNING] The following plugins are not marked @threadSafe in Apache HBase:
[WARNING] org.apache.maven.plugins:maven-site-plugin:3.7.1
[WARNING] Enable debug to see more precisely which goals are not marked @threadSafe.
[WARNING] *****************************************************************
{noformat}
Bug: Implement a special TestAppender to limit the size of test output
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Apr/22 11:04

Recently our build nodes were out of space several times. I've checked some of them, all are because of logging in an infinite loop which makes a very big log output.

I've already opened several issues for this, such as HBASE-26858, HBASE-26941, HBASE-26946.

So in general, I think we should have a way to limit the total size of a log output. So when we do something wrong in code it will not eat all the space soon and bring down our ci.
Bug: Compilation of master vs hadoop-3.3.2 fails
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Apr/22 20:51
Affects Version/s: 3.0.0-alpha-2

We noticed over on HBASE-26907 while adding Hadoop 3.3.2 to our nightly build matrix that we get a compilation failure when setting this as the hadoop.three version.

{noformat}
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-4305/yetus-general-check/src/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java:[832,29] package org.apache.htrace.core does not exist
[INFO] 1 error
[INFO] -------------------------------------------------------------
{noformat}
Bug: Replace fully qualified class names with import statements where possible
Status: Open | Priority: Major | Resolution: 
Created: 21/Apr/22 09:52

Over on HBASE-26954, it was noticed that our enforcer check missed a class. Turns out, the restrict-imports enforcer-rule only works on import statements. Let's take a pass for fully qualified classes uses outside of import statements and see if we can reduce their number, so as to avoid this type of mistake in the future.
Bug: Add support to hbase shell to remove coproc by its class name instead of coproc ID
Status: Open | Priority: Major | Resolution: 
Created: 15/Apr/22 22:02
Affects Version/s: 1.7.1

The syntax for removing coproc is as below:
  hbase> alter 't1', METHOD => 'table_att_unset', NAME => 'coprocessor$1'

We have to use coproc id to remove a coproc from a given table.

Consider the following scenario. Due to some bug in a coproc, we have to remove a given coproc from all the tables in a cluster. Every table can have different set of co-procs. For a given co-proc class, the coproc ID will not be same for all the tables in a cluster. This gets more complex if we want to remove co-proc from all the production clusters. 

Instead we can pass a co-proc class name to alter table command. So if a table has that co-proc, it will remove otherwise do nothing.
Bug: TestRaceBetweenSCPAndTRSP is flaky in master branch
Status: Open | Priority: Major | Resolution: 
Created: 13/Apr/22 04:32

Observed in [pre-commit check|https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4341/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt] of HBASE-26943
Bug: Fix LineLength warnings as much as possible if it can not be fixed by spotless
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Apr/22 05:45

One possible problem is the trailing comment, Like this

{code}
      assertTrue(initialRegions
        .contains(RegionReplicaUtil.getRegionInfoForReplica(mergedRegions.getFirst(), 1))); // this
                                                                                            // is
                                                                                            // the
                                                                                            // replica
                                                                                            // of
                                                                                            // the
                                                                                            // first
                                                                                            // region
{code}

Because of we limited line length, the format will be very strange and still has line length problem if some of the words are too long.

Let's move them from trailing to top of the statement.
Bug: Fix several indentation problems
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Apr/22 08:01

hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto/Encryption.java
hbase-endpoint/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpointTracing.java
hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RowResource.java
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRefreshRecoveredReplication.java
Bug: VerifyReplication should use region startKey to check PeerTable
Status: Open | Priority: Major | Resolution: 
Created: 04/Apr/22 15:12

 [https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java#L199]

Here we miss the case: peer table has redundant data, which are all smaller than the source table first rowkey
Bug: Fix missing braces warnings in TestProcedureMember
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Apr/22 05:40

Just like DefaultVisibilityExpressionResolver, we have this in TestProcedureMember

{code}
    if (member != null) try {
      member.close();
    } catch (IOException e) {
      e.printStackTrace();
    }
{code}
Bug: Rewrite the counting rows part in TestFromClientSide4
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Apr/22 05:39

Now we add an annotation to the variable declaration in the for loop, it will lead the following result after formatting
{code}
        for (@SuppressWarnings("unused")
        Result row : scanner) {
          nbRows++;
        }
{code}

It looks really strange and checkstyle also reports indentation error.

Let's just rewrite it an avoiding using annotation in this way, I do not know what is a better format for this scenario.
Bug: Fix missing braces warnings in DefaultVisibilityExpressionResolver
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Apr/22 05:23

It will cause incorrent indentation warnings, like this
{code}
      if (connection != null) try {
        connection.close();
      } catch (IOException ioe) {
        LOG.warn("Failed close of temporary connection", ioe);
      }
{code}
Bug: SteppingSplitPolicy and RegionSplitRestriction can result in illegal split noise
Status: Open | Priority: Minor | Resolution: 
Created: 02/May/22 15:16

I set this to minor because I don't think it causes any real harm aside from log noise. But we should probably fix it up.

Create a table with splits 0, 1, 2, 3, 4, 5. Then set {{hbase.regionserver.region.split_restriction.type}} to {{KeyPrefix}} and {{hbase.regionserver.region.split_restriction.prefix_length}} to 1.

If you do enough writes to these regions, the SteppingSplitPolicy will try to split them. So it will try to split, for example, 1 in half. This is possible without KeyPrefix restriction, but with the restriction the split point will be truncated to value 1. The split will be submitted to the master with splitRow = 1 and that will fail because "Split row is equal to start key: 1"

Setting up splits in this way, with the restriction, is useful for when you're adding a salt prefix to your rowkeys (i.e to better distribute timestamp rowkeys).
Bug: SFT seems to apparently tracking invalid/malformed store files
Status: Open | Priority: Minor | Resolution: 
Created: 10/May/22 20:08

Opening this on behalf of [~apurtell] , who first reported this issue on HBASE-26999: When running scale tests using ITLCC, the following errors were observed:
{noformat}
[00]2022-05-05 15:59:52,280 WARN [region-location-0] regionserver.StoreFileInfo:
Skipping hdfs://ip-172-31-58-47.us-west-2.compute.internal:8020/hbase/data/default/IntegrationTestLoadCommonCrawl/9eafc10e1b5a25532a4f0adf550828fc/c/9d07757144a7404fac02e161b5bd035e
because it is empty. HBASE-646 DATA LOSS?
...
[00]2022-05-05 15:59:52,320 WARN [region-location-2] regionserver.StoreFileInfo: 
Skipping hdfs://ip-172-31-58-47.us-west-2.compute.internal:8020/hbase/data/default/IntegrationTestLoadCommonCrawl/5322c54b9a899eae03cb16e956a836d5/c/184b4f55ab1a4dbc813e77aeae1343ae 
because it is empty. HBASE-646 DATA LOSS? {noformat}
 

From some discussions in HBASE-26999, it seems that SFT has wrongly tracked an incomplete/unfinished store file. 

For further context, follow the [comments thread on HBASE-26999|https://issues.apache.org/jira/browse/HBASE-26999?focusedCommentId=17533508&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17533508].