av9ash/GitBugs huggingface dataset:
Bug: hbase-shaded-mapreduce and hbase-shaded-client expose some of the same classes
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 30/Mar/22 18:23

We supply 2 primary artifacts for end-users to consume:
 * hbase-shaded-client, which is for general use
 * hbase-shaded-mapreduce, which is for use when you need to connect to hbase via mapreduce. For example, TableInputFormat

The problem is that these artifacts expose tons of duplicate classes. One example (among many) is org.apache.hadoop.hbase.Cell, which appears in both jars.

This may not be a problem if your projects are always very isolated – either doing mapreduce, or not. In that case you just depend in the one you need. Many users might exist in much more complicated environments where dependencies tend to bleed along more between projects. Here's an illustration:
 * Imagine a project FooService, which includes two modules FooServiceRestWeb (for the rest http resources) and FooServiceData (which includes DAOs for accessing data). FooServiceRestWeb depends on FooServiceData to access hbase.  In this case, FooServiceData should depend on hbase-shaded-client.
 * Now imagine another project FooPipeline, which has modules FooPipelineHadoop (with M/R jobs for processing data) and FooPipelineData (which has some DAOs for accessing data). In this case, FooPipelineData might depend on hbase-shaded-mapreduce since the context is intended for M/R.
 * The problem arises when suddenly we want to include some data from FooService into our pipeline. The most straightforward way to achieve this is by depending on FooServiceData,  which has all of he DAOs for that data but also depends on hbase-shaded-client. At this point you have a problem, because FooPipelineHadoop now depends on both hbase-shaded-mapreduce and hbase-shaded-client.

(Note, this obviously skirts around potential microservice solutions like only accessing FooService's data through the API... it's just for illustration, and it does come up.)

From a plain java perspective, having these 2 jars on the classpath is somewhat wasteful but not a huge issue since the implementations are all the same.

From a maven perspective, it's problematic because the maven dependency plugin will complain about the conflicting classes.

One potential fix is to add exclusions to the FooServiceData dependency, to avoid pulling in hbase-shaded-client. This works on a one-off basis but is much more painful in a large and complicated environment where this may come up hundreds of times.

A better fix in my opinion is to make hbase-shaded-mapreduce depend on hbase-shaded-client and then only expose the classes that aren't already exposed by the shaded client.

[~busbey] also mentioned a BOM being a potential solution, but I don't have experience with that.
Bug: Find unavailable regions by the startcode checking on hmaster start up and reassign them
Status: Open | Priority: Major | Resolution: 
Created: 25/Mar/22 09:23

Sometimes we have seen there are regions in open or opening state, but does not deployed on any rs and without procs for them, and after  checking the meta table, we find these startcode are expired. 

It is no easy to reproduce, may be caused by corner bug or user misoperation.

My approach is add some checking on hmaster start up, if the startcode of the regionLocation expired, and neither TRSP on region nor SCP on regionserver, then we should reassign the region, then we can resovle it easily just by restart hmaster. 

Hbck2 maybe also useful for some of them cases, but not easily for common user to use, especially the number of these regions not small and need to be recovery quickly.
Bug: Support meta replica LoadBalance mode for RegionLocator#getAllRegionLocations()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jan/22 21:56
Affects Version/s: 2.4.9

When HBase application restarts, its meta cache is empty. Normally, it will fill the meta cache one region at a time by scanning the meta region. This will cause huge pressure to the region server hosting meta during application restart. 

It can prefetching all region locations by calling RegionLocator#getAllRegionLocations().Meta replica LoadBalance mode is support in 2.4, it will be nice to load balance RegionLocator#getAllRegionLocations() to all meta replica regions so batch scan can spread across all meta replica regions.
Bug: Shell startup logs a bunch of noise
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Mar/22 16:22
Affects Version/s: 2.5.0

I noticed a behavior on branch-2.5 that is not present on master or branch-2: starting up the shell produces a bunch of extra zookeeper logging. The classes are {{zookeeper.ZooKeeper}} and {{zookeeper.ClientCnxn}}.
Bug: Should always set SFT implementation when cloning snapshot
Status: Open | Priority: Major | Resolution: 
Created: 14/Mar/22 15:15

Saw the TestCloneSnapshotProcedureFileBasedSFT failing several times

{noformat}
2022-03-14T11:23:13,782 INFO  [PEWorker-1] procedure2.ProcedureExecutor(1432): Finished pid=99, state=SUCCESS, hasLock=false; CloneSnapshotProcedure (table=testRecoverWithRestoreAclFlag snapshot=name: "snapshot-1647256973399"
table: "testCloneSnapshot"
creation_time: 1647256982366
type: FLUSH
version: 2
owner: ""
ttl: 0
max_file_size: 0
) in 6.9090 sec
2022-03-14T11:23:13,794 WARN  [PEWorker-1] procedure2.ProcedureExecutor$Testing(127): Toggle KILL before store update to: false
2022-03-14T11:23:13,794 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1777): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:MODIFY_TABLE_DESCRIPTOR_UPDATE, hasLock=true; InitializeStoreFileTrackerProcedure table=testRecoverWithRestoreAclFlag
2022-03-14T11:23:13,794 INFO  [PEWorker-1] procedure2.ProcedureExecutor(635): Stopping
2022-03-14T11:23:13,795 WARN  [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1997): Worker terminating UNNATURALLY null
java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:MODIFY_TABLE_DESCRIPTOR_UPDATE, hasLock=true; InitializeStoreFileTrackerProcedure table=testRecoverWithRestoreAclFlag
	at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1779) ~[classes/:?]
	at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1723) ~[classes/:?]
	at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1414) ~[classes/:?]
	at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:78) ~[classes/:?]
	at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1981) ~[classes/:?]
2022-03-14T11:23:14,012 WARN  [Time-limited test] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false
{noformat}

The CloneSnapshotProcedure is finished but then we get a InitializeStoreFileTrackerProcedure which messes up the test.

The InitializeStoreFileTrackerProcedure will be scheduled when rolling upgrade, where we do not have SFT set for a table. So typically it should not be schedule. Not sure how this could happen in the UT, need to dig more.

But anyway, when we clone a snapshot which was taken before we have SFT, it is possible the TableDescriptor does not have SFT implementation set, so we should set one for it.
Bug: ReplicationPeerManager#checkPeerExists should throw ReplicationPeerNotFoundException if peer doesn't exists
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Mar/22 17:49

ReplicationPeerManager#checkPeerExists should throw ReplicationPeerNotFoundException if peer doesn't exists. Currently it throws generic DoNotRetryIOException.
{code:java}
private ReplicationPeerDescription checkPeerExists(String peerId) throws DoNotRetryIOException {
  ReplicationPeerDescription desc = peers.get(peerId);
  if (desc == null) {
    throw new DoNotRetryIOException("Replication peer " + peerId + " does not exist");
  }
  return desc;
} {code}
Bug: Improve fidelity of RegionLocator spans
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jan/22 18:20

While backporting HBASE-26474 to branch-2, I realized that the nuance of tracing of region locator calls is lacking. In the current implementation, all calls to the region locator produce {{SpanKind.CLIENT}} spans. This is correct only for the code path where the client reaches out to a remote host to populate cache locations. This doesn't happen every time.

Instead, the top-level span should be {{SpanKind.INTERNAL}} and only when a region location must be requested remotely should we create a child span with {{SpanKind.CLIENT}}.
Bug: Proto files in hbase-protocol and hbase-shaded-protocol are out of sync
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 02/Feb/22 23:45
Affects Version/s: 2.5.0

Are we reserving hbase-protocol at a point in time and only doing new development in hbase-shaded-protocol? This is probably the case but I'm not seeing documentation anywhere about it. 

At the very least, if so, every file in hbase-protocol should carry a very prominent comment that it is a point in time snapshot and not receiving new development. 

If these files are supposed to be in sync, they are not. Noticed divergence in ClusterStatus.proto on branch-2. There's almost certainly more elsewhere.
Bug: splitRegion and mergeRegion admin RPCs are in hbase-protocol but not hbase-shaded-protocol
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 09/Mar/22 18:53

On parent issue [~gjacoby] indicates that splitRegion and mergeRegion admin RPCs are in hbase-protocol but not hbase-shaded-protocol.

It would seem functionally it is not a serious break, but is a compatibility concern.
Bug: Backport "HBASE-25681 Add a switch for server/table queryMeter" to branch-2.4
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Mar/22 18:58

https://issues.apache.org/jira/browse/HBASE-25687 backported this config to branch-2, but that didn't include 2.4 because it was already released by then. This backports to branch-2.4, because I think the config is valuable. I find in performance testing that the per-table metrics specifically here can have a non-trivial impact on throughput.
Bug: Avoid unnecessary replication suspending in RegionReplicationSink
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Feb/22 13:17
Affects Version/s: 3.0.0-alpha-2

It seems that the problem HBASE-26449 described still exists in following {{RegionReplicationSink.onComplete}},which is running in Netty's nioEventLoop. Assuming we have only one secondary replica, first we add the replica to the {{failed}} because a failure of replicating in following line 228, but before we enter the line 238, the flusher thread calls {{RegionReplicationSink.add}} and we clear the {{RegionReplicationSink.failedReplicas}} due to a flush all edit. When the Netty nioEventLoop continues to enter line 238, we still add a replica to the failedReplicas even though the {{maxSequenceId < lastFlushedSequenceId}}.

{code:java}
207 private void onComplete(List<SinkEntry> sent,
208         Map<Integer, MutableObject<Throwable>> replica2Error) {
                      ....
217        Set<Integer> failed = new HashSet<>();
218        for (Map.Entry<Integer, MutableObject<Throwable>> entry : replica2Error.entrySet()) {
219        Integer replicaId = entry.getKey();
220       Throwable error = entry.getValue().getValue();
221        if (error != null) {
222           if (maxSequenceId > lastFlushedSequenceId) {
                     ...
228             failed.add(replicaId);
229           } else {
                   ......

238       synchronized (entries) {
239           pendingSize -= toReleaseSize;
240           if (!failed.isEmpty()) {
241                failedReplicas.addAll(failed);
242                flushRequester.requestFlush(maxSequenceId);
243           }
                ......
253      }
254    }
{code}

What is worse, when we invoke {{RegionReplicationFlushRequester.requestFlush}}, the flushing may be skipped because in following {{RegionReplicationFlushRequester.flush}}, {{pendingFlushRequestSequenceId}} is less than {{lastFlushedSequenceId}}, so the only secondary replica is marked failed and requested flushing is skipped, the replication may suspend until next memstore flush :

{code:java}
private synchronized void flush(Timeout timeout) {
    pendingFlushRequest = null;
    if (pendingFlushRequestSequenceId >= lastFlushedSequenceId) {
      request();
    }
  }
{code}

I simulate this problem in the PR and my fix is double check the  {{if (maxSequenceId > lastFlushedSequenceId)}} in the {{synchronized}} block in {{RegionReplicationSink.onComplete}}.
Bug: TestBasicWALEntryStreamFSHLog testSizeOfLogQueue is flaky
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 10/Mar/22 20:19
Affects Version/s: 2.4.10

[ERROR] org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStreamAsyncFSWAL.testSizeOfLogQueue  Time elapsed: 0.056 s  <<< FAILURE!
java.lang.AssertionError: expected:<1> but was:<2>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream.testSizeOfLogQueue(TestBasicWALEntryStream.java:701)
Bug: Traces of async chains result in arbitrarily deep trace stack
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 28/Jan/22 23:09
Affects Version/s: 3.0.0-alpha-2

Trying out the latest {{master}} on a real cluster, traces of async operations do not report the way I expect. Actions that are sequential (for example, locate meta then locate region) do not render as trace peers. Instead the subsequent action is a child of the previous. I believe this means that we are not handling parent span context propagation in async callback chains. I've not yet tried branch-2 to see if it's applicable there as well.
Bug: Unify CallQueueTooBigException special pause with CallDroppedException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Mar/22 14:14

CallQueueTooBigException and CallDroppedException crop up in very similar circumstances – the former is thrown if the request cannot be enqueued because the queue is full; the latter is thrown when a call is dropped from the queue to make room for another call.

HBASE-17114 added a special pause feature, which allows pausing for a longer period of time when CallQueueTooBigException is encountered, vs the normal pause for other exceptions. The idea here is to help reduce load so the server can process its queue. We should extend this feature to encompass CallDroppedException for the same reason.

Currently the config is called "hbase.client.pause.cqtbe". We should probably deprecate that in favor of a more generic name.
Bug: Remove duplicate dependency declaration
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Mar/22 11:47
Affects Version/s: 2.5.0

On branch-2 derivatives, I noticed on PR builds of JDK11+Hadoop3 that running tests with {{-pl}} results in a warning from maven

{noformat}
[WARNING]                                                                                                                                                                                                   
[WARNING] Some problems were encountered while building the effective model for org.apache.hbase:hbase-build-configuration:pom:2.6.0-SNAPSHOT                                                               
[WARNING] 'profiles.profile[hadoop-3.0].dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.hadoop:hadoop-mapreduce-client-app:test-jar -> duplica
te declaration of version ${hadoop-three.version} @ org.apache.hbase:hbase:2.6.0-SNAPSHOT, /Users/ndimiduk/repos/apache/hbase/pom.xml, line 3816, column 23                                                 
[WARNING]                                                                                                                                                                                                   
[WARNING] Some problems were encountered while building the effective model for org.apache.hbase:hbase:pom:2.6.0-SNAPSHOT                                                                                   
[WARNING] 'profiles.profile[hadoop-3.0].dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.hadoop:hadoop-mapreduce-client-app:test-jar -> duplica
te declaration of version ${hadoop-three.version} @ line 3816, column 23                                                                                                                                    
[WARNING]                                                                                                                                                                                                   
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.                                                                                                 
[WARNING]                                                                                                                                                                                                   
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.                                                                                                  
[WARNING] 
{noformat}

I _think_ they're harmless, but we should clean them up.
Bug: on hbase shell, 'delete/deleteall' for a columnfamily is not working
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Mar/22 10:23
Affects Version/s: 2.4.11

on hbase shell,

'delete' or 'deleteall' for the whole columnFamily is not working properly.

 
{code:java}
hbase(main):026:0* put 'test', 'r1', 'f:1', 'a'
Took 0.0233 seconds
hbase(main):029:0> delete 'test', 'r1', 'f'
Took 0.0070 seconds
hbase(main):030:0> get 'test', 'r1'
COLUMN                                                                  CELL
 f:1                                                                    timestamp=1648114865022, value=a
1 row(s)
hbase(main):038:0> deleteall 'test', 'r1', 'f'
Took 0.0059 seconds
hbase(main):039:0> get 'test', 'r1'
COLUMN                                                                  CELL
 f:1                                                                    timestamp=1648114865022, value=a
1 row(s) {code}
 

looking inside of hbase-shell,

all delete/deleteall on hbase shell are converted to delete.addColumn/addColumns.

thus, delete/deleteall request without columnQualifier converted to 'null' qualifier deletion. (since HBASE-15616, null columnQualifier is possible)

 
{code:java}
        if column && all_version
          family, qualifier = parse_column_name(column)
          d.addColumns(family, qualifier, timestamp)
        elsif column && !all_version
          family, qualifier = parse_column_name(column)
          d.addColumn(family, qualifier, timestamp)
        end {code}
 

According to the description of help 'deleteall' and HBASE-9549
 "Users of the shell, MapReduce, REST, and Thrift who wish to interact with an *entire column family must use "family"* instead of "family:" (notice the omitted ':'). Including the ':' will be interpreted as an interaction with the empty qualifier in the "family" column family."
 
a column expression without ':' means whole family.
so in these cases, it's deletion request for the columnFamily,
and we should use addFamily/addFamilyVersion instead of addColumn/addColumns for deletion.
Bug: delete with null columnQualifier occurs NullPointerException when NewVersionBehavior is on
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Mar/22 13:39
Affects Version/s: 2.4.11

since  HBASE-15616, setting column qualifier as null is possible.

but when NewVersionBehavior is on, delete with null columnQualifier occurs NullPointerException.

 
{code:java}
@Test
public void testNullColumnQualifier() throws IOException {
  try (Table t = createTable()) {
    Delete del = new Delete(ROW);
    del.addColumn(FAMILY, null);
    t.delete(del);
    Result r = t.get(new Get(ROW)); //NPE happens.
    assertTrue(r.isEmpty());
  }
} {code}
{code:java}
 //output
Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.regionserver.querymatcher.NewVersionBehaviorTracker.add(NewVersionBehaviorTracker.java:214) at org.apache.hadoop.hbase.regionserver.querymatcher.NormalUserScanQueryMatcher.match(NormalUserScanQueryMatcher.java:73) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:627) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:157) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:6672) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6836) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6606) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:6583) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:6570) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2645) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2571) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42274) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:418) ... 3 more
{code}
Bug: Set java.io.tmpdir on mvn command when running jenkins job
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Mar/22 14:50

Recently on hbase1-10, it is easy to eat all the space of /tmp. I've checked several times, it is always the stderr*deferred file, which is the temp output of our UTs.

I think it means we have some problem for some of our UTs which could generate very big output, but if the tmp is full, it is not easy for us to find out the root cause as we need to delete the file to recover first...

Looking at the code of surefire

https://github.com/apache/maven-surefire/blob/68bca29b23d27dba3e06cf5e73a3fdc1a22598db/maven-surefire-common/src/main/java/org/apache/maven/plugin/surefire/report/Utf8RecodingDeferredFileOutputStream.java#L69

It just creates the file with jdk's Files.createTempFile, the location can be controlled by java.io.tmpdir, so let's point it to the target directory under the jenkins workspace to avoid wat all the spaces of /tmp first.
Bug: Remove warnings from meta replicas feature references in the HBase book
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Mar/22 13:29

Meta replicas is a new feature in HBase 2.4 and mentioned in "Use with caution" in the docs. Given that the feature and the related "async wal replication for meta" is actively used in production already, I'd like to remove these warnings from the docs.

With this change, users will have more confidence in the feature.
Bug: Adapt ConnectionRule for both sync and async connections
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Mar/22 12:23
Affects Version/s: 2.4.11

Backporting recent tracing work to branch-2 has presented some difficulties using the {{ConnectionRule}}. The issue comes down to {{AsyncConnection}} on master provides a {{toConnection}} method, which is not available on branch-2.

Start with a branch-2 implementation and then forward-port the API changes to master.