av9ash/GitBugs huggingface dataset:
Bug: Recover WAL encounter  KeeperErrorCode = NoNode cause RegionServer crash
Status: Open | Priority: Major | Resolution: 
Created: 28/Mar/23 13:41

20696 2022-07-07 14:51:52,638 ERROR.HRegionServer: ***** ABORTING region server ***: Failed to operate on replication queue *****
120697 org.apache.hadoop.hbase.replication.ReplicationException: Failed to set log position (serverName=***, queueId=***.regiongroup-0.1657176199216,position=55122787)
120698     at org.apache.hadoop.hbase.replication.ZKReplicationQueueStorage.setWALPosition(ZKReplicationQueueStorage.java:261)
120699     at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.lambda$logPositionAndCleanOldLogs$7(ReplicationSourceManager.java:529)
120700     at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.interruptOrAbortWhenFail(ReplicationSourceManager.java:476)
120701     at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:528)
120702     at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.updateLogPosition(ReplicationSourceShipper.java:271)
120703     at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.shipEdits(ReplicationSourceShipper.java:205)
120704     at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.run(ReplicationSourceShipper.java:120)
120705 Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
120706     at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
120707     at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1925)
120708     at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:1830)
120709     at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:668)
120710     at org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1797)
120711     at org.apache.hadoop.hbase.replication.ZKReplicationQueueStorage.setWALPosition(ZKReplicationQueueStorage.java:251)
120712     ... 6 more
Bug: Migrate meta locations from zookeeper to master data may not always possible if we migrate from 1.x HBase
Status: In Progress | Priority: Major | Resolution: 
Created: 09/Mar/23 17:51
Affects Version/s: 2.5.0

In HBase 1.x versions meta server location from zookeeper will be removed when the server stopped. In such cases migrating to 2.5.x branches may not create any meta entries in master data. So in case if we could not find the meta location from zookeeper we can get meta location from wal directories with .meta extension and add to master data.

{noformat}
  private void tryMigrateMetaLocationsFromZooKeeper() throws IOException, KeeperException {
    // try migrate data from zookeeper
    try (ResultScanner scanner =
      masterRegion.getScanner(new Scan().addFamily(HConstants.CATALOG_FAMILY))) {
      if (scanner.next() != null) {
        // notice that all replicas for a region are in the same row, so the migration can be
        // done with in a one row put, which means if we have data in catalog family then we can
        // make sure that the migration is done.
        LOG.info("The {} family in master local region already has data in it, skip migrating...",
          HConstants.CATALOG_FAMILY_STR);
        return;
      }
    }
    // start migrating
    byte[] row = CatalogFamilyFormat.getMetaKeyForRegion(RegionInfoBuilder.FIRST_META_REGIONINFO);
    Put put = new Put(row);
    List<String> metaReplicaNodes = zooKeeper.getMetaReplicaNodes();
    StringBuilder info = new StringBuilder("Migrating meta locations:");
    for (String metaReplicaNode : metaReplicaNodes) {
      int replicaId = zooKeeper.getZNodePaths().getMetaReplicaIdFromZNode(metaReplicaNode);
      RegionState state = MetaTableLocator.getMetaRegionState(zooKeeper, replicaId);
      info.append(" ").append(state);
      put.setTimestamp(state.getStamp());
      MetaTableAccessor.addRegionInfo(put, state.getRegion());
      if (state.getServerName() != null) {
        MetaTableAccessor.addLocation(put, state.getServerName(), HConstants.NO_SEQNUM, replicaId);
      }
      put.add(CellBuilderFactory.create(CellBuilderType.SHALLOW_COPY).setRow(put.getRow())
        .setFamily(HConstants.CATALOG_FAMILY)
        .setQualifier(RegionStateStore.getStateColumn(replicaId)).setTimestamp(put.getTimestamp())
        .setType(Cell.Type.Put).setValue(Bytes.toBytes(state.getState().name())).build());
    }
    if (!put.isEmpty()) {
      LOG.info(info.toString());
      masterRegion.update(r -> r.put(put));
    } else {
      LOG.info("No meta location available on zookeeper, skip migrating...");
    }
  }
{noformat}
Bug: Support for Hadoop's LDAP Authentication mechanism (Web UI only)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Mar/23 06:24

Hadoop's AuthenticationFilter has changed and now has support for ldap mechanism too. HBase still uses an older version tightly coupled with kerberos and spnego as the only auth mechanisms. HADOOP-12082 has added support for multiple auth handlers including LDAP. On trying to use Hadoop's AuthenticationFilterInitializer in hbase.http.filter.initializers, there is a casting exception as HBase requires it to extend org.apache.hadoop.hbase.http.FilterInitializer.
Bug: Enhance quotas to consume blockBytesScanned rather than response size
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Mar/23 15:22

As of HBASE-27558 we now apply quota.getReadAvailable() to max block bytes scanned by scans/multis. This issue enhances further so that we can track read size consumed in Quotas based on block bytes scanned rather than response size. In this mode, quotas would end-to-end be based on blockBytesScanned.

Right now we call quota.addGetResult or addScanResult. This would just be a matter of no-oping those calls, and calling RpcCall.getBlockBytesScanned() in Quota.close() instead.
Bug: Optimize reference counting in off-heap ByteBuff
Status: Open | Priority: Major | Resolution: 
Created: 17/Mar/23 13:11

In HBASE-27710 we uncovered a performance regression in reference counting of ByteBuff. This was especially prominent in on-heap buffers when doing a simple HFile.Reader iteration of a file. For that case, we saw a 4x regression when reference counting was in play.

It stands to reason that this same regression exists in off-heap buffers, and I've run a microbenchmark which indeed shows the same issue. With existing reference counting, scanning a 20gb hfile takes 40s. With an optimized version, scanning the same file takes 20s. We don't typically see this in profiling live regionservers where so much else goes on, but optimizing this would eliminate some cpu cycles.

It's worth noting that netty saw this same regression a few years ago: [https://github.com/netty/netty/pull/8895]. Hat tip to [~zhangduo] for pointing this out.

One of the fixes there was to copy some internal code from deeper in the ref counting, so that the call stack was smaller and inlining was possible. We can't really do that.

Another thing they did was add a boolean field in their CompositeByteBuffer, which gets set to true when the buffer is recycled. So they don't need to do reference counting on every operation, instead they can just check a boolean.

I tried adding a boolean to our RefCnt.java, and it indeed fixes the regression. The problem is, due to class alignment issues in java, adding this boolean field increases the heap size of RefCnt from 24 to 32 bytes. This seems non-trivial given it's used in bucket cache where there could be many millions of them.

I think we can get around this by simply nulling out the recycler in RefCnt after it has been called. Then, instead of doing a boolean check we can do a null check. This performs similarly to the boolean, but without any extra memory.
Bug: Use hasPathCapability to support recoverLease, setSafeMode, isFileClosed for non-HDFS file system
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Mar/23 22:39
Affects Version/s: 2.4.16

after HADOOP-18671 , we will change the hbase-asyncfs to use use hasPathCapability to support recoverLease, setSafeMode, isFileClosed for non-HDFS file system instead of directly casting only HDFS in RecoverLeaseFSUtils
Bug: Hbase shell non interactive stalls
Status: Open | Priority: Major | Resolution: 
Created: 22/Mar/23 13:02
Affects Version/s: 2.4.8

The following bash shell script invocations may stall or not `hbase shell -n` indefinitely (=until timeout). This might be how tty / pipes interact with the hbase shell. Opening this to better understand either if this is a bug or limitation in hbase shell, or what workarounds are there. The below samples and execution is with hbase 2.4.8, whilst in a containerized HBase 2.4.11 the stall issues don't appear.

{color:#00875a}Works executed in CLI bash{color}
{code:java}
function check_alter_hbase () {
  HBASE_SHELL_CMD="hbase shell -n"
  hbase_state=$( echo "describe 'my_table'" | $HBASE_SHELL_CMD )
  echo $hbase_state
}
export -f check_alter_hbase
timeout 30s bash -c check_alter_hbase{code}
{color:#de350b}Doesn't work as bash script, stalls until timeout ** {color}
{code:java}
# cat stall.sh
function check_alter_hbase () {
  HBASE_SHELL_CMD="hbase shell -n"
  hbase_state=$( echo "describe 'my_table'" | $HBASE_SHELL_CMD )
  echo $hbase_state
}
export -f check_alter_hbase
timeout 30s bash -c check_alter_hbase
# echo $?
124{code}
{color:#172b4d}Running the same bash stall logic in HBase 2.4.11 doesn't seem to reproduce the issue, though there is a slight difference that we have containers there (e.g. pod hbasemaster-0).{color}

{color:#00875a}Works executed in CLI as bash script{color}
{code:java}
# cat works.sh
function check_alter_hbase () {
  HBASE_SHELL_CMD="hbase shell -n"
  hbase_state=$( echo "describe 'my_table'" | $HBASE_SHELL_CMD )
  echo $hbase_state
}
export -f check_alter_hbase
timeout --foreground 30s bash -c check_alter_hbase
# bash works.sh
Table my_table is ENABLED ...
# echo $?
0{code}
{color:#ff0000}Doesn't work as internal init script, stalls (called indirectly not via CLI){color}
{code:java}
function check_alter_hbase () {
  HBASE_SHELL_CMD="hbase shell -n"
  hbase_state=$( echo "describe 'my_table'" | $HBASE_SHELL_CMD )
  echo $hbase_state
}
export -f check_alter_hbase
timeout --foreground 30s bash -c check_alter_hbase
{code}
Running the same bash stall logic in HBase 2.4.11 doesn't seem to reproduce the issue, though there is a slight difference that we have containers there (e.g. pod hbasemaster-0).
Bug: hbase-spark bulkload stage directory limited to hdfs only
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Feb/23 18:36
Affects Version/s: connector-1.0.0

It's impossible to set up the staging directory for bulkload operation in spark-hbase connector to any other filesystem different from hdfs. That might be a problem for deployments where hbase.rootdir points to cloud storage. In this case, an additional copy task from hdfs to cloud storage would be required before loading hfiles to hbase.
Bug: Support hbase-connectors compilation with HBase 2.5.3, Hadoop 3.2.4 and Spark 3.2.3
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Feb/23 05:16

Goal is to allow hbase-connectors to compile with:
 * HBase: 2.5.3
 * Hadoop: 3.2.4 and
 * Spark: 3.2.3

We could also discuss if we want to bump the versions of the above mentioned in the pom itself,

or just want to let spark connector compile with above components as the JIRA title says.
Bug: Support steal job queue mode for read RPC queues of RWQueueRpcExecutor
Status: Open | Priority: Major | Resolution: 
Created: 30/Mar/23 04:14
Affects Version/s: 2.5.3

Currently, the RPC queues are distinguished by request type, under most circumstances of RWQueueRpcExecutor, there are write queues and read queues, while reads queues are always divided by get requests and scan requests. The reason why we isolate the scan requests and get requests is that we do not want large scans block small gets.

Since the handler resources for a regionserver is limited and we can't dynamicly change the handler ratio by the ratio of requests. We should both keep large scan and the small gets be isolated, and let the idle handlers for the samller ratio scans to handle some gets when the gets handlers are busy.

This steal queue idea can also be used in other circumstances, e.g. idle read handler steal jobs from write queus.
Bug: HBase token expired because the connection was not released
Status: Open | Priority: Major | Resolution: 
Created: 22/Feb/23 03:01

I used hbase-spark to read hbase data in spark streaming application, and the token expired error occurred after 7 days.

Error:
{code:java}
23/02/20 19:29:07 INFO AsyncRequestFutureImpl: id=38643, table=XXX, attempt=11/16, failureCount=1ops, last exception=java.io.IOException: Call to XXX.XXX/XXX.XXX.XXX.XXX:60020 failed on local exception: java.io.IOException: org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): Token has expired on XXX.XXX,60020,1675839912342, tracking started null, retrying after=10009ms, operationsToReplay=1 {code}
HBase RpcClient has expired connections that have not been cleaned up.

arthas command: 
{code:java}
watch org.apache.hadoop.hbase.ipc.AbstractRpcClient getConnection "target.connections.values.{token.toString}" -x 2 -n 2 {code}
result:
{code:java}
@ArrayList[
    @String[Kind: HBASE_AUTH_TOKEN, Service: a88deda4-d74c-483e-af8b-0fb9de7c7594, Ident: ((username=XXX, keyId=575, issueDate=1676858059133, expirationDate=1677462859133, sequenceNumber=772469))],
    ...
    @String[Kind: HBASE_AUTH_TOKEN, Service: a88deda4-d74c-483e-af8b-0fb9de7c7594, Ident: ((username=XXX, keyId=573, issueDate=1676656459093, expirationDate=1677261259093, sequenceNumber=587447))],
    ...
    @String[Kind: HBASE_AUTH_TOKEN, Service: a88deda4-d74c-483e-af8b-0fb9de7c7594, Ident: ((username=XXX, keyId=571, issueDate=1676454859019, expirationDate=1677059659019, sequenceNumber=430605))],
    ...
]{code}
The hbase connection seems to be cleaned up only after being idle for a period of time.
{code:java}
org.apache.hadoop.hbase.spark.HBaseConnectionCache#connectionMap
org.apache.hadoop.hbase.ipc.AbstractRpcClient#connections {code}
Bug: Bump commons-lang3 to 3.12.0, surefire to 3.0.0 and dependency manage reload4j to 1.2.25 to support jdk11
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Feb/23 07:49
Affects Version/s: connector-1.0.0

When trying to run the tests with Java 11, I get:
{noformat}
[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.733 s <<< FAILURE! - in org.apache.hadoop.hbase.spark.TestJavaHBaseContext
[ERROR] org.apache.hadoop.hbase.spark.TestJavaHBaseContext  Time elapsed: 0.721 s  <<< ERROR!
java.lang.ExceptionInInitializerError
    at org.apache.hadoop.hbase.spark.TestJavaHBaseContext.setUpBeforeClass(TestJavaHBaseContext.java:86)
Caused by: java.lang.NullPointerException
    at org.apache.hadoop.hbase.spark.TestJavaHBaseContext.setUpBeforeClass(TestJavaHBaseContext.java:86)[ERROR] org.apache.hadoop.hbase.spark.TestJavaHBaseContext  Time elapsed: 0.721 s  <<< ERROR!
java.lang.NullPointerException
    at org.apache.hadoop.hbase.spark.TestJavaHBaseContext.tearDownAfterClass(TestJavaHBaseContext.java:106)
{noformat}
The same test runs fine with Java 8.
Bug: Clean up hbase-connector dependencies and assembly
Status: Open | Priority: Major | Resolution: 
Created: 24/Mar/23 04:59

We ship a lot of external jars in the /lib directory of the assembly.
I know that for Spark none of those are needed, because we take everything from the Spark classpath.
I am not familiar with the Kafka connector, but according to the readme, that one also adds the jars from 'hbase classpath' , which includes most/all of the external JARs in the lib directory.
Bug: scan table is slow when enter hbase shell first time
Status: Open | Priority: Major | Resolution: 
Created: 29/Mar/23 03:54
Affects Version/s: 2.4.14

When we test HBase 2.4.14 on our cluster...

I find we  scan a table is slow when enter hbase shell first time.

step like this:

hbase shell

scan 'hbase:meta',\{LIMIT=>1}

the cost time is more than HBase 1.3

Any one can help compare the two versions(HBase 2.x vs HBase 1.x) in first scan process?

thanks~

 

-------

I find the the “Future<Result> f = cs.poll(timeBeforeReplicas, TimeUnit.MICROSECONDS);” in org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas#call 

cost above 200+ms
Bug: Should support single call queue mode for RPC handlers while separating by request type
Status: Open | Priority: Major | Resolution: 
Created: 02/Mar/23 12:03
Affects Version/s: 2.5.3

Currently we not only seperate call queues by request type, e.g. read, write, scan, but also distinguish queues for handlers by the config `hbase.ipc.server.callqueue.handler.factor`, whose description is as follows,
{code:java}
Factor to determine the number of call queues.
  A value of 0 means a single queue shared between all the handlers.
  A value of 1 means that each handler has its own queue. {code}
But I think what we want is not only one queue for all the requests, or each handler has its own queue. We also want each request type has its own queue.

Distinguishing queues in the same type of requests will make some handlers too iddle but some handlers too busy under current balanced/random RPC executor framework. For the extrem case, each handler has its own queue, then if a large request comes for a handler, duing to he executor dispath calls without considering the queue size or the state of the handler, the afterwards coming requests will be queued until the handler complete the large slow request. While other handlers may process small requests quickly, but they can not help or grab calls from the busy queue, they must stay and wait it own queue jobs coming. Then we can see the queue time of some requests are long but there are iddle handlers.

We can also see these circumstances, that the queue time of calls is too larger than the process time, sometimes twice or more. Restarting the slow RS will make these problems disappear. 

By using single call queue for each request type, we can fully use the handler resources.
Bug: Test failures on branch-2.4 with corrupted exclude list
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/23 15:01
Affects Version/s: 2.4.16

Nightly builds and PRs on branch-2.4 are failing with an invalid exclude list.

Executed unit test command:
{code:java}
/opt/maven/bin/mvn --batch-mode -Dmaven.repo.local=/home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-4944/yetus-m2/hbase-branch-2.4-patch-0 --threads=4 -Djava.io.tmpdir=/home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-4944/yetus-jdk8-hadoop2-check/src/target -DHBasePatchProcess -PrunAllTests -Dtest.exclude.pattern=**/replication.regionserver.TestMetaRegionReplicaReplicationEndpoint.java,**/client.TestMetaRegionLocationCache.java,**/master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks.java,**/replication.TestZKReplicationQueueStorageWARNING: All illegal access operations will be denied in a future release.java,**/replication.regionserver.TestBasicWALEntryStreamFSHLog.java -Dsurefire.firstPartForkCount=0.5C -Dsurefire.secondPartForkCount=0.5C clean test -fae {code}
The latest exclude list contains "WARNING: All illegal access operations will be denied in a future release" and maven treats this as a new parameter. As a result unit tests are failing on CI that rely on the exclude list.

[https://ci-hbase.apache.org/job/HBase-Find-Flaky-Tests/job/branch-2.4/lastSuccessfulBuild/artifact/output/excludes/*view*/]
{noformat}
**/replication.regionserver.TestMetaRegionReplicaReplicationEndpoint.java,**/client.TestMetaRegionLocationCache.java,**/master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks.java,**/replication.TestZKReplicationQueueStorageWARNING: All illegal access operations will be denied in a future release.java,**/replication.regionserver.TestBasicWALEntryStreamFSHLog.java {noformat}
Bug: Update the list of prefetched files upon region movement
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Mar/23 15:38
Affects Version/s: 3.0.0-alpha-3

Currently, we maintain a list of Hfiles on disk for which prefetch is complete and avoid prefetching those files after a restart. But we don't handle cases where blocks are evicted from the cache. This ticket is for updating the list of prefetched files when the region is moved but the blocks from that region are not evicted.
Bug: Include EventType and ProcedureV2 pid in logging via MDC
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Mar/23 13:17
Affects Version/s: 2.4.17

Tracing the distributed actions of ProcedureV2 {{RemoteProcedure}} execution is painful. We are pretty good about logging the {{proc_id}} most of the time, but we don't catch them all. Now that we're up on log4j2, let's use the MDC feature to include some basic information universally.
Bug: Normalizer can skip picking presplit regions while preparing merge plans
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 25/Feb/23 05:06

Normalizer can be used to merge the regions which become emtpy post TTL expiry.
But it's picking the presplit regions as well. We can skip picking presplit regions while preparing the merge plans by looking at the number of storefiles along with size. Presplit regions have zero storefiles where as the TTL expired regions have 1 storefile with zero size.
Bug: Race conditions in BlockingRpcConnection
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Mar/23 14:47

We've been experiencing strange timeouts since upgrading to hbase2 client. We use BlockingRpcConnection for now until we migrate our auth stack to native TLS. In diagnosing the timeouts, I noticed a few issues in this class:
 # Most importantly, there is a race condition which can result in a case where a BlockingRpcConnection instance has 2 reader threads running. In this case, both are competing for the socket and it causes weird timeouts and in some cases corrupted response (i.e. InvalidProtocolBufferException)
 # The waitForWork loop does not properly handle interruption. When it gets interrupted, if the above race condition occurs, the waitForWork loop ends up forever being in a tight loop. The "wait()" call instantly throws InterruptedException, and we set interrupted state back and restart the loop. So no waiting is occurring anymore.

The race condition is somewhat rare, only occurring in certain failure scenarios on our highest volume clients. But when it happens, a low level of errors will forever be thrown for the affected server connection until the client is bounced.
Bug: Remove numRegions in Region Metrics
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Mar/23 10:39

After HBASE-27681, I'd like to refactor region metrics, Then found this issue.

We have a metric named `numRegions` in Region Metrics, which I think it's a regionServer metric, but not a region metric.

And we do have a `regionCount` metric in RegionServer Metrics.

So I think we should delete the `numRegions`.
Bug: Add biggest cell related info into web ui
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Mar/23 12:42

There are some disadvantages to large cell, such as can't be cached or cause memory fragmentation, but currently user can't easily to find them out.

My proposal is save len and key of the biggest cell into fileinfo of hfile, and shown on web ui, including two places.

1: Add "Len Of Biggest Cell" into main page of regionServer, in here we can find out which regions has large cell by sorting.

2: Add "Len Of Biggest Cell" and "Key Of Biggest Cell" into region page, in here we can find out the exactly key and the hfile.