av9ash/GitBugs huggingface dataset:
Bug: Blocked on SyncFuture in AsyncProtobufLogWriter#write
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jan/22 13:47

Ran into an issue on hbase 2.4.6, I think related to HBASE-26679. Individual writes are blocking on SyncFuture, which never gets completed. Eventually (5m) the writes timeout and fail. But the regionserver hung on like this basically forever until I killed it about 14 hours later. While 26679 may fix the hang bug, I think we should have additional protection against such zombie states. In this case I think what happened is that the rollWAL was requested due to failed appends, but it also hung forever. See the below stack trace:

 
{code:java}
Thread 240 (regionserver/host:60020.logRoller):
  State: WAITING
  Blocked count: 38
  Waited count: 293
  Waiting on java.util.concurrent.CompletableFuture$Signaller@13342c6d
  Stack:
    java.base@11.0.5/jdk.internal.misc.Unsafe.park(Native Method)
    java.base@11.0.5/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
    java.base@11.0.5/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796)
    java.base@11.0.5/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128)
    java.base@11.0.5/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823)
    java.base@11.0.5/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998)
    app//org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:189)
    app//org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeMagicAndWALHeader(AsyncProtobufLogWriter.java:202)
    app//org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:170)
    app//org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:113)
    app//org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:669)
    app//org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:130)
    app//org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:841)
    app//org.apache.hadoop.hbase.wal.AbstractWALRoller$RollController.rollWal(AbstractWALRoller.java:268)
    app//org.apache.hadoop.hbase.wal.AbstractWALRoller.run(AbstractWALRoller.java:187) {code}
 

The wall roller thread was stuck on this wait seemingly forever, so it was never able to roll the wal and get writes working again. I think we should add a timeout here, and abort the regionserver if a WAL cannot be rolled in a timely manner.
Bug: Avoid waiting to clear buffer usage of ReplicationSourceShipper when aborting the RS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Mar/22 10:05
Affects Version/s: 2.4.10

HBASE-24813 introduced the clear of buffer used in replication source shipper, but there is sleep in the method, if the variable sleepForRetries has a large value, and there are many wal groups, the aborting of RS may last a long time, but we should only do some necessary things in the aborting progress.
{code:java}
void clearWALEntryBatch() {
  long timeout = System.currentTimeMillis() + this.shipEditsTimeout;
  while(this.isAlive() || this.entryReader.isAlive()){
    try {
      if (System.currentTimeMillis() >= timeout) {
        LOG.warn("Shipper clearWALEntryBatch method timed out whilst waiting reader/shipper "
          + "thread to stop. Not cleaning buffer usage. Shipper alive: {}; Reader alive: {}",
          this.source.getPeerId(), this.isAlive(), this.entryReader.isAlive());
        return;
      } else {
        // Wait both shipper and reader threads to stop
        Thread.sleep(this.sleepForRetries);
      }
    } catch (InterruptedException e) {
      LOG.warn("{} Interrupted while waiting {} to stop on clearWALEntryBatch. "
          + "Not cleaning buffer usage: {}", this.source.getPeerId(), this.getName(), e);
      return;
    }
  }
...... {code}
Bug: Mark RpcExecutor as IA.LimitedPrivate COPROC and PHOENIX
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Mar/22 11:56
Affects Version/s: 2.4.10

{{RpcExecutor}}, an abstract base class, is marked as {{IA.Private}}. However, it has several subclasses that are marked as {{IA.LimitedPrivate(COPROC, PHOENIX)}}. I think that the base class needs to match the highest exposure level of any of its subclasses.
Bug: SplitTableRegionProcedure calls openParentRegions() at a wrong state during rollback.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Mar/22 19:59
Affects Version/s: 2.4.10

Changed the issue title and description for the scope of the work. 

there is a bug in handling Rollback in SplitTableRegionProcedure.

[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/SplitTableRegionProcedure.java#L304]

[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/SplitTableRegionProcedure.java#L385]
{code:java}
In the state machine:


        case SPLIT_TABLE_REGION_CLOSE_PARENT_REGION:
          addChildProcedure(createUnassignProcedures(env));
          // Comments from HX:
          // createUnassignProcedures() can throw out IOException. If this happens,
          // it wont reach state SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGION and no parent regions
          // is closed as all created UnassignProcedures are rolled back. If it rolls back with
          // state SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, no need to call openParentRegion(),
          // otherwise, it will result in OpenRegionProcedure for an already open region.
          setNextState(SplitTableRegionState.SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS);
          break;


In the rollback,


        case SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS:
          // Doing nothing, in SPLIT_TABLE_REGION_CLOSE_PARENT_REGION,
          // we will bring parent region online
          break;
        case SPLIT_TABLE_REGION_CLOSE_PARENT_REGION:
          // Comments from HX: 
          // OpenParentRegion() should not be called here as explained above.
          openParentRegion(env);
          break; {code}
Bug: Disable Checks API callback from test results in PRs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Mar/22 10:46

Often as not, PR builds will succeed but be reported on the PR as a failure. The reason is represented in Jenkins pipeline as a warning in the "Archive Junit-formatted test results", with the message

{noformat}
[2022-03-01T22:05:49.166Z] Recording test results
[2022-03-01T22:05:50.627Z] [Checks API] No suitable checks publisher found.
{noformat}

This is confusing and discouraging to contributors.
Bug: Increments submitted by 1.x clients will be stored with timestamp 0 on 2.x+ clusters
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jan/22 22:52

We discovered this in our work on upgrading from ~1.2.0 to 2.4.6. A 1.2.0 client will submit an Increment, without specifying a timestamp. This should be translated as LATEST_TIMESTAMP, but an hbase 2 server will store it with a timestamp of 0.

This is because the QualifierValue timestamp proto field is optional, and is not set in branch-1 MutationProtos: [https://github.com/apache/hbase/blob/branch-1/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java#L218-L219.] In branch-1, the timestamp is actually ignored entirely: [https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L8435-L8456.] So this is not an issue there.

As of HBASE-18546, the timestamp is no longer ignored, but the usage of QualiferValue.getTimestamp() in deserialization is not wrapped in a .hasTimestamp() call: [https://github.com/apache/hbase/blob/7d8dc3524981a04e82ff236bb3dc18d36ebc82b4/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java#L886-L909]. The default value for non-present optionals is defined by the underlying primitive, so for longs that is 0. Since the Increment's timestamp is now respected by HRegion, it causes the erroneous 0 value to be stored: [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L4180] (updateLatestStamp only overwrites the cell timestamp if it's LATEST_TIMESTAMP, which in this case it has been deserialized as 0)

I think this might only apply to clients who do not use a KeyValueCodec.
Bug: BufferedDataBlockEncoder$OffheapDecodedExtendedCell.deepClone throws UnsupportedOperationException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Feb/22 15:50
Affects Version/s: 2.4.10

BufferedDataBlockEncoder$OffheapDecodedExtendedCell.deepClone throws an unsupportedException.

However, org.apache.hadoop.hbase.regionserver.HRegion.get(Get, boolean, long, long)

calls the method:
{code:java}
      // Copy EC to heap, then close the scanner.
      // This can be an EXPENSIVE call. It may make an extra copy from offheap to onheap buffers.
      // See more details in HBASE-26036.
      for (Cell cell : tmp) {
        results.add(cell instanceof ByteBufferExtendedCell ?
          ((ByteBufferExtendedCell) cell).deepClone(): cell);
      } {code}

According to the comment above, this is probably caused by HBASE-26036.
Bug: Integrate user-experience for hbase-client
Status: Reopened | Priority: Major | Resolution: 
Created: 13/Jan/22 22:22

Today, we have two mechanism in order to get the tokens needed to authenticate:
 # Kerberos, we rely on a Kerberos ticket being present in a well-known location (defined by JVM properties) or via programmatic invocation of UserGroupInformation
 # Delegation tokens, we rely on special API to be called (our mapreduce API) which loads the token into the current UserGroupInformation "context" (the JAAS PrivilegedAction).

The JWT bearer token approach is very similar to the delegation token mechanism, but HBase does not generate this JWT (as we do with delegation tokens). How does a client provide this token to the hbase-client (i.e. {{ConnectionFactory.getConnection()}} or a {{UserGroupInformation}} call)? We should be mindful of all of the different "entrypoints" to HBase ({{{}hbase ...{}}} commands, {{java -cp}} commands, Phoenix commands, Spark comands, etc). Our solution should be effective for all of these approaches and not require downstream changes.

*Update*

The environment variable approach is done, but I have a new idea which will support token renewal as well:

Implement a generic JwtTokenProvider interface with a default file-based provider.

Read the token from a specific file similar to what happens in Kerberos. Token renewal is not part of this ticket.
Bug: Define user experience for JWT renewal
Status: Open | Priority: Major | Resolution: 
Created: 13/Jan/22 22:27

We need to define what our level of support will be for an HBase application which must run longer than the lifetime of a JWT token.

The JWT 2.0 RFCs mention different kinds of tokens, notably a Refresh token may be helpful [https://datatracker.ietf.org/doc/html/rfc8693]

This is inter-twined with HBASE-26667. For example, if we maintained a Refresh token in the client, we would have to build in logic (like we have for Kerberos credentials) to automatically launch a thread and know where to obtain a new JWT token from.

*Idea*

Once the JwtTokenProvider interface and file-based provider is ready, implement token renewal logic.

Port FileWatcher class from the ZooKeeper project and watch for file changes.
Bug: Add native TLS encryption support to RPC server/client
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/22 22:17
Affects Version/s: 2.4.12

Today, HBase must complete the SASL handshake (saslClient.complete()) prior to turning on any RPC encryption (hbase.rpc.protection=privacy, sasl.QOP=auth-conf).

This is a problem because we have to transmit the bearer token to the server before we can complete the sasl handshake. This would mean that we would insecurely transmit the bearer token (which is equivalent to any other password) which is a bad smell.

Ideally, if we can solve this problem for the oauth bearer mechanism, we could also apply it to our delegation token interface for digest-md5 (which, I believe, suffers the same problem).

The plan is to port Server/Client TLS implementation from the ZooKeeper project. It's a Netty based solution which looks like the best fit for NettyRpc client/server.
Bug: Add dynamic configuration support for system coprocessors
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Mar/22 01:13
Affects Version/s: 3.0.0-alpha-3

[Dynamic Configuraiton|https://hbase.apache.org/book.html#dyn_config] is very helpful that the operator can keep the JVM for regionserver or master running without restarting it. 

With this feature, this task aims to extend the scope of on demend configuration to system coprocessors including REGION, USER REGION, REGIONSERVER, MASTER such that we could save time on rolling restart with a quicker update.
Bug: Update Hadoop3 versions for JEP 223 compliance
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 30/Mar/22 12:28
Affects Version/s: 2.4.12

It happened that my JDK version upgraded to 11.0.14.1. Running unit tests involving the HDFS mini cluster now fails with a stack trace that ends with

{noformat}
Caused by: java.lang.IllegalArgumentException: Invalid Java version 11.0.14.1                                                                                                                               
        at org.eclipse.jetty.util.JavaVersion.parseJDK9(JavaVersion.java:71)                                                                                                                                
        at org.eclipse.jetty.util.JavaVersion.parse(JavaVersion.java:49)                                                                                                                                    
        at org.eclipse.jetty.util.JavaVersion.<clinit>(JavaVersion.java:43)
{noformat}

We are using hadoop-3.2.0, which uses jetty-9.3.24. This is a Jetty issue has been fixed upstream in Jetty via https://github.com/eclipse/jetty.project/issues/2090. Hadoop has upgraded its Jetty version to 9.4.20 in HADOOP-16152, which is available as of hadoop-3.2.2.
Bug: TestProcedureSchedulerConcurrency fails in pre commit
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Feb/22 07:56

Saw this on the jenkins page. Seems like a test issue.

{noformat}
Exception in thread "Thread-10" java.util.ConcurrentModificationException
	at java.base/java.util.ArrayDeque.nonNullElementAt(ArrayDeque.java:271)
	at java.base/java.util.ArrayDeque$DeqIterator.next(ArrayDeque.java:701)
	at java.base/java.util.AbstractCollection.toString(AbstractCollection.java:472)
	at java.base/java.lang.String.valueOf(String.java:2951)
	at java.base/java.lang.StringBuilder.append(StringBuilder.java:168)
	at org.apache.hadoop.hbase.procedure2.ProcedureEvent.toString(ProcedureEvent.java:134)
	at java.base/java.lang.String.valueOf(String.java:2951)
	at java.base/java.lang.StringBuilder.append(StringBuilder.java:168)
	at org.apache.hadoop.hbase.procedure2.TestProcedureSchedulerConcurrency$2.run(TestProcedureSchedulerConcurrency.java:130)
{noformat}
Bug: Introduce custom attributes to threads
Status: Open | Priority: Major | Resolution: 
Created: 22/Feb/22 16:02

The idea is to introduce custom attributes to a thread. We will be passing this from thread to newly spawned thread. More details can be found here: https://docs.google.com/document/d/1b6CJt7_E8NU5fe41yeZZhuVPx-5SDx7x1jNtIRmnR_g/edit#heading=h.bv19tq8gxgcv
Bug: Introduce a little RAMBuffer for bucketcache to reduce gc and improve throughput
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jan/22 12:05

In branch-1, BucketCache just allocate new onheap bytebuffer to construct new HFileBlock when get cached blocks. This rough allocation increases the GC pressure for those "hot" blocks. 
Here introduce a RAMBuffer for those "hot" blocks in BucketCache. The thought is simple. The RAMBuffer is an timeout expiring cache. When a Multi-level block is read twice, we cache it in the RAMBuffer. When the block timeout in the cache (e.g. 60s), that means the block is not being accessed in 60s, we evict it. Not like LRU, we do not cache block when the whole RAMBuffer size reaches to a threshold (to fit different workload, the threshold is dynamic). This will prevent the RAMBuffer from being churned.


{panel:title=The performance of RAMBuffer with its hit ratio is 100%}
!Hit 100%.png|height=250|width=250!
{panel}

I also did a YCSB performance test. 

The circumstance is:
Size of BucketCache: 40 GB
Target table size: 112 GB
Properties:
!Properties.png|height=250|width=250!

The operation distribution of YCSB workload is latest.

Client Side Metrics
See the attachment ClientSideMetrics.png

Server Side GC:
The current bucket cache triggered 217 GCs, which costs 2.74 minutes in total.
With RAMBuffer, the server side had 210 times GC and 2.56 minutes in total.

As the master & branch-2 using ByteBufferAllocator to manage the bucketcache memory allocation, the RAMBuffer may not have GC improvement as much as branch-1.
Bug: Backport HBASE-26688 to branch-1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jan/22 08:15
Affects Version/s: 1.7.1

Backport issue [HBASE-26688|https://issues.apache.org/jira/browse/HBASE-26688] to branch-1 to eliminate the client crash when using threads shared EMPTY_RESULT.
Bug: Reimplement master local region initialization to better work with SFT
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/22 07:55

It is not like a normal region where we have a TableDescriptor so it can store the SFT implementation of its own. In the current implementation, if we change the global SFT configuration, the SFT implementation of the master local reigon will be changed and cause data loss.

First I think we could hard coded it to use DefaultSFT. The region is small and will not cause too much performance impact. Then we could find a way to manage the SFT implementation of it.

====== Update ======

The initialization of master local region depends on renaming, which can not work well on OSS. So we should also change it. The basic idea is to touch a '.initialized' file to indicate it is initialized. Need to consider how to migrate from the existing master local region where it does not have this file.

And we could also store the TableDescriptor on file system, so we can determine whether this is a SFT change. If so, we should do the migration before actually opening the master local region.
Bug: RegionServer JVM crash when compact mob table
Status: Open | Priority: Major | Resolution: 
Created: 11/Mar/22 08:32

When compact a mob table, RS JVM may crash or failed to do compaction as the following logs:
{code:java}
2022-03-11T16:18:44,089 ERROR [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45525-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(675): Compaction failed Request=regionName=t1,,1646986716811.964618e679a2434aa7d27018baef8154., storeName=A, fileCount=2, fileSize=2.0 M (1010.2 K, 1010.2 K), priority=1, time=1646986723135java.io.IOException: Mob compaction failed for region: 964618e679a2434aa7d27018baef8154    at org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.performCompaction(DefaultMobStoreCompactor.java:574) ~[classes/:?]    at org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:365) ~[classes/:?]    at org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.compact(DefaultMobStoreCompactor.java:225) ~[classes/:?]    at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:125) ~[classes/:?]    at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1141) ~[classes/:?]    at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2442) ~[classes/:?]    at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:656) ~[classes/:?]    at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:702) ~[classes/:?]    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_292]    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_292]    at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_292]Caused by: java.io.IOException: Added a key not lexically larger than previous. Current cell = org.apache.hadoop.hbase.PrivateCellUtil$ValueAndTagRewriteByteBufferExtendedCell@565d5bac, prevCell = user00000000000000000000/A:filed01/1646986721047/Put/vlen=0/mvcc=0    at org.apache.hadoop.hbase.util.BloomContext.sanityCheck(BloomContext.java:63) ~[classes/:?]    at org.apache.hadoop.hbase.util.BloomContext.writeBloom(BloomContext.java:54) ~[classes/:?]    at org.apache.hadoop.hbase.regionserver.StoreFileWriter.appendGeneralBloomfilter(StoreFileWriter.java:296) ~[classes/:?]    at org.apache.hadoop.hbase.regionserver.StoreFileWriter.append(StoreFileWriter.java:315) ~[classes/:?]    at org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.performCompaction(DefaultMobStoreCompactor.java:464) ~[classes/:?]    ... 10 more {code}
It is the same problem as [HBASE-25929|https://issues.apache.org/jira/browse/HBASE-25929], because DefaultMobStoreCompactor overwrite performCompaction method of DefaultCompactor.
Bug: BufferedDataBlockEncoder.OnheapDecodedCell value can get corrupted
Status: Reopened | Priority: Major | Resolution: 
Created: 17/Mar/22 06:12

In our production cluster we observed the cell value is modified after successful scanner read. After analyzing we observed OnheapDecodedCell is not created properly.

We create OnheapDecodedCell with complete valAndTagsBuffer underlying array.

{code:java}
 return new OnheapDecodedCell(Bytes.copy(keyBuffer, 0, this.keyLength),
          currentKey.getRowLength(), currentKey.getFamilyOffset(), currentKey.getFamilyLength(),
          currentKey.getQualifierOffset(), currentKey.getQualifierLength(),
          currentKey.getTimestamp(), currentKey.getTypeByte(), valAndTagsBuffer.array(),
          valAndTagsBuffer.arrayOffset() + vOffset, this.valueLength, memstoreTS, tagsArray,
          tOffset, this.tagsLength);
{code}

Here we are passing valAndTagsBuffer.array() for value extraction.


The underlying array will be modified if it is altered anywhere.
Bug: Allow builders of hbase to specify hadoop inclusion mode for hbase-shaded-client
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 31/Mar/22 15:57

HBASE-20333 added a new shaded artifact, hbase-shaded-client-byo-hadoop. This is useful in environments where they already depend on hadoop so pulling in the hadoop classes from hbase-shaded-client would be extreme.

I think the split that [~busbey] decided on there definitely made sense for the general case (with byo-hadoop being the specially named artifact). But some of us may work in environments where the default case is the byo-hadoop case. For those environments, it's a little confusing and verbose to have to refer to the hbase client as hbase-shaded-client-byo-hadoop everywhere. For example, we have well over 500 poms directly pulling in hbase client and many more pulling it in transitively. We'll want to use byo-hadoop everywhere in our environment, but that's an implementation detail that the teams owning those 500 poms don't need to know or care about.

Using profiles I think we could provide an optional activation which allows someone building their own hbase to decide whether hbase-shaded-client should include hadoop or not. This way normal users using the artifacts published to maven central are unaffected, but super users building their own have some extra flexibility.