av9ash/GitBugs huggingface dataset:
Bug: Compact L0 files for cold regions using StripeCompactionPolicy
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Nov/21 13:11
Affects Version/s: 2.0.0

For cold regions, the L0 files can be all expired after a long while. We should compact and delete all TTL expired data in them if the count of them is less than the configured min files to compact in L0.
Bug: The default rpc timeout 200ms is too small for replicating meta edits
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Dec/21 13:54

For most meta edits, we will call refreshStoreFiles, which is time consuming. When running tests in HBASE-26487, it is very easy to timeout and cause the replicating to pause for a while.

I think for replicating meta edits, we should have a larger timeout value.
Bug: [hbase-thirdparty] Exclude the original protobuf-java jar when shading
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Nov/21 13:25

We have a patched version of protobuf and we will compile it, and then in shade we also include the original protobuf-java, where we will get lot of warnings

{noformat}
WARNING] hbase-shaded-protobuf-3.5.1.jar, protobuf-java-3.17.1.jar define 670 overlapping class and resource:
[WARNING]   - META-INF/MANIFEST.MF
[WARNING]   - com.google.protobuf.AbstractMessage
[WARNING]   - com.google.protobuf.AbstractMessage$Builder
[WARNING]   - com.google.protobuf.AbstractMessage$BuilderParent
[WARNING]   - com.google.protobuf.AbstractMessageLite
[WARNING]   - com.google.protobuf.AbstractMessageLite$Builder
[WARNING]   - com.google.protobuf.AbstractMessageLite$Builder$LimitedInputStream
[WARNING]   - com.google.protobuf.AbstractMessageLite$InternalOneOfEnum
[WARNING]   - com.google.protobuf.AbstractParser
[WARNING]   - com.google.protobuf.AbstractProtobufList
[WARNING]   - 660 more...
[WARNING] maven-shade-plugin has detected that some class files are
[WARNING] present in two or more JARs. When this happens, only one
[WARNING] single version of the class is copied to the uber jar.
[WARNING] Usually this is not harmful and you can skip these warnings,
[WARNING] otherwise try to manually exclude artifacts based on
[WARNING] mvn dependency:tree -Ddetail=true and the above output.
[WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
[WARNING] hbase-shaded-protobuf-3.5.1-sources.jar, protobuf-java-3.17.1-sources.jar define 202 overlapping resources: 
[WARNING]   - META-INF/MANIFEST.MF
[WARNING]   - com/google/protobuf/AbstractMessage.java
[WARNING]   - com/google/protobuf/AbstractMessageLite.java
[WARNING]   - com/google/protobuf/AbstractParser.java
[WARNING]   - com/google/protobuf/AbstractProtobufList.java
[WARNING]   - com/google/protobuf/AllocatedBuffer.java
[WARNING]   - com/google/protobuf/Android.java
[WARNING]   - com/google/protobuf/Any.java
[WARNING]   - com/google/protobuf/AnyOrBuilder.java
[WARNING]   - com/google/protobuf/AnyProto.java
[WARNING]   - 192 more...
[WARNING] maven-shade-plugin has detected that some class files are
[WARNING] present in two or more JARs. When this happens, only one
[WARNING] single version of the class is copied to the uber jar.
[WARNING] Usually this is not harmful and you can skip these warnings,
[WARNING] otherwise try to manually exclude artifacts based on
[WARNING] mvn dependency:tree -Ddetail=true and the above output.
[WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
{noformat}

Maybe this is the case why we bundle the incorrect protobuf classes and cause the failure when running with JDK8.
Bug: ArrayIndexOutOfBoundsException in KeyValueUtil.copyToNewKeyValue()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Dec/21 12:53
Affects Version/s: 2.2.7

While investigating a Phoenix crash, I've found a possible problem in KeyValueUtil.

When using Phoenix, we need configure (at least for older versions) org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec as a WAL codec in HBase.

This codec will eventually serialize standard (not phoenix specifc WAL entries) to the WAL file, and internally converts the Cell objects to KeyValue objects, by building a new byte[].

This fails with an ArrayIndexOutOfBoundsException, because the we allocate a byte[] the size of Cell.getSerializedSize(), and it seems that we are processing a Cell that does not actually serialize the column family and later fields. 
However, we are building a traditional KeyValue object for serialization, which does serialize them, hence we run out of bytes.

I think that since we are writing a KeyValue, we should not rely of the getSerializedSize() method of the source cell, but rather calculate the backing array size based on how KeyValue expects its data to be serialized.

The stack trace for reference:

{noformat}
java.lang.ArrayIndexOutOfBoundsException: 9787
        at org.apache.hadoop.hbase.util.Bytes.putByte(Bytes.java:502)
        at org.apache.hadoop.hbase.KeyValueUtil.appendKeyTo(KeyValueUtil.java:142)
        at org.apache.hadoop.hbase.KeyValueUtil.appendToByteArray(KeyValueUtil.java:156)
        at org.apache.hadoop.hbase.KeyValueUtil.copyToNewByteArray(KeyValueUtil.java:133)
        at org.apache.hadoop.hbase.KeyValueUtil.copyToNewKeyValue(KeyValueUtil.java:97)
        at org.apache.phoenix.util.PhoenixKeyValueUtil.maybeCopyCell(PhoenixKeyValueUtil.java:214)
        at org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec$IndexKeyValueEncoder.write(IndexedWALEditCodec.java:218)
        at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.append(ProtobufLogWriter.java:59)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.doAppend(FSHLog.java:294)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.doAppend(FSHLog.java:65)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.appendEntry(AbstractFSWAL.java:931)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1075)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:964)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:873)
        at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:129)
        at java.lang.Thread.run(Thread.java:748)
{noformat}

Note that I am still not sure exactly what triggers this bug, one possibility is org.apache.hadoop.hbase.ByteBufferKeyOnlyKeyValue
Bug: Set storage policy of recovered edits  when wal storage type is configured
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Dec/21 10:40

In our cluster, we has many SSD and a little HDD.  (Most table configured storage policy ONE_SSD, and all wals is configured ALL_SSD)

when all cluster down, It's difficult to recovery cluster. Because HDD Disk IO bottleneck (Almost all disk io util is 100%).

I think the most hdfs operation when recovery is split wal to recovered edits dir, And read it.

And it goes better when i stop hbase and set all recovered.edits to ALL_SSD.

So we can get benifit of recovery time if we set recovered.edits dir to better storage like WAL.

Now i reuse config item  hbase.wal.storage.policy to set recovered.edits storage type. Because I did not find a scenario where they use different storage Policy
Bug: Add expiration for long time vacant scanners in Thrift2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Dec/21 08:14

In thrift1 implementation, the ThriftHBaseSerivceHandler holds a Cache with an expire time after access. This will make the long time vacant scanners be collected by gc. However, in thrift2 we do not have this feature and only use a map to store scanners and we need the client close the scanner manually. If not, the expired scanners will live in memory forever. In this case, I applied the cache expiration to thrift2 service.
Bug: Optimize decodeNumeric in OrderedBytes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Dec/21 12:53

Currently, when we decode a byte array to a big decimal, there are plenty of BigDecimal operations, which result in a lot of only once used BigDecimal objects. Furthermore, the BigDecimal calculations are slow. We could boost this function by using String concatenation and point movement of BigDecimal. The JMH benchmark is uploaded to the attachment.

Also, I added a UT to test the encoding / decoding correctness of 200 random test samples.

The logic of the process is same as the Optimization of the encode function in the ticket [HBASE-26566|https://issues.apache.org/jira/browse/HBASE-26566]
Bug: Should find a way to clear the replication queue for a legacy region_replica_replication peer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Dec/21 13:33

When rolling upgrading, we will delete the legacy region_replica_replication peer. But since the old region servers still use this peer for replicating, we can not delete all the replication queues.

We need to find a way to deal with these legacy replication queues after upgrading.
Bug: Consider rolling upgrading from old region replication framework
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Nov/21 15:51

In the old implementation, we will cerate a replication peer for region replication. And in the new framework, we do not need the peer any more, so we have deleted the RegionReplicaReplicationEndpoint class.

But when rolling upgrading, the peer is still there so we will try to load it and it will lead to a class not found exception. We need to handle it.

And during the upgrading, some of the region servers may not have the correct replicateToReplica method yet. Maybe we could try to catch the no such method error and try to fallback to use replay method. Although it is not perfect, but finally a flush can fix everything.
Bug: Should not always clear all the failed replicas when getting a flush all request
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Nov/21 16:05

As described in this section in the design doc:

https://docs.google.com/document/d/1WPSBwRqIPgTrPRM2wtzAe-3XUvA9S7auWy3Fy8xuZLc/edit#heading=h.8nfip05gosr

We need to make sure that all the edits which have been sent to the broken replica must have a less sequence id than the flush sequence number of the commit flush edit, then we are safe to recover it. Otherwise we may miss some edit.

The current implementation is not enough, we need to improve it.
Bug: Make sure we do not flush a region too frequently
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Nov/21 02:08

In HBASE-26412, once we found there is an error replicating the edits, we will trigger a flush, and in HBASE-26413, we are likely to also trigger a flush when the pending edits are too large.

We should find a way to avoid flushing the region too much.

And a failure of replicating usually means the region is moved or the target region server is crashed, only a flush after the secondary replica is online will be useful, so flushing the region lots of times before the scedondary replica online is just useless.
Bug: The logic of the method incrementIV in Encryption class has problem
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Dec/21 01:53

{code:java}
public static void incrementIv(byte[] iv, int v) {
    int length = iv.length;
    boolean carry = true;
    // TODO: Optimize for v > 1, e.g. 16, 32
    do {
      for (int i = 0; i < length; i++) {
        if (carry) {
          iv[i] = (byte) ((iv[i] + 1) & 0xFF);
          carry = 0 == iv[i];
        } else {
          break;
        }
      }
      v--;
    } while (v > 0);
  }
{code}
The carry flag is outside the while loop and the inner loop check the carry value is true, so that when it was set to false, it will make the rest iterations busy spinning. 
And the logic becomes whatever the v is, the IV will be incremented by (1 or 2). As the description in todo, this function should support increment value > 1. So that this is not what we expect I think.
Bug: Snapshot referenced data files are deleted when delete a table with merge regions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Dec/21 08:33

In our cluster, we have a feature: take a snapshot when delete a table.

But when we restore the snapshot, we found that some data files are deleted.

The problem is that, when delete a table with merge regions, HBase only archive regions in meta, and merged parent regions are deleted in file system which contain data files in snapshot .
Bug: Netty exclusion through ZooKeeper not effective as intended
Status: Open | Priority: Major | Resolution: 
Created: 17/Dec/21 21:14
Affects Version/s: 2.4.8

Picking up where [~psomogyi] has been digging this week. We've been seeing an issue where MiniDFS-based tests fail to start due to missing netty classes.

HBASE-25969 seems to have intended to remove transitive Netty but was ineffective (at least for hadoop.profile=3.0). The dependency exclusion was for {{io.netty:netty}} and {{io.netty:netty-all}} but ZooKeeper 3.5.7 transitively depends on {{netty-handler}} and {{netty-transport-native-epoll}}  (per [https://search.maven.org/artifact/org.apache.zookeeper/zookeeper/3.5.7/jar)]

The funny part is that we _should_ have seen failures in any hbase unit test using MiniDFS because we excluded netty and netty-all in HBASE-25969, but because we missed the exclusions, we still keep running.

The intent of HBASE-25969 was good, but I think we need to revisit the execution. We need netty dependencies on the scope=test classpath. We just want to keep them off the scope=compile classpath (out of our client and server jars).

disclaimer: I have not yet looked at 2.5.x or master yet to see if this also affects them.
Bug: Upgrade to log4j 2.16.0
Status: Resolved | Priority: Major | Resolution: Done
Created: 14/Dec/21 09:27

Log4j just release version 2.16.0 where jndi is turned off by default. Based on the release announcement it is not required to fix CVE-2021-44228 but recommended. 

[https://lists.apache.org/thread/d6v4r6nosxysyq9rvnr779336yf0woz4]
Bug: Missing junit when executing IntegrationTestRegionReplicaReplication
Status: Open | Priority: Major | Resolution: 
Created: 17/Dec/21 15:55

{noformat}
Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
	at org.apache.hadoop.hbase.IntegrationTestIngest.initTable(IntegrationTestIngest.java:101)
	at org.apache.hadoop.hbase.IntegrationTestIngest.setUpCluster(IntegrationTestIngest.java:92)
	at org.apache.hadoop.hbase.IntegrationTestBase.setUp(IntegrationTestBase.java:170)
	at org.apache.hadoop.hbase.IntegrationTestBase.doWork(IntegrationTestBase.java:153)
	at org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:153)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.hadoop.hbase.IntegrationTestRegionReplicaReplication.main(IntegrationTestRegionReplicaReplication.java:225)
Caused by: java.lang.ClassNotFoundException: org.junit.Assert
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 7 more
{noformat}
Bug: Protobuf Services should be resolved by their full name
Status: Open | Priority: Major | Resolution: 
Created: 16/Dec/21 18:44
Affects Version/s: 2.5.0

I noticed this while investigating test failures resulting from HBASE-26542. The test in question ({{TestRpcServerSlowConnectionSetup}}) builds a connection header by populating the {{serviceName}} field with the service descriptor's {{getFullName()}} method. The value returned by {{getFullName()}} is the protobuf service's package and name. However, down in {{RpcServer}}, we resolve the service named by the connection header using the service descriptor's {{getName()}} method. {{getName()}} returns the protobuf service's name, without the package.

It is my opinion that this is a bug in the {{RpcServer}}, it should resolve services using their fully qualified name.
Bug: StoreHotnessProtector may block Replication
Status: Open | Priority: Major | Resolution: 
Created: 14/Dec/21 19:32

I'm upgrading from hbase1 to hbase2, and I'm still in my QA environment where load is very low. Even still, I've noticed some bad interaction between Replication and the StoreHotnessProtector.

The ReplicationSink collects edits from the WAL and executes them in batches via the normal HTable interface. Despite the name of this property, the max batch sizes are based on "hbase.rpc.rows.warning.threshold" which has a default of 5000. 

The StoreHotnessProtector defaults to allowing 10 concurrent writes (of 100 columns or more) to a Store, or 20 concurrent "prepares" of said writes. The Prepare part is what causes issues here. When a batch mutate comes in, the RS first takes a lock on all rows in the batch. This happens in HRegion#lockRowsAndBuildMiniBatch, and the writes are recorded as "preparing" in StoreHotnessProtector before acquiring the lock. This recording basically increments a counter, and throws an exception if that counter goes over 20.

Back in HRegion#lockRowsAndBuildMiniBatch, the exception is caught and recorded in the results for any items that failed. Any items that succeed continue on to write,  unless the write is atomic, in which case it immediately throws an exception.

This response gets back to the client, which automatically handles retries. With enough retries, the batch call will eventually succeed because each retry contains fewer and fewer writes to handle. Assuming you have enough retries, this is basically enforcing an automatic chunking of of a batch write into sub-batches of 20. Again, this only affects writes that hit more than 100 columns (by default).

At this point I'll say that this in general seems overly aggressive, especially since the StoreHotnessProtector doesn't actually do any checks for actual load on the RS. You could have a totally idle RegionServer and submit a single batch of 100 Puts with 101 columns each – if you don't have at least 5 retries configured, the batch will fail.

Back to ReplicationSink, the default batch size is 5000 Puts and the default retries is 4. For a table with wide rows (which might cause replication to try to sink Puts with more than 100 columns), it becomes basically impossible to replicate because the number of retries is not nearly enough to move through a batch of up to 5000, 20 at a time.
Bug: Unexpected crash of meta RegionServer causes the cluster out of service
Status: Open | Priority: Major | Resolution: 
Created: 04/Nov/21 03:12
Affects Version/s: 1.7.1

We have a cluster of two HMasters, C3HM1 and C3HM2, and three RegionServers, C3RS1, C3RS2, C3RS3. 
We use an external ZooKeeper cluster which is a pseudo-distributed cluster:

{code:java}
  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>C3hb-zk</value>
  </property>
  <property>
    <name>hbase.zookeeper.property.clientPort</name>
    <value>11181</value>
  </property>
{code}

For other HBase options, we use the default settings. The buggy scenario is as follows:

1. Start the cluster, C3HM1 becomes the active master;
2. C3RS2 crashes right before creating the znode "/hbase/meta-region-server" on ZooKeeper;

{code:java}
[org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:665), org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:644), org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndWatch(ZKUtil.java:1182), org.apache.hadoop.hbase.zookeeper.MetaTableLocator.setMetaLocation(MetaTableLocator.java:464), org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:2182), org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:329)]
{code}

3. The meta server is still not online after 10 minutes. The data of znode "/hbase/master" is C3HM1.

And the bug does not appear on HBase-2.4.5.
Bug: Unexpected crash makes client receive error exception
Status: Open | Priority: Major | Resolution: 
Created: 19/Oct/21 10:37
Affects Version/s: 1.7.1

There is a table {{mytable}} in a HBase cluster with two HMasters and two RegionServers.

1. Current active HMaster is HMaster1;
2. Client disabled the table {{mytable}};
3. {{mytable}} is in disabled state;
4. Client request to truncate {{mytable}};
5. HMaster1 tries to truncate {{mytable}}. For the TruncateTableState {{TRUNCATE_TABLE_ASSIGN_REGIONS}}, HMaster1 assigns the regions of the table and mark the table as Enabling;
6. HMaster1 crashes;
7. HMaster2 becomes active;
8. Client gets info that the truncate operation failed due to {{TableNotDisabledException}}, while the table has already be truncated.

In HMaster2, we got following info:

{code:java}
INFO [ProcedureExecutor-12] procedure2.ProcedureExecutor: Rolledback procedure TruncateTableProcedure (table=mytable preserveSplits=true) id=9 owner=myhbase state=ROLLEDBACK exec-time=4.6030 sec exception=org.apache.hadoop.hbase.TableNotDisabledException: mytable
{code}

It seems that the client receives such error message due to HMaster's recovery process is recovering the truncate table process. But for client, it should not get this error message caused by HMaster failover. Similarly, for create table, disable table request, client can get java.util.concurrent.TimeoutException caused by HMaster failover.
Bug: Passing an invalid DURABILITY when creating a table enters an endless loop of retries
Status: Open | Priority: Major | Resolution: 
Created: 07/Dec/21 18:25

As part of our Hbase2 upgrade, our automation copies the HTableDescriptor from a CDH5 cluster into the HBase2 cluster, then kicks off replication. During our testing we encountered a misconfigured table, which had a DURABILITY => 'DEFAULT', when the correct value is 'USE_DEFAULT'.

In hbase 1.x, any invalid value encountered by Durability.valueOf is try/caught and results in the default value of USE_DEFAULT. So this misconfiguration caused no pain in cdh5.

In hbase 2.x+, the IllegalArgumentException from Durability.valueOf is no longer caught. This is probably a good thing, but unfortunately it caused the CreateTableProcedure to fail in a way that resulted in an endless loop of retries, with no backoff.

This may be a general issue with CreateTableProcedure – there should probably be a pre-step which validates the HTableDescriptor and terminally fails if invalid.

Additionally, does it make sense to have a backoff on the retry of procedures? The vary rapid retry of this procedure actually caused HDFS issues because it was creating many thousands of .regioninfo files in rapid succession, enough to lag replication and cause DataNodes to be considered bad, which caused RegionServers to abort due to failed WAL writes.