av9ash/GitBugs huggingface dataset:
Bug: Add HBase Thrift API to get all table names along with whether it is enabled or not
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Sep/21 08:48
Affects Version/s: 2.2.7

When loading Hue HBase page, hue will firstly get all the table names in one thrift call, then hue will send a request as a new connection to Hbase thrift server for each table to check if the table is enabled or not. When the number of table becomes big, for example a few hundreds. Hue will have the chance to fail to show the table list, and it shows an error message on Hue web UI "Api Error: Unable to authenticate". In Hbase thrift server log, we can see error message, below is an example from test environment:
======================================
2020-07-02 21:54:01,025 INFO org.apache.hadoop.hbase.thrift.ThriftHttpServlet: Failed to authenticate with HTTP/[...]@HADOOP.COM kerberos principal
2020-07-02 21:54:01,025 ERROR org.apache.hadoop.hbase.thrift.ThriftHttpServlet: Kerberos Authentication failed
org.apache.hadoop.hbase.thrift.HttpAuthenticationException: java.lang.reflect.UndeclaredThrowableException
......
Caused by: java.lang.reflect.UndeclaredThrowableException
......
Caused by: org.apache.hadoop.hbase.thrift.HttpAuthenticationException: Kerberos authentication failed:
......
Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34))
......
Caused by: KrbException: Request is a replay (34)
======================================
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

My current understanding of the cause is Hue sends requests too fast so that it has the chance to send 2 requests with the same Kerberos authenticator (timestamp is same). So Hbase thrift server will think this is a replay attack. We came up with this assumption based on tcpdump observation and this document, https://web.mit.edu/kerberos/krb5-latest/doc/basic/rcache_def.html

If HBase Thrift server can have an API call that Hue can call it once and get all table names with whether is enabled or not, we can avoid above issue.
Bug: Do not store meta region location as permanent state on zookeeper
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Aug/21 05:03

As it breaks one of our design rules

https://hbase.apache.org/book.html#design.invariants.zk.data

We used to think hbase should be recovered automatically when all the data on zk (except the replication data) are cleared, but obviously, if you clear the meta region location, the cluster will be in trouble, and need to use operation tools to recover the cluster.

So here, along with the ConnectionRegistry improvements, we should also consider move the meta region location off zookeeper.
Bug: jenkins script for caching artifacts should verify cached file before relying on it
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Aug/21 20:53
Affects Version/s: 2.5.0

Nightly builds of branch-2 have been failing on the "Packaging and Integration" stage since 2021-07-22, commit 8bc180c7737bbd1792851004625c552896bf57c6.

The first failing build was https://ci-hadoop.apache.org/blue/organizations/jenkins/HBase%2FHBase%20Nightly/detail/branch-2/304/pipeline/12
Bug: The default value of 'hbase.hregion.memstore.mslab.indexchunksize.percent' should be 0
Status: Open | Priority: Major | Resolution: 
Created: 27/Jul/21 09:41
Affects Version/s: 2.4.0

The default value of {{hbase.hregion.memstore.mslab.indexchunksize.percent}} introduced by HBASE-24892 is 0.1, but we use {{DefaultMemStore}} by default , which has no IndexChunk and {{ChunkCreator.indexChunksPool}} is useless(IndexChunk is only used by {{CompactingMemStore}} and {{CellChunkMap}}, ), so the {{hbase.hregion.memstore.mslab.indexchunksize.percent}} should be  0 by default to save memory space. Only when we use {{CompactingMemStore}} and {{CellChunkMap}}, it is meaningful to set {{hbase.hregion.memstore.mslab.indexchunksize.percent}} by user. 
Howerver, because existing bug in {{ChunkCreator}}, it  depends on HBASE-26142
Bug: Limit count and size of L0 files compaction in StripeCompactionPolicy
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Aug/21 08:56

When selecting L0 files in the stripe store file manager to compact, they all will be selected. This is the key problem. No file count control and no compaction size control for L0 files compactions now. If the compaction size is large, e.g. some TBs, then the L0 compaction will need a lot of time to complete. 

Since L0 files not only contains the recently flushed files, bulk loaded files will also be put into L0. And what's more, when opening a daughter region, if the parent stripes can not be rebuild in the daughter, all the files will be put to L0.

So when there are large enough files in L0, there will exists a quite long compaction for all the L0 files. If the compaction speed less than the file flush speed to L0, larger compactions afterwards. This is a big problem especially in bulkloading files.
Bug: Region server and master could use itself as ConnectionRegistry
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Aug/21 12:43

As they already cached everything for connection registry in memory, the cluster connection can fetch the in memory data directly instead of go to zookeeper again.

This is an optimization to reduce the pressure on zookeeper.

For MasterRegistry, we do not want to use it as the ConnectionRegistry for our cluster connection because:

    // We use ZKConnectionRegistry for all the internal communication, primarily for these reasons:

    // - Decouples RS and master life cycles. RegionServers can continue be up independent of

    //   masters' availability.

    // - Configuration management for region servers (cluster internal) is much simpler when adding

    //   new masters or removing existing masters, since only clients' config needs to be updated.

    // - We need to retain ZKConnectionRegistry for replication use anyway, so we just extend it for

    //   other internal connections too.

The above comments are in our code, in the HRegionServer.cleanupConfiguration method.

But since now, masters and regionservers both implement the ClientMetaService interface, we are free to just let the ConnectionRegistry to make use of these in memory information directly, instead of going to zookeeper again.
Bug: BucketCache could not free BucketEntry which restored from persistence file
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Sep/21 10:57
Affects Version/s: 2.4.6

HBASE-21957 reserved a {{BucketEntry}} ctor which has no {{RefCnt#recycler}}, and when {{BucketCache}} is restored from a persistence file, it use this no  {{RefCnt#recycler}}, {{BucketEntry}} ctor. But from HBASE-21957, {{BucketEntry}} is freed through {{RefCnt#recycler}} , so this no  {{RefCnt#recycler}}  {{BucketEntry}} could not be freed even if corresponding  {{HFileBlock}} is evicted from {{BucketCache}}.
Bug: Hbase scan setMaxResultsPerColumnFamily not giving right results
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Sep/21 10:13
Affects Version/s: 2.1.9

scan.setMaxResultsPerColumnFamily(1) is giving less number of row keys
vs when it is not set in the scan, on a table that has multiple column families and multiple qualifier or multiple versions for each cell 

Docs for setMaxResultsPerColumnFamily:
{code:java}
/**
 * Set the maximum number of values to return per row per Column Family
 * @param limit the maximum number of values returned / row / CF
 */{code}
Bug: Shell 'load' command behavior has changed
Status: Open | Priority: Major | Resolution: 
Created: 08/Sep/21 23:20
Affects Version/s: 2.4.5

After upgrade fro 2.3.x to 2.4.x, we noticed that the behavior of the shell's {{load}} command has changed.

Given a file of commands, i.e.
{noformat}
$ echo 'list' > /tmp/file
{noformat}

2.3.x:

{noformat}
$ hbase shell
hbase(main):001:0> load '/tmp/file'
TABLE                                                                                                                                                                                                       
TestTable                                                                                                                                                                                                   
1 row(s)
Took 0.3076 seconds                                                                                                                                                                                         
=> true
hbase(main):002:0>
{noformat}

branch-2.4:

{noformat}
$ hbase shell
hbase:001:0> load '/tmp/file'

Traceback (most recent call last):
NameError (undefined local variable or method `list' for main:Object)
{noformat}
Bug: Balancer run is improperly triggered by accuracy error of double comparison
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Sep/21 19:09

{code:java}
protected synchronized boolean areSomeRegionReplicasColocated(Cluster c) {
  regionReplicaHostCostFunction.init(c);
  if (regionReplicaHostCostFunction.cost() > 0) {
    return true;
  }
  regionReplicaRackCostFunction.init(c);
  if (regionReplicaRackCostFunction.cost() > 0) {
    return true;
  }

{code}
The values are in double data type. Balancer could get stuck in constant runs and unnecessary moves.
{code:java}
2021-09-24 12:02:41,943 INFO org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Running balancer because at least one server hosts replicas of the same region.
2021-09-24 12:01:42,878 INFO org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished computing new moving plan. Computation took 2400001 ms to try 3048341 different iterations.  Found a solution that moves 81 regions; Going from a computed imbalance of 1.7429830473781883E-4 to a new imbalance of 1.6169961756947032E-4. 
{code}
 we should use COST_EPSILON instead of 0 for double comparison.
Bug: Backport backup/restore to branch-2
Status: Open | Priority: Major | Resolution: 
Created: 27/Sep/21 17:35

I was discussing this great feature with [~rda3mon] on Slack. His company is using this on their fork of hbase 2.1. We're working on upgrading to 2.4 now, and have our own home grown backup/restore system which is not as sophisticated as the native solution. If this solution was backported to branch-2, we would strongly consider adopting it as we finish up our upgrade.

It looks like this was originally cut from 2.0 due to release timeline pressures: https://issues.apache.org/jira/browse/HBASE-19407, and now suffers from a lack of community support. This might make sense since it only exists in 3.x, which is not yet released.

It would be great to backport this to branch-2 so that it reach a wider audience and adoption
Bug: Recovered ReplicationSource not terminated
Status: Open | Priority: Major | Resolution: 
Created: 29/Sep/21 12:18
Affects Version/s: 1.2.6

Recovered ReplicationSource not terminated after replicated all wal.

after all task done, it just remove source，but does not terminate it. so some resources(such as zk connection)  in replication endpoint leaks.
{code:java}
//代码占位符
if (replicationQueueInfo.isQueueRecovered()) {
  // use synchronize to make sure one last thread will clean the queue
  synchronized (workerThreads) {
    Threads.sleep(100);// wait a short while for other worker thread to fully exit
    boolean allOtherTaskDone = true;
    for (ReplicationSourceWorkerThread worker : workerThreads.values()) {
      if (!worker.equals(this) && worker.isAlive()) {
        allOtherTaskDone = false;
        break;
      }
    }
    if (allOtherTaskDone) {
      manager.closeRecoveredQueue(this.source);
      LOG.info("Finished recovering queue " + peerClusterZnode
          + " with the following stats: " + getStats());
    }
  }
}{code}
Bug: Add tracing support for HTable and sync connection on branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/21 03:59
Affects Version/s: 2.5.0

Split from HBASE-25853, the original change of HBASE-22120 does not include tracing support for sync client, if we decide to support it in branch-2, we will need to come up our own solution. 

this change should cover mainly the classes of HTable, ConnectionImplementation, and HRegionLocator

    * RPC and IPC tracing supported by HBASE-26125
    * WAL tracing supported by HBASE-26131
Bug: Improve data structure and algorithm for BalanceClusterState to improve computation speed for large cluster
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Aug/21 00:16

With ~800 node and ~500 regions per node on our large production cluster, balancer cannot complete within hours even after we just add 2% servers after maintenance. 

The unit tests with larger number of regions are taking longer and longer with changes to balancer with recent changes too, evident with the increment of the time limit recent PR's included.

It is time to replace some of the data structure for better time complexity including:

int[][] regionsPerServer; // serverIndex -> region list

int[][] regionsPerHost; // hostIndex -> list of regions
 int[][] regionsPerRack; // rackIndex -> region list

// serverIndex -> sorted list of regions by primary region index
 ArrayList<HashSet<Integer>> primariesOfRegionsPerServer;

// hostIndex -> sorted list of regions by primary region index
 int[][] primariesOfRegionsPerHost;

// rackIndex -> sorted list of regions by primary region index
 int[][] primariesOfRegionsPerRack;

Areas of algorithm improvement include:
 # O(n ) to O(1) time to  lookup or update per server/host/rack for every move test iteration.(n = number of regions per server/host/rack).
 # O(n ) to O(1) time for reserse lookup of region index from primary index.
 # Recomputation of primaryRegionCountSkewCostFunction reduced from O(n ) to O(1)
Bug: Improve Performance of HBCK when specifying a subset of tables
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Sep/21 23:08
Affects Version/s: hbase-operator-tools-1.1.0

On clusters with a large number of tables, listing all table directories from the filesystem can take a lot of time (especially on S3). We should avoid listing all the tables if we pass in a subset of the tables
Bug: Create an option to reintroduce BlockCache to mapreduce job
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Sep/21 18:49
Affects Version/s: 2.4.6

In HBASE-21498 (see [this commit|https://github.com/apache/hbase/commit/27a0f205c52f83fe7500ee2ffc6cf6582f565a63#diff-8a3e39e6df1afe47811fc17702da598fe0d80496d66e579bea4bd224c6d8da03R218], it change the behavior that only region server can initialize on-heap BlockCache/LruBlockCache, this should be the right change for HMaster. 

Other downstream dependency that uses getScanner from a file-based region and read HStore/HFile lost the BlockCache for caching INDEX/LEAF_INDEX (at least still a problem with HBase-2.4) after this change (it worked before) and caused performance impact with 2x slower.

One way to bring back the performance is to allow non-RS and non-HMaster can use a compact version of blockcache with smaller memory and less hbase internal configuration. 

Or if we can find a way to cache or skip reading the same {{LEAF_INDEX}} when scanning the DATA block with HFile.
Bug: Remove MetaTableLocator usages in non-migration code
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Sep/21 15:24

After HBASE-26193, we store the meta region location in master local region and zookeeper is just a mirror, but in our code base, sometimes we still get the location from zookeeper.

We should change them all to use the data from master local region, i.e, by reading from meta location cache or assignment manager.
Bug: Allow HashTable/SyncTable to perform rawScan when comparing cells
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Sep/21 10:37
Affects Version/s: 2.2.7

We had noticed some of our customers having inconsistent results when running HashTable/SyncTable in scenarios described previously by HBASE-21596. As discussed on HBASE-21596, one alternative to avoid such inconsistencies, is to set 'NEW_VERSION_BEHAVIOUR' introduced by HBASE-15968. That, however, does rely on the order of operations, which can't be guaranteed when standard replication (not serial replication) is being used, which is the case on the majority of our customers.

A possible workaround is to place delete markers to every single cell version returned on a raw scan, and setting a high value to the scan number of versions to be read.
Bug: Add an option to use region location from meta table in TableSnapshotInputFormat
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Sep/21 16:40
Affects Version/s: 2.3.6

TableSnapshotInputFormat currently calculates block locality of a region to decide the best location to run the task. While this works for a small scale table snapshot, we found that for a table snapshot with many regions, the locality calculation takes too much time. 

In the case of a table with high locality, we can use region location from meta table to decide a snapshot region's location. Add an option to support it.
Bug: Add support to run balancer overriding current config
Status: Open | Priority: Major | Resolution: 
Created: 05/Aug/21 19:41

At times we want a one-time run for balancer with specific aggressive configs such as recovering from a failure of half cluster. Currently some config are loaded dynamically but some don't. And it could be error prone when we try to restore the configs.

With a dry run feature coming, we want to let user run a dry run with the overriding config to choose the config and then run it. The implementation becomes more straightforward if we allow overidding configs for both config and actual runs.
Bug: Fix some obvious bugs in MultiByteBuff.put
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Aug/21 06:41
Affects Version/s: 2.4.5

MultiByteBuff.put(int destOffset, ByteBuff src, int srcOffset, int length) has some obvious bugs:
* It seems mix up {{items}} in {{src}} {{MutiByteBuff}} and {{items}} in the  {{dest}} {{MultiByteBuff}} , just as line 749 and line  754 illustrated.  The logic is only right when  src {{ByteBuff}} is also a {{MultiByteBuff}} and byte size of every {{ByteBuffer}} in {{src.items}} has exactly the same size as every {{ByteBuffer}} in the {{dest.items}},but looking the usage of this method in the hbase project, obviously the assumption is not right.
{code:java}
746     public MultiByteBuff put(int offset, ByteBuff src, int srcOffset, int length) {
747         checkRefCount();
748         int destItemIndex = getItemIndex(offset);
749         int srcItemIndex = getItemIndex(srcOffset);
750         ByteBuffer destItem = this.items[destItemIndex];
751         offset = offset - this.itemBeginPos[destItemIndex];
752
753        ByteBuffer srcItem = getItemByteBuffer(src, srcItemIndex);
754        srcOffset = srcOffset - this.itemBeginPos[srcItemIndex];
            ...
{code}
       


* If src is {{SingleByteBuff}} and its remaining space is fewer than length，when remaining space is exhausted, this {{MultiByteBuff.put}} method would not throw any exception and continue to put src {{ByteBuff}} once again from position 0 because following {{MultiByteBuff.getItemByteBuffer}} ignores index paramter for  {{SingleByteBuff}} . Obviously, this behavior is much strange and unexpected.
  {code:java}
       private static ByteBuffer getItemByteBuffer(ByteBuff buf, int index) {
             return (buf instanceof SingleByteBuff) ? buf.nioByteBuffers()[0]
                   : ((MultiByteBuff) buf).items[index];
        }
   {code} 

Why seems tests is OK with too much bugs? Because in normal cases, we just use {{SingleByteBuff}} not {{MultiByteBuff}}.
Bug: Backport HBASE-25811 for fixing nightly tests with error of `NoClassDefFoundError: io/opentelemetry/api/GlobalOpenTelemetry`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Aug/21 21:04
Affects Version/s: 2.5.0

see nightly error [https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/313/artifact/output-integration/hadoop-2/mr-importtsv.err/*view*/]
{code:java}
Caused by: java.lang.NoClassDefFoundError: io/opentelemetry/api/GlobalOpenTelemetry
 at org.apache.hadoop.hbase.trace.TraceUtil.getGlobalTracer(TraceUtil.java:33)
 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:398)
 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328)
 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:92)
 at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:589)
 at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:45761)
 at org.apache.hadoop.hbase.client.ClientServiceCallable.doGet(ClientServiceCallable.java:50)
 at org.apache.hadoop.hbase.client.HTable$1.rpcCall(HTable.java:377)
 at org.apache.hadoop.hbase.client.HTable$1.rpcCall(HTable.java:372)
 at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
 at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:108)
 ... 32 more
Caused by: java.lang.ClassNotFoundException: io.opentelemetry.api.GlobalOpenTelemetry
 at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
 ... 43 more
{code}
Bug: Backport HBASE-25778 "The tracinig implementation for AsyncConnectionImpl.getHbck is incorrect" to branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/21 03:40
Affects Version/s: 2.5.0

17/17 commits of HBASE-22120, original commit [{{f36e153}}|https://github.com/apache/hbase/commit/f36e1539648bbaee84c626fd54d1605baebf3c5a]
Bug: Backport HBASE-23762 "Add documentation on how to enable and view tracing with OpenTelemetry" to branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/21 03:39
Affects Version/s: 2.5.0

16/17 commits of HBASE-22120, original commit [{{be4503d}}|https://github.com/apache/hbase/commit/be4503d9f82f044fbfce21c8a42d0b1684607238]
Bug: Backport HBASE-25733 "Upgrade opentelemetry to 1.0.1" to branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/21 03:39
Affects Version/s: 2.5.0

15/17 commits of HBASE-22120, original commit [{{b714889}}|https://github.com/apache/hbase/commit/b71488998970a3353086a34736ed1edab527f673]
Bug: Backport HBASE-25732 "Change the command line argument for tracing after upgrading opentelemtry to 1.0.0" to branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/21 03:39
Affects Version/s: 2.5.0

14/17 commits of HBASE-22120, original commit [{{8df9beb}}|https://github.com/apache/hbase/commit/8df9bebdd367d52a32b08c18a7cf4f9c2d712071]
Bug: Backport HBASE-25723 "Temporarily remove the trace support for RegionScanner.next" to branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/21 03:37
Affects Version/s: 2.5.0

13/17 commits of HBASE-22120, original commit [{{7f90c22}}|https://github.com/apache/hbase/commit/7f90c2201f6a17d2e2d031505c35ae7c2b1ed7ea]
Bug: Backport HBASE-25616 "Upgrade opentelemetry to 1.0.0" to branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/21 03:37
Affects Version/s: 2.5.0

12/17 commits of HBASE-22120, original commits [{{8399293}}|https://github.com/apache/hbase/commit/8399293e21127df3ffdcb757242e4cb5964c7e99]
Bug: Backport HBASE-25617 "Revisit the span names" to branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/21 03:37
Affects Version/s: 2.5.0

_11/17 commits of HBASE-22120, original commit [{{8d68f8c}}|https://github.com/apache/hbase/commit/8d68f8cd1c8613be1b499eaa99f46806b2743294]_