av9ash/GitBugs huggingface dataset:
Bug: Add BucketAllocator bucket size statistic logging
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jul/22 14:29

BucketCache places blocks into configurably sized buckets based on the block size. The default sizes aim to provide broad coverage for common block sizes, but should probably be tuned for certain use-cases. However, we provide no way for operators to gain insight into the distribution of buckets.

There already exists a BucketAllocator#logStatistics method, but it is not called anywhere. I suggest that we hook that up in BucketCache#logStats (which is called periodically by a stats thread). We can go from there.

Looking at the IndexStatistics used in that method, it looks like a good start. One thing I'd like to add is a count of freeBuckets and completelyFreeBuckets per index. I think this will be useful for indicating how much more wiggle room we have for redistributing buckets among the various block sizes.
Bug: Add metrics for evaluating cost and effectiveness of bloom filters
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jul/22 01:05

Bloom filters can be costly for some tables, easily resulting in an aggregate memory footprint of many GBs. It's currently hard to monitor for that cost on a per-table basis. You can view {{staticBloomSize}} in JMX, but that is for the whole server. Otherwise you must manually sum the values using the regionserver UI.  We can add this (as well as staticIndexSize) to the per-table metrics.

Additionally, it can be hard to know how effective those bloom filters are. I think the easiest way to measure that is to count bloomFilterRequests and bloomFilterNegativeResults. With these metrics in hand, one can have an easier time deciding how much memory they want to give to their L1 cache and/or whether they want to disable blooms on a table.
Bug: Remove unnecessary usage of CachedBlocksByFile from RS UI
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jul/22 13:36

We have started deploying regionservers with large BucketCaches, with over 1M blocks in the cache. I noticed that our RS UI was loading pretty slowly, and I also noticed the following warning:
{quote}*The stats below are incomplete!* We ran into our accounting limit of 1000000 blocks. Up the configuration {_}hbase.ui.blockcache.by.file.max{_}.{quote}
Tracking that down, it seems like the original implementation back in 2014 for the BlockCache statistics used CachedBlocksByFile to calculate counts, sizes, etc. But over time this was improved to be backed by CacheStats, for example in HBASE-11573, HBASE-15635, HBASE-18532, etc.

At this point the cbsbf variable holding the CachedBlocksByFile object is not actually used by anything on the RS UI. Instead those per-file statistics have been moved to other endpoints, available via the "View block cache as JSON" and "as JSON by file" links.

Removing the creation of the cbsbf variable makes a dramatic improvement on RS UI load times for large caches. Prior to the change, it consistently took 4-5s to load the UI. Afterward it takes a fraction of a second.
Bug: Rs shut down because of DroppedSnapshotException when memstore flushing
Status: Open | Priority: Major | Resolution: 
Created: 22/Sep/22 06:20
Affects Version/s: 1.0.0

After memstore be flushed, flush marker will be writed to WAL.
But current wal being rolled now, the DroppedSnapshotException will happen.

      if (wal != null) {
        // write flush marker to WAL. If fail, we should throw DroppedSnapshotException
        FlushDescriptor desc = ProtobufUtil.toFlushDescriptor(FlushAction.COMMIT_FLUSH,
          getRegionInfo(), flushOpSeqId, committedFiles);
        WALUtil.writeFlushMarker(wal, this.htableDescriptor, getRegionInfo(),
          desc, sequenceId, true);
      }
    } catch (Throwable t) {
.......

      DroppedSnapshotException dse = new DroppedSnapshotException("region: " +
          Bytes.toStringBinary(getRegionName()));
      dse.initCause(t);
      status.abort("Flush failed: " + StringUtils.stringifyException(t));
      throw dse;
Bug: Unnecessary replication to secondary region replicas should avoid when WAL.sync throws Exception
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Aug/22 14:06
Affects Version/s: 3.0.0-alpha-4

As HBASE-27230 and HBASE-27223 said, if {{WAL.sync}} throws an exception, we should abort the region server to avoid the data inconsistent between the primary region and secondary region replicas, so we have no need to replicate to secondary region replicas when  {{WAL.sync}} throws an exception.
Bug: Add query rate metric for filtered reads
Status: Open | Priority: Major | Resolution: 
Created: 19/Sep/22 23:19

We have very useful metrics ServerReadQueryPerSecond and ServerWriteQueryPerSecond. The benefit of these over readRequestCount and writeRequestCount are that they are stable over the life of the regionserver. The readRequestCount and writeRequestCount can be heavily skewed when regions move around and the counters get decremented. We have filteredReadRequestCount which has the same issue. It'd be nice to have a stable rate for this metric, as I find that filtered reads are almost as costly as non-filtered.
Bug: Add metrics reporting on multi-cf reads
Status: Open | Priority: Major | Resolution: 
Created: 19/Sep/22 23:16

Data in column families are stored in separate files. When a read comes in to a table with multiple column families, by default it will fetch data from all column families unless the specific column/family was requested. These reads are more expensive, especially in multigets.

As an operator, it's hard to know when these are coming in. It would be useful to have a metric.
Bug: Add an option to reject multi-cf bulkloads
Status: Open | Priority: Major | Resolution: 
Created: 19/Sep/22 23:14

Doing a multi-cf bulkload requires taking a write lock on the region. When bulkloading many regions or doing many bulkloads to individual regions, this can cause a sort of denial of service for the region or entire table.

For those deployments which prefer to prioritize uptime, we should provide the ability to disallow these multi-cf bulkloads. It would be good to make this configurable via hbase-site.xml and overridden by TableDescriptor.
Bug: split or merge removed region under snapshot
Status: Open | Priority: Major | Resolution: 
Created: 12/Sep/22 22:48
Affects Version/s: 2.4.5

We run into snapshot failures for one table with large number of regions. The event sequence is like the following:

 
 # Snapshot process lists all regions for one table.
 # Normalize kicks in to split some regions for the table under snapshot.
 # split finishes and major compaction finishes. The parent region is moved to archive.
 # When the Snapshot processes the parent region, it does not exist and snapshot fails.

Since snapshot process acquires the table lock, but there is no table lock acquired in split or merge process, they crash into each other.
Bug: The trace related assertions are flaky for async client tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Sep/22 06:07

https://ci-hbase.apache.org/job/HBase-Flaky-Tests/job/master/4167/testReport/junit/org.apache.hadoop.hbase.client/TestAsyncTableScanner/testScanWrongColumnFamily_0__table_raw__scan_normal_/

The failure message is kinda unreadable... I guess the problem is we do not wait enough time as the completion of the span can be executed concurrently with normal scan operation.

And also I saw this in the test code

{code}
    // RawAsyncTableImpl never invokes the callback to `onScanMetricsCreated` -- bug?
{code}

This is not a bug as you need to manually enable scan metrics by calling Scan.setScanMetricsEnabled(true).

Let me also fix this.
Bug: Fix some tests hung by CompactSplit.requestCompactionInternal ignoring compactionsEnabled check
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Sep/22 12:10
Affects Version/s: 3.0.0-alpha-3

When compaction is disabled, {{CompactSplit.compactionsEnabled}} is set to false and {{CompactSplit.longCompactions}} and {{CompactSplit.shortCompactions}} is shutdown.
But for  {{CompactSplit.requestCompactionInternal}} , it only check {{CompactSplit.compactionsEnabled}} in {{CompactSplit.selectCompaction}}, when {{selectNow}} is false,  {{CompactSplit.selectCompaction}}  is not invoked and would submit the  compaction task to {{CompactSplit.shortCompactions}},after HBASE-27332, because {{CompactSplit.shortCompactions}} is shut down, {{RejectedExecutionException}} would throw.

For test such as {{TestRegionSplit.testSplitTableRegion}}, because compaction is disabled, this problem would make the new region after split could not open, so it make the test hang.
Bug: Enforcer phase EvaluateBeanShell fails occasionally
Status: Open | Priority: Major | Resolution: 
Created: 25/Aug/22 12:12

For example, from https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/412/General_20Nightly_20Build_20Report/

{noformat}
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ hbase-shaded-client-byo-hadoop ---
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.EvaluateBeanshell failed with message:
Couldn't evaluate condition: File license = new File("/home/jenkins/jenkins-home/workspace/HBase_Nightly_branch-2.4/component/hbase-shaded/hbase-shaded-client/target/maven-shared-archive-resources/META-INF/LICENSE");

                    // Beanshell does not support try-with-resources,
                    // so we must close this scanner manually
                    Scanner scanner = new Scanner(license);

                    while (scanner.hasNextLine()) {
                      if (scanner.nextLine().startsWith("ERROR:")) {
                        scanner.close();
                        return false;
                      }
                    }
                    scanner.close();
                    return true;
[INFO] No sources to compile
{noformat}
Bug: HBase shell hang for a minute when quiting
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Aug/22 14:47
Affects Version/s: 2.5.0

Found this when trying the 2.5.0 RC. The way to reproduce

Use './bin/hbase shell' to connect to a cluster, and then run these two commands

{code}
count "hbase:meta"
quit
{code}

We will hang about 1 minute before the final exit of the shell process.
Bug: Improvements of HMaster web ui
Status: Open | Priority: Major | Resolution: 
Created: 25/Aug/22 13:47
Affects Version/s: 2.4.13

The HMaster web ui open very slowly when there are many regions in the cluster, it may takes above 3min to load region and table messages， The RIT and table on home page is very slow, because we load all info on one time...

we need improve page loading speed of HMaster home page...
Bug: RoundRobinTableInputFormat main should extend `AbstractHBaseTool` for proper configuration parsing
Status: Open | Priority: Major | Resolution: 
Created: 29/Aug/22 12:42
Affects Version/s: 2.3.5

Noticed while reviewing the compatibility report for 2.5.0RC1. `RoundRobinTableInputFormat` ships with an undocumented main method that prints out input split diagnostic information. That main does some basic configuration parsing but it inconsistent with how parsing is normally done. If the tool is going to ship a main, it should do paring properly, make use of `AbstractHBaseTool`.
Bug: RSGroupMappingScript#getRSGroup has thread safety problem
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/22 09:34

We are using version 1.4.12 and met a problem in table creation phase some time. The master branch also has this problem. The error message is:

{code:java}
2022-07-26 19:26:20.122 [http-nio-8078-exec-24,d2ad4b13b542b6fb] ERROR HBaseServiceImpl - hbase create table: xxx:xxxx failed. (HBaseServiceImpl.java:116)
java.lang.RuntimeException: org.apache.hadoop.hbase.constraint.ConstraintException: org.apache.hadoop.hbase.constraint.ConstraintException: Default RSGroup (default
default) for this table's namespace does not exist.
{code}
The rsgroup here should be one 'default' but not two consecutive 'default'.  The code to get RSGroup from a mapping script is:

{code:java}
String getRSGroup(String namespace, String tablename) {
      if (rsgroupMappingScript == null) {
        return null;
      }
      String[] exec = rsgroupMappingScript.getExecString();
      exec[1] = namespace;
      exec[2] = tablename;
      try {
        rsgroupMappingScript.execute();
      } catch (IOException e) {
        // This exception may happen, like process doesn't have permission to run this script.
        LOG.error("{}, placing {} back to default rsgroup", e.getMessage(),
          TableName.valueOf(namespace, tablename));
        return RSGroupInfo.DEFAULT_GROUP;
      }
      return rsgroupMappingScript.getOutput().trim();
    }
{code}
here the rsgourpMappingScript could be executed by multi-threads.

To test it is a multi-thread issue, I ran a piece of code locally and found that the hadoop ShellCommandExecutor is not thread-safe (I run the code with hadoop 2.10.0 and 3.3.2). So that we should make this method synchronized. Besides, I found that this issue is retained in master branch also.

The test code is attached and my rsgroup mapping script is very simple:

{code:java}
#!/bin/bash

namespace=$1
tablename=$2
echo default
{code}
The reproduced screenshot is also attached.
Bug: Backport "HBASE-26810 Add dynamic configuration support for system coprocessors" to branch-2.5
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 12/Aug/22 04:27
Affects Version/s: 2.5.0

Backport HBASE-26810 dynamic configuration for system coprocessor to branch-2.5 , this have been done in master and branch-2, so let's see if we can have it in branch-2.5 before 2.5.0 or 2.5.1 release
Bug: Fix RC calculation in create-release
Status: Open | Priority: Major | Resolution: 
Created: 23/Aug/22 15:52
Affects Version/s: 3.0.0-alpha-4

Working on 2.5.0RC1, there exits a 2.5.0RC0 in the repo already. `create-release` is proposing 2.5.0RC2 as the tag.
Bug: Configuration to allow client fallback to non-SSL if handshake fails
Status: Open | Priority: Major | Resolution: 
Created: 23/Aug/22 13:32

[~zhangduo] brought up a great idea when reviewing the new native TLS docs. Since HMasters and RegionServers are both clients and servers, currently it requires multiple cluster restarts to transition an existing cluster to TLS:
{quote}1. Enable {{hbase.server.netty.tls.enabled}} only, with supportPlaintext=true. After restart, servers will _accept_ TLS, but not _send_ TLS. 

2. Additionally enable {{hbase.client.netty.tls.enabled}} on servers, keeping supportPlaintext=true. After restart, servers will now _accept and send_ TLS.

3. Remove supportPlaintext=true. After restart, servers will reject requests if not TLS.

Clients can be updated to use {{hbase.client.netty.tls.enabled}} either after step 1 or 2, but before 3.
{quote}
This could be simplified by allowing the client-side to also fallback to plaintext if handshake fails:
{quote}Theoretically at client side it is possible. In the NettyRpcConnection implementation, before the ssl handshake finishes, we will not send any data out. So if we want to fallback to non ssl communication, we could just remove the ssl handler and set up the connection without encryption. But probably at server side we will just close the connection...
{quote}
We'd need to test to see how the server-side handles this downgrade, per the last sentence above.
Bug: Add options to consider compressed size when delimiting blocks during hfile writes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Aug/22 12:13
Affects Version/s: 3.0.0-alpha-4

In HBASE-27232 we had modified "hbase.writer.unified.encoded.blocksize.ratio" property soo that it can allow for the encoded size to be considered when delimiting hfiles blocks during writes.

-Here we propose two additional properties,"hbase.block.size.limit.compressed" and  "hbase.block.size.max.compressed" that would allow for consider the compressed size (if compression is in use) for delimiting blocks during hfile writing. When compression is enabled, certain datasets can have very high compression efficiency, so that the default 64KB block size and 10GB max file size can lead to hfiles with very large number of blocks.- 

-In this proposal, "hbase.block.size.limit.compressed" is a boolean flag that switches to compressed size for delimiting blocks, and "hbase.block.size.max.compressed" is an int with the limit, in bytes for the compressed block size, in order to avoid very large uncompressed blocks (defaulting to 320KB).-

Note: As of 15/08/2022, the original proposal above has been modified to define a pluggable strategy for predicating block compression rate. Please refer to the release notes for more details.
Bug: Adding a trigger for Stochastica Balancer to safeguard for upper bound outliers.
Status: Open | Priority: Major | Resolution: 
Created: 14/Aug/22 01:02

In large clusters, if one outlier has a lot of regions, the calculated imbalance for  RegionCountSkewCostFunction is quite low and often fails to trigger the balancer.

For example, a node with twice average count on a 400-node cluster only produce an imbalance of 0.004 < 0.02(current default threshold to trigger balancer). An empty node also have similar effect but we have a safeguard in place. https://issues.apache.org/jira/browse/HBASE-24139

We can add a safeguard for this so  we don't have to lower threshold on larger clusters that makes the balancer more sensitive to other minor imbalances.
Bug: Fix build failure against Hadoop 3.3.4 due to added dependency on okhttp
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Aug/22 14:04

{noformat}
$ mvn clean install -DskipTests -Phadoop-3.0 -Dhadoop-three.version=3.3.4
...
[ERROR] Found artifact with unexpected contents: '/home/rocky/srcs/hbase/hbase-shaded/hbase-shaded-mapreduce/target/hbase-shaded-mapreduce-3.0.0-alpha-4-SNAPSHOT.jar'
    Please check the following and either correct the build or update
    the allowed list with reasoning.

    okhttp3/
    okhttp3/Request$Builder.class
    ...
    kotlin/
    kotlin/collections/
    kotlin/collections/ArraysUtilJVM.class
    ...
{noformat}
Bug: bin/hbase still use slf4j-log4j while reload4j in place
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/22 00:54
Affects Version/s: 2.4.13

For the bin/hbase file, there are some class path set which will handle the slf4j-log4j before, but now in hbase branch 2, we upgrade to reload4j, but the hbase file not upgrade, for example, if we run mapredcp:

hbase mapredcp:

/usr/lib/hbase/lib/shaded-clients/hbase-shaded-mapreduce-2.4.12.jar:/usr/lib/hbase/lib/client-facing-thirdparty/audience-annotations-0.5.0.jar:/usr/lib/hbase/lib/client-facing-thirdparty/commons-logging-1.2.jar:/usr/lib/hbase/lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar:/usr/lib/hbase/lib/client-facing-thirdparty/jcl-over-slf4j-1.7.33.jar:/usr/lib/hbase/lib/client-facing-thirdparty/jul-to-slf4j-1.7.33.jar:/usr/lib/hbase/lib/client-facing-thirdparty/reload4j-1.2.19.jar:/usr/lib/hbase/lib/client-facing-thirdparty/slf4j-api-1.7.33.jar:/usr/lib/hbase/lib/client-facing-thirdparty/slf4j-reload4j-1.7.33.jar


Among above, /usr/lib/hbase/lib/client-facing-thirdparty/slf4j-reload4j-1.7.33.jar should be excluded but as slf4j-log4j before

This will impact the downstream users
Bug: CME in AuthManager causes region server crash
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Aug/22 14:29

java.util.ConcurrentModificationException
    at java.util.HashMap$HashIterator.nextNode(HashMap.java:1445)
    at java.util.HashMap$KeyIterator.next(HashMap.java:1469)
    at org.apache.hadoop.hbase.security.access.AuthManager.authorizeNamespace(AuthManager.java:313)
    at org.apache.hadoop.hbase.security.access.AuthManager.authorizeUserNamespace(AuthManager.java:297)
    at org.apache.hadoop.hbase.security.access.AuthManager.authorizeUserTable(AuthManager.java:405)
    at org.apache.hadoop.hbase.security.access.AuthManager.authorizeUserTable(AuthManager.java:372)
    at org.apache.hadoop.hbase.security.access.AccessChecker.permissionGranted(AccessChecker.java:711)
    at org.apache.hadoop.hbase.security.access.AccessController.permissionGranted(AccessController.java:325)
    at org.apache.hadoop.hbase.security.access.AccessController.internalPreRead(AccessController.java:1337)
    at org.apache.hadoop.hbase.security.access.AccessController.preGetOp(AccessController.java:1422)
    at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:866)
    at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:863)
    at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:581)
    at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:655)
    at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preGet(RegionCoprocessorHost.java:863)
    at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2905)
    at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2844)
    at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42274)
    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:458)
    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
Bug: Should stop autoRead and skip all the bytes when rpc request too big
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Aug/22 06:09

We want to write RequestTooBigException back to client so we will not close the channel immediately after we foud a too big request, but then we will not skip all the bytes so it will casue the decode still running, and also client could still send more bytes which generate a lot of loads on the rpc server, before we finally close the channel.
Bug: The implementation of TestReplicationStatus.waitOnMetricsReport is incorrect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Aug/22 14:49

{code}
  private List<ReplicationLoadSource> waitOnMetricsReport(int greaterThan, ServerName serverName)
    throws IOException {
    ClusterMetrics metrics = hbaseAdmin.getClusterMetrics(EnumSet.of(Option.LIVE_SERVERS));
    List<ReplicationLoadSource> list =
      metrics.getLiveServerMetrics().get(serverName).getReplicationLoadSourceList();
    while (list.size() <= greaterThan) {
      Threads.sleep(1000);
    }
    return list;
  }
{code}

Obviously the while loop is just a dummy one as the list will not be updated in the loop...
Bug: Rewrite NettyRpcServer to decode rpc request with netty handler
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jul/22 14:20

This is a follow on issue of HBASE-26708.

The memory leaks happens in the saslReadAndProcess method. It does not follow a typical netty style so it is more likely to introduce bugs.

Since we agree that NettyRpcServer is the future and SimpleRpcServer should be deprecated and finally removed, let's first implement dedicated rpc request decoder for netty.
Bug: TestPerTableCFReplication.testParseTableCFsFromConfig is broken because of  ReplicationPeerConfigUtil.parseTableCFsFromConfig
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/22 12:00
Affects Version/s: 3.0.0-alpha-4

HBASE-27203 has modified {{ReplicationPeerConfigUtil.parseTableCFsFromConfig}} which causes 
{{TestPerTableCFReplication.testParseTableCFsFromConfig}} broken.