av9ash/GitBugs huggingface dataset:
Bug: Introduce a LogRollProcedure
Status: Open | Priority: Minor | Resolution: 
Created: 25/Apr/22 05:21

The current log-rolling for all regionservers is based in ZK. Here is an attempt to reimplement it with procedure v2.

Here are some requirements about the implementation.
The procedure can be introduced as a new feature. It should remain fully compatible with previous implementations. Also, this feature can be disabled by the configuration. Currently we only use the logroll procedure when taking a backup job, so I think all code logic should be implemented in the hbase-backup module as much as possible(I'm not sure if this is the right way to do it. If you have any suggestions, please let me know).

Here are some details about the implementation.
*LogRollProcedure*
The LogRollProcedure is used to roll WAL for all the regionservers in the cluster. It acquires the shared lock of the backup system table.
*RSLogRollProcedure*
The RSLogRollProcedure is used to schedule a RSLogRollRemoteProcedure for each regionserver. When the subprocedure returns, the RSLogRollProcedure will check the logrolling result in the backup system table. If failed, The RSLogRollProcedure will schedule a new RSLogRollRemoteProcedure to retry.
*RSLogRollRemoteProcedure*
The RSLogRollRemoteProcedure is used to send the log roll request to the remote server.

This is only the first version implementation, any suggestions and feedbacks are appreciated.
Bug: Speed up the loading of table descriptor from filesystem
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Jun/22 13:09
Affects Version/s: 3.0.0-alpha-3

If there are a large number of tables in the HBase cluster, it will take a long time to fully load the table descriptor  from filesystem for the first time.
In our production cluster, there were 50000 + tables. It took several minutes to load the table descriptor from filesystem. This problem seriously affects the performance of HMaster active/standby switchover.

We should support concurrent loading to solve this problem.
Bug: Brotli compression support
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Apr/22 23:27

I was about to document the compression improvements upcoming in 2.5.0 and realized we lacked support for one of the major compression formats in use today. (Arguably brotli is not often used to compress data at rest, but is widely supported among web browsers and HTTP servers and clients.) 

Brotli support for Java is provided by Apache 2 licensed [Brotli4j|https://github.com/hyperxpro/Brotli4j] available in Maven central.

It is not difficult to add it, so let's add it. There might be some advantage to opting for brotli compression when storing web data given the precomputed static dictionary of such content included in its specification.
Bug: Emit source metrics for BlockCacheExpressHitPercent
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Jun/22 20:36
Affects Version/s: 2.0.0

[https://github.com/apache/hbase/blob/d447fa01ba36a11d57927b78cce1bbca361b1d52/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java#L346-L400]
{code:java}
public double getHitCachingRatio() {
  double requestCachingCount = getRequestCachingCount();
  if (requestCachingCount == 0) {
    return 0;
  }
  return getHitCachingCount() / requestCachingCount;
} {code}
This code is responsible for the metric {{{}BlockCacheExpressHitPercent{}}}. The metric represents the percentage of requests which were cacheable, but not found in the cache. Unfortunately, since the counters are process-level counters, the ratio is for the lifetime of the process. This makes it less useful for looking at cache behavior during a smaller time period.

The underlying counters are {{hitCachingCount}} and {{{}missCachingCount{}}}. Having access to the underlying counters allows for offline computation of the same metric for any given time period. But these counters are not emitted today from {{{}MetricsRegionServerWrapperImpl.java{}}}.

Compare this to {{hitCount}} and {{missCount}} which are emitted as metrics {{blockCacheHitCount}} and {{{}blockCacheMissCount{}}}. But these are raw counts for the cache, which include requests that are not cacheable. The cacheable metrics are more interesting, since it can be common to miss on a request which is not cacheable.

Interestingly, these metrics are emitted regularly as part of a log line in {{{}StatisticsThread.logStats{}}}.

We should emit blockCache{{{}HitCachingCount{}}} and {{blockCacheMissCachingCount}} along with the current metrics.
Bug: CME in TestRegionNormalizerWorkQueue
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/May/22 05:21
Affects Version/s: 2.5.0

FAILURE! - in org.apache.hadoop.hbase.master.normalizer.TestRegionNormalizerWorkQueue

{noformat}
[ERROR] org.apache.hadoop.hbase.master.normalizer.TestRegionNormalizerWorkQueue.testTake  Time elapsed: 0.065 s  <<< ERROR!
java.util.concurrent.ExecutionException: java.util.ConcurrentModificationException
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
	at org.apache.hadoop.hbase.master.normalizer.TestRegionNormalizerWorkQueue.testTake(TestRegionNormalizerWorkQueue.java:211)
Caused by: java.util.ConcurrentModificationException
	at java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
	at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:742)
	at org.apache.hadoop.hbase.master.normalizer.RegionNormalizerWorkQueue.take(RegionNormalizerWorkQueue.java:192)
	at org.apache.hadoop.hbase.master.normalizer.TestRegionNormalizerWorkQueue.lambda$testTake$3(TestRegionNormalizerWorkQueue.java:192)
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
	at java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1632)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
{noformat}
Bug: TestProcedureSchedulerConcurrency.testConcurrentWaitWake is flaky
Status: Resolved | Priority: Minor | Resolution: Cannot Reproduce
Created: 29/May/22 19:35
Affects Version/s: 2.5.0

This one might be architecture specific. Pending repro on x86_64. 

org.apache.hadoop.hbase.procedure2.TestProcedureSchedulerConcurrency.testConcurrentWaitWake
  Run 1:
TestProcedureSchedulerConcurrency.testConcurrentWaitWake:60->testConcurrentWaitWake:145->Object.wait:-2 » Interrupted 
  Run 2: PASS
Bug: TestCoprocessorEndpointTracing.traceAsyncTableEndpoint is flaky
Status: Resolved | Priority: Minor | Resolution: Cannot Reproduce
Created: 29/May/22 19:31
Affects Version/s: 2.5.0

org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpointTracing.traceAsyncTableEndpoint 
  Run 1: TestCoprocessorEndpointTracing.traceAsyncTableEndpoint:210 
Expected: a collection containing (SpanKind with a name that a string containing "COPROC_EXEC" and SpanKind with a parentSpanId that "d2a7bb4f52ee70a2" and SpanData with StatusCode that is <OK>)
     but: SpanKind with a name that a string containing "COPROC_EXEC" name was "ZKConnectionRegistry.getMetaRegionLocations", SpanKind with a name that a string containing "COPROC_EXEC" name was "AsyncRegionLocator.getRegionLocation", SpanKind with a name that a string containing "COPROC_EXEC" name was "hbase.pb.RegionServerStatusService/RegionServerReport", SpanKind with a name that a string containing "COPROC_EXEC" name was "RpcServer.process", SpanKind with a name that a string containing "COPROC_EXEC" name was "hbase.pb.RegionServerStatusService/RegionServerReport", SpanKind with a name that a string containing "COPROC_EXEC" name was "Region.getScanner", SpanKind with a name that a string containing "COPROC_EXEC" name was "hbase.pb.ClientService/Scan", SpanKind with a name that a string containing "COPROC_EXEC" name was "RegionScanner.close", SpanKind with a name that a string containing "COPROC_EXEC" name was "RpcServer.process", SpanKind with a name that a string containing "COPROC_EXEC" name was "AsyncRegionLocator.getRegionLocation", SpanKind with a name that a string containing "COPROC_EXEC" name was "hbase.pb.ClientService/ExecService", SpanKind with a name that a string containing "COPROC_EXEC" name was "RpcServer.process", SpanKind with a name that a string containing "COPROC_EXEC" name was "AsyncRegionLocator.getRegionLocation", SpanKind with a name that a string containing "COPROC_EXEC" name was "SCAN hbase:meta", SpanKind with a name that a string containing "COPROC_EXEC" name was "hbase.pb.ClientService/Scan", SpanKind with a name that a string containing "COPROC_EXEC" name was "traceAsyncTableEndpoint" 
  Run 2: PASS
Bug: TestMemStoreLAB is flaky
Status: Resolved | Priority: Minor | Resolution: Cannot Reproduce
Created: 29/May/22 19:29
Affects Version/s: 2.5.0

org.apache.hadoop.hbase.regionserver.TestMemStoreLAB.testLABChunkQueue
  Run 1: TestMemStoreLAB.testLABChunkQueue » OutOfMemory GC overhead limit exceeded
  Run 2: TestMemStoreLAB.testLABChunkQueue » OutOfMemory GC overhead limit exceeded
  Run 3: PASS

org.apache.hadoop.hbase.regionserver.TestMemStoreLAB.testLABRandomAllocation
  Run 1: TestMemStoreLAB.testLABRandomAllocation » OutOfMemory GC overhead limit exceeded
  Run 2: TestMemStoreLAB.testLABRandomAllocation » OutOfMemory GC overhead limit exceeded
  Run 3: PASS

org.apache.hadoop.hbase.regionserver.TestMemStoreLAB.testLABThreading
  Run 1: TestMemStoreLAB.testLABThreading:166 » Runtime Deferred
  Run 2: TestMemStoreLAB.testLABThreading:166 » Runtime Deferred
  Run 3: PASS
Bug: Quotas causes too much load on meta for large clusters
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Apr/22 20:51

We've been upgrading larger clusters to hbase 2, and ran into this issue where two equivalent clusters (one on 1.2 and the other on 2.4.x) running almost identical workloads, but the hbase2 cluster was doing way more meta requests.

One of the reasons seems to be from https://issues.apache.org/jira/browse/HBASE-21820, which added an updateQuotaFactors method to the QuotaCache QuotaRefreshChore. This new method  calls MetaTableAcessor.scanMeta for every table in the cluster. This is happening on every regionserver, so if you have many tables and/or many regionservers you can easily start sending lots of traffic to meta. One way to offset this is to reduce frequency of hbase.quota.refresh.period, but this means you are less able to quickly respond to changes in load.

We should figure out a caching or notification mechanism that can reduce the need scan meta so much. It seems like we're mainly scanning meta to get a count of total regions per table, which should not be changing so often.
Bug: SnapshotInfo --snapshot param is marked as required even when trying to list all snapshots
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Apr/22 20:33
Affects Version/s: 2.4.11

SnapshotInfo –-list-snapshots lists all existing snapshots and doesn't need any filter, however, --snapshot param is marked as required, causing list to fail if this param is not defined. 

Also, the help description is a bit confusing about which options should be used together.
Bug: The batch size of cleaning expired mob files should have an upper bound
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Jun/22 09:14
Affects Version/s: 2.4.12

Currently the cleaning logic for expired mob files is adding all the deletable files in one directory to a list in memory and then archiving all the files in the list. But when there are millions of files need to delete, the list will be huge and make great heap memory pressure to the master.
Bug: Optimize debug output log of ConstantSizeRegionSplitPolicy class. 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Jun/22 00:48
Affects Version/s: 2.4.12

https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java#L103
https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java#L111

The debug log should be changed to "Should split because {} size={}, sizeToCheck={}{}".
Bug: Add index and bloom filter statistics of LruBlockCache on rs web UI
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Apr/22 12:26

When _CombinedBlockCache_ is configured, _LruBlockCache_ is used for index/bloom/meta block cache, and _BucketCache_ is used for data block cache. 

Index and bloom filter statistics on rs web UI will be helpful.
Bug: Splitting region may become RIT for 20 minutes when meta region is moved during split
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 15/Jun/22 07:38
Affects Version/s: 1.4.13

When meta region is moved during region split, the spitting region may become RIT(Region in Transition) for 20 minutes. And then the RegionServer has the RIT will fail. We are facing the issue during RegionServer rolling restart on HBase 1.4.13. I found that HBase 1.7.1 has the same issue, too. It is OK on HBase 2.4.12.

This is the fix.
Bug: TestUpdateRSGroupConfiguration.testCustomOnlineConfigChangeInRSGroup is flaky
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/May/22 19:37
Affects Version/s: 2.5.0

org.apache.hadoop.hbase.rsgroup.TestUpdateRSGroupConfiguration.testCustomOnlineConfigChangeInRSGroup
  Run 1: TestUpdateRSGroupConfiguration.testCustomOnlineConfigChangeInRSGroup:106 expected:<1000> but was:<0>
  Run 2: PASS
Bug: Encryption data contains checksum
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 07/Jun/22 07:13

in HFileBlockDefaultDecodingContext.prepareDecofing， we decoding data from onDiskBlock.

The onDiskBlock contains byte iv length, iv data and encrypted block data if cryptoContext is not null.

The encrypted block data contains two part, first part is the real encrypted block data, the second part is checksum.

when we the decrypt method(BlockIOUtils.readFullyWithHeapBuffer)，the checkSum will be put into the decryptionStream. if we use others padding mode which likes pkcs5padding, the decrytion will throw an error "Input Length must be a multiple of 16 when decrypting with a padded cipher", because of the checkSum is not a multiple of 16.

so we need to remove the checksum when we doing decrytion. 

why we will not get error when we use AES/CTR/NOPPDING, because CTR is not sensitive about the block size. but the other alg such as CBC or ECB, they are sensitive about block size.
Bug: IntegrationLoadTestCommonCrawl async load improvements
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Jun/22 22:41

ITLCC improvements:

- Use an async client and work stealing executor for parallelism during loads.

- Remove the verification read retries, these are not that effective during replication lag anyway.

- Increase max task attempts because S3 might throttle.

- Implement a side task that exercises Increments by extracting urls from content and updating a cf that tracks referrer counts. These are not validated at this time. It could be possible to log the increments, sum them with a reducer, and then verify the total, but this is left as a future exercise.
Bug: Improve TestMultiRespectsLimits
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Jun/22 18:04

TestMultiRespectLimits submits puts with empty values due to what appears to be a simple mistake. See https://github.com/apache/hbase/blob/c0e7e98ec71e77a9bfccf8f59e0c7c59aa70c828/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMultiRespectsLimits.java#L156

Refer to https://issues.apache.org/jira/browse/HBASE-26856?focusedCommentId=17558469&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17558469 for more context.
Bug: support commons-crypto version 1.1.0
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Jun/22 07:41

Now the version of Commons-crypto in hbase is 1.0.0,  at its version 1.1.0 , commons-rypto  did some modification in the code level to add the support for AARCH64 and compatible with openssl 1.1.x
Bug: Add namespace column family to UNDELETABLE_META_COLUMNFAMILIES
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Jun/22 12:50
Affects Version/s: 3.0.0-alpha-1

If we delete the namespace family from hbase:meta, clusters can also be problematic. So I think we should also add the namespace family to the family list which can not be deleted.
Bug: graceful_stop.sh should support specifing server port
Status: Open | Priority: Minor | Resolution: 
Created: 27/Jun/22 06:27

Right now the graceful_stop.sh script just support specify server hostname, while RS  are not necessarily deployed with default configuration,  the script is unavailable in this situation.

graceful_stop.sh is based on org.apache.hadoop.hbase.util.RegionMover, the code of RegionMover support specifing port.
Bug: Support unit test pattern matching again
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/May/22 00:34
Affects Version/s: 2.5.0

Sometimes we want to run a build with a subset of unit tests to be executed over multiple modules, like:

{{mvn clean install -DskipITs -Dtest=Test*Encode*}}

which will execute some tests in common, some in server, etc. but not in checkstyle, which fails

{noformat}
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M6:test
(default-test) on project hbase-checkstyle:
No tests matching pattern "Test*Encode*" were executed!
(Set -Dsurefire.failIfNoSpecifiedTests=false to ignore this error.) -> [Help 1]
[ERROR] 
{noformat}

We already set {{failIfNoTests}} to false in the surefire configuration. Apparently {{failIfNoSpecifiedTests}} is also necessary to support the above. When {{failIfNoSpecifiedTests}} is specified in the configuration or on the command line, surefire behaves as expected.

This behavior has changed in branch-2.5 and up. 

Open to suggestions for other solutions.
Bug: Region Visualizer is blank if master and regionserver are on the same host
Status: Open | Priority: Minor | Resolution: 
Created: 29/May/22 19:15
Affects Version/s: 2.5.0

This is similar to HBASE-27066 reported by [~tomscut] . HBASE-27066 was applied to branch-2.5 before this test.

I have a test configuration where all daemons are running on the same host: the NN, a DN, one ZK peer, a master, a regionserver, the test client (ltt). 

Go to [http://localhost:8110/master-status,] the UI is functioning normally except the Region Visualizer is empty, even though there is one table active. 

{noformat}
hbase:001:0> list_regions 'cluster_test'
                             SERVER_NAME |                                                           REGION_NAME |  START_KEY |    END_KEY |  SIZE |      REQ |   LOCALITY |
 --------------------------------------- | --------------------------------------------------------------------- | ---------- | ---------- | ----- | -------- | ---------- |
 buildbox.localdomain,8120,1653851244365 |         cluster_test,,1653804780151.bd73ddb4f821e5242da509d81f708a17. |            |   33333333 |  8258 |        0 |        1.0 |
 buildbox.localdomain,8120,1653851244365 | cluster_test,33333333,1653804780151.0c213c693af504e7da65e9b77e0e01dc. |   33333333 |   66666666 |  8258 |        0 |        1.0 |
 buildbox.localdomain,8120,1653851244365 | cluster_test,66666666,1653804780151.85cd0bd82ab83afe3effeb0f8615185a. |   66666666 |   99999999 |  8254 |        0 |        1.0 |
 buildbox.localdomain,8120,1653851244365 | cluster_test,99999999,1653804780151.6996cee2e337abdaa66bcd639aad2a52. |   99999999 |   cccccccc |  8254 |        0 |        1.0 |
 buildbox.localdomain,8120,1653851244365 | cluster_test,cccccccc,1653804780151.c84740760cf19c558353142cd26aae9b. |   cccccccc |            |  8259 |        0 |        1.0 |
 5 rows
Took 0.3243 seconds
{noformat}                                                                                                                                                                           


 !Screen Shot 2022-05-29 at 12.14.03 PM.png!
Bug: TestReplicationSource.testReplicationSourceInitializingMetric is flaky
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/May/22 01:05
Affects Version/s: 2.5.0

1 second appears to be not enough time for slow or loaded test hosts:
{noformat}
TestReplicationSource.testReplicationSourceInitializingMetric  Time elapsed: 1.021 s  <<< FAILURE!
java.lang.AssertionError: Waiting timed out after [1,000] msec
	at org.junit.Assert.fail(Assert.java:89)
	at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:203)
	at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:135)
	at org.apache.hadoop.hbase.replication.regionserver.TestReplicationSource.testReplicationSourceInitializingMetric(TestReplicationSource.java:576)
{noformat}
Bug: In the HFileBlock class, the log printing fspread/fsread cost time unit should be milliseconds
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Jun/22 03:33
Affects Version/s: 2.3.7

In the HFileBlock class

 
{code:java}
protected HFileBlock readBlockDataInternal(FSDataInputStream is, long offset, long onDiskSizeWithHeaderL, boolean pread, boolean verifyChecksum, boolean updateMetrics, boolean intoHeap) throws IOException {
  ...
  long startTime = EnvironmentEdgeManager.currentTime();
  ...
  long duration = EnvironmentEdgeManager.currentTime() - startTime;
  if (updateMetrics) {
    HFile.updateReadLatency(duration, pread);
  }
  ...
  LOG.trace("Read {} in {} ns", hFileBlock, duration);
  ...
}{code}
But Environment EdgeManager.currentTime returns milliseconds.
Bug: CellComparator should extend Serializable
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/May/22 18:05
Affects Version/s: 2.4.12

Prior to HBASE-18945, CellComparator was a class which implemented Serializable. In that jira, an interface was extracted with the intent of making the implementation IA.Private. This broke some downstream consumers, so the new CellComparator interface was made IA.Public in HBASE-19160.

One remaining breakage from that change is that if someone is using CellComparator in spark it will fails because it's not serializable. Everything in spark needs to be serializable, and the old CellComprator class used to extend Serializable. There was some small discussion in HBASE-18945 about removing Serializable, but I don't think anyone thought of this use-case.

The failure is:

Job aborted due to stage failure: Task not serializable: java.io.NotSerializableException: org.apache.hadoop.hbase.CellComparatorImpl

We should make CellComparator extend Serializable, or have CellComparatorImpl implement it.
Bug: Lower some DEBUG level logs in ReplicationSourceWALReader to TRACE
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 31/May/22 18:46
Affects Version/s: 2.5.0

See https://issues.apache.org/jira/browse/HBASE-21505?focusedCommentId=17544085&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17544085

{quote}
For ReplicationSourceWALReader#addEntryToBatch, empty Edit entry log was changed from TRACE/DEBUG as part of this Jira to improve the visibility apparently:
...
One of the clusters where I am running heavy workload, I keep seeing these logs with very high frequency
{quote}
Bug: The layout of the Web interface is broken
Status: Open | Priority: Minor | Resolution: 
Created: 26/May/22 01:36

I deployed a cluster with version 3.0.0-alpha-3-SNAPSHOT. The layout of the Web interface is broken.

Version 2.3.2:

!image-2022-05-26-09-36-20-478.png|width=895,height=101!

{color:#ff0000}Version 3.0.0-alpha-3-SNAPSHOT:{color}

!image-2022-05-26-09-35-42-909.png|width=870,height=92!