av9ash/GitBugs huggingface dataset:
Bug: NettyRpcServer may hange if it should skip initial sasl handshake
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jun/23 06:09
Affects Version/s: 2.6.0

{{NettyRpcServer}} may hange if  it should skip initial sasl handshake when server does not enable security  and client enables security,  I think this problem is caused by two reasons：
* For Server:
  The type of the response is  {{RpcResponse}}, but for {{NettyRpcServerPreambleHandler}},when it  send {{RpcResponse}} ,{{NettyRpcServerResponseEncoder}} does not exist, so {{RpcResponse}} messages cannot be sent.

* For Client
  When {{NettyHBaseSaslRpcClientHandler}} receives {{SaslUtil.SWITCH_TO_SIMPLE_AUTH}}, it does not remove {{SaslChallengeDecoder}} and {{NettyHBaseSaslRpcClientHandler}}, so the latter responses are considered to be incorrect.
Bug: Add client side table metrics for rpc calls and request latency.
Status: Open | Priority: Major | Resolution: 
Created: 10/May/23 03:17
Affects Version/s: 3.0.0-alpha-3

Now we can get client metrics to see number of calls or latency of related RPC calls like get, mutate, scan etc...

But more often, we are more concerned about table level metrics, so we need to subdivide them.
Bug: Add support for default user quotas
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Apr/23 20:10

If someone sets a quota with USER => 'all' (or maybe '*'), treat that as a default quota for each individual user. When a request comes from a user, it will lookup current QuotaState based on username. If one doesn't exist, it will be pre-filled with whatever the 'all' quota was set to. Otherwise, if you then define a quota for a specific user that will override whatever default you have set for that user only.
Bug: support quota user overrides
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Apr/23 15:27

The below is the original idea that started this work, but not what we actually landed on. See the first comment from [~rmdmattingly] and the release note for that.

 

Old description:
{quote}Currently we provide the ability to define quotas for namespaces, tables, or users. On multi-tenant clusters, users may be broken down into groups based on their use-case. For us this comes down to 2 main cases:
 # Hadoop jobs – it would be good to be able to limit all hadoop jobs in aggregate
 # Proxy APIs - this is common where upstream callers don't hit hbase directly, instead they go through one of many proxy api's.  For us we have a custom auth plugin which sets the username to the upstream caller name. But it would still be useful to be able to limit all usage from some particular proxy API in aggregate.

I think this could build upon the idea for Connection attributes in HBASE-27657. Basically when a Connection is established we can set an attribute (i.e. quotaGrouping=hadoop or quotaGrouping=MyProxyAPI).  In QuotaCache, we can add a {{getQuotaGroupLimiter(String groupName)}} and also allow someone to define quotas using {{set_quota TYPE => THROTTLE, GROUP => 'hadoop', LIMIT => '100M/sec'}}

I need to do more investigation into whether we'd want to return a simple group limiter (more similar to table/namespace handling) or treat it more like the USER limiters which returns a QuotaState (so you can limit by-group-by-table).

We need to consider how GROUP quotas interact with USER quotas. If a user has a quota defined, and that user is also part of a group with a quota defined, does the request need to honor both quotas? Maybe we provide a GROUP_BYPASS setting, similar to GLOBAL_BYPASS?
{quote}
Bug: Use ADMIN_QOS in MasterRpcServices for regionserver operational dependencies
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jun/23 22:50
Affects Version/s: 2.4.10

Analysis of a recent production incident is not yet complete but an item of note is an apparent deadlock. Imagine you are gracefully draining a regionserver by way of a flurry of moveRegion requests. The handler for moveRegion submits a TRSP and then waits on its future without timeout. Imagine that there are sufficient number of moveRegion requests to tie up the normal priority master RPC pool. Now imagine that all of those requests are waiting on TRSPs pending on a regionserver that is concurrently bounced or maybe it fails. The TRSPs are blocked in REGION_STATE_TRANSITION_CLOSE because the target regionserver terminated before responding to the close requests, blocking the moveRegion requests, blocking the RPC handlers. The regionserver restarts and tries to check in, but cannot report to the master because there are no free normal priority handlers to handle it. It seems not correct to have the regionserver operational dependencies (regionServerStartup, regionServerReport, and reportFatalRSError) contending with normal priority requests.

They should be made ADMIN_QOS priority to avoid this case.
Bug: Rework kuttl image as a yetus precommit base
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jun/23 15:03

Due to permissions issues on build workers (INFRA-24621), I think we'll benefit by shifting the perspective a little on how the kuttl image is used in CI. The current PRs establish the kuttl image as a (relatively) small, self-contained utility image. When it is invoked in CI, we run yetus with docker-in-docker support, and launch tests by calling `docker container run ... kuttl ...`. So, docker is invoked from within the Yetus pre-commit docker container. Rather, I want to implement the kuttl image as extending from the yetus image (or yetus-base, I'm not sure yet). That way we don't need to run pre-commit with docker-in-docker mode and the precommit process can invoke kuttl directly.

The existing kuttl image build is too sophisticated to be invoked by Yetus as a provided docker image (it makes use of build-args). It may also be too sophisticated to be run in a Jenkins worker (it makes use of buildx, INFRA-24704). So I guess we need to publish the image someplace and use the published tag via the Yetus `--docker-tag` flag.
Bug: Report heap used by BucketCache as a jmx metric
Status: Open | Priority: Major | Resolution: 
Created: 25/May/23 21:08

The BucketCache can take a non-trivial amount of heap, especially for very large cache sizes. For example, we have a server with 500k blocks in the bucket cache, and according to a heap dump it was holding around 260mb. One needs to account for this when determining the size of heap to use, so we should report it.

The major contributors I saw were the offsetLock, blocksByHFile set, and backingMap.
Bug: Fix build error against Hadoop 3.3.5
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/May/23 01:25

Build with hadoop 3.3.5 will fail with following messages.

Some packages are not included in shaded jar.
{code:java}
$ mvn clean install -DskipTests -Phadoop-3.0 -Dhadoop-three.version=3.3.5
...
[INFO] --- exec:1.6.0:exec (check-jar-contents-for-stuff-with-hadoop) @ hbase-shaded-with-hadoop-check-invariants ---
[ERROR] Found artifact with unexpected contents: '/home/yamasakisua/hbase/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.17.jar'
    Please check the following and either correct the build or update
    the allowed list with reasoning.    com/
    com/sun/
    com/sun/jersey/
    com/sun/jersey/json/ {code}
Bug: Prepare the slides for introducing new features in HBase on CommunityOverCode Asia 2023
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jun/23 02:53

Title:
What's new in the recent and upcoming HBase releases

Description:

Apache HBase™ is the Hadoop database, a distributed, scalable, big data store.
The HBase community is preparing new major release 3.0.0 and new minor release 2.6.0, with some brand new features. In this presentation, we will introduce these new features, about how they benefit our users and how we implement them in HBase. Like:
- OpenTelemetry Tracing: HBASE-22120, HBASE-26419
- The new region replication framework: HBASE-26233
- Move the logging framework from log4j to log4j2: HBASE-19577
- Cloud native supports
  - Introduce StoreFileTracker for better object storage support: HBASE-26067, HBASE-26584
  - K8s deployment support: HBASE-27827, in progress
  - No persistent data on zookeeper
    - Move meta location off zookeeper: HBASE-26193
    - Table based replication queue storage: HBASE-27109
    - File system based replication peer storage: HBASE-27110
  - redeploy cluster with only root directory on object storage: HBASE-26245
- Future?
  - HBase on Ozone
    - WAL on Ozone: HBASE-27740
  - Fully cloud native, no other self deployed services other than HBase itself
    - New WAL implementation: BookKeeper? Embedded WAL service?
Bug: Add client side table metrics for rpc calls and request latency.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/May/23 03:17
Affects Version/s: 3.0.0-alpha-3

Now we can get client metrics to see number of calls or latency of related RPC calls like get, mutate, scan etc...

But more often, we are more concerned about table level metrics, so we need to subdivide them.
Bug: Eliminate the 'WARNING: package jdk.internal.util.random not in java.base' when running UTs with jdk11
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/May/23 13:42

When fixing a jdk17 issue, we add a

--add-opens java.base/jdk.internal.util.random=ALL-UNNAMED

line in our pom when running UTs with jdk11+, so it will also be applied for jdk11, but we only need this for jdk17, on jdk11, it will generate a warning, as described in the title.

We should try some maven magics to fix this.
Bug: Introduce kubernetes deployment
Status: In Progress | Priority: Major | Resolution: 
Created: 04/May/23 11:46

As per the [discussion|https://lists.apache.org/thread/fgxyk4y32xnhzr5prdmhfjkfpk15g5jx] on the dev list, introduce a basic harness for deploying ZooKeeper, HDFS, and HBase on Kubernetes.
Bug: Skip zero length bytes of WALKey and WALEntry cells to avoid replication stuck with WAL compression
Status: Open | Priority: Major | Resolution: 
Created: 14/Jun/23 05:19
Affects Version/s: 2.4.17

We found both the two replication stuck circumstances on our production production clusters, which enables 'hbase.regionserver.wal.enablecompression' and 'hbase.regionserver.wal.value.compression.type'. 

The zero length of WALKey and WALEntry cells should be skipped.
Bug: TimeoutIOException: Failed to get sync result after 300000 ms for txid=16920651960, WAL system stuck?
Status: Open | Priority: Major | Resolution: 
Created: 09/May/23 06:44
Affects Version/s: 2.2.6

A node under a RsGroup (only one table), at a certain moment, the write call queue is blocked, and the blocking time starts, and the reading and writing qps of this table are all reduced to 0, and the client cannot read and write the table, RS call At the point in time when queue blocking starts, the following errors are continuously reported in the log:
 
2023-05-08 12:42:27,310 ERROR [MemStoreFlusher.2] regionserver.MemStoreFlusher: Cache flush failed for region user_feature_v2,eacf_1658057555,1660314723816.2376cc2326b5372131cc530b115d959a.
org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=16920651960, WAL system stuck?
        at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:155)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:743)
        at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:625)
        at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:602)
        at org.apache.hadoop.hbase.regionserver.HRegion.doSyncOfUnflushedWALChanges(HRegion.java:2754)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2691)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2549)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2523)
        at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2409)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:611)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:580)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$1000(MemStoreFlusher.java:68)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:360)
        at java.lang.Thread.run(Thread.java:748)
The data in the node memstore cannot be flushed to the WAL file, other indicators of the node are normal, and HDFS is not under pressure. After restarting the blocked node, the table returned to normal.
Bug: Disable hdfs readahead for pread reads
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/May/23 21:04

In https://issues.apache.org/jira/browse/HBASE-17914, a flag was introduced {{{}hbase.store.reader.no-readahead{}}}. The default is false, so readahead is enabled. This flag is used for creating the default store reader (i.e. the one used by PREAD reads). Stream readers don't use this flag, instead they always pass -1.

When that flag is true, we pass a readahead value of 0 to FSDataInputStream.setReadahead. When the flag is false, we pass -1 which triggers hdfs default behavior. The default behavior is to use a readahead of 4MB.

It seems to me that we don't want readahead for PREAD reads, and especially not such a large readahead. Our default block size is 64kb, which is much smaller than that. A PREAD read is not likely to do sequential IO, so not likely to utilize the cached readahead buffer.

I set no-readahead to true in a few of our clusters and in each case saw a massive reduction in disk IO and thus increase in throughput. I load tested this in a test cluster which does fully random reads of ~300 byte rows on a dataset which is 20x larger than memory. The load test was able to achieve nearly double the throughput.

As a follow-on, we might consider tuning the readahead for STREAM reads. 4mb seems way too big for many common workloads.
Bug: Interrupt BucketCachePersister thread when BucketCache is shutdown
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/May/23 19:51
Affects Version/s: 2.6.0

Recent changes with BucketCachePersister caused this unit test  [TestVerifyBucketCacheFile.testRetrieveFromFile()|https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5222/3/testReport/org.apache.hadoop.hbase.io.hfile.bucket/TestVerifyBucketCacheFile/precommit_checks___yetus_jdk8_Hadoop2_checks___testRetrieveFromFile_1__blockSize_16_384__bucketSizes__I_1151e434_/] to become flaky.

The BucketCachePersister thread keeps running even after BucketCache.shutdown() is run and we do not clear the contents of the backingMap during the shutdown if cache persistence is enabled causing this test to fail. BucketCachePersister should be interrupted in BucketCache.shutdown().
Bug: A random data generator tool leveraging bulk load.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/23 16:50

As of now, there is no data generator tool in HBase leveraging bulk load. Since bulk load skips client writes path, it's much faster to generate data and use of for load/performance tests where client writes are not a mandate.
{*}Example{*}: Any tooling over HBase that need x TBs of HBase Table for load testing.

{*}Requirements{*}:
1. Tooling should generate RANDOM data on the fly and should not require any pre-generated data as CSV/XML files as input.
2. Tooling should support pre-splited tables (number of splits to be taken as input).
3. Data should be UNIFORMLY distributed across all regions of the table.

*High-level Steps*
1. A table will be created (pre-splited with number of splits as input)
2. The mapper of a custom Map Reduce job will generate random key-value pair and ensure that those are equally distributed across all regions of the table.
3. [HFileOutputFormat2|https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java] will be used to add reducer to the MR job and create HFiles based on key value pairs generated by mapper. 
4. Bulk load those HFiles to the respective regions of the table using [LoadIncrementalFiles|https://hbase.apache.org/2.2/devapidocs/org/apache/hadoop/hbase/tool/LoadIncrementalHFiles.html]

*Results*
We had POC for this tool in our organization, tested this tool with a 11 nodes HBase cluster (having HBase + Hadoop services running). The tool generated:
1. *100* *GB* of data in *6 minutes*
2. *340 GB* of data in *13 minutes*
3. *3.5 TB* of data in *3 hours and 10 minutes*

*Usage*
hbase org.apache.hadoop.hbase.util.bulkdatagenerator.BulkDataGeneratorTool -mapper-count 100 -table TEST_TABLE_1 -rows-per-mapper 1000000 -split-count 100 -delete-if-exist -table-options "NORMALIZATION_ENABLED=false"
Bug: The chunk created by mslab may cause memory fragement and lead to fullgc
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Apr/23 02:38

The default size of chunk is 2m, when we use G1, if heapRegionSize equals 4m, these chunks are allocated as humongous objects, exclusively allocating one region, then the remaining 2m become memory fragement.

Lots of memory fragement may lead to fullgc even if the percent of used heap not high enough.

I have tested to reduce the chunk size to 2047k(2m-1k, a bit lesser than half of heapRegionSize), there was no repeat of the above.

BTW, in G1, humongous objects are objects larger or equal the size of half a region, and the heapRegionSize is automatically calculated based on the heap size parameter if not explicitly specified.
Bug: ConnectionImplementation#locateRegionInMeta should pause and retry when taking user region lock failed
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/May/23 01:45
Affects Version/s: 2.4.17

It just throws exception and skips the pause and retry logic when 
ConnectionImplementation#takeUserRegionLock fails. In some circumstances, no pause and retry by outer logic will make next ConnectionImplementation#takeUserRegionLock still fails, since all the threads simultaneously grab the lock.
Bug: Expose a getter on Connection/AsyncConnection for getting public access to connection metrics
Status: Open | Priority: Major | Resolution: 
Created: 25/May/23 13:32

Currently the MetricsConnection metrics are only exposed via jmx. This is not very easy to deal with on the client side. In our environment we have 10s of thousands of client instances and we do not expose a jmx port. If we did, it'd be painful to try fetching from so many (and constantly changing) instances. 

Many environments have their own monitoring stack and their own way of shipping deployables metrics to their monitoring. We should provide a way to get access to these metrics within the process. It's possible (but extremely cumbersome) to query jmx beans from within the process. We did this in my environment and I would recommend against it. Let's provide direct access to the metrics.

I think we probably want to keep MetricsConnection IA.Private, but maybe we can create a new MetricsConnectionSnapshot and provide a getter on the Connection/AsyncConnection.
Bug: HMaster.getCompactionState can happen NPE when region state is closed
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/May/23 15:11
Affects Version/s: 2.3.7

Following steps to reproduce:

1 create table 
{code:java}
create 'hbase_region_test', 'info', SPLITS => ['3', '7']  {code}
 

2 write data
{code:java}
//代码占位符
put 'hbase_region_test', '10010', 'info:name', 'Tom'
put 'hbase_region_test', '20010', 'info:name', 'Tom'
put 'hbase_region_test', '30010', 'info:name', 'Tom'
put 'hbase_region_test', '40010', 'info:name', 'Tom'
put 'hbase_region_test', '50010', 'info:name', 'Tom'
put 'hbase_region_test', '60010', 'info:name', 'Tom'
put 'hbase_region_test', '70010', 'info:name', 'Tom'
put 'hbase_region_test', '80010', 'info:name', 'Tom'
put 'hbase_region_test', '90010', 'info:name', 'Tom' {code}
 

3 closed a region of hbase_region_test by unassign 

 

4 calling method HMaster.getCompactionState:    

At this step, we can trigger this method to be called by opening hbase UI page about 'hbase_region_test' table detailes and getting the compaction state of 'hbase_region_test' samply

 

5 HMaster print NPE logs about HMaster.getCompactionState
Bug: Skipping compact for this region if the table disable compaction
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jun/23 15:41
Affects Version/s: 2.3.7

CompactionChecker.chore() is called periodically for compacting. as follow:
{code:java}
//代码占位符
protected void chore() {
    for (Region r : this.instance.onlineRegions.values()) {
        // Skip compaction if region is read only
        if (r == null || r.isReadOnly()) {
            continue;
        }


        HRegion hr = (HRegion) r;
        for (HStore s : hr.stores.values()) {
            // ...
        }
    }
} {code}
Just skip compact for the region if table disable compaction.
Bug: Eliminate duplicate code for  getNonRootIndexedKey in HFileBlockIndex
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jun/23 03:59
Affects Version/s: 3.0.0-alpha-4

The code to get the {{midkey}} by {{midLeafBlock}} in  {{HFileBlockIndex.CellBasedKeyBlockIndexReader.midkey}} method is almost the same as  {{HFileBlockIndex.BlockIndexReader.getNonRootIndexedKey}}, the code is somewhat complicated, we could use the {{HFileBlockIndex.BlockIndexReader.getNonRootIndexedKey}} directly in  {{HFileBlockIndex.CellBasedKeyBlockIndexReader.midkey}}.
Bug: Asyncfs may print too many WARN logs when replace writer
Status: Open | Priority: Major | Resolution: 
Created: 18/May/23 07:55

When we use Asyncfs for WAL, could see many WARN logs like this:

!image-2023-05-18-15-51-33-823.png!

namenode#complete is a relatively heavy operation, and sometimes it takes seconds.

So we can reduce this log, just as what [DFSOutputStream#completeFile|https://github.com/apache/hadoop/blob/706d88266abcee09ed78fbaa0ad5f74d818ab0e9/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java#L975]] do.
Bug: Report memstore on-heap and off-heap size as jmx metrics in sub=Memory bean
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jun/23 23:03

Currently we only report "memStoreSize" jmx metric in sub=Memory bean. There are "Memstore On-Heap Size" and "Memsotre Off-Heap Size" in the RS UI. It would be useful to report them in JMX.

In addition, "memStoreSize" metric under sub=Memory is 0 for some reason (while that under sub=Server is not). Need to do some digging to see if it is a bug.