av9ash/GitBugs huggingface dataset:
Bug: HDFS erasure coding support for table data dirs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Nov/23 15:02

[Erasure coding|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html] (EC) is a hadoop-3 feature which can drastically reduce storage requirements, at the expense of locality. At my company we have a few hbase clusters which are extremely data dense and take mostly write traffic, fewer reads (cold data). We'd like to reduce the cost of these clusters, and EC is a great way to do that since it can reduce replication related storage costs by 50%.

It's possible to enable EC policies on sub directories of HDFS. One can manually set this with {{{}hdfs ec -setPolicy -path /hbase/data/default/usertable -policy xxxx{}}}. This can work without any hbase support.

One problem with that is a lack of visibility by operators into which tables might have EC enabled. I think this is where HBase can help. Here's my proposal:
 * Add a new TableDescriptor and ColumnDescriptor field ERASURE_CODING_POLICY
 * In ModifyTableProcedure preflightChecks, if ERASURE_CODING_POLICY is set, verify that the requested policy is available and enabled via DistributedFileSystem.
getErasureCodingPolicies().
 * During ModifyTableProcedure, add a new state for MODIFY_TABLE_SYNC_ERASURE_CODING_POLICY.
 ** When adding or changing a policy, use DistributedFileSystem.
setErasureCodingPolicy to sync it for the data and archive dir of that table (or column in table)
 ** When removing the property or setting it to empty, use DistributedFileSystem.
unsetErasureCodingPolicy to remove it from the data and archive dir.

Since this new API is in hadoop-3 only, we'll need to add a reflection wrapper class for managing the calls and verifying that the API is available. We'll similarly do that API check in preflightChecks.
Bug: Upgrade zookeeper to a newer version
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Oct/23 13:52

See this discussion thread:

https://lists.apache.org/thread/gl3lx2btg59ysocsspn8pjq77w0sl3op

I prefer we upgrade zookeeper version to 3.8.x for HBase 2.6+.
Bug: Release 3.0.0-beta-1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Dec/23 08:07

We have passed a 10B ITBLL run after HBASE-28199 gets in, so I think it is time to cut the 3.0.0-beta-1 now.

There are still some improvments such as HBASE-28240, which will target to 3.0.0-beta-2.
Bug: hbck -o should not allow bypassing pre transit check by default
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Oct/23 01:36
Affects Version/s: 2.4.17

When operator uses hbck assigns or unassigns with "-o", the override will also skip pre transit checks. While this is one of the intentions with "-o", the primary purpose should still be to only unattach existing procedure from RegionStateNode so that newly scheduled assign proc can take exclusive region level lock.

We should restrict bypassing preTransitCheck by only providing it as site config.

If bypassing preTransitCheck is configured, only then any hbck "-o" should be allowed to bypass this check, otherwise by default they should go through the check.

 

It is important to keep "unset of the procedure from RegionStateNode" and "bypassing preTransitCheck" separate so that when the cluster state is bad, we don't explicitly deteriorate it further e.g. if a region was successfully split and now if operator performs "hbck assigns \{region} -o" and if it bypasses the transit check, master would bring the region online and it could compact store files and archive the store file which is referenced by daughter region. This would not allow daughter region to come online.

Let's introduce hbase site config to allow bypassing preTransitCheck, it should not be doable only by operator using hbck alone.

 

"-o" should mean "override" the procedure that is attached to the RegionStateNode, it should not mean forcefully skip any region transition validation checks.
Bug: Bump jruby to 9.4.8.0 to fix snakeyaml CVE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Dec/23 09:43

As a follow up of HBASE-28249, we want to bump to latest 9.4.x line here. 

This release line drops critical snakeyaml CVE ({*}org.yaml : snakeyaml : 1.33{*} having [CVE-2022-1471|https://nvd.nist.gov/vuln/detail/CVE-2022-1471]) from our classpath with following change along with several other bugs/fixes: 
 * The Psych YAML library is updated to 5.1.0. This version switches the JRuby extension to SnakeYAML Engine, avoiding CVEs against the original SnakeYAML and updating YAML compatibility to specification version 1.2. [#6365|https://github.com/jruby/jruby/issues/6365], [#7570|https://github.com/jruby/jruby/issues/7570], [#7626|https://github.com/jruby/jruby/pull/7626]

NOTE: JRuby 9.4.x targets Ruby 3.1 compatibility instead of Ruby 2.6 which 9.3.x were having!
Bug: Decouple RIT list management from TRSP invocation
Status: Open | Priority: Major | Resolution: 
Created: 17/Oct/23 01:52
Affects Version/s: 2.5.6

Operators bypassed some in progress TRSPs leading to a state where some regions were persistently in transition but hidden. Because the master builds its list of regions in transition by tracking TRSP, the bypass of TRSP removed the regions from the RIT list. 

Although I can see from reading the code this is the expected behavior, it is surprising for operators and should be changed. Operators expect that regions that should be open but are not appear the master's RIT list, provided by /rits.jsp, the output of the shell's 'rit' command, and in ClusterStatus.

We should only remove a region from the RIT map when assignment reaches a suitable terminal state.
Bug: Introduce table metric for blockedRequestCount
Status: Open | Priority: Major | Resolution: 
Created: 27/Nov/23 23:04
Affects Version/s: 2.4.17

If compaction is disabled temporarily to allow stabilizing hdfs load, we can forget re-enabling the compaction. This can result into flushes getting delayed for "hbase.hstore.blockingWaitTime" time (90s). While flushes do happen eventually after waiting for max blocking time, it is important to realize that any cluster cannot function well with compaction disabled for significant amount of time.

 

We would also block any write requests until region is flushed (90+ sec, by default):
{code:java}
2023-11-27 20:40:52,124 WARN  [,queue=18,port=60020] regionserver.HRegion - Region is too busy due to exceeding memstore size limit.
org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, regionName=table1,1699923733811.4fd5e52e2133df1e347f32c646f23ab4., server=server-1,60020,1699421714454, memstoreSize=1073820928, blockingMemStoreSize=1073741824
    at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4200)
    at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3264)
    at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3215)
    at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:967)
    at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:895)
    at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2524)
    at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36812)
    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2432)
    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:311)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:291) {code}
 

Delayed flush logs:
{code:java}
LOG.warn("{} has too many store files({}); delaying flush up to {} ms",
  region.getRegionInfo().getEncodedName(), getStoreFileCount(region),
  this.blockingWaitTime); {code}
Suggestion: Introduce regionserver metric (MetricsRegionServerSource) for the num of flushes getting delayed due to too many store files.
Bug: Master should recover if meta region state is inconsistent
Status: Open | Priority: Major | Resolution: 
Created: 09/Nov/23 19:38
Affects Version/s: 2.4.17

During active master initialization, before we set master as active (i.e. {_}setInitialized(true){_}), we need both meta and namespace regions online. If the region state of meta or namespace is inconsistent, active master can get stuck in the initialization step:
{code:java}
private boolean isRegionOnline(RegionInfo ri) {
  RetryCounter rc = null;
  while (!isStopped()) {
...
...
...
    // Check once-a-minute.
    if (rc == null) {
      rc = new RetryCounterFactory(Integer.MAX_VALUE, 1000, 60_000).create();
    }
    Threads.sleep(rc.getBackoffTimeAndIncrementAttempts());
  }
  return false;
}
 {code}
In one of the recent outage, we observed that meta was online on a server, which was correctly reflected in meta znode, but the server starttime was different. This means that as per the latest transition record, meta was marked online on old server (same server with old start time). This kept active master initialization waiting forever and some SCPs got stuck in initial stage where they need to access meta table before getting candidate for region moves.

The only way out of this outage is for operator to schedule recoveries using hbck for old server, which triggers SCP for old server address of meta. Since many SCPs were stuck, the processing of new SCP too was taking some time and manual restart of active master triggered failover, and new master was able to complete SCP for old meta server, correcting the meta assignment details, which eventually marked master as active and only after this, we were able to see real large num of RITs that were hidden so far.

We need to let master recover from this state to avoid manual intervention.
Bug: Fix the miss count in one of CombinedBlockCache getBlock implementations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Nov/23 21:28
Affects Version/s: 2.4.17

In one of the CombinedBlockCache.getBlock(getBlock(cacheKey,cachingrepeat,updateCacheMetrics) we always compute a miss in L1 if the passed block is of type DATA. We should compute the miss in one of the caches only, not both.
Bug: Expose region cached size over JMX metrics and report in the RS UI
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Dec/23 23:03
Affects Version/s: 2.4.17

With large file based bucket cache, the prefetch executor can take long time to complete cache all of the dataset. It would be useful to report how much % of regions data is already cached, in order to give an idea of how much work prefetch executor has done.

This PRs adds jmx metrics for region cache % and also reports the same in the RS UI "Store File Metrics" tab as below:

!Screenshot 2023-12-06 at 22.58.17.png|width=658,height=114!
Bug: Split the  queueCallTime, processCallTime and totalCallTime according to queue type when using RWQueueRpcExecutor
Status: Open | Priority: Major | Resolution: 
Created: 26/Nov/23 16:16
Affects Version/s: 2.5.0

When using RWQueueRpcExecutor, in order to monitor the running status of the read and write queues separately, it is necessary to monitor the queueTime and processingTime of the request according to the queue type. In this way, the queue's queueCallTime, processCallTime and totalCallTime  metrics are split according to the read and write queue types. So Add the following metrics：
 # queueReadCallTime
 # processReadCallTime
 # totalReadCallTime
 # queueWriteCallTime
 # processWriteCallTime
 # totalWriteCallTime
 # queueScanCallTime
 # processScanCallTime
 # totalScanCallTime

 

These metrics are important references for subsequent modification of parameters hbase.ipc.server.callqueue.read.ratio and hbase.ipc.server.callqueue.scan.ratio.
Bug: Auto create configured namespaces
Status: Open | Priority: Major | Resolution: 
Created: 04/Dec/23 15:11

During startup, the HMaster will create the default and system namespaces automatically. To simplify the management of common namespaces, it would be beneficial to offer a configuration option that operators can use to ensure that additional namespaces are created during startup. This would eliminate the need to wrap createTable calls in checkAndCreateNamespace or provide separate cluster bootstrap functionality to guarantee that the namespace is created.
Bug: Region reopen procedure should support some sort of throttling
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Nov/23 14:48

The mass reopening of regions caused by a table descriptor modification can be quite disruptive. For latency/error sensitive workloads, like our user facing traffic, we need to be very careful about when we modify table descriptors, and it can be virtually impossible to do it painlessly for busy tables.

It would be nice if we supported configurable batching/throttling of reopenings so that the amplitude of any disruption can be kept relatively small.
Bug: It's impossible to re-enable the quota table if it gets disabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Nov/23 20:11

HMaster.enableTable tries to read the quota table. If you disable the quota table, this fails. So then it's impossible to re-enable it. The only solution I can find is to delete the table at this point, so that it gets recreated at startup, but this results in losing any quotas you had defined.  We should fix enableTable to not check quotas if the table in question is hbase:quota.
Bug: Tables to which Stripe Compaction policy is applied cannot be forced to trigger Major Compaction.
Status: Open | Priority: Major | Resolution: 
Created: 29/Nov/23 06:37
Affects Version/s: 2.2.6

    There is a table and the Stripe Compaction strategy is applied. Each region has an average value of 40G and is divided into 8 Stripes. Each Stripe is 5G. The business deletes a large amount of data. Manually triggering major compaction on the entire table and a single region does not work and cannot be selected.

    After reading the source code, the merging strategy applied under each Stripe is ExploringCompactionPolicy. This strategy has a key point. It filters the Store file list of a single Stripe. In the candidate file list, as long as there is a file that is too large in size and meets the condition, fileSize > (totalFileSize - fileSize) * (hbase.hstore.compaction.ratio default value 1.2), files will not be filtered out to participate in major compaction.

    It is necessary to support a forced merging mechanism. For scenarios where a large amount of data is deleted, or where bulkload exists, you can explicitly pass in a parameter such as foreMajor when manually triggering the major, and then perform forced Major Compaction in Stripe units to support the data. Clean up.
Bug: Possible data loss in WAL after RegionServer crash
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Dec/23 16:06

We recently had a production incident:
 # RegionServer crashes, but local DataNode lives on
 # WAL lease recovery kicks in
 # Namenode reconstructs the block during lease recovery (which results in a new genstamp). It chooses the replica on the local DataNode as the primary.
 # Local DataNode reconstructs the block, so NameNode registers the new genstamp.
 # Local DataNode and the underlying host dies, before the new block could be replicated to other replicas.

This leaves us with a missing block, because the new genstamp block has no replicas. The old replicas still remain, but are considered corrupt due to GENSTAMP_MISMATCH.

Thankfully we were able to confirm that the length of the corrupt blocks were identical to the newly constructed and lost block. Further, the file in question was only 1 block. So we downloaded one of those corrupt block files and hdfs {{hdfs dfs -put -f}} to force that block to replace the file in hdfs. So in this case we had no actual data loss, but it could have happened easily if the file was more than 1 block or the replicas weren't fully in sync prior to reconstruction.

In order to avoid this issue, we should avoid writing WAL blocks too the local datanode. We can use CreateFlag.NO_WRITE_LOCAL for this. Hat tip to [~weichiu] for pointing this out.

During reading of WALs we already reorder blocks so as to avoid reading from the local datanode, but avoiding writing there altogether would be better.
Bug: move_servers_namespaces_rsgroup is not changing the new rs group in namespace description
Status: Open | Priority: Major | Resolution: 
Created: 12/Oct/23 07:40

{noformat}

hbase:024:0> list_rsgroups
NAME                                                          SERVER / TABLE
 rs1                                                          server hostname2:16020
                                                              table hbase:meta
                                                              table hbase:acl
                                                              table hbase:namespace
                                                              table hbase:rsgroup
 default                                                      server hostname3:16020
                                                              table ns_R:ta
 tenant_group1
 tenant_group                                                 server hostname1:16020
                                                              table ns:tb1
                                                              table ns:tab
                                                              table ns:t2
4 row(s)
Took 0.0129 seconds
hbase:025:0> move_servers_namespaces_rsgroup 'rs1',['hostname1:16020'], ['ns']
Took 0.0302 seconds
hbase:026:0> list_rsgroups
NAME                                                          SERVER / TABLE
 rs1                                                          server hostname1:16020
                                                              server hostname2:16020
                                                              table ns:tb1
                                                              table ns:tab
                                                              table hbase:meta
                                                              table hbase:acl
                                                              table ns:t2
                                                              table hbase:namespace
                                                              table hbase:rsgroup
 default                                                      server hostname2:16020
                                                              table ns_R:ta
 tenant_group1
 tenant_group
4 row(s)
Took 0.0140 seconds
hbase:027:0> describe_namespace 'ns'
DESCRIPTION
{NAME => 'ns', hbase.rsgroup.name => 'tenant_group'}
Quota is disabled
Took 0.0093 seconds
hbase:028:0>

{noformat}
Bug: Region Canary can take lot more time If any region (except the first region) starts with delete markers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Nov/23 23:34

In CanaryTool.java, Canary reads only the first row of the region using [Get|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L520C33-L520C33] for any region of the table. Canary uses [Scan with FirstRowKeyFilter for table scan|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L530] If the said region has empty start key (This will only happen when region is the first region for a table)

With -[HBASE-16091|https://issues.apache.org/jira/browse/HBASE-16091]- RawScan was [implemented|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L519-L534] to improve performance for regions which can have high number of delete markers. Based on currently implementation, [RawScan is only enabled|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L519] if region has empty start-key (or region is first region for the table).

RawScan doesn't work for rest of the regions in the table except first region. Also If the region has all the rows or majority of the rows with delete markers, Get Operation can take a lot of time. This is can cause timeouts for CanaryTool.
Bug: Flaky test: TestFanOutOneBlockAsyncDFSOutput (Part 2)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Dec/23 15:52

The following test sometimes fails for me when running locally with Maven:

TestFanOutOneBlockAsyncDFSOutput.testRecover()

I can't really figure out the reason, but it's probably a side effect of the preceding test: testConnectToDatanodeFailed(). This test also restarts one of the datanodes in the MiniDFS cluster just like testRecover() and it somehow causes the failure.
{noformat}
java.lang.AssertionError: flush should fail
	at org.junit.Assert.fail(Assert.java:89)
	at org.apache.hadoop.hbase.io.asyncfs.TestFanOutOneBlockAsyncDFSOutput.testRecover(TestFanOutOneBlockAsyncDFSOutput.java:154)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method){noformat}
The flush() call is testRecover() should fail, because we restart one of the DN in the DFS cluster which is expected to brake connection. It succeeds though if the preceding test already restarted a DN. No matter which DN we restart, even if they're different, the error occurs.

I also tried to add CLUSTER.waitDatanodeFullyStarted() at the end of testConnectToDatanodeFailed(), looks like it made the tests slightly more stable, but didn't help fully.
Bug: Flaky test: TestTableShell
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Dec/23 15:22

The test is running the following Ruby commands:
{noformat}
# Instert test data
@test_table.put(1, "x:a", 1)
@test_table.put(2, "x:raw1", 11)
@test_table.put(2, "x:raw1", 11)
@test_table.put(2, "x:raw1", 11)
@test_table.put(2, "x:raw1", 11)
{noformat}
And validate the versions with:
{noformat}
args = { VERSIONS => 10, RAW => true } # Since 4 versions of row with rowkey 2 is been added, we can use any number >= 4 for VERSIONS to scan all 4 versions.
num_rows = 0
@test_table._scan_internal(args) do # Raw Scan
  num_rows += 1
end
# 5 since , 1 from row key '1' and other 4 from row key '4'
assert_equal(num_rows, 5,
             'Num rows scanned without RAW/VERSIONS are not 5')
{noformat}
Which sometimes (almost always on fast machines) fails, because it only finds 3 versions out of 4. I believe this due to commands are running too fast and inserts data with same timestamp, so HBase cannot distinguish them. I'd like to add some sleep between puts to fix it.
Bug: Infinite waiting on lock acquisition by snapshot can result in unresponsive master
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Dec/23 05:39
Affects Version/s: 2.4.17

When a region is stuck in transition for significant time, any attempt to take snapshot on the table would keep master handler thread in forever waiting state. As part of the creating snapshot on enabled or disabled table, in order to get the table level lock, LockProcedure is executed but if any region of the table is in transition, LockProcedure could not be executed by the snapshot handler, resulting in forever waiting until the region transition is completed, allowing the table level lock to be acquired by the snapshot handler.

In cases where a region stays in RIT for considerable time, if enough attempts are made by the client to create snapshots on the table, it can easily exhaust all handler threads, leading to potentially unresponsive master. Attached a sample thread dump.

Proposal: The snapshot handler should not stay stuck forever if it cannot take table level lock, it should fail-fast.

!image.png!