av9ash/GitBugs huggingface dataset:
Bug: BuckeCache.evictBlocksByHfileName won't work after a cache recovery from file
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Mar/24 20:39
Affects Version/s: 2.6.0

HBASE-27313, HBASE-27686 and HBASE-27743 have extended BucketCache persistent cache capabilities to make it resilient to RS crashes or non graceful stops, when using file based ioengine for BucketCache.

BucketCache maintains two main collections for tracking blocks in the cache: backingMap and blocksByHFile. The former is used as the main index of blocks for the actual cache, whilst the latter is a set of all blocks in the cache ordered by name, in order to conveniently and efficiently retrieve the list of all blocks from a single file in the BucketCache.evictBlocksByHfile method.

 

The problem is that at cache recovery time, we are populating the blocksByHFile set, which causes any calls to BucketCache.evictBlocksByHfile method to not evict any blocks, once we have recovered the cache from the cache persistence file (for instance, after a n RS restart).
Bug: Integration of time-based priority logic of bucket cache in prefetch functionality of HBase.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Mar/24 10:38

This Jira tracks the integration of the framework of APIs (implemented in HBASE-28465) related to data tiering into prefetch logic of HBase. The implementation should filter out the cold data and enable the prefetching of hot data into bucket cache.

Thanks,

Janardhan
Bug: HFilePrettyPrinter thrown NoSuchMethodError about MetricRegistry
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jan/24 06:35
Affects Version/s: 2.5.6

{{When I view the contents of the HFile through storeFile.jsp or hbase hfile -p -v xxxx, it will throw an exception similar to the following:}}
 
{{HTTP ERROR 500 java.lang.NoSuchMethodError: 'com.codahale.metrics.Histogram com.codahale.metrics.MetricRegistry.histogram(java.lang.String, com.codahale.metrics.MetricRegistry$MetricSupplier)'}}
||URI:|/storeFile.jsp|
||STATUS:|500|
||MESSAGE:|java.lang.NoSuchMethodError: 'com.codahale.metrics.Histogram com.codahale.metrics.MetricRegistry.histogram(java.lang.String, com.codahale.metrics.MetricRegistry$MetricSupplier)'|
||SERVLET:|org.apache.hadoop.hbase.generated.regionserver.storeFile_jsp|
||CAUSED BY:|java.lang.NoSuchMethodError: 'com.codahale.metrics.Histogram com.codahale.metrics.MetricRegistry.histogram(java.lang.String, com.codahale.metrics.MetricRegistry$MetricSupplier)'|

I found that there were conflicting files in the lib folder, specifically metrics-core-3.0.1.jar and metrics-core-3.2.6.jar. After deleting metrics-core-3.0.1.jar, the functionality returned to normal.
Bug: Add trust/key store type to ZK TLS settings handled by HBase
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Feb/24 16:32
Affects Version/s: 2.4.17

Let's add the following settings as well. Last time we missed it.
{noformat}
zookeeper.ssl.keyStore.type
zookeeper.ssl.trustStore.type{noformat}
Handle them in hbase-site.xml as:
{noformat}
hbase.zookeeper.property.ssl.keyStore.type
hbase.zookeeper.property.ssl.trustStore.type{noformat}
Bug: do-release-docker fails to setup gpg agent proxy if proxy container is slow to start
Status: Open | Priority: Major | Resolution: 
Created: 22/Mar/24 14:24

In do-release-docker.sh we spin up the gpg-agent-proxy container and then immediately run ssh-keyscan and then immediately run ssh. Despite having {{{}set -e{}}}, both of these can fail without failing the script. This manifests as a really hard to debug failure in the hbase-rm container with "gpg: no gpg-agent running in this session"

With some debugging I realized that the ssh tunnel had not been created. looking at the logs, the gpg-agent-proxy.ssh-keyscan file is empty and the gpg-proxy.ssh.log shows a Connection refused error.

You'd think these would fail the script, but they don't for different reasons:
 # ssh-keyscan output is piped through sort. Running ssh-keyscan directly returns an error code, but piping it through sort turns it into a success code.
 # ssh is executed in background with {{{}&{}}}, which similarly loses the error code

I think we should add a step prior to ssh-keyscan which waits until port 62222 is available. I'm not sure how to retain the error codes in the above 2 commands, but can try to look into that as well.
Bug: Introduce a version field in file based tracker record
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Mar/24 14:43

Per the discussion around HBASE-27826 and the related design doc, we all agree that we should add version field to store file tracker, so when downgrading, we will know that we will miss something when reading a tracker file with higher version and fail the initialization, instead of ignore it silently and may cause data loss.
Bug: Decommissioned hosts should be rejected by the HMaster
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Feb/24 13:59

We had an issue with a cluster, internally at HubSpot, where a decommissioned RegionServer was still being picked up by the HMaster. The host the RegionServer was living on was impaired, and we couldn't correctly kill the RegionServer, so the HMaster would periodically hear back from the host and remove it from its dead host's list.

We would like to implement a fix so that this doesn't happen. We're thinking of adding a boolean flag to the Decommission RegionServer Admin API that signifies ignoring the startcode of the servername, when the boolean is True the host will be rejected every time it comes back even if it had a different startcode.
Bug: Quotas should have a configurable minimum wait interval
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 08/Mar/24 18:26

At my day job we're attempting to rollout read size throttling by default for thousands of distinct users across hundreds of multi-tenant clusters.

During our rollout we've observed that throttles with a 1 second refill interval will yield relatively tiny wait intervals disproportionately often. From what we've seen, wait intervals are <=5ms on approximately 20-50% of our RpcThrottlingExceptions; this could sound theoretically promising if latency is your top priority. But, in reality, this makes it very difficult to configure a throttle tolerant HBase client because retries become very prone to near-immediate exhaustion, and throttled clients quickly saturate the cluster's RPC layer with rapid-fire retries.

One can combat this with the FixedIntervalRateLimiter, but that's a very heavy handed approach from latency's perspective, and can still yield tiny intervals that exhaust retries and erroneously fail client operations under significant load.

With this in mind, I'm proposing that we introduce a configurable minimum wait interval for quotas, defaulted to 0. This would make quotas much more usable at scale from our perspective.
Bug: Fix BackupSystemTable Scans 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Mar/24 15:35

When calculating which WALs should be included in an incremental backup the backup system does a prefix scan for the last roll log timestamp. This uses the backup root in the prefix (<prefix><backupRoot>.) If you happen have multiple backup roots where one is a root of the other you'll get inaccurate results. 

 

Since the rowkey is <prefix><backupRoot><null separator><server> let's modify the prefix scan to be <prefix><backupRoot><null separator>.
Bug: Remove hbase-examples from hbase-assembly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Mar/24 05:32
Affects Version/s: 3.0.0-beta-1

hbase-assembly is supposed to contain programming examples for HBase.
However, it is added to the assembly API, and becomes part of the HBase distrbutions.

On hand this adds some potentially useful components and coprocessors to HBase, on the other hand many of those are not production quality, and were never meant to be used as-is.

It also adds the Curator libraries to the general HBase classpath, which are used but by but a single example.

Removing hbase-examples from the assembly would fix both problems.
Bug: Cast without checking in TestCacheConfig
Status: Open | Priority: Major | Resolution: 
Created: 20/Mar/24 21:56

*1. What happened*
After setting `hfile.block.cache.policy=AdaptiveLRU`, running test `org.apache.hadoop.hbase.io.hfile.TestCacheConfig#testBucketCacheConfigL1L2Setup` results in a casting error.
 

*2. Where's the problem*
In`PooledRemoteClient`:
{code:java}
    FirstLevelBlockCache lbc = cbc.l1Cache;
...
    // Force evictions by putting in a block too big.
    final long justTooBigSize = ((LruBlockCache) lbc).acceptableSize() + 1;{code}
However, lbc does not necessarily need to be a LruBlockCache, leading to a potential `ClassCastException`
 

*3. How to reproduce*
1. set `hfile.block.cache.policy` to `AdaptiveLRU `
2. run `org.apache.hadoop.hbase.io.hfile.TestCacheConfig#testBucketCacheConfigL1L2Setup`
you should observe
{code:java}
java.lang.ClassCastException: class org.apache.hadoop.hbase.io.hfile.LruAdaptiveBlockCache cannot be cast to class org.apache.hadoop.hbase.io.hfile.LruBlockCache (org.apache.hadoop.hbase.io.hfile.LruAdaptiveBlockCache and org.apache.hadoop.hbase.io.hfile.LruBlockCache are in unnamed module of loader 'app')
at org.apache.hadoop.hbase.io.hfile.TestCacheConfig.testBucketCacheConfigL1L2Setup(TestCacheConfig.java:342)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
 

We would like to provide a patch if this is confirmed.
Bug: Remove Curator dependency from hbase-endpoint
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Mar/24 05:09
Affects Version/s: 3.0.0-beta-1

TestRpcControllerFactory used to erronously import some classes from Curator's relocated Guava.
This has been fixed now, but the (compile scope) dependency for curator has not been removed.

I propose removing the dependency in master, branch-3 and branch-2.
Bug: Bounded leak of FSDataInputStream buffers from checksum switching
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jan/24 22:07

In FSDataInputStreamWrapper, the unbuffer() method caches an unbuffer instance the first time it is called. When an FSDataInputStreamWrapper is initialized, it has hbase checksum disabled.

In HFileInfo.initTrailerAndContext we get the stream, read the trailer, then call unbuffer. At this point, checksums have not been enabled yet via prepareForBlockReader. So the call to unbuffer() caches the current non-checksum stream as the unbuffer instance.

Later, in initMetaAndIndex we do a similar thing. This time, prepareForBlockReader has been called, so we are now using hbase checksums. When initMetaAndIndex calls unbuffer(), it uses the old unbuffer instance which actually has been closed when we switched to hbase checksums. So that call does nothing, and the new no-checksum input stream is never unbuffered.

I haven't seen this cause an issue with normal hdfs replication (though haven't gone looking). It's very problematic for Erasure Coding because DFSStripedInputStream holds a large buffer (numDataBlocks * cellSize, so 6mb for RS-6-3-1024k) that is only used for stream reads NOT pread. The FSDataInputStreamWrapper we are talking about here is only used for pread in hbase, so those 6mb buffers just hang around totally unused but unreclaimable. Since there is an input stream per StoreFile, this can add up very quickly on big servers.
Bug: Quota estimates are too optimistic for large scans
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Feb/24 19:52

Let's say you're running a table scan with a throttle of 100MB/sec per RegionServer. Ideally your scans are going to pull down large results, often containing hundreds or thousands of blocks.

You will estimate each scan as costing a single block of read capacity, and if your quota is already exhausted then the server will evaluate the backoff required for your estimated consumption (1 block) to be available. This will often be ~1ms, causing your retries to basically be immediate.

Obviously it will routinely take much longer than 1ms for 100MB of IO to become available in the given configuration, so your retries will be destined to fail. At worst this can cause a saturation of your server's RPC layer, and at best this causes erroneous exhaustion of the client's retries.

We should find a way to make these estimates a bit smarter for large scans.
Bug: Allow Action and Policies of ServerKillingMonkey to be configurable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Mar/24 16:41

Currently for ServerKillingMonkeyFactory, actions and policies have hardcoded timeouts.
{code:java}
    Action[] actions1 = new Action[] {
      new RestartRandomRsExceptMetaAction(60000),
      new RestartActiveMasterAction(5000),
      // only allow 2 servers to be dead
      new RollingBatchRestartRsAction(5000, 1.0f, 2, true),
      new ForceBalancerAction(),
      new GracefulRollingRestartRsAction(gracefulRollingRestartTSSLeepTime),
      new RollingBatchSuspendResumeRsAction(rollingBatchSuspendRSSleepTime,
          rollingBatchSuspendtRSRatio)
    }; {code}
and
{code:java}
    return new PolicyBasedChaosMonkey(properties, util,
      new CompositeSequentialPolicy(new DoActionsOncePolicy(60 * 1000, actions1),
        new PeriodicRandomActionPolicy(60 * 1000, actions1)),
      new PeriodicRandomActionPolicy(60 * 1000, actions2));
  } {code}
We should allow these to be configurable too.
Bug: Add support for using mapreduce sort in HFileOutputFormat2
Status: Open | Priority: Major | Resolution: 
Created: 13/Mar/24 15:49

Currently HFileOutputFormat2 uses CellSortReducer, which attempts to sort all of the cells of a row in memory using a TreeSet. There is a warning in the javadoc "If lots of columns per row, it will use lots of memory sorting." This can be problematic for WALPlayer, which uses HFileOutputFormat2. You could have reasonably sized row which just gets lots of edits in the time period of WALs being replayed, and that would cause an OOM. We are seeing this in some cases with incremental backups.

MapReduce has built-in sorting capabilities which are not limited to sorting in memory. It can spill to disk as necessary to sort very large datasets. We can get this capability in HFileOutputFormat2 with a couple changes:
 # Add support for a KeyOnlyCellComparable type as the map output key
 # When configured, use job.setSortComparatorClass(CellWritableComparator.class) and job.setReducerClass(PreSortedCellsReducer.class)
 # Update WALPlayer to have a mode which can output this new comparable instead of ImmutableBytesWritable

CellWritableComparator exists already for the Import job, so there is some prior art.
Bug: Potential Data loss issue for the HFLog
Status: Open | Priority: Major | Resolution: 
Created: 20/Feb/24 22:47

 I am going through the WAL implementation HFLog, and based on the code reading, as my understanding there maybe potential data loss for example in below scenarios: # Append record A3 already into the stream, and but the syncFuture S3 does not reach to the SyncRunner yet
 # The SyncRunner run the sync and call sync on writer, but failed, all the data in stream lost including A3
 # Because the syncFuture S3 does not reach to the syncRunner, it can not release it and mark it failed
 # In next run() of the SyncRunner, the new sync() succeeded, it will release and mark S3 as succeeded, but actually it should failed which cause data loss.

Can anyone help to check above above is true? Am I missing anything regarding this process?
!Screenshot 2024-02-19 at 10.53.20.png!
Bug: TableNotFoundException when executing 'hbase hbck'
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Feb/24 11:38

*Reproduction:*
Starting hbase cluster
Executing command 'hbase hbck'
TableNotFoundException occur aboubt '{color:#FF0000}hbase:replication{color}'
{code:java}
//代码占位符
org.apache.hadoop.hbase.replication.ReplicationException: failed to listAllQueues
        at org.apache.hadoop.hbase.replication.TableReplicationQueueStorage.listAllQueues(TableReplicationQueueStorage.java:289)
        at org.apache.hadoop.hbase.util.hbck.ReplicationChecker.getUnDeletedQueues(ReplicationChecker.java:77)
        at org.apache.hadoop.hbase.util.hbck.ReplicationChecker.checkUnDeletedQueues(ReplicationChecker.java:105)
        at org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixReplication(HBaseFsck.java:2575)
        at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:803)
        at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:3752)
        at org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:3551)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:97)
        at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3536)
Caused by: org.apache.hadoop.hbase.TableNotFoundException: hbase:replication {code}
*Problem analysis:*
hbase does not create the table 'hbase:replication' until the first add_peer operation would be executed.
However, 'hbase hbck' operation would check the 'hbase:replication', So if this table has not been created, we would get TableNotFoundException.
Bug: Set correct Result to RegionActionResult for successful Put/Delete mutations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Mar/24 00:59

While returning response of multi(), RSRpcServices build the RegionActionResult with Result or Exception (ClientProtos.ResultOrException). It sets the Exception to this class in all cases where the operation fails with corresponding exception types e.g. NoSuchColumnFamilyException or FailedSanityCheckException etc.

In case of atomic mutations Increment and Append, we add the Result object to ClientProtos.ResultOrException, which is used by client to retrieve result from the batch API: {_}Table#batch(List<? extends Row> actions, Object[] results){_}.

Phoenix performs atomic mutation for Put using _preBatchMutate()_ endpoint. Hence, returning Result object with ResultOrException is important for the purpose of returning the result back to the client as part of the atomic operation. Even if Phoenix returns the OperationStatus (with Result) to MiniBatchOperationInProgress, since HBase uses the empty Result for the Success case, the client would not be able to get the expected result.
{code:java}
case SUCCESS:
  builder.addResultOrException(
    getResultOrException(ClientProtos.Result.getDefaultInstance(), index));
  break; {code}
If OperationStatus returned by _Region#batchMutate_ has valid Result object, it should be used by RSRpcServices while returning the response.
Bug: RpcThrottlingException messages should describe the throttled access pattern
Status: Open | Priority: Major | Resolution: 
Created: 08/Mar/24 20:22

Right now we catch RpcThrottlingExceptions and have some debug logging in [RegionServerRpcQuotaManager|https://github.com/apache/hbase/blob/98eb3e01b352684de3c647a6fda6208a657c4607/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/RegionServerRpcQuotaManager.java#L234-L236] — this is okay support for in depth explanations, but is not trivially transparent to users.

For example, at my day job we have proxy APIs which sit between HBase and many microservices. We throttle these microservices in isolation, but the RpcThrottlingExceptions appear to be indiscriminate in the stdout of the proxy API.

If we added the given username, table, and namespace to RpcThrottlingException messages then understanding the nature and specificity of any given throttle violation should be much more straightforward. Given that quotas/throttling is most useful in a multi-tenant environment, I would anticipate this being a pretty universal usability pain point.

It would be a bit more complicated, but we should also consider including more information about the rate limiter which has been violated. For example, what is the current configured read size limit that we've exceeded?
Bug: Improve quota RateLimiter synchronization
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Feb/24 13:22

We've been experiencing RpcThrottlingException with 0ms waitInterval. This seems odd and wasteful, since the client side will immediately retry without backoff. I think the problem is related to the synchronization of RateLimiter.

The TimeBasedLimiter checkQuota method does the following:
{code:java}
if (!reqSizeLimiter.canExecute(estimateWriteSize + estimateReadSize)) {
  RpcThrottlingException.throwRequestSizeExceeded(
    reqSizeLimiter.waitInterval(estimateWriteSize + estimateReadSize));
} {code}
Both canExecute and waitInterval are synchronized, but we're calling them independently. So it's possible under high concurrency for canExecute to return false, but then waitInterval returns 0 (would have been true)

I think we should simplify the API to have a single synchronized call:
{code:java}
long waitInterval = reqSizeLimiter.tryAcquire(estimateWriteSize + estimateReadSize);
if (waitInterval > 0) {
  RpcThrottlingException.throwRequestSizeExceeded(waitInterval);
}{code}
Bug: WALPrettyPrinter fails with "java.io.EOFException: EOF while reading message size"
Status: Open | Priority: Major | Resolution: 
Created: 07/Mar/24 13:38
Affects Version/s: 2.6.0

While testing WALPrettyPrinter fails with "java.io.EOFException: EOF while reading message size"
{code:java}
hbase wal -p /apps/hbase/data/WALs/HOSTNAME,16020,1709296677041/HOSTNAME%2C16020%2C1709296677041.1709815102928{code}
{code:java}
Writer Classes: ProtobufLogWriter AsyncProtobufLogWriter SecureProtobufLogWriter SecureAsyncProtobufLogWriter
Cell Codec Class: org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec
Exception in thread "main" java.io.EOFException: EOF while reading message size
    at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.parseDelimitedFrom(ProtobufUtil.java:3727)
    at org.apache.hadoop.hbase.regionserver.wal.ProtobufWALStreamReader.next(ProtobufWALStreamReader.java:56)
    at org.apache.hadoop.hbase.wal.WALStreamReader.next(WALStreamReader.java:42)
    at org.apache.hadoop.hbase.wal.WALPrettyPrinter.processFile(WALPrettyPrinter.java:297)
    at org.apache.hadoop.hbase.wal.WALPrettyPrinter.run(WALPrettyPrinter.java:516)
    at org.apache.hadoop.hbase.wal.WALPrettyPrinter.main(WALPrettyPrinter.java:429){code}
cc [~nihaljain.cs]
Bug: Add ofs (Ozone Filesystem) support for acquireDelegationToken
Status: Open | Priority: Major | Resolution: 
Created: 05/Mar/24 15:40

Currently acquireDelegationToken is hardcoded for checking swebhdfs, webhdfs and hdfs(Refer below or [here|[https://github.com/apache/hbase/blob/4f97ece9f5ab9288ea44f5842be55a4dbaa866e0/hbase-server/src/main/java/org/apache/hadoop/hbase/security/token/FsDelegationToken.java#L62-L84]]).
We should add support for Ozone also. 
{code:java}
public void acquireDelegationToken(final FileSystem fs) throws IOException {
  String tokenKind;
  String scheme = fs.getUri().getScheme();
  if (SWEBHDFS_SCHEME.equalsIgnoreCase(scheme)) {
    tokenKind = SWEBHDFS_TOKEN_KIND.toString();
  } else if (WEBHDFS_SCHEME.equalsIgnoreCase(scheme)) {
    tokenKind = WEBHDFS_TOKEN_KIND.toString();
  } else if (HDFS_URI_SCHEME.equalsIgnoreCase(scheme)) {
    tokenKind = HDFS_DELEGATION_KIND.toString();
  } else {
    LOG.warn("Unknown FS URI scheme: " + scheme);
    // Preserve default behavior
    tokenKind = HDFS_DELEGATION_KIND.toString();
  }

  acquireDelegationToken(tokenKind, fs);
} {code}
This can impact jobs like BulkLoad on a secure environment.

Thanks [~bszabolcs] for the debug help!