av9ash/GitBugs huggingface dataset:
Bug: AsyncProcess inconsistent exception thrown for operation timeout
Status: Open | Priority: Major | Resolution: 
Created: 11/Feb/24 14:35
Affects Version/s: 2.0.0

I'm not sure if I'll get to this, but wanted to log it as a known issue.

AsyncProcess has a design where it breaks the batch into sub-batches based on regionserver, then submits a callable per regionserver in a threadpool. In the main thread, it calls waitUntilDone() with an operation timeout. If the callables don't finish within the operation timeout, a SocketTimeoutException is thrown. This exception is not very useful because it doesn't give you any sense of how many calls were in progress, on which servers, or why it's delayed.

Recently we've been improving the adherence to operation timeout within the callables themselves. The main driver here has been to ensure we don't erroneously clear the meta cache for operation timeout related errors. So we've added a new OperationTimeoutExceededException, which is thrown from within the callables and does not cause a meta cache clear. The added benefit is that if these bubble up to the caller, they are wrapped in RetriesExhaustedWithDetailsException which includes a lot more info about which server and which action is affected. 

Now we've covered most but not all cases where operation timeout is exceeded. So when exceeding operation timeout it's possible sometimes to see a SocketTimeoutException from waitUntilDone, and sometimes see OperationTimeoutExceededException from the callables. It will depend on which one fails first. It may be nice to finish the swing here, ensuring that we always throw OperationTimeoutExceededException from the callables.

The main remaining case is in the call to locateRegion, which hits meta and does not honor the call's operation timeout (instead meta operation timeout). Resolving this would require some refactoring of ConnectionImplementation.locateRegion to allow passing an operation timeout and having that affect the userRegionLock and meta scan.
Bug: BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully cached
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Mar/24 15:21
Affects Version/s: 2.6.0

Noticed that TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning was flakey, failing whenever the block eviction happened while prefetch was still ongoing.

In the test, we pass an instance of BucketCache directly to the cache config, so the test is actually placing both data and meta blocks in the bucket cache. So sometimes, the test call BucketCache.notifyFileCachingCompleted after the it has already evicted two blocks.  

Inside BucketCache.notifyFileCachingCompleted, we iterate through the backingMap entry set, counting number of blocks for the given file. Then, to consider whether the file is fully cached or not, we do the following validation:
{noformat}
if (dataBlockCount == count.getValue() || totalBlockCount == count.getValue()) {
  LOG.debug("File {} has now been fully cached.", fileName);
  fileCacheCompleted(fileName, size);
}  {noformat}
But the test generates 57 total blocks, 55 data and 2 meta blocks. It evicts two blocks and asserts that the file hasn't been considered fully cached. When these evictions happen while prefetch is still going, we'll pass that check, as the the number of blocks for the file in the backingMap would still be 55, which is what we pass as dataBlockCount.

As BucketCache is intended for storing data blocks only, I believe we should make sure BucketCache.notifyFileCachingCompleted only accounts for data blocks. Also, the TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning should be updated to consistently reproduce the eviction concurrent to the prefetch.
Bug: Interrupt cache prefetch thread when a heap usage threshold is reached
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jan/24 15:19
Affects Version/s: 2.4.17

Mostly critical when using non heap cache implementations, such as offheap or file based. If the cache medium is too large and there are many blocks to be cached, it may create a lot of cache index object in the RegionServer heap. We should have guardrails to preventing caching from exhausting available heap.
Bug: Make Delay prefetch property to be dynamically configured
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/24 05:08
Affects Version/s: 2.4.17

Make the prefetch delay configurable. The prefetch delay is associated to hbase.hfile.prefetch.delay configuration. There are some cases where configuring hbase.hfile.prefetch.delay would help in achieving better throughput.
Bug: Build hbase-thirdparty with JDK17
Status: Open | Priority: Major | Resolution: 
Created: 20/Feb/24 07:42

hbase-thirdparty fails to build with JDK 17. It fails for Unsafe module with following:
{code:java}
[INFO] Apache HBase Third-Party Libs ...................... SUCCESS [  5.138 s]
[INFO] Apache HBase Patched and Relocated (Shaded) Protobuf SUCCESS [ 17.888 s]
[INFO] Apache HBase Relocated (Shaded) Netty Libs ......... SUCCESS [02:19 min]
[INFO] Apache HBase Relocated (Shaded) netty-tcnative Libs  SUCCESS [ 46.429 s]
[INFO] Apache HBase Relocated (Shaded) GSON Libs .......... SUCCESS [  1.021 s]
[INFO] Apache HBase Relocated (Shaded) Third-party Miscellaneous Libs SUCCESS [  8.435 s]
[INFO] Apache HBase Relocated (Shaded) Jetty Libs ......... SUCCESS [  4.972 s]
[INFO] Apache HBase Relocated (Shaded) Jersey Libs ........ SUCCESS [ 11.526 s]
[INFO] Apache HBase Relocated (Shaded) jackson-jaxrs-json-provider SUCCESS [  1.421 s]
[INFO] Apache HBase Unsafe Wrapper ........................ FAILURE [  0.331 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  03:57 min
[INFO] Finished at: 2024-02-16T07:20:20Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.11.0:compile (default-compile) on project hbase-unsafe: Compilation failure: Compilation failure: 
[ERROR] /workspace/hbase/hbase-unsafe/src/main/java/org/apache/hadoop/hbase/unsafe/HBaseUnsafeInternal.java:[259,18] cannot find symbol
[ERROR]   symbol:   method defineClass(java.lang.String,byte[],int,int,java.lang.ClassLoader,java.security.ProtectionDomain)
[ERROR]   location: variable UNSAFE of type sun.misc.Unsafe
[ERROR] /workspace/hbase/hbase-unsafe/src/main/java/org/apache/hadoop/hbase/unsafe/HBaseUnsafeInternal.java:[263,18] cannot find symbol
[ERROR]   symbol:   method defineAnonymousClass(java.lang.Class<capture#1 of ?>,byte[],java.lang.Object[])
[ERROR]   location: variable UNSAFE of type sun.misc.Unsafe
[ERROR] -> [Help 1]
{code}
The code does not compile as there is a breaking change in JKD17 around unsafe usage:
 * [https://bugs.openjdk.org/browse/JDK-8266760]
 * [https://bugs.openjdk.org/browse/JDK-8199699]

Need to raise a patch with changes for HBaseUnsafeInternal implementation such that it is compatible (with JDK8) and still works for JDK11 and JDK17.
Bug: HBase backup yarn queue parameter ignored
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Feb/24 11:48
Affects Version/s: 2.6.0

It seems the parameter to specify the yarn queue for HBase backup (`-q`) is ignored:
{code:java}
hbase backup create full hdfs:///tmp/backups/hbasetest/hbase -q hbase-backup {code}
gets executed on the "default" queue.

Setting the queue through the configuration does work.
Bug: Introduce a close method for memstore for release active segment
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/24 09:10

Per the analysis in parent issue, we will always have an active segment in memstore even if it is empty, so if we do not call close on it, it will lead to a netty leak warning message.

Although there is no real memory leak for this case, we'd better still fix it as it may hide other memory leak problem.
Bug: [JDK17] Update hbase-env.sh with alternates to JVM flags which are no longer supported
Status: In Progress | Priority: Major | Resolution: 
Created: 20/Feb/24 08:07

Some JVM flags like {{{}-XX:+PrintGCDetails{}}}, {{-XX:+PrintGCDateStamps}} etc. are no longer supported with JDK 17 and hbase would fail to start if these are passed. 
We should do an audit and update [https://github.com/apache/hbase/blob/master/conf/hbase-env.sh] to capture alternate/fix. 

Will refer following for a fix/repalacement:
 * [https://stackoverflow.com/questions/54144713/is-there-a-replacement-for-the-garbage-collection-jvm-args-in-java-11]
 * [https://docs.oracle.com/javase/9/tools/java.htm#GUID-BE93ABDC-999C-4CB5-A88B-1994AAAC74D5__CONVERTGCLOGGINGFLAGSTOXLOG-A5046BD1]
Bug: Warn: Cache key had block type null, but was found in L1 cache
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Feb/24 22:32
Affects Version/s: 2.4.18

I'm ITBLL testing branch-2.6 and am seeing lots of these warns. This is new to me. I would expect a warn to be on the rare side or be indicative of a problem, but unclear from the code.

cc [~wchevreuil]
Bug: create-release should spotless:apply after making any file changes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Mar/24 10:09
Affects Version/s: 4.0.0-alpha-1

Looks like the release notes generator can sometimes leave whitespace in its changes, as is currently the case on branch-2.5. We should be a little more careful about this in the create-release scripts. Anything that performs and commits some change should run spotless:apply over the affected files.
Bug: Return too slow when scanning a table with non-existing REGION_REPLICA_ID
Status: Open | Priority: Major | Resolution: 
Created: 17/Mar/24 02:09

It takes more than 8 seconds to get the error result, and it's always like this every thime.
 
Reproduction Step:
 
1. Creating a table with region replica
create 't01', 'info', \{REGION_REPLICATION => 2}
 
2. Scanning this table with a non-existing REGION_REPLICA_ID by CONSISTENCY.TIMELINE
scan 't01', \{CONSISTENCY => "TIMELINE", REGION_REPLICA_ID => 2}
 
3. we would take more than 8s to get the error result.
{code:java}
hbase:010:0> scan 't01', {CONSISTENCY => "TIMELINE", REGION_REPLICA_ID => 2}ROW                                     COLUMN+CELL
ERROR: No location for t01, row='', locateType=CURRENT, replicaId=2
For usage try 'help "scan"'
Took 8.2830 seconds {code}
Bug: Finish 2.4.18 release
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Mar/24 06:28

# Release the artifacts on repository.apache.org
# Move the binaries from dist-dev to dist-release
# Add xml to download page(via HBASE-28473)
# Push tag 2.4.18RCx as tag rel/2.4.18 (/)
# Release 2.4.18 on JIRA https://issues.apache.org/jira/projects/HBASE/versions/12353080
# Add release data on https://reporter.apache.org/addrelease.html?hbase
# Send announcement email
Bug: Integration of time-based priority caching into compaction paths.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Mar/24 10:53

The time-based priority caching is dependent on the date-tiered compaction that structures store files in date-based tiered layout. This Jira tracks the changes are needed for the integration of this compaction strategy with the data-tiering to enable appropriate caching of hot data in the cache, while the code data can remain the cloud storage.

Thanks,

Janardhan
Bug: Incremental backup can fail if log gets archived while WALPlayer is starting up
Status: Open | Priority: Major | Resolution: 
Created: 27/Mar/24 21:01

We had incremental backup fail with FileNotFoundException for a file in the WALs directory. Upon investigation, the log had been archived a few mins earlier. WALInputFormat's record reader has support for falling back on an archived path:
{code:java}
} catch (IOException e) {
  Path archivedLog = AbstractFSWALProvider.findArchivedLog(logFile, conf);
  // archivedLog can be null if unable to locate in archiveDir.
  if (archivedLog != null) {
    openReader(archivedLog);
    // Try call again in recursion
    return nextKeyValue();
  } else {
    throw e;
  }
} {code}
But the getSplits method has different handling:
{code:java}
try {
  List<FileStatus> files = getFiles(fs, inputPath, startTime, endTime);
  allFiles.addAll(files);
} catch (FileNotFoundException e) {
  if (ignoreMissing) {
    LOG.warn("File " + inputPath + " is missing. Skipping it.");
    continue;
  }
  throw e;
} {code}
This ignoreMissing variable was added in HBASE-14141 and is enabled via 
wal.input.ignore.missing.files which is defaulted to false and never set. Looking at the comment and reviewboard history of HBASE-14141 I think there might have been some confusion about where to handle these missing files, and this got lost in the shuffle.
 
I would prefer not to ignore missing hfiles. I think that could result in some weird behavior:
 * RegionServer has 10 archived and 30 not-yet-archived WALs needing to be backed up
 * The process starts, and while it's running 1 of those 30 WALs gets archived. That would get skipped due to FileNotFoundException
 * But the remaining 29 would be backed up

This scenario could cause some data consistency issues if this incremental backup is restored. We missed some edits in the middle of applied edits from other WALs.

So I do think failing as we do today is necessary for consistency, but unrealistic in a live cluster. The solution is to try finding the missing file in the archived directory. Backups has a coprocessor which will not allow the archived file to be cleaned up until it's backed up, so I think it's safe to say that a WAL is either definitely in WALs or oldWALs.
Bug: Support to skip Kerberos authentication for metric endpoints
Status: Open | Priority: Major | Resolution: 
Created: 06/Jan/24 05:25
Affects Version/s: 2.5.5

Make HBase support to skip Kerberos authentication for metric endpoints. (e.g. /jvm, /prometheus, /metrics)

Since HBase uses KerberoAuthenticationHandler.java, whitelist configuration can be used like [HADOOP-16527|https://issues.apache.org/jira/browse/HADOOP-16527].
Bug: Use connection url to specify the connection registry information
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Mar/24 12:26

As describe in this email from [~ndimiduk]

https://lists.apache.org/thread/98wqlkqvlnmpx3r7yrg9mw4pqz9ppofh

The first advantage here is that, we can encode the connection registry implementation in the scheme of the connection url, so for replication, we can now support cluster key other than zookeeper, which is important for us to remove zookeeper dependency on our public facing APIs.
Bug: Time Based Priority for BucketCache
Status: In Progress | Priority: Major | Resolution: 
Created: 28/Mar/24 04:59

This Jira introduces the feature of time-based data tiering in HBase to optimize storage efficiency and access performance by segregating data based on its recency. By keeping recent data in the bucket cache (backed by faster storage types like SSDs) and evicting older data, the system aims to provide a more flexible control over the cache allocation and eviction logic via configuration, allowing for defining time priorities for cached data. 

The need for a more extensive cache allocation mechanism becomes even more critical on HBase deployments where cache access reflects on significant performance gains, such as when using cloud storage as the underlying file system.

The data is segregated into hot or cold categories based on its age. The recent data within a specific time range (configured as hot-data-age) is treated as hot and is stored in the cache, while the older data is stored and accessed from the file system.

This feature intends to provide the TCO gains by optimizing the utilization of high cost bucket cache. Perfect fit for the use cases that have the date-based data writes while the scans focus on the recently written data.

Please find the detailed design document of the feature attached with the Jira.

Thanks,

Janardhan
Bug: CompressionTest hangs when run over a Ozone ofs path
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Mar/24 08:15

If we run the Compression test over HDFS path, it works fine:
{code:java}
hbase org.apache.hadoop.hbase.util.CompressionTest hdfs://ns1/tmp/dir1/dir2/test_file.txt snappy
24/03/20 06:08:43 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
24/03/20 06:08:43 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
24/03/20 06:08:43 INFO impl.MetricsSystemImpl: HBase metrics system started
24/03/20 06:08:43 INFO metrics.MetricRegistries: Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
24/03/20 06:08:43 INFO compress.CodecPool: Got brand-new compressor [.snappy]
24/03/20 06:08:43 INFO compress.CodecPool: Got brand-new compressor [.snappy]
24/03/20 06:08:44 INFO compress.CodecPool: Got brand-new decompressor [.snappy]
SUCCESS {code}
The command exits, but when the same is tried over a ofs path, the command hangs.
{code:java}
hbase org.apache.hadoop.hbase.util.CompressionTest ofs://ozone1710862004/test-222compression-vol/compression-buck2/test_file.txt snappy
24/03/20 06:05:19 INFO protocolPB.OmTransportFactory: Loading OM transport implementation org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransportFactory as specified by configuration.
24/03/20 06:05:20 INFO client.ClientTrustManager: Loading certificates for client.
24/03/20 06:05:20 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
24/03/20 06:05:20 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
24/03/20 06:05:20 INFO impl.MetricsSystemImpl: HBase metrics system started
24/03/20 06:05:20 INFO metrics.MetricRegistries: Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
24/03/20 06:05:20 INFO rpc.RpcClient: Creating Volume: test-222compression-vol, with om as owner and space quota set to -1 bytes, counts quota set to -1
24/03/20 06:05:20 INFO rpc.RpcClient: Creating Bucket: test-222compression-vol/compression-buck2, with bucket layout FILE_SYSTEM_OPTIMIZED, om as owner, Versioning false, Storage Type set to DISK and Encryption set to false, Replication Type set to server-side default replication type, Namespace Quota set to -1, Space Quota set to -1
24/03/20 06:05:21 INFO compress.CodecPool: Got brand-new compressor [.snappy]
24/03/20 06:05:21 INFO compress.CodecPool: Got brand-new compressor [.snappy]
24/03/20 06:05:21 WARN impl.MetricsSystemImpl: HBase metrics system already initialized!
24/03/20 06:05:21 INFO metrics.MetricRegistries: Loaded MetricRegistries class org.apache.ratis.metrics.dropwizard3.Dm3MetricRegistriesImpl
24/03/20 06:05:22 INFO compress.CodecPool: Got brand-new decompressor [.snappy]
SUCCESS 
.
.
.<It hangs unless exited>{code}
The command doesnt exit.
Attaching the jstack of the process below:
[^hbase_ozone_compression.jstack]

cc: [~weichiu]
Bug: HFileOutputFormat2 ClassCastException with s3 magic committer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Mar/24 12:36

In hadoop3 there's the s3 magic committer which can speed up s3 writes dramatically. In HFileOutputFormat2.createRecordWriter we cast the passed in committer as a FileOutputCommitter. This causes a class cast exception when the s3 magic committer is enabled:

 
{code:java}
Error: java.lang.ClassCastException: class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter cannot be cast to class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter {code}
 

We can cast to PathOutputCommitter instead, but its only available in hadoop3+. So we will need to use reflection to work around this in branch-2.
Bug: Add metric for GetClusterStatus request count.
Status: Open | Priority: Major | Resolution: 
Created: 05/Jan/24 22:46

We have been bitten multiple times by GetClusterStatus request overwhelming HMaster's memory usage. It would be good to add a metric for the total GetClusterStatus requests count.

In almost all of our production incidents involving GetClusterStatus request, HMaster were running out of memory with many clients call this RPC in parallel and the response size is very big.

In hbase2 we have [ClusterMetrics.Option|https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/ClusterMetrics.java#L164-L224] which can reduce the size of the response.

It would be nice to add another metric to indicate if the response size of GetClusterStatus is greater than some threshold (like 5MB)
Bug: Integration of time-based priority caching logic into cache evictions.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Mar/24 10:46

When the time-based priority caching is enabled, then, the block evictions triggered when the cache is full, should use the time-based priority caching framework APIs to detect the cold files and evict the blocks of those files first. This ensures that the hot data remains in cache while the cold data is evicted from cache.

Thanks,

Janardhan
Bug: Missing null check of rpcServer.scheduler.executor causes NPE with invalid value of hbase.client.default.rpc.codec
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Mar/24 21:58

When `hbase.client.default.rpc.codec` is set to `org.apache.hadoop.hbase.regionserver.wal.SecureWALCellCodec` the method `createRpcClientRTEDuringConnectionSetup` gives a NoSuchMethodException for the provided default codec value. 
 
Since there was an exception, the code inside try block is not executed which contained the below code line 
{code:java}
try (AbstractRpcClient<?> client = createRpcClientRTEDuringConnectionSetup(clientConf))
{       rpcServer.start();       . . .     }
{code}
 


When finally block is executed which contains 
{code:java}
finally
{       rpcServer.stop();     }
{code}
 


The method `rpcServer.stop()` eventually makes call to `executor.shutdown()`. A null pointer exception is returned as  `rpcServer.scheduler.executor`  was never instantiated, which happens inside `rpcServer.start()`
 
{code:java}
java.lang.NullPointerException
at org.apache.hadoop.hbase.ipc.FifoRpcScheduler.stop(FifoRpcScheduler.java:71)
at org.apache.hadoop.hbase.ipc.SimpleRpcServer.stop(SimpleRpcServer.java:441)
at org.apache.hadoop.hbase.ipc.AbstractTestIPC.testRTEDuringConnectionSetup(AbstractTestIPC.java:194){code}
Bug: Expose DelegatingRpcScheduler as IA.LimitedPrivate
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jan/24 17:26

We have DelegatingRpcScheduler in src/test of hbase-server. RpcScheduler itself is IA.LimitedPrivate, and in 2.6.0 we are pushing an incompatible change from HBASE-27144.

We can limit the impact of breaking changes like this by exposing DelegatingRpcScheduler to users. Users can extend this class and only override the pieces that they care about, thus reducing the surface area of compatibility issues.