av9ash/GitBugs huggingface dataset:
Bug: Avoid resetting bucket cache during restart if inconsistency is observed for some blocks.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Oct/24 11:39
Affects Version/s: 2.6.0

During the execution of persistence of backing map into the persistence file, the backing map is not guarded by a lock against caching of blocks and block evictions.

Hence, some of the block entries in the backing map in persistence may not be consistent with the bucket cache.
During, the retrieval of the backing map from persistence if an inconsistency is detected, the complete bucket cache is discarded and is rebuilt.

One of the errors, seen, is, as mentioned below:


{code:java}
2024-09-30 08:58:33,840 WARN org.apache.hadoop.hbase.io.hfile.bucket.BucketCache: Can't restore from file[/hadoopfs/ephfs1/bucketcache.map]. The bucket cache will be reset and rebuilt. Exception seen:
org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocatorException: Couldn't find match for index 26 in free list
        at org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator$Bucket.addAllocation(BucketAllocator.java:140)
        at org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator.<init>(BucketAllocator.java:406)
        at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.retrieveFromFile(BucketCache.java:1486)
        at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.lambda$startPersistenceRetriever$0(BucketCache.java:377)
        at java.base/java.lang.Thread.run(Thread.java:840) {code}

This retrieval can be optimised to only discard the inconsistent entries in the persistent backing map and retain the remaining entries. The bucket cache validator will throw away the inconsistent entry from the backing map.

Thanks,
Janardhan
Bug: hbase 2.5.10 execute "./bin/hbase shell"  occurred an error  jar-bootstrap.rb
Status: Open | Priority: Major | Resolution: 
Created: 16/Oct/24 06:37
Affects Version/s: 2.5.10

{code:java}
$ $HBASE_HOME/bin/hbase shell -n
  This file has been superceded by packaging our ruby files into a jar
  and using jruby's bootstrapping to invoke them. If you need to
  source this file fo some reason it is now named 'jar-bootstrap.rb' and is
  located in the root of the file hbase-shell.jar and in the source tree at
  'hbase-shell/src/main/ruby' {code}
Bug: Ranger doesn't work with hbase 2.6 when using hadoop 3.3.6
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 08/Oct/24 12:21
Affects Version/s: 2.5.0

Hello, 
I have this issue on my stack
{code:java}
2024-09-23T08:12:51.336Z - ERROR [PolicyRefresher(serviceName=hbase-tdp)-196:Errors@168] - The following errors and warnings have been detected with resource and/or provider classes:   SEVERE: Missing dependency for method public void com.sun.jersey.json.impl.provider.entity.JSONRootElementProvider.setConfiguration(com.sun.jersey.core.util.FeaturesAndProperties) at parameter at index 0   SEVERE: Missing dependency for method public void com.sun.jersey.json.impl.provider.entity.JSONListElementProvider.setConfiguration(com.sun.jersey.core.util.FeaturesAndProperties) at parameter at index 0   SEVERE: Missing dependency for method public void com.sun.jersey.json.impl.provider.entity.JSONRootElementProvider.setConfiguration(com.sun.jersey.core.util.FeaturesAndProperties) at parameter at index 0   SEVERE: Missing dependency for method public void com.sun.jersey.json.impl.provider.entity.JSONRootElementProvider.setConfiguration(com.sun.jersey.core.util.FeaturesAndProperties) at parameter at index 0   SEVERE: Missing dependency for field: javax.ws.rs.core.UriInfo com.sun.jersey.server.impl.template.ViewableMessageBodyWriter.ui   SEVERE: Missing dependency for field: com.sun.jersey.spi.template.TemplateContext com.sun.jersey.server.impl.template.ViewableMessageBodyWriter.tc   SEVERE: Missing dependency for method public void com.sun.jersey.json.impl.provider.entity.JSONListElementProvider.setConfiguration(com.sun.jersey.core.util.FeaturesAndProperties) at parameter at index 0   SEVERE: Missing dependency for method public void com.sun.jersey.json.impl.provider.entity.JacksonProviderProxy.setFeaturesAndProperties(com.sun.jersey.core.util.FeaturesAndProperties) at parameter at index 1   SEVERE: Missing dependency for method public void com.sun.jersey.json.impl.provider.entity.JSONJAXBElementProvider.setConfiguration(com.sun.jersey.core.util.FeaturesAndProperties) at parameter at index 0   SEVERE: Missing dependency for method public void com.sun.jersey.json.impl.provider.entity.JSONJAXBElementProvider.setConfiguration(com.sun.jersey.core.util.FeaturesAndProperties) at parameter at index 0
 {code}
The issue is that jersey-json and jersey-jar are missing on the hbase plugin

The solution is to add this lines to distro/src/main/assembly/hbase-agent.xml under 
lib/ranger-hbase-plugin-impl
{code:java}
<include>com.sun.jersey:jersey-json:jar:${jersey-bundle.version}</include>Line58added              
<include>com.sun.jersey:jersey-server:jar:${jersey-bundle.version}</include>Line59added  
<include>com.sun.jersey:jersey-servlet:jar:${jersey-bundle.version}</include> {code}
Bug: Add metrics around FSFT backed file reads and updates
Status: Open | Priority: Major | Resolution: 
Created: 07/Oct/24 14:45

With HBASE-27826, number of interactions via SFT layer would increase, as all the interactions of StoreFiles going forward would require StoreFileTracker context and that would increase the number of reads to inherent backed file of FileBaseStoreFileTracker, this would add metrics on number of times we read/update backed file
Bug: Performance regression from 2.5.8 to 2.6.0 when seeking SEEK_NEXT_USING_HINT to a next column family.
Status: Open | Priority: Major | Resolution: 
Created: 04/Oct/24 13:42
Affects Version/s: 2.6.0

We have a custom hbase filter that seeks (SEEK_NEXT_USING_HINT) to a next column family (called "cf" in our case) based on data in a cell in a prior column family (called "bf_slicing"). We upgraded to hbase 2.6.0 from 2.5.8, the change in this ticket https://issues.apache.org/jira/browse/HBASE-27788 caused a significant performance degradation (from instant seeking to the next family to traversing the entire bf_slicing family).

We traced the cause to the following:

When comparing families here, the 'cf' family is ordered lower than 'bf_slicing' due to its length, causing the first column family ("bf_slicing") to be fully traversed. The offending code is here: [https://github.com/apache/hbase/pull/5171/files#diff-1ec9654ed8e00f46e11430fc726f8351db59597723efa0bf1e268196f00244c6R54]

The original story (HBASE_27788) mentions no seeking should be done outside a column family, but our use case seems legitimate in the data model, so we think this is a bug.

 The attached screenshot shows a jprofiler dump with invocation counts showing it is traversing all data (going to 12M cells in this case), this is on hbase 2.6.0

[^Screenshot from 2024-10-04 15-52-32.png]
Bug: Explore adopting Revapi for compatibility reporting in PRs and RCs
Status: Open | Priority: Major | Resolution: 
Created: 04/Oct/24 07:55

The first release candidate of 2.6.1 had quite a few incompatible changes. It would be great to surface issues at the time when changes are bring introduced, instead of at release time. Our current API Compatibility report is quite nice, but is driven by a program that carries a restrictive license and so we haven't been able to deploy it widely. It happens that I was looking at another ASF project this week and noticed that they do have compatibility checking in pre-commit, powered by [Revapi|https://revapi.org/], which has the Apache v2 license.
Bug: Automate organization of pom.xml using spotless
Status: Open | Priority: Major | Resolution: 
Created: 03/Oct/24 09:37

In our codebase, dependencies in POM files should be auto organized such that test dependencies are listed after compile dependencies. To ensure this is consistently applied across the project, we will utilize the {{sortPom}} functionality provided by the Spotless plugin.

See discussion here: [https://github.com/apache/hbase/pull/6258#discussion_r1784337008]

 

CC: [~ndimiduk], [~stoty]
Bug: RefCnt Leak error when closing a HalfStoreFileReader
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Oct/24 11:34
Affects Version/s: 2.7.0

In HBASE-28596 we have added ability for references to get resolved to the original file blocks in bucket cache. As part of this, we had to modify HalfStoreFileReader.close method, to create a scanner and seek to boundary cell in order to get the related offset and calculate the limiting offset for blocks we want to evict. We missed close the scanner instance there, which then cause the refcount leaks reported as below:

{noformat}
2024-09-25 14:24:51,292 ERROR org.apache.hbase.thirdparty.io.netty.util.ResourceLeakDetector: LEAK: RefCnt.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:  
Created at:
        org.apache.hadoop.hbase.nio.RefCnt.<init>(RefCnt.java:59)
        org.apache.hadoop.hbase.nio.RefCnt.create(RefCnt.java:54)
        org.apache.hadoop.hbase.nio.ByteBuff.wrap(ByteBuff.java:550)
        org.apache.hadoop.hbase.io.ByteBuffAllocator.allocate(ByteBuffAllocator.java:357)
        org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine.read(FileIOEngine.java:134)
        org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getBlock(BucketCache.java:666)
        org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.getBlock(CombinedBlockCache.java:98)
        org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.getCachedBlock(HFileReaderImpl.java:1102)
        org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1287)
        org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1248)
        org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$CellBasedKeyBlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:318)
        org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:670)
        org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:623)
        org.apache.hadoop.hbase.io.HalfStoreFileReader.close(HalfStoreFileReader.java:368)
        org.apache.hadoop.hbase.regionserver.HStore.removeCompactedfiles(HStore.java:2352)
        org.apache.hadoop.hbase.regionserver.HStore.closeAndArchiveCompactedFiles(HStore.java:2314)
        org.apache.hadoop.hbase.regionserver.CompactedHFilesDischargeHandler.process(CompactedHFilesDischargeHandler.java:41)
        org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
{noformat}
Bug: Resuming a release build from stage "publish-dist" fails due to missing CHANGES.md and RELEASENOTES.md
Status: Open | Priority: Major | Resolution: 
Created: 01/Oct/24 06:58

The do-release scripts seem to be built to support resuming a partial release from a "stage". I found that resuming a release at the "publish-dist" stage fails due to missing CHANGES.md and RELEASENOTES.md. The script appears to assume the files are present in the output directory, which won't be the case if the output directory is cleaned after the previous failed attempt. These files are populated via the "tag" stage.
Bug: Refactor BackupHFileCleaner + fix test
Status: In Progress | Priority: Minor | Resolution: 
Created: 12/Dec/24 15:54
Affects Version/s: 2.6.0

The TestBackupHFileCleaner test is broken, as it uses a different API to register bulk loads than the API that is actually used to register bulk loads during backups. The test also incorrectly closes the FS of the HBaseTestingUtil, causing this test to block for about 5 minutes during shutdown.

Both the test and BackupHFileCleaner itself are overly convoluted and should be cleaned up. This is another intermediate step towards fixing HBASE-28715.
Bug: [HBCK2] filesystem command always report region hole and doesn't exit automatically
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Oct/24 07:54

1. When executing hbck2 filesystem command, it always reports region hole for all region like:
{code:java}
ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo and region dir in hdfs to plug the hole.
ERROR: Found inconsistency in table hbase:meta
ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo and region dir in hdfs to plug the hole.
ERROR: Found inconsistency in table hbase:namespace{code}
  The reason for this problem is that the region directories are not loaded at all. In hbck1, the process will load all region directories by default.

2. After the check is completed, the process will not exit automatically because the hbasefsck threads are not daemon threads.
Bug: Fix memStoreSize metric for a table and add heap and off-heap size metrics
Status: In Progress | Priority: Minor | Resolution: 
Created: 21/Nov/24 03:17

The "memStoreSize" metric for a table is incorrect as it reports the sum of the data size, the heap size, and the off-heap size, while "memStoreSize" metric for a region or a region server is just the data size.

https://github.com/apache/hbase/blob/e082405d8e7fa5c9b89e7af3153f022731168ecb/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsTableWrapperAggregateImpl.java#L79-L80

We should fix it to report only the data size for consistency and add additional metrics, similarly to HBASE-27892, not to lose the extra information for understanding the flush behavior.

* {{memStoreHeapSize}}
* {{memStoreOffHeapSize}}
Bug: Prefetch thread shouldn't run for master store
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Nov/24 10:43
Affects Version/s: 2.6.1

The master store is hosted on Master processes. Since masters don't have a BlockCache, we shouldn't run a prefetch thread at all, when opening the master store region.

Currently, this is causing a NoSuchElementException to be logged in master logs, which although harmless, can be confusing for operators:
{noformat}
2024-10-23 15:23:29,236 WARN org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch path=s3a://odx-qe-bucket/cc-odx-coy6zg/cod-udzhubgp5l5k/hbase/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/336a510d9f9a472c9e2a8f3e00352e3b, offset=0, end=82012
java.util.NoSuchElementException: No value present
	at java.base/java.util.Optional.get(Optional.java:143)
	at org.apache.hadoop.hbase.io.hfile.HFilePreadReader$1.run(HFilePreadReader.java:73)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecute {noformat}
Bug: Test results for backwards compatibility tests are not distinguishable in Jenkins reports
Status: Open | Priority: Minor | Resolution: 
Created: 07/Nov/24 08:22

The failures are listed like this:
 * [health checks / yetus jdk17 hadoop3 backwards compatibility checks / org.apache.hadoop.hbase.regionserver.wal.TestAsyncWALReplay.(?)|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1201/testReport/junit/org.apache.hadoop.hbase.regionserver.wal/TestAsyncWALReplay/health_checks___yetus_jdk17_hadoop3_backwards_compatibility_checks_______2/]

We can determine which version failed by checking the console output of the individual maven jobs, but this is not ideal.
Bug: Run Hadoop3 backward compatibility checks in parallel
Status: Open | Priority: Minor | Resolution: 
Created: 25/Oct/24 05:53

Adding the backwards compatibility cheks has increased the Nightly check runtime from ~9 hours to ~12 hours. My initial estimate for running the devTests was two hours, but it's actually four hours on the test runners.

Run the backwards compatibility tests in parallel.

Also consider running the older JDK8,JDK11 tests in parallel, perhaps by moving everything into a matrix section.
Bug: Random object created and used only once in org.apache.hadoop.hbase.util.LoadTestKVGenerator.getValueForRowColumn(int, byte[][])
Status: Open | Priority: Minor | Resolution: 
Created: 16/Oct/24 18:08

An unrelated spotbug warning popped up in one of precommit test. [https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6342/4/artifact/yetus-general-check/output/branch-spotbugs-hbase-server-warnings.html]

 
h2. Bad practice Warnings
||Code||Warning||
|DMI|Random object created and used only once in org.apache.hadoop.hbase.util.LoadTestKVGenerator.getValueForRowColumn(int, byte[][])|
| |[Bug type DMI_RANDOM_USED_ONLY_ONCE (click for details)|https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6342/4/artifact/yetus-general-check/output/branch-spotbugs-hbase-server-warnings.html#DMI_RANDOM_USED_ONLY_ONCE]
In class org.apache.hadoop.hbase.util.LoadTestKVGenerator
In method org.apache.hadoop.hbase.util.LoadTestKVGenerator.getValueForRowColumn(int, byte[][])
Called method java.util.Random.nextBytes(byte[])
At LoadTestKVGenerator.java:[line 111]|
h1. Details
h2. DMI_RANDOM_USED_ONLY_ONCE: Random object created and used only once

This code creates a java.util.Random object, uses it to generate one random number, and then discards the Random object. This produces mediocre quality random numbers and is inefficient. If possible, rewrite the code so that the Random object is created once and saved, and each time a new random number is required invoke a method on the existing Random object to obtain it.

If it is important that the generated Random numbers not be guessable, you _must_ not create a new Random for each random number; the values are too easily guessable. You should strongly consider using a java.security.SecureRandom instead (and avoid allocating a new SecureRandom for each random number needed).
Bug: Skip excessive evaluations of LINK_NAME_PATTERN and REF_NAME_PATTERN regular expressions
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Oct/24 13:02
Affects Version/s: 2.6.0

To test if a file is a link file, HBase checks if its file name matches the regex
{code:java}
^(?:((?:[_\p{Digit}\p{IsAlphabetic}]+))(?:\=))?((?:[_\p{Digit}\p{IsAlphabetic}][-_.\p{Digit}\p{IsAlphabetic}]*))=((?:[a-f0-9]+))-([0-9a-f]+(?:(?:_SeqId_[0-9]+_)|(?:_del))?)$
{code}
To test if an HFile has a "reference name," HBase checks if its file name matches the regex
{code:java}
^([0-9a-f]+(?:(?:_SeqId_[0-9]+_)|(?:_del))?|^(?:((?:[_\p{Digit}\p{IsAlphabetic}]+))(?:\=))?((?:[_\p{Digit}\p{IsAlphabetic}][-_.\p{Digit}\p{IsAlphabetic}]*))=((?:[a-f0-9]+))-([0-9a-f]+(?:(?:_SeqId_[0-9]+_)|(?:_del))?)$)\.(.+)$
{code}
Matching against these big regexes is computationally expensive. HBASE-27474 introduced (in 2.6.0) [code in a hot path|https://github.com/apache/hbase/blob/1602c531b245b4d455b48161757cde2ec3d1930b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java#L1716] in {{HFileReaderImpl}} that checks whether an HFile is a link or reference file while deciding whether to cache blocks from that file. In flamegraphs taken at my company during performance tests, this meant that these regex evaulations take 2-3% of the CPU time on a busy RegionServer.

Later, the hot-path invocation of the regexes was removed in HBASE-28596 in branch-2 and later, but not branch-2.6, so only the 2.6.x series suffers the performance regression. Nonetheless, all invocations of these regexes are still unnecessarily expensive and can be fast-failed easily.

The link name pattern contains a literal "=", so any string that does not contain a "=" can be assumed to not match the regex. The reference name pattern contains a literal ".", so any string that does not contain a "." can be assumed to not match the regex. This optimization is mostly helpful in 2.6.x, but is valid in all branches.

Running performance tests of this optimization removed the regex evaluations from my flamegraphs entirely, and reduced query latency by 15%. Some charts are attached.
Bug: Remove redundant hadoop-two.version specification in shaded poms
Status: Resolved | Priority: Trivial | Resolution: Duplicate
Created: 07/Nov/24 06:30

On branch-2.x, we explicitly set the hadoop-hdfs version to ${hadoop-two.version}
We don't do that for the rest of the hadoop artifacts, and the managed version is already set to 2.10.2 when using the Hadoop 2 profile, so this is unneccessary.
Bug: Introduce a general 'local region' to store data on master
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 21/May/20 06:20

We already have a local region to store the procedure data and when implementing HBASE-11288, splittable meta, we are thinking of also storing the data for root table in a local region.

Now in the patch for HBASE-24388, we introduced another local region to store the data for root table, but maybe it is better to store the procedure data and root table together in a single region(with different families).

And this should be done before 2.3.0, to prevent shipping the procedure store region out in a release. Set it a blocker for 2.3.0.

Patch will be available soon.
Bug: The behavior of RegionInfoBuilder.newBuilder(RegionInfo) is strange
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 03/Jun/20 07:25

In this method, we will copy all the fields, including RegionName and EncodedRegionName of the RegionInfo, so after we changed some other fields like start key or stop key or replica id, and then create a new RegionInfo, the RegionName of the RegionInfo will not be regenerated and cause problem.