av9ash/GitBugs huggingface dataset:
Bug: Modify default value of hbase.bucketcache.persistence.chunksize to 10K
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/24 11:11
Affects Version/s: 2.7.0

Currently, the default value of the configuration parameter

"hbase.bucketcache.persistence.chunksize" is 10 million (10000000). 
This is the number of block entries that are processed during the persistence of bucket-cache backing map to the persistence file. During the testing, it was found that, this high number of chunksize resulted in high heap utilisation in region servers leading to longer GC pauses which also led to server crashes intermittently.

When the value of this configuration is set to 10K(10000), the cache remains stable. No GC delays are observed. Also no server crashes are observed.
The jmap outputs collected against the regionservers showed reduced memory utilisation from 4.5-5GB to 1-1.5GB for the objects related to persistence code.

Hence, we need to adjust the default value of this configuration parameter to 10K.

Thanks,
Janardhan
Bug: Exception handling during retrieval of bucket-cache from persistence.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/24 04:41
Affects Version/s: 2.7.0

During the retrieval of bucket cache from the persistence file during the startup, it was observed that, if an exception, other than, the IOException occurs, the bucket cache internal members remain uninitialised and cause the bucket to remain unusable. The exception is not logged in the trace file and the retrieval thread exits without initialising the bucket-cache.

Also, the NullPointerExceptions are seen when, trying to use the cache.
{code:java}
2024-09-10 14:33:30,020 ERROR org.apache.hadoop.hbase.io.hfile.bucket.BucketCache: WriterThread encountered error
java.lang.NullPointerException
        at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$RAMQueueEntry.writeToCache(BucketCache.java:1975)
        at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.doDrain(BucketCache.java:1298) {code}
 
{code:java}
2024-09-13 07:01:05,964 ERROR org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: Error getting metrics from source RegionServer,sub=Server
java.lang.NullPointerException
        at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getFreeSize(BucketCache.java:1819)
        at org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.getFreeSize(CombinedBlockCache.java:179)
        at org.apache.hadoop.hbase.regionserver.MetricsRegionServerWrapperImpl.getBlockCacheFreeSize(MetricsRegionServerWrapperImpl.java:308)
        at org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceImpl.addGaugesToMetricsRecordBuilder(MetricsRegionServerSourceImpl.java:525)
        at org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceImpl.getMetrics(MetricsRegionServerSourceImpl.java:333) {code}
All type of exceptions need to be handled gracefully.
All types of exceptions must be logged to the trace file.
The bucket cache needs to reinitialised and made usable.

Thanks,
Janardhan
Bug: ExportSnapshot failed if reference file presented
Status: Open | Priority: Major | Resolution: 
Created: 30/Aug/24 09:48
Affects Version/s: 2.4.14

ExportSnapshot tasks failed due to FNFE.  

If there are reference files in data.manifest, they are converted to referred files, so same referred files are handled in multi map tasks. It will cause FNFE if the same file copied in different map tasks concurrently.
Bug: Introduce row statistics coprocessor example
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/24 18:25
Affects Version/s: 2.0.0

Goal:

Add a coprocessor to HBase that allows administrators to track high level statistics on the rows and cells in their HBase tables. Administrators can load this coprocessor into their RegionServers if they wish to gain more visibility into the shape of their data in HBase.

At my day job, we've leveraged the statistics from this coprocessor to automatically configure more optimal block sizes and smarter compaction schedules for our fleet of nearly 200 HBase clusters.

Context:

Since HBase tables can store terabytes or even petabytes of data, HBase administrators often have incomplete information about the data stored in their HBase tables. Without a comprehensive understanding of the shape of their data, it can be difficult for administrators to configure clusters for a desired level of performance and/or reliability. Row statistics have the potential to supercharge HBase management.

[Design doc|https://docs.google.com/document/d/1oaNAZUER5zO8yivmzRBVAMmL6r2cYiJn9YCbDe14LMw/edit#heading=h.nch5d72p27ex]
Bug: hbase-connectors fails to build with hbase 2.6.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/24 11:31

hbase-connectors fails to build with hbase 2.6.0

{code:java}
[INFO] Reactor Summary for Apache HBase Connectors 1.1.0-SNAPSHOT:
[INFO] 
[INFO] Apache HBase Connectors ............................ SUCCESS [  4.377 s]
[INFO] Apache HBase - Kafka ............................... SUCCESS [  0.116 s]
[INFO] Apache HBase - Model Objects for Kafka Proxy ....... SUCCESS [  3.222 s]
[INFO] Apache HBase - Kafka Proxy ......................... FAILURE [  8.305 s]
[INFO] Apache HBase - Spark ............................... SKIPPED
[INFO] Apache HBase - Spark Protocol ...................... SKIPPED
[INFO] Apache HBase - Spark Protocol (Shaded) ............. SKIPPED
[INFO] Apache HBase - Spark Connector ..................... SKIPPED
[INFO] Apache HBase - Spark Integration Tests ............. SKIPPED
[INFO] Apache HBase Connectors - Assembly ................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  16.703 s
[INFO] Finished at: 2024-08-17T11:29:20Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project hbase-kafka-proxy: Compilation failure
[ERROR] /workspaces/hbase-connectors/kafka/hbase-kafka-proxy/src/main/java/org/apache/hadoop/hbase/kafka/KafkaBridgeConnection.java:[169,31] <anonymous org.apache.hadoop.hbase.kafka.KafkaBridgeConnection$1> is not abstract and does not override abstract method setRequestAttribute(java.lang.String,byte[]) in org.apache.hadoop.hbase.client.TableBuilder {code}
Bug: Backup restores are broken if the backup has moved locations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Sep/24 17:57
Affects Version/s: 2.6.1

My company runs a few hundred HBase clusters. We want to take backups everyday in one public cloud region, and then use said cloud's native replication solution to "backup our backups" in a secondary region. This is how we plan for region-wide disaster recovery.

This system should work, but doesn't because of the way that BackupManifests are constructed.

Backing up a bit (no pun intended): when we replicate backups verbatim, the manifest file continues to point to the original backup root. This shouldn't matter, because when taking a restore one passes a RestoreRequest to the RestoreTablesClient — and this RestoreRequest includes a BackupRootDir field. This works as you would expect initially, but eventually we build a BackupManifest that fails to interpolate this provided root directory and, instead, falls back to what it finds on disk in the backup (which would point back to the primary backup location, even if reading a replicated backup).

To fix this, I'm proposing that we properly interpolate the request's root directory field when building BackupManifests.
Bug: Concurrent execution of normalizer operations on tables in RegionNormalizerWorker
Status: Open | Priority: Major | Resolution: 
Created: 14/Aug/24 08:10

Recently, I have been managing the large tables in the HBase cluster by enabling the normalizer to set the size of the regions and keep the number of regions within a reasonable range.
The current code retrieves tables from RegionNormalizerWorkQueue and performs normalization operations on the tables in series. When there are multiple large tables in a cluster that need to be managed, the efficiency will be very low.
I have confirmed that each split or merge plan generated by each table during the normalizer process will be limited by RateLimiter, so I think it is reasonable to perform table normalizers concurrently.
In terms of implementation, create a thread pool for executing tasks in RegionNormalizerWorker, with a default value of 1 for the number of thread pools, and provide a parameter that can be configured to other values.
Bug: ParseException may occur when getting the fileDate of the mob file recovered through snapshot
Status: Open | Priority: Major | Resolution: 
Created: 24/Sep/24 10:58
Affects Version/s: 2.4.13

The task ExpiredMobFileCleaner may occur ParseException when parsing MOB file recovered through snapshot, causing these expired MOB file cannot be deleted.

 

The Reason:
The task ExpiredMobFileCleaner obtain the MOB file creation time by parsing the MOB filename.
In regular MOB table, the 32nd to 40th characters of the MOB filename indicate the file creation time, ExpiredMobFileCleaner can get the creation time of MOB file by obtaining these characters.
However, in MOB tables recovered through snapshot, the format of MOB filename is tableName-mobregionaname-hfilename，so ExpiredMobFileCleaner may not be able to obtain the creation time of MOB file by obtaining the characters at the above location. So, in this situation, ParseException will occur, causing these expired MOB file cannot be deleted finally.
Bug: No data available when scan cross-region and setReversed(true)  in Spark on HBase sc.newAPIHadoopRDD
Status: Open | Priority: Major | Resolution: 
Created: 13/Jul/24 08:20
Affects Version/s: 2.2.3

Using below scala code to scan data in Spark on HBase:

    val scan = new Scan()
    scan.withStartRow(Bytes.toBytes(startKey))
    scan.withStopRow(Bytes.toBytes(stopKey))

    if (reversed) {
      scan.setReversed(true)
    }
    val conf = ConnectionFactory.createConnection.getConfiguration
    conf.set(TableInputFormat.INPUT_TABLE, tableName)
    conf.set(TableInputFormat.SCAN, TableMapReduceUtil.convertScanToString(scan))

   val rdd = sc.newAPIHadoopRDD(conf, classOf[TableInputFormat], classOf[ImmutableBytesWritable], classOf[Result])

 

1.When scan cross-region  without reversed=true, the scan can be performed normally and the result can be obtained.

2.When scan do not cross-region but with reversed=true, the scan can be performed normally and the result can be obtained.

3.When scan cross-region  with reversed=true,  the result is empty.
Bug: Should call ProcedureSchduler.completionCleanup for non-root procedure too
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Sep/24 14:46

Per the discussion in this PR

https://github.com/apache/hbase/pull/6247

And the related issue HBASE-28830, it seems incorrect that we only call cleanup for non-root procedures.

This issue aims to see if there are any issues we call this method for every procedure.
Bug: Add a metric of the amount of data written to WAL to determine the pressure of replication
Status: Open | Priority: Major | Resolution: 
Created: 18/Sep/24 09:55

Add a metric of the amount of data written to WAL to determine the pressure of replication.

Combined with the replication shipped size metric, the user can determine how many RegionServers are needed to meet the data(WAL) writing requirements, that is to achieve the goal of no replication lag.

 

In the postAppend method of ReplicationSourceWALActionListener, the amount of WAL data that needs to be replicated is counted, and the filter logic in replication is reused.

Alos add a hbase.replication.walAppendsize.metric.enable=false config to disable this metric, if it has some side effect.
Bug: Protect hbase:meta from any hotspotting regions by balancing them to different region servers
Status: Open | Priority: Major | Resolution: 
Created: 09/Aug/24 04:53

During some of the incidents, it is observed that when hbase:meta is colocated with high load (or hotspotting) region, that makes meta unavailable and causes availability issues. This item is to provide a way to load balance such regions such that meta is not impacted.
Bug: Change the generic type for ObserverContext from 'RegionCoprocessorEnvironment' to '? extends RegionCoprocessorEnvironment' in RegionObserver
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Sep/24 13:54

This will be a breaking change for coprocessor implementation, but the ability of region observer is not changed, so I think it is OK to include this in 3.0.0 release, as we have already changed the coprocessor protobuf to the relocated one, which already breaks lots of coprocessors.
Bug: AsyncFSWAL is broken when running against hadoop 3.4.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jul/24 02:20

{noformat}
2024-07-10T10:09:33,161 ERROR [master/localhost:0:becomeActiveMaster {}] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(258): Couldn't properly initialize access to HDFS internals. Please update your WAL Provider to not make use of the 'asyncfs' provider. See HBASE-16110 for more information.
java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.beginFileLease(long,org.apache.hadoop.hdfs.DFSOutputStream)
        at java.lang.Class.getDeclaredMethod(Class.java:2675) ~[?:?]
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createLeaseManager(FanOutOneBlockAsyncDFSOutputHelper.java:175) ~[hbase-asyncfs-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.<clinit>(FanOutOneBlockAsyncDFSOutputHelper.java:252) ~[hbase-asyncfs-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]
        at java.lang.Class.forName0(Native Method) ~[?:?]
        at java.lang.Class.forName(Class.java:375) ~[?:?]
        at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:149) ~[classes/:?]
        at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:174) ~[classes/:?]
        at org.apache.hadoop.hbase.wal.WALFactory.<init>(WALFactory.java:262) ~[classes/:?]
        at org.apache.hadoop.hbase.wal.WALFactory.<init>(WALFactory.java:231) ~[classes/:?]
        at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:383) ~[classes/:?]
        at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:135) ~[classes/:?]
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1003) ~[classes/:?]
        at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2524) ~[classes/:?]
        at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:613) ~[classes/:?]
        at org.apache.hadoop.hbase.trace.TraceUtil.lambda$tracedRunnable$2(TraceUtil.java:155) ~[hbase-common-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]
        at java.lang.Thread.run(Thread.java:840) ~[?:?]
{noformat}
Bug: Support partial results in AggregateImplementation and AsyncAggregationClient
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Aug/24 16:41
Affects Version/s: 2.6.0

Currently there is a gap in the coverage of HBase's quota-based workload throttling. Requests sent by {{[Async]AggregationClient}} reach {{AggregateImplementation}}. This then executes Scans in a way that bypasses the quota system. We see issues with this at Hubspot where clusters suffer under this load and we don't have a good way to protect them.

In this ticket I'm teaching {{AggregateImplementation}} to optionally stop scanning when a throttle is violated, and send back just the results it has accumulated so far. In addition, it will send back a row key to {{AsyncAggregationClient}}. When the client gets a response with a row key, it will sleep in order to satisfy the throttle, and then send a new request with a scan starting at that row key. This will have the effect of continuing the work where the last request stopped.

This feature will be unconditionally enabled by {{AsyncAggregationClient}} once this ticket is finished. {{AggregateImplementation}} will not assume that clients support partial results, however, so it can keep supporting older clients. For clients that do not support partial results, throttles will not be respecting, and results will always be complete.

This feature was [first proposed on the mailing list|https://lists.apache.org/thread/1vqnxb71z7swq2cogz4qg3cn6b10xp4v]. Builds on work in HBASE-28346.
Bug: Missing dependency
Status: Resolved | Priority: Major | Resolution: Not A Bug
Created: 23/Sep/24 09:33
Affects Version/s: 2.6.0

Hello, I'm using hbase 2.6 with hadoop 3.3.6 and ranger 2.5 and i have the following errors in my logs :
{code:java}
2024-09-23T08:12:51.336Z - ERROR [PolicyRefresher(serviceName=hbase-tdp)-196:Errors@168] - The following errors and warnings have been detected with resource and/or provider classes:   SEVERE: Missing dependency for method public void com.sun.jersey.json.impl.provider.entity.JSONRootElementProvider.setConfiguration(com.sun.jersey.core.util.FeaturesAndProperties) at parameter at index 0   SEVERE: Missing dependency for method public void com.sun.jersey.json.impl.provider.entity.JSONListElementProvider.setConfiguration(com.sun.jersey.core.util.FeaturesAndProperties) at parameter at index 0   SEVERE: Missing dependency for method public void com.sun.jersey.json.impl.provider.entity.JSONRootElementProvider.setConfiguration(com.sun.jersey.core.util.FeaturesAndProperties) at parameter at index 0   SEVERE: Missing dependency for method public void com.sun.jersey.json.impl.provider.entity.JSONRootElementProvider.setConfiguration(com.sun.jersey.core.util.FeaturesAndProperties) at parameter at index 0   SEVERE: Missing dependency for field: javax.ws.rs.core.UriInfo com.sun.jersey.server.impl.template.ViewableMessageBodyWriter.ui   SEVERE: Missing dependency for field: com.sun.jersey.spi.template.TemplateContext com.sun.jersey.server.impl.template.ViewableMessageBodyWriter.tc   SEVERE: Missing dependency for method public void com.sun.jersey.json.impl.provider.entity.JSONListElementProvider.setConfiguration(com.sun.jersey.core.util.FeaturesAndProperties) at parameter at index 0   SEVERE: Missing dependency for method public void com.sun.jersey.json.impl.provider.entity.JacksonProviderProxy.setFeaturesAndProperties(com.sun.jersey.core.util.FeaturesAndProperties) at parameter at index 1   SEVERE: Missing dependency for method public void com.sun.jersey.json.impl.provider.entity.JSONJAXBElementProvider.setConfiguration(com.sun.jersey.core.util.FeaturesAndProperties) at parameter at index 0   SEVERE: Missing dependency for method public void com.sun.jersey.json.impl.provider.entity.JSONJAXBElementProvider.setConfiguration(com.sun.jersey.core.util.FeaturesAndProperties) at parameter at index 0
 {code}

Do you have any ideas on how I can resolve that ?
Bug: Migrate to GitHub Checks API for PR Precommit
Status: Open | Priority: Major | Resolution: 
Created: 18/Sep/24 09:01

Our PR pre-commit check is configured to leave comments on the PR with its build results. Especially for our 2.x branches, it is leaving upwards of 4 comments per build. This results in a lot of bot spam, which is distracting from human conversations. I've mitigated the issue via HBASE-28642, but it would be better if the bot didn't leave comments, but instead used the [Checks API|https://docs.github.com/en/rest/checks]. I believe that Yetus is already able to do this, and is configured to do so. I think we need to engage with INFRA to get our auth token adjusted.
Bug: [hbase-thirdparty] Finish hbase-thirdparty 4.1.9 release
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Sep/24 15:05

Release the artifacts on repository.apache.org
Move the binaries from dist-dev to dist-release
Push tag 4.1.9RC0 as tag rel/4.1.9
Release thirdparty-4.1.9 on JIRA https://issues.apache.org/jira/projects/HBASE/versions/12355077
Add release data on https://reporter.apache.org/addrelease.html?hbase
Send announcement email
Bug: apache.hadoop.hbase.DoNotRetryIOException: java.lang.UnsatisfiedLinkError: org-apache.hadoop.util.NativeCodeLoa der.buildSupportsSnappy()Z
Status: Open | Priority: Major | Resolution: 
Created: 12/Aug/24 12:59
Affects Version/s: 2.4.16

.When I create a snappy compression table, it appeared exception:

ERROR: org. apache.hadoop.hbase.DoNotRetryIOException: java.lang.UnsatisfiedLinkError: org-apache.hadoop.util.NativeCodeLoa

der.buildSupportsSnappy()Z Set hbase.table.sanity.checks to false at conf or table descriptor if you want to bypass sanity checks

at org.apache.hadoop.hbase.util.TableDescriptorChecker.warnOrThrowExceptionForFailure(TableDescriptorChecker.java:

337）

at org.apache.hadoop.hbase.util.TableDescriptorChecker.checkCompression(TableDescriptorChecker-java:304) at org-apache.hadoop.hbase.util.TableDescriptorChecker.sanityCheck(TableDescriptorChecker-java:114)

at org. apache.hadoop.hbase.master.HMaster. createrableHMaster-Java: 2094)

at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:696)

at org.apache.hadoop.hbase.shaded.protobuf-generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.

java)

b b b

b
 - b

at org.apache.hadoop.hbase. ipc.RpcServer.call(RpcServer.java:387)

at org-apache.hadoop.hbase.ipc.CallRunner.run(CallRunner-java:132)

at org-apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369)

at org-apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349)

Caused by: org.apache.hadoop.hbase.DoNotRetryI0Exception: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCod

eLoader.buildSupportsSnappy()Z

at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:100)

at org.apache.hadoop.hbase.util.TableDescriptorChecker.checkCompression(TableDescriptorChecker.java:300)

... 8 more

Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z

 

I viewed the hbase documation, which hbase-2.4.16 is compatibled with hadoop 3.3.3

But hadoop 3.3.3 moves to use {{lz4-java}} and {{snappy-java}} instead of requring the native libraries of these to be installed on the systems running Hadoop.So i packaged hbase-2.4.16 by mvn {-}P hadoop{-}3.0, it still can not create snappy table.The problem is same as above.
When I changed hadoop dependecy to hadoop-3.3.3.,the problem was fixed.
Bug: Backport "HBASE-20653 Add missing observer hooks for region server group to MasterObserver" to branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Sep/24 09:02
Affects Version/s: 2.5.10

Currently the following region server group operations don't have corresponding hook in MasterObserver :

* getRSGroupInfo
* getRSGroupInfoOfServer
* getRSGroupInfoOfTable
* listRSGroup

This JIRA is to 

* add them to MasterObserver
* add pre/post hook calls in RSGroupAdminEndpoint thru master.getMasterCoprocessorHost for the above operations
* add corresponding tests to TestRSGroups (in similar manner to that of HBASE-20627)