av9ash/GitBugs huggingface dataset:
Bug: Failed to start HBase MiniCluster when running UT with JDK 17 in hbase-hbck2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Aug/24 12:29
Affects Version/s: hbase-operator-tools-1.2.0

I noted that we have already supported to build hbase-operator-tools with JDK17.
[HBASE-28381] Support building hbase-operator-tools with JDK17 - ASF JIRA (apache.org)
 
But, when running UT in the hbase-hbck2 project using JDK17, an error occurs when starting the hbase MiniCluster cluster.
 
java.lang.NoSuchFieldException: modifiers
        at java.lang.Class.getDeclaredField(Class.java:2610) ~[?:?]
        at org.apache.hadoop.hbase.fs.HFileSystem.addLocationsOrderInterceptor(HFileSystem.java:338) [hbase-server-2.4.4.jar:2.4.4]
        at org.apache.hadoop.hbase.fs.HFileSystem.addLocationsOrderInterceptor(HFileSystem.java:295) [hbase-server-2.4.4.jar:2.4.4]
        at org.apache.hadoop.hbase.fs.HFileSystem.<init>(HFileSystem.java:100) [hbase-server-2.4.4.jar:2.4.4]
        at org.apache.hadoop.hbase.regionserver.HRegionServer.initializeFileSystem(HRegionServer.java:767) [hbase-server-2.4.4.jar:2.4.4]
        at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:654) [hbase-server-2.4.4.jar:2.4.4] 
 
 
The reason is that the 'modifiers' field is not accessible in JDK17.
Since HBase 2.5.0 supports JDK 17, I updated the HBase dependencies, and the test cases can pass successfully.
Bug: Exclude pom file of jaxws-ri in output tarball
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jul/24 03:26

Permission denied...
Not sure what is the real problem.

{noformat}
17:21:17  Building a binary tarball from the source tarball succeeded.
[Pipeline] echo
17:21:17  unpacking the hbase bin tarball into 'hbase-install' and the client tarball into 'hbase-client'
[Pipeline] sh
17:21:18  tar: /jaxws-ri-2.3.2.pom: Cannot open: Permission denied
17:21:20  tar: Exiting with failure status due to previous errors
Post stage
[Pipeline] stash
17:21:20  Warning: overwriting stash ‘srctarball-result’
17:21:20  Stashed 2 file(s)
[Pipeline] sshPublisher
17:21:20  SSH: Current build result is [FAILURE], not going to run.
[Pipeline] sh
17:21:20  Remove /home/jenkins/jenkins-home/workspace/HBase_Nightly_master/output-srctarball/hbase-src.tar.gz for saving space
[Pipeline] archiveArtifacts
17:21:20  Archiving artifacts
[Pipeline] archiveArtifacts
17:21:20  Archiving artifacts
[Pipeline] archiveArtifacts
17:21:20  Archiving artifacts
[Pipeline] archiveArtifacts
17:21:20  Archiving artifacts
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
17:21:20  Failed in branch packaging and integration
{noformat}
Bug: hbase-table-reporter shoud use the hbase.version declared in the parent pom.xml and use junit for UT
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Aug/24 12:07
Affects Version/s: 1.2.0

In hbase-operator-tools project, the submodule hbase-table-reporter shoud use the hbase.version declared in the parent pom, rather than using 2.1.1 defined by itself.
And, we need to add junit dependency and use junit for UT in this submodule.
Bug: Implement asynchronous bucket-cache recovery from persistence.
Status: Open | Priority: Major | Resolution: 
Created: 08/Aug/24 10:23

During the recovery of bucket cache from persistence, we need to rebuild a transient in-memory structure (blockByHfile) which has the blocks sorted according to their filenames.

This transient structure population is done during the server start-up during the recreation of bucket cache from persistence. This process can be time-consuming if bucket-cache has large number of blocks. (in millions). This population is executed inline with the server restart and this blocks the server from being available for several minutes. This makes the server restart inconvenient for the external users.

The need is to make the cache retrieval asynchronous to server restart by spawning a new thread that rebuilds cache from the persistence, while the server continues with its restart. This prevents an unnecessary delay in the server restart.

Note that the bucket cache may not available immediately for usage and will only be ready to use after the data is repopulated from persistence into memory.

Thanks,
Janardhan
Bug: [hbase-thirdparty] Bump dependencies for hbase-thirdparty-4.1.8 release
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Aug/24 19:56

Upgrade the dependencies,
 * guava 33.1.0-jre -> 33.2.1-jre
        [https://mvnrepository.com/artifact/com.google.guava/guava]
 * commons-cli 1.6.0 -> 1.8.0
        [https://mvnrepository.com/artifact/commons-cli/commons-cli]
 * error_prone_annotations 2.26.1 -> 2.28.0
        [https://mvnrepository.com/artifact/com.google.errorprone/error_prone_annotations]
 * gson 2.10.1 -> 2.11.0
        [https://mvnrepository.com/artifact/com.google.code.gson/gson]
 * jetty 9.4.54.v20240208 -> 9.4.55.v20240627
        [https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-server]
 * jersey 2.42 -> 2.43
        [https://mvnrepository.com/artifact/org.glassfish.jersey.core/jersey-client]
 * jackson-jaxrs-json-provider 2.17.0 -> 2.17.2
        [https://mvnrepository.com/artifact/com.fasterxml.jackson.jaxrs/jackson-jaxrs-json-provider]
Bug: HBase-Nightly-s390x Build failures
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jul/24 06:01

Hi [~zhangduo]
This is regarding recent [s390x CI failures|https://ci-hbase.apache.org/job/HBase-Nightly-s390x/] .
The install.log and junit.log has got below output -
{code:java}
/tmp/jenkins18056117051185954087.sh: line 12: /home/jenkins/tools/maven/latest3//bin/mvn: No such file or directory{code}
 Upon checking the machine stats it seems like the Apache Maven path is not getting set properly, since the mvn_home outputs -
{code:java}
MAVEN_HOME: /home/jenkins/tools/maven/latest3/{code}
 where as the mvn_version outputs the following -
{code:java}
[1mApache Maven 3.6.3[m Maven home: /usr/share/maven Java version: 11.0.23, vendor: Ubuntu, runtime: /usr/lib/jvm/java-11-openjdk-s390x Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "5.4.0-174-generic", arch: "s390x", family: "unix"{code}
 Could you please help us get this fixed? 
Thanks.
Bug: Change the generic type of List in InternalScanner.next
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jul/24 13:25

Plan to change it from List<Cell> to List<? super ExtendedCell>, so we could pass both List<Cell> and List<ExtendedCell> to it, or even List<RawCell> for coprocessors.

This could save a lot of casting in our main code.

This is an incompatible change for coprocessors, so it will only go into branch-3+, and will be marked as incompatible change.
Bug: FNFE may occur when accessing the region.jsp of the replica region
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jul/24 12:08
Affects Version/s: 2.4.13

On hbase UI, we can get the details of storefiles in region region by accessing region.jsp.

However, When hbase table enables the region replication, the replica region may reference deleted storefile due to it dosen't refresh in a timely manner, so in this case, we would get FNFE when openning the region.jsp of the region.

 

java.io.FileNotFoundException: File file:/home/gl/code/github/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/tmp/hbase/data/default/t01/e073c6b7c05eadda3f91d5b9692fc98d/info/5c52361153044b89aa61090cd5497998.4433b98ccf6b4a011ab03fc4a5e38a1a does not exist at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:915) at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:1236) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:905) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:462) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:462) at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:1881) at org.apache.hadoop.hbase.generated.regionserver.region_jsp._jspService(region_jsp.java:97) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:111) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
Bug: Should wipe out all the output directories before unstash in nightly job
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jul/24 02:57

For master and branch-3, we do not have jdk8 and jdk11 stages but we can still see there are comments on jira which include these stages's results.

I think the problem is that, in the 'init health results' stage, we want to stash some empty results but actually there are some build results for previous builds there so we stash some non empty results.

We should wipe out these directories first before stash them.
Bug: CompactionTool fails with NPE when mslab is enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jul/24 06:29
Affects Version/s: 2.5.9

While using the CompactionTool, NPE is observed.

*Command:*
{code:java}
hbase org.apache.hadoop.hbase.regionserver.CompactionTool  -major <table-path>{code}
*Exception Details:*
{code:java}
Exception in thread "main" java.lang.NullPointerException
        at org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.recycleChunks(MemStoreLABImpl.java:296)
        at org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.lambda$new$0(MemStoreLABImpl.java:109)
        at org.apache.hadoop.hbase.nio.RefCnt.deallocate(RefCnt.java:95)
        at org.apache.hbase.thirdparty.io.netty.util.AbstractReferenceCounted.handleRelease(AbstractReferenceCounted.java:86)
        at org.apache.hbase.thirdparty.io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
        at org.apache.hadoop.hbase.nio.RefCnt.release(RefCnt.java:84)
        at org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.close(MemStoreLABImpl.java:269)
        at org.apache.hadoop.hbase.regionserver.Segment.close(Segment.java:143)
        at org.apache.hadoop.hbase.regionserver.AbstractMemStore.close(AbstractMemStore.java:381)
        at org.apache.hadoop.hbase.regionserver.HStore.closeWithoutLock(HStore.java:723)
        at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:795)
        at org.apache.hadoop.hbase.regionserver.CompactionTool$CompactionWorker.compactStoreFiles(CompactionTool.java:171)
        at org.apache.hadoop.hbase.regionserver.CompactionTool$CompactionWorker.compactRegion(CompactionTool.java:137)
        at org.apache.hadoop.hbase.regionserver.CompactionTool$CompactionWorker.compactTable(CompactionTool.java:129)
        at org.apache.hadoop.hbase.regionserver.CompactionTool$CompactionWorker.compact(CompactionTool.java:118)
        at org.apache.hadoop.hbase.regionserver.CompactionTool.doClient(CompactionTool.java:374)
        at org.apache.hadoop.hbase.regionserver.CompactionTool.run(CompactionTool.java:424)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.hadoop.hbase.regionserver.CompactionTool.main(CompactionTool.java:460){code}
*Fix Suggestions:*
Initialize the ChunkCreator in CompactionTool when hbase.hregion.memstore.mslab.enabled is enabled.
Bug: RegionSizeCalculator ignored the size of memstore, which leads Spark miss data
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jul/24 03:50
Affects Version/s: 2.5.10

RegionSizeCalculator only considers the size of StoreFile and ignores the size of MemStore. For a new region that has only been written to MemStore and has not been flushed, will consider its size to be 0.

When we use TableInputFormat to read HBase table data in Spark.
{code:java}
spark.sparkContext.newAPIHadoopRDD(
    conf,
    classOf[TableInputFormat],
    classOf[ImmutableBytesWritable],
    classOf[Result])
}{code}
Spark defaults to ignoring empty InputSplits, which is determined by the configuration  "{{{}spark.hadoopRDD.ignoreEmptySplits{}}}".
{code:java}
private[spark] val HADOOP_RDD_IGNORE_EMPTY_SPLITS =
  ConfigBuilder("spark.hadoopRDD.ignoreEmptySplits")
    .internal()
    .doc("When true, HadoopRDD/NewHadoopRDD will not create partitions for empty input splits.")
    .version("2.3.0")
    .booleanConf
    .createWithDefault(true) {code}
The above reasons lead to Spark missing data. So we should consider both the size of the StoreFile and the MemStore in the RegionSizeCalculator.
Bug: Metrics for ConnectionRegistry API's need to be added
Status: Open | Priority: Major | Resolution: 
Created: 24/Jul/24 08:09
Affects Version/s: 2.4.17

For now, no metrics are being pushed for connection registry API's. We need at least some basic metrics for API's- 

requestCount - number of requests from client

failureCount - number of requests where we give failed response

response time - time took to respond to request
Bug: Rpc ConnectionRegistry APIs should have timeout
Status: Open | Priority: Major | Resolution: 
Created: 19/Jul/24 05:59
Affects Version/s: 2.4.18

ConnectionRegistry are some of the most basic metadata APIs that determine how clients can interact with the servers after getting required metadata. These APIs should timeout quickly if they cannot serve metadata in time.

Similar to HBASE-28428 introducing timeout for Zookeeper ConnectionRegistry APIs, we should also introduce timeout (same timeout values) for Rpc ConnectionRegistry APIs as well. RpcConnectionRegistry uses HBase RPC framework with hedge read fanout mode.

We have two options to introduce timeout:
 # Use RetryTimer to keep watch on CompletableFuture and make it complete exceptionally if timeout is reached (similar proposal as HBASE-28428).
 # Introduce separate Rpc timeout config for AbstractRpcBasedConnectionRegistry as the rpc timeout for generic RPC operations (hbase.rpc.timeout) could be higher.
Bug: The expired snapshot can be read by CopyTable or ExportSnapshot﻿
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jul/24 12:35
Affects Version/s: 2.4.13

We can get data of the expired snapshot through the following way.
{code:java}
hbase org.apache.hadoop.hbase.mapreduce.CopyTable --snapshot expired_snapshot --new.name my_table{code}
 And we did not check if the snapshot is expired when we export a snaoshot by ExportSnapshot tool.
Bug: Move our official slack channel from apache-hbase.slack.com to the one in the-asf.slack.com
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jul/24 06:17

According to this thread in the mailing list

https://lists.apache.org/thread/cyr8vfxvfqm2srz7m1kkp4mkk015r8wx

Let's do the move.
Bug: Should support different license name for 'Apache License, Version 2.0'
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jul/24 15:43

Starting from v29 of apache parent pom, the license name was changed from 'Apache License, Version 2.0' to 'Apache-2.0'.

We should support it so we do not need to set lots of exceptions in supplemental-models.xml.
Bug: Use hasReferences api of SFT in RegionSplitter
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Sep/24 17:06

HBASE-28564 Adds apis in SFT layer for all interaction related to Reference files - create, read, hasReferences 
This Jira refactors RegionSplitter code to use the hasReferences api of SFT instead of HRegionFileSystem.hasReferences()
Bug: [hbase-connectors] Add OpenLineage reporting support for Spark connector
Status: Open | Priority: Minor | Resolution: 
Created: 04/Sep/24 15:26

We would like to enhance the `hbase-spark` connector by integrating [OpenLineage|https://openlineage.io/] support. OpenLineage is a powerful framework for tracking and managing data lineage across various data systems. Adding this capability to the `hbase-spark` connector will allow users to automatically capture lineage information during data processing tasks, improving data governance, transparency, and auditability.
Bug: Add support for non replica actions to AsyncRequestFutureImpl.isActionComplete
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 07/Aug/24 16:50
Affects Version/s: 2.5.10

The current isActionComplete method we have in AsyncRequestFutureImpl is only designed to support replica actions, it would be useful to have isActionComplete support non replica actions that could be reused for HBASE-27781 / [#6144|https://github.com/apache/hbase/pull/6144/files] and in other paths where non replica actions are being handled. 

Since isActionComplete currently only has one caller we can move the replica action check to the caller method instead of having the check be inside isActionComplete.
Bug: Backport "HBASE-18382 [Thrift] Add transport type info to info server" to branch-2
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Sep/24 06:02

It would be really helpful to know if the Thrift server was started using the HTTP or binary transport. Any additional info, like QOP settings for SASL etc. would be great too. Right now the UI is very limited and shows {{true/false}} for, for example, {{Compact Transport}}. It'd suggest to change this to show something more useful like this:

{noformat}
Thrift Impl Type: non-blocking
Protocol: Binary
Transport: Framed
QOP: Authentication & Confidential
{noformat}

or

{noformat}
Protocol: Binary + HTTP
Transport: Standard
QOP: none
{noformat}
Bug: [hbase-thirdparty] Enable infra automation: autolink to Jira and 'pull-request-available' label
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Aug/24 17:42

Patterns like HBASE-28786 will be clickable, so it's easier to go to the Jira page.

{{.asf.yaml}} provides an option to do this via config: See [https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-AutolinksforJira]

Even without manual configuring this already works for hbase somehow. See [https://github.com/apache/hbase/commits/master/]
Maybe configured at repo level in INFRA?

But same does not work for hbase-connectors and other repos. Will use this config to fix the issue.

Also will enable label feature for 'pull-request-available', same as hbase main repo: HBASE-28325
Bug: Add missing permission check for updateRSGroupConfig in branch-2
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Sep/24 11:26
Affects Version/s: 2.7.0

Found this during HBASE-28867, we do not have security check for updateRSGroupConfig in branch-2. See [https://github.com/apache/hbase/blob/0dc334f572329be7eb2455cec3519fc820c04c25/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java#L450]

Same check exists in master [https://github.com/apache/hbase/blob/52082bc5b80a60406bfaaa630ed5cb23027436c1/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java#L2279]
Bug: NoMethodError undefined method assignment_expression?
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 20/Sep/24 00:02

After HBASE-28250 Bump jruby to 9.4.8.0 to fix snakeyaml CVE after every command the message "NoMethodError undefined method assignment_expression?" is printed. 
This is called from code copied from https://github.com/ruby/irb/blob/v1.4.2/lib/irb.rb . The fix is to also copy over the definition of `assignment_expression`.
Bug: Create Snapshot V1 reference files using SFT
Status: Open | Priority: Minor | Resolution: 
Created: 19/Sep/24 17:50

HBASE-28564 adds apis to handler Reference files in SFT for create, read, hasReferences.
This Jira refactors Snapshot code (V1 manifest) to create reference files using SFT and make the Snapshot layer SFT aware
Bug: Should check if table exists firstly when executing ExpiredMobFileCleaner
Status: Open | Priority: Minor | Resolution: 
Created: 09/Sep/24 11:45
Affects Version/s: 2.4.13

We would get the following error message when specifing an non_existing table whild executing ExpiredMobFileCleaner.

./bin/hbase org.apache.hadoop.hbase.mob.ExpiredMobFileCleaner  t04_mob01 info

Exception in thread "main" org.apache.hadoop.hbase.TableNotFoundException: t04_mob01
        at java.base/java.lang.Thread.getStackTrace(Thread.java:1610)
        at org.apache.hadoop.hbase.util.FutureUtils.setStackTrace(FutureUtils.java:144)
        at org.apache.hadoop.hbase.util.FutureUtils.rethrow(FutureUtils.java:163)
        at org.apache.hadoop.hbase.util.FutureUtils.get(FutureUtils.java:186)
        at org.apache.hadoop.hbase.client.AdminOverAsyncAdmin.getDescriptor(AdminOverAsyncAdmin.java:172)
        at org.apache.hadoop.hbase.mob.ExpiredMobFileCleaner.run(ExpiredMobFileCleaner.java:99)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82)
        at org.apache.hadoop.hbase.mob.ExpiredMobFileCleaner.main(ExpiredMobFileCleaner.java:75)
        at --------Future.get--------(Unknown Source)
        at org.apache.hadoop.hbase.client.RawAsyncHBaseAdmin.lambda$getDescriptor$24(RawAsyncHBaseAdmin.java:654)
        at org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:71)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
        at org.apache.hadoop.hbase.client.AsyncMasterRequestRpcRetryingCaller.lambda$doCall$4(AsyncMasterRequestRpcRetryingCaller.java:80)
        at org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:71)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
        at org.apache.hadoop.hbase.client.RawAsyncHBaseAdmin$1.run(RawAsyncHBaseAdmin.java:467)
        at org.apache.hbase.thirdparty.com.google.protobuf.RpcUtil$1.run(RpcUtil.java:56)

Maybe it is better to return concise information when table does not exist by checking table exists firstly.
Bug: Change the output of DatanodeInfo in the log to the hostname of the datanode
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Aug/24 10:19

Now,  DatanodeInfo will be output in the print log. When we are troubleshooting and searching for slow datanode nodes, we need to convert IP addresses to hostnames, which is quite cumbersome.
I think the output log should has good readability, so it would be better to output the hostname of the datanode.