av9ash/GitBugs huggingface dataset:
Bug: Add coprocessor hook to authorize user based on client SSL certificate chain
Status: In Progress | Priority: Major | Resolution: 
Created: 01/Nov/24 21:49
Affects Version/s: 2.6.1

In order to authorize the connected user based on the provided SSL client certificate chain, a new coprocessor hook is needed for Master and RegionServer, because currently there's no way to abort connection by a coprocessor when or after the user is authenticated.
Bug: s390x Nightly jobs are not running
Status: Open | Priority: Major | Resolution: 
Created: 16/Dec/24 06:20

Hi [~zhangduo]
This is regarding the s390x jobs that have been disabled - [https://ci-hbase.apache.org/job/HBase-Nightly-s390x/]. 
The last job ([https://ci-hbase.apache.org/job/HBase-Nightly-s390x/605/console]) does produce a result with some flaky test fails. Most of them fails due to timeout issue. 
Please let me know if there is something I can do at my end to help re-enable s390x jobs in CI.
Bug:  Force CF compatibility during incremental backup
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Oct/24 14:41
Affects Version/s: 2.6.2

Incremental backups can be taken even if the table descriptor of the current table does not match the column families of the full backup for that same table. When restoring the table, we choose to use the families of the full backup. This can cause the restore process to fail if we add a column family in the incremental backup that doesn't exist in the full backup. The bulkload process will fail because it is trying to write column families that don't exist in the restore table. 

 

I think the correct solution here is to prevent incremental backups from being taken if the families of the current table don't match those of the full backup. This will force users to instead take a full backup.
Bug: Provide HBase Docker Image for Integration Testing on Docker Hub or Github Packages
Status: Open | Priority: Major | Resolution: 
Created: 06/Dec/24 08:47

* I noticed that documents like [https://hbase.apache.org/book.html#_get_started_with_hbase] require batch shell commands to initialize the HBase environment.
 *  It would be nice if HBase could provide a runtime Docker image on Docker Hub or Github Packages. This would help writing unit tests for third-party libraries like *testcontainers-java* .

 - Some early investigation came from https://issues.apache.org/jira/browse/PHOENIX-6912 and https://issues.apache.org/jira/browse/PHOENIX-6919 .

 

Previous test images:
[https://github.com/F21/hbase-phoenix-all-in-one]
 [https://github.com/cloudera/opdb-docker]
Bug: Optimize read performance for accumulated delete markers on the same row or cell
Status: Patch Available | Priority: Major | Resolution: 
Created: 23/Dec/24 06:01
Affects Version/s: 2.5.10

I have been confronted with a problem that some Get operations take several seconds. 
!screenshot-2.png!

The reason was founded that users Put and Delete on some rows repeatedly. As delete markers are accumulated on the same row or cell, Get operations slow down. It can be reproduced by follow HBase shell commands.
{code:java}
create 'test', 'c'

java_import org.apache.hadoop.hbase.client.Delete
java_import org.apache.hadoop.hbase.TableName
java_import java.lang.System

con = @hbase.instance_variable_get(:@connection)
table = con.getTable(TableName.valueOf('test'))

1000.times do |i|
  # batch 10000 deletes with different timestamps every 10 seconds
  now = System.currentTimeMillis()
  dels = 10000.times.map do |i|
    del = Delete.new(Bytes.toBytes('row'))
    del.addFamily(Bytes.toBytes('c'), now + i)
  end
  table.delete(dels)
  sleep(10)
  puts "i - #{i}"
  get 'test', 'row'
end
{code}
{code:java}
i - 0
COLUMN                                                                                      CELL
0 row(s)
Took 0.0251 seconds
...
i - 10
COLUMN                                                                                      CELL
0 row(s)
Took 0.0412 seconds
...
i - 20
COLUMN                                                                                      CELL
0 row(s)
Took 0.0760 seconds
...
i - 30
COLUMN                                                                                      CELL
0 row(s)
Took 0.1014 seconds
...
i - 40
COLUMN                                                                                      CELL
0 row(s)
Took 0.1616 seconds
...
{code}
But the performance of Get operations can be optimized by using SEEK_NEXT_COL.
{code:java}
i - 1
COLUMN                                                                                      CELL
0 row(s)
Took 0.0087 seconds
...
i - 11
COLUMN                                                                                      CELL
0 row(s)
Took 0.0077 seconds
...
i - 21
COLUMN                                                                                      CELL
0 row(s)
Took 0.0087 seconds
...
{code}
Please review the PR.
[https://github.com/apache/hbase/pull/6557]
Bug: StoreFileInfo.isMobRefFile() would return true if the file is mob file not a reference mob file
Status: Open | Priority: Major | Resolution: 
Created: 15/Nov/24 14:53

StoreFileInfo.isMobRefFile() is used to determine whether a file is a reference MOB file created through snapshot.

But StoreFileInfo.isMobRefFile() may mistakenly identify the real MOB file as a reference MOB file created through snapshot.

For example:

A reference MOB file, the name is t02_mob=34916d9a943a3b1fdb4a2bbeef2e3e97-d41d8cd98f00b204e9800998ecf8427e20241112464aebcdfc354fbb834d58900de1226d_6424974cfc2ba56e1c6f43c34e833a8e, StoreFileInfo.isMobRefFile() would return true for this name;


A real MOb file, the name is d41d8cd98f00b204e9800998ecf8427e202411128ebcda68f97943cb80b64a00a0841a8d_6424974cfc2ba56e1c6f43c34e833a8e，
StoreFileInfo.isMobRefFile() would also return true for this name.
Bug: Optimize unnecessary type castings in Scan and Get setter methods
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Nov/24 16:43

In the setter methods of Scan and Get, we invoke the super class method, type cast and return the object.
{code:java}
public Scan setColumnFamilyTimeRange(byte[] cf, long minStamp, long maxStamp) {
return (Scan) super.setColumnFamilyTimeRange(cf, minStamp, maxStamp);
}{code}
 

In profiler reports it's observed that this typecasting does take some additional cpu cycles. we can optimize this  by adding an explicit return statement with current object. 
{code:java}
public Scan setColumnFamilyTimeRange(byte[] cf, long minStamp, long maxStamp) {
super.setColumnFamilyTimeRange(cf, minStamp, maxStamp);
return this;
}{code}
Bug: Refactor direct interactions of HFilelink file creations to SFT interface
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Nov/24 15:54

HBASE-27826 introduces tracking of Link/Reference files with StoreFileTracker interface and with that, we can have the FileBasedStoreFileTracker implementation to track Link/Reference files only using .filelist file itself and not create actual link/ref files (virtual links).

To support the same, we need to refactor all the direct interactions of Ref/Link file creations to SFT layer and this change 
- adds a createHFilelink() api to StoreFileTrackerBase.
- Moves all the creations of HFileLink (except for Snapshot which will be handled in HBASE-28863) to SFT layer
Bug: Bump jruby to 9.4.9.0 to fix rexml CVE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Nov/24 12:21

JRuby 9.4.9.0 has been released on November 04 2024

See release notes at [https://www.jruby.org/2024/11/04/jruby-9-4-9-0.html]

This release drops moderate rexml [CVE-2024-49761|https://github.com/advisories/GHSA-2rxp-v6pw-ch6m] [ |https://github.com/advisories/GHSA-2rxp-v6pw-ch6m]from our classpath with following change along with several other bugs/fixes: 
 * REXML was updated to 3.3.9 to get recent fixes and to address [CVE-2024-49761|https://github.com/advisories/GHSA-2rxp-v6pw-ch6m], a ReDOS vulnerability. Only users parsing unsanitized XML with REXML are affected. [#8396|https://github.com/jruby/jruby/pull/8396]

 

Also, based on our internal sonatype scanner we had following CVE due to rexml in 9.4.8.0:
|THREAT|SECURITY ISSUE|CVSS SCORE|COMPONENT|
|8|[CVE-2024-35176|https://nvd.nist.gov/vuln/detail/CVE-2024-35176]|7.5|rexml 3.2.5|
|8|[CVE-2024-41123|https://nvd.nist.gov/vuln/detail/CVE-2024-41123]|7.5|rexml 3.2.5|
|8|[CVE-2024-41946|https://nvd.nist.gov/vuln/detail/CVE-2024-41946]|7.5|rexml 3.2.5|
|7|[CVE-2024-39908|https://nvd.nist.gov/vuln/detail/CVE-2024-39908]|5.3|rexml 3.2.5|

Let's see which all CVEs we get rid of with this change!
Bug: Amount of region cached in the region metrics not updated for a region immediately after it is flushed with cacheOnWrite turned on
Status: In Progress | Priority: Major | Resolution: 
Created: 18/Dec/24 06:31
Affects Version/s: 2.6.1

An issue has been observed where if a new region is created and flushed with the configuration "hbase.rs.cacheblocksonwrite" is set to true, then, the region metrics does not reflect the amount of region cached for this new region. It starts reflecting the correct metric when this region is reopened for whatever reason.

 

The steps to reproduce this issue are as follows:
 * Create a table {{create 'tprefetch', 'cf'}}
 * {{{}{}}}Insert a record in the table {{put 'tprefetch', 'row1', 'cf:col1', 'aaaaa'}}
 * {{{}{}}}{{{}flush 'tprefetch'{}}}

After the flush, query the region metrics for this region and it will show that the region is 05 cached. This leads to the cacheAwareLoadBalancer take incorrect decisions whenever a region server is restarted.

{{}}
Bug: Deleted data reappears: Files generated by compaction interruption or failure cause deleted data to reappear
Status: Open | Priority: Major | Resolution: 
Created: 16/Oct/24 09:09
Affects Version/s: 0.98.0

Encountered a ghost data issue in the online production environment: A business batch request simultaneously includes put and delete operations. Deleted data has a very small probability of reappearing after some time, and the problematic data is within a single region. The primary cluster has a backup cluster, and after the problematic rowkey appears in the primary cluster, it cannot be found in the backup cluster.
Bug: Modify the "Incremental" Command to Leverage Existing Backed-Up WALs for Continuous Backup
Status: Open | Priority: Major | Resolution: 
Created: 21/Nov/24 12:37
Affects Version/s: 2.6.0

The incremental backup collects all the WALs generated since the last full or incremental backup and processes them into HFiles for optimized restoration. With the new continuous backup feature, the necessary WALs are already available at the backup location and are used directly, eliminating the need to retrieve them from the source cluster.

The incremental backup command itself remains unchanged.

If previous backups are part of a continuous backup: Uses the WALs already backed up in the backup location.

If previous backups are not part of a continuous backup: Operates as a traditional incremental backup, collecting WALs from the source cluster.
Bug: Get asyncfs working with custom SASL mechanisms
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Nov/24 18:22

There are a lot of SASL changes in Hadoop 3.5.

We need to somehow handle this.

There are two changes we need to make:

- Correctly handle SASL mechanisms that do not support QOP, like SCRAM
- Correctly handle SASL mechanisms that do send a non-empty initial response.
Bug: Upgrade JAX-WS to to 2.3.7
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Dec/24 07:15
Affects Version/s: 2.5.10

The current version is old and pulls in a lot of CVEs.

-The jaxws-ri artifact is for the binary assembly, jaxws-rt is the one to be used from maven.-
-(Actual dependencies should be identical)-
Bug: Change the default Hadoop 3 version to 3.4.1 on branch-2.5 and branch-2.6
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Nov/24 09:43

We've already done this on the non-release branches.
Opening a separate ticket to track doing the same on the non-release branches.

To recap the discussion: 
Duo and I +1 this change.
Nick has reservations, and has suggesed involving [~apurtell@yahoo.com] in the dicussion.
Bug: Backport missing UI patches to branch-2.5
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Dec/24 13:41

While working on HBASE-28832, we noticed that some UI-related patches were not merged to branch-2.5, branch-2.6 or branch-2 which makes it harder to backport HBASE-28832 changes.

It was suggested to identify specific JIRAs that need to be backported. Reason: it's better/easier to do that work than to maintain and enhance divergence between the branches.
Bug: Refactor assembly creation to use only DependencySets and move cached classpath creation to a new module
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Dec/24 12:22

There are a few problem with the current assembly creation:

* The assembly and cached classpath creation is done from the same module, even though they need to include different JARs.

*  To filter out the test jars, the descriptor uses a combination of moduleSets and dependencySets.
* However, the results are less than ideal, as some test-related JARs are still included in the assembly, while some others have multiple versions of the same jar added to the assembly.

Separating the assembly and cached classpath creation lets us build the assembly without the extra test dependencies and duplicate versions in a much simpler and more robust manner.

Relying heavily on the maven dependency mechanism solves the problem of duplicate versions, and not depending on the test-related modules at all solves the problem of lingering test dependencies.
Bug: Backport missing UI patches to branch-2.6
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Dec/24 07:24

While working on HBASE-28832, we noticed that some UI-related patches were not merged to branch-2.5, branch-2.6 or branch-2 which makes it harder to backport HBASE-28832 changes.

It was suggested to identify specific JIRAs that need to be backported. Reason: it's better/easier to do that work than to maintain and enhance divergence between the branches.
Bug: Extending the "Full Backup" Command to Enable Continuous Backup Support
Status: In Progress | Priority: Major | Resolution: 
Created: 21/Nov/24 12:33
Affects Version/s: 2.6.0

Enhance the Full Backup command to support continuous backup functionality. This includes:
 * Adding new options/parameters to the backup create command that are required for continuous backup.
 * Creating the replication peer and replicating WALs to the specified storage location, in addition to the usual full backup process.
Bug: Backport missing UI patches to branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Dec/24 07:30

While working on HBASE-28832, we noticed that some UI-related patches were not merged to branch-2.5, branch-2.6 or branch-2 which makes it harder to backport HBASE-28832 changes.

It was suggested to identify specific JIRAs that need to be backported. Reason: it's better/easier to do that work than to maintain and enhance divergence between the branches. 

 

The following Jira patches were backported to branch-2 in this Jira:
 * HBASE-24624 Optimize table.jsp code
 * HBASE-25402 Sorting order by start key or end key is not considering empty start key/end key
 * HBASE-28778 NPE may occur when opening master-status or table.jsp or procedure.jsp while Master is initializing
 * HBASE-20452 Master UI: Table merge button should validate required fields before submit
Bug: Poll SSL cert file for changes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Oct/24 19:15
Affects Version/s: 2.6.1

At my company we have an issue with our HBase servers not reloading TLS certificate files after they change on disk. We run our HMasters inside Kubernetes Pods, and define our certificate contents as Kubernetes Secrets. Then, the Secrets are projected into the HMaster containers as files. When the value of a Secret changes, the file changes automatically. However, Kubernetes does some complicated indirection, and does not change the files directly. It swaps a new directory in with new files in it.

HBase sets up a WatchService on the directory containing the TLS cert. For example, at my company, the cert is at {{{}/etc/hadoop/conf/ssl/cert/server-chain.pem{}}}. Then, events from that WatchService are delivered to a [handler method|https://github.com/apache/hbase/blob/836630422df2776287a860eff9d7104c3eca0582/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto/tls/X509Util.java#L530] which contains this check:
{code:java}
Path eventFilePath = dirPath.resolve((Path) event.context());
if (filePath.equals(eventFilePath)) {
 shouldResetContext = true;
}{code}
Debug logs show why this conditional is never true:
{code:java}
2024-10-21T17:48:13,659 [FileChangeWatcher-server-chain.pem] DEBUG org.apache.hadoop.hbase.io.FileChangeWatcher: Got file changed event: ENTRY_CREATE with context: ..2024_10_21_17_48_13.2471317370
2024-10-21T17:48:13,659 [FileChangeWatcher-server-chain.pem] DEBUG org.apache.hadoop.hbase.io.FileChangeWatcher: Got file changed event: ENTRY_CREATE with context: ..2024_10_21_17_48_13.2471317370
2024-10-21T17:48:13,660 [FileChangeWatcher-server-chain.pem] DEBUG org.apache.hadoop.hbase.io.crypto.tls.X509Util: Ignoring watch event and keeping previous default SSL context. Event kind: ENTRY_CREATE with context: ..2024_10_21_17_48_13.2471317370
.... {code}
The watcher events have a variety of files attached to them, but none of them are {{{}server-chain.pem{}}}, so HBase thinks they are not relevant.

I propose that we simply remove the condition inspecting the file name that was changed, and always reload the SSL context if a watcher event fires. This may lead to unnecessary reloads, but that will be harmless.
Bug: Performance regression in hot reads after split/merge
Status: Open | Priority: Major | Resolution: 
Created: 02/Dec/24 20:29
Affects Version/s: 2.6.1

We noticed a significant performance regression which comes from HBASE-27474. In that ticket, logic is added so that we don't cache blocks that exist within a reference or a link if compactions are enabled.

The issue we noticed is that we had a cluster which had compactions enabled, but compactions were a bit delayed. During the time, there were some regions which were recently split/merged and they contained references. This cluster is very hot reads and relies heavily on bloom filters. I noticed through profiles that we were spending a lot of time fetching BLOOM_CHUNK blocks from hdfs. This is almost never the case since we continually rightside the block cache to ensure all blooms are cached. In fact, we had no evictions at the time. So why weren't they getting cached?

With trace logging enabled I noticed that all of the blocks being read over and over happened to come from hfiles that looked to be references. This led me to the ticket in question.

This feels like a very serious regression, as it leads to substantantial impact to both hdfs and hbase in terms of request times and GC time and the host becomes fully hosed. I sort of wonder if we should revert that issue, or at the very least make it configurable. I'm not sure how to preserve the intended behavior of the ticket while also protecting the regionserver performance. In our case this happened for bloom blocks, but it could just as easily happen to a hot data block.
Bug: Introduce a new WALProvider to generate WAL files consumed by Replication
Status: Open | Priority: Major | Resolution: 
Created: 11/Dec/24 08:15

Under our current Replication implementation, all WAL files will be consumed and the data will be filtered by a table attribute 
REPLICATION_SCOPE_GLOBAL
 
We are using 2.4.11 version in our production env. And we have a small group table to use the replication to do CDC, however with the growth of the number of other non-replicated tables, we met the log queue accumulation issue and the data lag grew to hours level. 
 
To solve the issue we write the tables, which has CF marked the replication_scope to global, to separated WAL files and let the replication only consume those files. The feature solved our problems and push it to the community. 

In our env, we have 19 tables and only 8 of them need to be replicated. 

There are write heavy tables in the the rest of 11. So that our business benefits a lot from this feature.

 
The attached screenshot shows the improvement.
Bug: The RegionServer metrics of WALs is incorrect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Dec/24 11:06

RegionServer Metrics of wals (file count and file size) is incorrect.
We should should recalculate every time, rather than continuously accumulating them.

 

// MetricsRegionServerWrapperImpl.RegionServerMetricsWrapperRunnable.run()

List<WALProvider> providers = regionServer.getWalFactory().getAllWALProviders();
for (WALProvider provider : providers) {
    numWALFiles += provider.getNumLogFiles();
    walFileSize += provider.getLogFileSize();
}
Bug: Add a banner to the masterUI when disable_rpc_throttle is active
Status: Open | Priority: Major | Resolution: 
Created: 17/Dec/24 09:34

When quotas are enabled, rpc throttling is also enabled. There's a shell command to disable rpc throttles, added in HBASE-21159. As we do with the balancer, we should display a warning banner on the master UI when the quota system is enabled but the {{disable_rpc_throttle}} has been activated.
Bug: Depend on jaxws-rt instead jaxws-ri
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Dec/24 09:01
Affects Version/s: 2.5.10

Jaxws-ri pulls in extra dependencies that are not needed for either the rest server or client:

 _jakarta.mail-api-1.6.3.jar_
 _jakarta.persistence-api-2.2.2.jar_
 _jaxb-jxc-2.3.2.jar_
 _jaxb-xjc-2.3.2.jar_
 _jaxws-eclipselink-plugin-2.3.2.jar_
 _jaxws-rt-2.3.2.jar_
 _jaxws-tools-2.3.2.jar_
 _org.eclipse.persistence.asm-2.7.4.jar_
 _org.eclipse.persistence.core-2.7.4.jar_
 _org.eclipse.persistence.moxy-2.7.4.jar_
 _org.eclipse.persistence.sdo-2.7.4.jar_
 _sdo-eclipselink-plugin-2.3.2.jar_

However replacing jaxws-rt with jaxws-ri adds the following duplicate dependencies to 
the assembly:

stax2-api-4.1.jar
woodstox-core-5.1.0.jar

The assembly already has several duplicate dependencies, which is tracked in HBASE-29002 .
Bug: Restore bulkload testing to branch-2 branches
Status: Open | Priority: Major | Resolution: 
Created: 13/Dec/24 14:54
Affects Version/s: 2.5.0

While backporting HBASE-28767 I noticed that {{TestIncrementalBackupWithBulkLoad}} wasn't present on branch-2. This is the only backups test that wasn't present. I tried to restore it but found that {{TestIncrementalBackupWithBulkLoad}} depends on {{TestBulkLoadHFiles}}, which also doesn't exist on branch-2.

We should ensure basic test coverage of the bulkload features on branch-2 release lines and consider reintroducing these tests (and others?)
Bug: Remove unrelocated JAX-WS from assembly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Dec/24 09:38
Affects Version/s: 3.0.0-beta-2

HBase-rest uses thirdparty jax-ws.

The only thing requiring unrelocated JAX-WS is hbase-it. (and probably even that could be replaced with the thirdparty version)

There is no need to ship jax-ws in the assembly if we don't ship hbase-it.