av9ash/GitBugs huggingface dataset:
Bug: Per-table or per-CF compression codec setting overrides
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Oct/21 01:32
Affects Version/s: 2.5.0

This won't work as expected today...
{noformat}
hbase> create 'sometable', \
  { NAME => 'somefamily', VERSIONS => 1000, COMPRESSION => 'ZSTD' }, \
  CONFIGURATION => { 'hbase.io.compress.zstd.level' => '9' }
{noformat}
... but it should. We get and retain Compressor instances in HFileBlockDefaultEncodingContext, and could in theory call Compressor#reinit when setting up the context, to update compression parameters like compression level and buffer size per the ambient configuration, but we do not plumb through the CompoundConfiguration from the Store into HFileBlockDefaultEncodingContext. Instead can only update codec parameters globally in system site conf files.

This is actually pretty important for algorithms like ZSTD, which offers more than 20 different compression levels, where at level 1 it is almost as fast at compression as LZ4, and where at levels > 19 it utilizes computationally expensive techniques to rival LZMA at compression ratio (and poor compression speed). It is very likely that the ZSTD level you'd want to employ for a given table's data will vary by use case.
Bug: Missing server side debugging on failed SASL handshake
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Oct/21 21:00

In trying to debug some problems with the pluggable authentication, I noticed that we are eating the IOException without logging it (at any level) in ServerRpcConnection.

This makes it super hard to debug when that pluggable interface has a problem because the context gets lost (clients just get a pretty useless DNRIOE).
Bug: RegionServer leaks DFSInputStreams due to not closing region after warmup
Status: Open | Priority: Minor | Resolution: 
Created: 07/Oct/21 17:44

I'm working on HDFS-16262, in which we add tracking of DFSInputStreams. The DFSInputStreams are registered on creation and de-registered on call to close(). As part of testing I noticed that extra DFSInputStreams were being created and never closed. I tracked it down to HBASE-15441 in which the r.close() was removed here: [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L7480]

This has not been a huge issue because DFSInputStreams don't currently keep resources unless you do a non-positional read or use extended read buffers. So for the warmup case, the DFSInputStream is created and then eventually GC'd.

That said, the contract of a Closeable is typically that you should close them to release resources. So we're kind of lucking out that it's not strictly necessary until you do a read, and that could change at any time (like in HDFS-16262).

I think we should either add back a call to r.close(), or maybe it's possible to skip creating the Reader altogether for warmup. It doesn't look like the reader itself is necessary for warmup, but I could be missing something.
Bug: TestCacheOnWrite and TestCompactingToCellFlatMapMemStore use too many file descriptors
Status: Resolved | Priority: Trivial | Resolution: Cannot Reproduce
Created: 11/Dec/21 20:36
Affects Version/s: 2.5.0

This is somewhat test environment specific but an indication that TestCacheOnWrite could stand to go on a resource diet. Likewise, TestCompactingToCellFlatMapMemStore

Apache Maven 3.8.3 (ff8e977a158738155dc465c6a97ffaf31982d739)
Java version: 1.8.0_312, vendor: Azul Systems, Inc., runtime: /Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre
OS name: "mac os x", version: "12.0.1", arch: "aarch64", family: "mac"

Representative output from TestCacheOnWrite:

[ERROR] org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testCachingDataBlocksThresholdDuringCompaction[35]  Time elapsed: 2.113 s  <<< ERROR!
java.lang.IllegalStateException: failed to create a child event loop
	at org.apache.hbase.thirdparty.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88)
	at org.apache.hbase.thirdparty.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:60)
	at org.apache.hbase.thirdparty.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:49)
	at org.apache.hbase.thirdparty.io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
	at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:86)
	at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:81)
	at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:68)
	at org.apache.hadoop.hbase.wal.NettyAsyncFSWALConfigHelper.getEventLoopConfig(NettyAsyncFSWALConfigHelper.java:64)
	at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.doInit(AsyncFSWALProvider.java:80)
	at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.init(AbstractFSWALProvider.java:125)
	at org.apache.hadoop.hbase.wal.SyncReplicationWALProvider.init(SyncReplicationWALProvider.java:115)
	at org.apache.hadoop.hbase.wal.WALFactory.<init>(WALFactory.java:207)
	at org.apache.hadoop.hbase.wal.WALFactory.<init>(WALFactory.java:178)

Caused by: java.io.IOException: Too many open files
	at java.base/sun.nio.ch.IOUtil.makePipe(Native Method)
	at java.base/sun.nio.ch.KQueueSelectorImpl.<init>(KQueueSelectorImpl.java:86)
	at java.base/sun.nio.ch.KQueueSelectorProvider.openSelector(KQueueSelectorProvider.java:35)
	at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:173)
	... 55 more
Bug: The message of StoreTooBusy is confused
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 15/Dec/21 10:49

 

When check Store limit. We both check parallelPutToStoreThreadLimit and parallelPreparePutToStoreThreadLimit. 
{code:java}
if (store.getCurrentParallelPutCount() > this.parallelPutToStoreThreadLimit
    || preparePutCount > this.parallelPreparePutToStoreThreadLimit) {
  tooBusyStore = (tooBusyStore == null ?
      store.getColumnFamilyName() :
      tooBusyStore + "," + store.getColumnFamilyName());
} {code}
But we only print Above parallelPutToStoreThreadLimit only. 

 

 
{code:java}
if (tooBusyStore != null) {
  String msg =
      "StoreTooBusy," + this.region.getRegionInfo().getRegionNameAsString() + ":" + tooBusyStore
          + " Above parallelPutToStoreThreadLimit(" + this.parallelPutToStoreThreadLimit + ")";
  if (LOG.isTraceEnabled()) {
    LOG.trace(msg);
  }
  throw new RegionTooBusyException(msg);
}{code}
It confused me a lot time ...... 

Just add message of parallelPreparePutToStoreThreadLimit
Bug: Add auth method information to AccessChecker audit log
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 30/Nov/21 05:44

If we turn on audit logging, authentication events are logged with auth method information  (e.g. KERBEROS, TOKEN etc) like below.

{noformat}
2021-11-30 14:15:07,417 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Auth successful for PRINCIPAL@REALM (auth:KERBEROS)
{noformat}

However, authorization event logs which are emitted by AccessChecker class don't contain auth method information.

I need this info to filter out audit log lines generated by MapReduce jobs, auth method = TOKEN case.
Bug: Unreadable table names after HBASE-24605
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 18/Nov/21 05:00
Affects Version/s: 3.0.0-alpha-1

During the fix HBASE-24605 (Break long region names in the web UI) the 'word break' rule was applied to the 'table-striped' style, so literally, all tables in Master Web UI were affected. The most noticeable is the 'User Tables' when table names, as well as descriptions, and even State are wrapped in a very ugly way. 
We have two options here: 
1. Fix the user tables only as it was done previously for the procedures table
2. Apply 'word break' to those tables that require that. 
Since most of the users are comfortable with the current UI we may go through the first way, so the changes would affect only the User table. 
Sample screenshots before and after are attached.
Bug: Revert REST protobuf package to org.apache.hadoop.hbase.rest
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 11/Jul/24 19:31

In Hbase 3+, the package name of the REST messages has been renamed to org.apache.hadoop.hbase.shaded.rest from org.apache.hadoop.hbase.rest 

These definitions are only used by REST, and have nothing to do with standard HBase RPC communication.

I propose reverting the package name.
We may also want to move the protobuf definitions back to the hbase-rest module.
Bug: BackupLogCleaner cleans required WALs when using multiple backuproots
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 02/Jul/24 14:01
Affects Version/s: 2.6.0

The BackupLogCleaner is responsible for avoiding the deletion of WAL/logs that still need to be included in a future backup.

The logic to decide which files can be deleted does not work correctly when multiple backup roots are used. Each backup root has a different chain of backups (full, incremental1, incremental2, ...). So, if any chain requires a log, it should be preserved. This is not the case.

The result is that logs could be incorrectly deleted, resulting in data loss in backups.

PR incoming with test & fix.
Bug: when a procedure on a table executed as a child procedure, further table procedure operations on that table are blocked forever waiting to acquire the table procedure lock 
Status: Resolved | Priority: Critical | Resolution: Implemented
Created: 10/Sep/24 14:59
Affects Version/s: 2.5.10

when a procedure on a table executed as a child procedure, furthur table procedure operations on that table are blocked forever waiting to aquire the table procedure lock .

This issue occur due to not clearing of table lock for the table procedures submitted as child procedures after the changes in HBASE-28683,
Bug: BackupLogCleaner unsafe for multi-root backups
Status: Resolved | Priority: Critical | Resolution: Duplicate
Created: 12/Sep/24 09:58

The BackupLogCleaner is responsible for keeping WALs around that are not yet backed-up.

However, it does not keep track of multi-root backups. I.e. if there are 2 backups roots R1 and R2, where R1 was backed up 1 week ago, and R2 today, the WALS needed for R1 could be deleted (= data loss in the R1 backups).
Bug: Hadoop check for hadoop 3.4.0 is failing
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 08/Jul/24 11:32

In hadoop 3.4.0, hadoop common depends on org.bouncycastle:bcprov-jdk15on, where we an enforcer rule to force depending on org.bouncycastle:*-jdk18on.

We should exlude org.bouncycastle:bcprov-jdk15on from hadoop.

And also remove direct references of protobuf 2.5 in our asyncfs code.
Bug: Upgrade from bootstrap 3.4.1 to non vulnerable version 5.3.3
Status: Patch Available | Priority: Major | Resolution: 
Created: 11/Sep/24 07:27

Bootstrap 3.4.1 was released in 13 Feb, 2019 and there has been no new 3.x release since then. This version of bootstrap has multiple medium CVEs reported recently. 
See [https://security.snyk.io/package/npm/bootstrap/3.4.1] for details.

CVE List:
 * [https://www.cve.org/CVERecord?id=CVE-2024-6484]
 * [https://www.cve.org/CVERecord?id=CVE-2024-6485]

Related Github Issue/Advisory:
 * [https://github.com/twbs/bootstrap/issues/40692]
 * [https://github.com/advisories/GHSA-9mvj-f7w8-pvh2]

Based on synk.io the only non-vulnerable version seems to be in 5.x line.
Upgrading from 3.x to 4.x itself would be substantial work. So may be we would have to move step by step i.e. migrate from 3.x to 4.x and then 4.x to 5.x.

This JIRA is to capture all sub-task needed to achieve same.
Bug: Backup delete incorrectly handles bulk loaded entries
Status: Open | Priority: Major | Resolution: 
Created: 08/Jul/24 12:12
Affects Version/s: 2.6.0

To delete a backup, `BackupAdminImpl#deleteBackup` calls `BackupSystemTable#deleteBulkLoadedRows` on all rows returned by `BackupSystemTable#readBulkLoadedFiles(backupId)`.

This last method comes down to a scan over the bulk load table for all rows matching "bulk:BACKUPID:".

However, current code never writes rows of that format. The only methods that do so are `BackupSystemTable#writeBulkLoadedFiles` and `BackupSystemTable#createPutForBulkLoadedFile`, which are only used in test code.

Note that the bulkload rows that do get written (see `BackupObserver`) only get deleted when creating incremental backups, but suffer from HBASE-28706.

Consequences: I suspect that if you do some bulk loads, create a full backup, and delete that backup before doing any incremental backup, those entries in the bulk load table will never get deleted, meaning those HFiles can never get deleted (because `BackupLogCleaner` blocks their deletion).
Bug: Parallelize the archival of compacted files 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/24 06:53
Affects Version/s: 2.5.10

While splitting a region in hbase it has to cleanup compacted files for bookkeeping.
 
Currently we do it sequentially and that is good enough because for hdfs as it is a fast operation. When we do the same in s3 it becomes a issue. We need to paralleize this to make it faster. 
{code:java}
// code placeholder
for (File file : toArchive) {
      // if its a file archive it
      try {
        LOG.trace("Archiving {}", file);
        if (file.isFile()) {
          // attempt to archive the file
          if (!resolveAndArchiveFile(baseArchiveDir, file, startTime)) {
            LOG.warn("Couldn't archive " + file + " into backup directory: " + baseArchiveDir);
            failures.add(file);
          }
        } else {
          // otherwise its a directory and we need to archive all files
          LOG.trace("{} is a directory, archiving children files", file);
          // so we add the directory name to the one base archive
          Path parentArchiveDir = new Path(baseArchiveDir, file.getName());
          // and then get all the files from that directory and attempt to
          // archive those too
          Collection<File> children = file.getChildren();
          failures.addAll(resolveAndArchive(fs, parentArchiveDir, children, start));
        }
      } catch (IOException e) {
        LOG.warn("Failed to archive {}", file, e);
        failures.add(file);
      }
    } {code}
Bug: Implement proper string builder for MoveRegionRequest in ProtobufUtil.getShortTextFormat 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Sep/24 04:57

While looking into an issue where I was trying to debug MoveRegion, came across this responseTooSlow log where the params aren't properly build for MoveRegionRequest. Would be good to properly formulate this so that we know the regionName and other details.
{code:java}
ipc.RpcServer - (responseTooSlow): {"call":"MoveRegion(org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MoveRegionRequest)","starttimems":"<>","responsesize":"0","method":"MoveRegion","param":"TODO: class org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MoveRegionRequest","processingtimems":830247,"client":"<>","queuetimems":0,"class":"HMaster"}{code}
Bug: Manage hbase-thirdparty transitive dependencies via BOM pom
Status: Patch Available | Priority: Major | Resolution: 
Created: 25/Sep/24 12:07

Despite the intentions to the contrary, there are several places where we need the version of a dependency managed in hbase-thirdparty to match an import in the main product (and maybe also in our other repos). Right now, this is managed via comments in the poms, which read "when this changes there, don't for get to update it here...". We can do better than this.

I think that hbase-thirdparty could publish a BOM pom file that can be imported into any of the downstream hbase projects that make use of that release of hbase-thirdparty. That will centralize management of these dependencies in the hbase-thirdparty repo.

This blog post has a nice write-up on the idea, https://www.garretwilson.com/blog/2023/06/14/improve-maven-bom-pattern
Bug: Change the default Hadoop3 version to 3.4.1, and add tests to make sure HBase works with earlier supported Hadoop versions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Sep/24 07:07
Affects Version/s: 2.6.0

Discussed on the mailing list:
https://lists.apache.org/thread/orc62x0v2ktvj26ltvrqpfgzr94ncswn
Bug: Use region server configuration for evicting the cache while unassigning a region
Status: Reopened | Priority: Major | Resolution: 
Created: 04/Sep/24 04:47
Affects Version/s: 2.6.0

While unassigning a region because of either a region movement or split/merge, the master node sends a RPC to the region server where the region is currently hosted. As part of this RPC, it also tells the region server whether the blocks for the region being unassigned should be evicted from the cache. This is done by passing a boolean parameter. The value for this parameter is found from couple of configuration parameters seen on the master node. For example, "hbase.rs.evictblocksonsplit" (if the region is being unassigned because of split) or "hbase.rs.evictblocksonclose" (if the region is being moved from this region server to some other region server).

With support for dynamic configuration, these configuration parameter values may differ between the master node and the worker nodes.

Since, caching is managed by the worker nodes, we should use the configuration values as seen on the worker nodes rather than the values seen on the master node.

The master node should just tell the worker nodes if the region unassignment is happening because of a split or a region movement and the worker nodes should take the appropriate decision about evicting from cache depending on the configuration seen there.
Bug: NPE may occur when opening master-status or table.jsp or procedure.jsp while Master is initializing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Aug/24 12:19

The reason:

For table.jsp, NPE may occur when calling  master.getConnection() while Master is initializing.
asyncClusterConnection will only be initialized when HMaster call method setupClusterConnection,
so before that, asyncClusterConnection is null, at this moment,  we would get NPE if opening table.jsp.

procedure.jsp and master-status may also encounter NPE for the similar reason.

 

Error Message：

java.lang.NullPointerException: Cannot invoke "org.apache.hadoop.hbase.procedure2.ProcedureExecutor.getProcedures()" because "procExecutor" is null at org.apache.hadoop.hbase.generated.master.procedures_jsp._jspService(procedures_jsp.java:76) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:111) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.apache.hbase.thirdparty.org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450) at org.apache.hbase.thirdparty.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.apache.hbase.thirdparty.org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) at org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:117) at org.apache.hbase.thirdparty.org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

 

2024-08-13T20:41:40,056 WARN  [qtp463313451-80] server.HttpChannel: /master-status
java.lang.NullPointerException: Cannot invoke "org.apache.hadoop.hbase.master.assignment.RegionStateNode.isInState(org.apache.hadoop.hbase.master.RegionState$State[])" because "rsn" is null
        at org.apache.hadoop.hbase.master.http.MasterStatusServlet.getMetaLocationOrNull(MasterStatusServlet.java:80) ~[hbase-server-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]
        at org.apache.hadoop.hbase.master.http.MasterStatusServlet.doGet(MasterStatusServlet.java:60) ~[hbase-server-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) ~[javax.servlet-api-3.1.0.jar:3.1.0]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
        at org.apache.hbase.thirdparty.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[hbase-shaded-jetty-4.1.7.jar:?]
        at org.apache.hbase.thirdparty.org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) ~[hbase-shaded-jetty-4.1.7.jar:?]
        at org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:117) ~[hbase-http-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]
Bug: SFT's BrokenStoreFileCleaner may cause data loss
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Sep/24 22:01
Affects Version/s: 2.5.10

When having this BrokenStoreFileCleaner enabled, one of our customers has run into a data loss situation, probably due to a race condition between regions getting moved out of the regionserver while the BrokenStoreFileCleaner was checking this region's files eligibility for deletion. We have seen that the file got deleted by the given region server, around the same time the region got closed on this region server. I believe a race condition during region close is possible here:

1) In BrokenStoreFileCleaner, for each region online on the given RS, we get the list of files in the store dirs, then iterate through it [1]; 
2) For each file listed, we perform several checks, including this one [2] that checks if the file is "active"
The problem is, if the region for the file we are checking got closed between point #1 and #2, by the time we check if the file is active in [2], the store may have already been closed as part of the region closure, so this check would consider the file as deletable.

One simple solution is to check if the store's region is still open before proceeding with deleting the file.

[1] https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/BrokenStoreFileCleaner.java#L99
[2] https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/BrokenStoreFileCleaner.java#L133
Bug: table level wal appendSize and replication source metrics not correctly shown in /jmx response
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 18/Sep/24 03:20

Found 2 metrics did not display in the /jmx http interface response, table level wal appendSize and table level replication source.

I suspect it's because the metric name contains a colon : 

!image-2024-09-18-11-21-10-279.png|width=521,height=161!

 

!image-2024-09-18-11-21-20-295.png|width=521,height=282!

 

after modify the table name string to "Namespace_$namespace_table_$table, the metric really display correctly in the /jmx response
Bug: Align the fixed issues in git and jira
Status: In Progress | Priority: Major | Resolution: 
Created: 17/Sep/24 13:39

The API cleanup is almost done, so we are entering the last steps for releasing the final 3.0.0.

We should try our best to align the fixed issues in git and jira, especially for a major release.

It will be a huge project, so file a jira issue to track this work.

Plan to do this in the next few weeks, waiting for my 192GB machine to ready so I can start to run ITBLL and upgrading tests locally(Actually I've already bought everything for the machine but it failed to boot with 192G DDR5 RAM so I sent the motherboard back...)