av9ash/GitBugs huggingface dataset:
Bug: Partition BackupSystemTable queries
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jun/24 16:13
Affects Version/s: 2.6.0

When successfully taking an incremental backup, one of our final steps is to delete bulk load metadata from the system table for the bulk loads that needed to be captured in the given backup. This means that we will basically truncate the entire bulk loads system table in a single batch of the deletes after successfully taking an incremental backup. This logic occurs in {{{}BackupSystemTable#deleteBulkLoadedRows{}}}:
{code:java}
/*
 * Removes rows recording bulk loaded hfiles from backup table
 * @param lst list of table names
 * @param rows the rows to be deleted
 */
public void deleteBulkLoadedRows(List<byte[]> rows) throws IOException {
  try (Table table = connection.getTable(bulkLoadTableName)) {
    List<Delete> lstDels = new ArrayList<>();
    for (byte[] row : rows) {
      Delete del = new Delete(row);
      lstDels.add(del);
      LOG.debug("orig deleting the row: " + Bytes.toString(row));
    }
    table.delete(lstDels);
    LOG.debug("deleted " + rows.size() + " original bulkload rows");
  }
} {code}
Depending on your usage, one may run tons of bulk loads between backups, so this design is needlessly fragile. We should partition these deletes so that we never erroneously fail a backup due to this.
Bug: Don't clean bulk load system entries until backup is complete
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jun/24 17:05
Affects Version/s: 2.6.0

I've been thinking through the incremental backup order of operations, and I think we delete rows from the bulk loads system table too early and, consequently, make it possible to produce a "successful" incremental backup that is missing bulk loads.

To summarize the steps here, starting in {{{}IncrementalTableBackupCilent#execute{}}}:
 # We take an incremental backup of the WALs generated since the last backup
 # We ensure any bulk loads done since the last backup are appropriately represented in the new backup by going through the system table and copying the appropriate files to the backup directory
 # We delete all of the system table rows which told us about these bulk loads
 # We generate a backup manifest and mark the backup as complete

If we began deleting any of the system table rows regarding bulk loads, but fail in steps 3 and 4 before we are able to mark the backup as complete, then we'll be in a precarious spot. If we retry an incremental backup then it may succeed, but it would not know to persist the bulk loaded files for which we have already deleted system table references.

We could consider this issue an extension or replacement of https://issues.apache.org/jira/browse/HBASE-28084 in some ways, depending on what solution we land on. I think that we could fix this specific issue by reordering the bulk load table cleanup, but there will always be gotchas like this. Maybe it is simpler to require that the next backup be a full backup after any incremental failure.
Bug: An unbounded backup failure message can cause an irrecoverable state for the given backup
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jun/24 12:51
Affects Version/s: 2.6.0

The BackupInfo class has a failedMsg field which is a string of unbounded length. When a DistCp job fails then its failure message contains all of its source paths, and its failure message gets propagated to this failedMsg field on the given BackupInfo.

If a DistCp job has enough source paths, then this will result in backup status updates being rejected:
{noformat}
java.lang.IllegalArgumentException: KeyValue size too large
        at org.apache.hadoop.hbase.client.ConnectionUtils.validatePut(ConnectionUtils.java:513)
        at org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:1095)
        at org.apache.hadoop.hbase.client.HTable.lambda$put$3(HTable.java:564)
        at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187)
        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:563)
        at org.apache.hadoop.hbase.backup.impl.BackupSystemTable.updateBackupInfo(BackupSystemTable.java:292)
        at org.apache.hadoop.hbase.backup.impl.BackupManager.updateBackupInfo(BackupManager.java:376)
        at org.apache.hadoop.hbase.backup.impl.TableBackupClient.failBackup(TableBackupClient.java:243)
        at org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:317)
        at org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:603)
        at com.hubspot.hbase.recovery.core.backup.BackupManager.lambda$runBackups$2(BackupManager.java:145){noformat}
Without the ability to update the backup's state, it will never be returned as a failed backup by the client. This means that any mechanisms designed for repairing or cleaning failed backups won't work properly.

I think that a simple fix here would be fine: we should truncate the failedMsg field to a reasonable maximum size.
Bug: Remove vulnerable dependencies: slf4j-log4j12 and log4j:log4j
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Apr/24 13:23

slf4j-log4j12 is a bridge from SLF4J to Log4j 1.x.

Since log4j 1.x is vulnerable , so this needs to be removed.

 

It is to be replaced with the log4j-slf4j-impl dependency, which is a bridge from SLF4J to Log4j 2.x.
Bug: IndexOutOfBoundsException when executing HBCK2 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Apr/24 12:33

Reproduction
 
Execute the following command:
{code:java}
//代码占位符
${HBASE_HOME}/bin/hbase --config /etc/hbase-conf hbck -j ~/hbase-operator-tools/hbase-hbck2/target/hbase-hbck2-1.0.0-SNAPSHOT.jar {code}
we would get IndexOutOfBoundsException, as following.
{code:java}
//代码占位符
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -1    at java.lang.String.substring(String.java:1967)    at org.apache.logging.log4j.util.PropertiesUtil.partitionOnCommonPrefixes(PropertiesUtil.java:555)    at org.apache.logging.log4j.core.config.properties.PropertiesConfigurationBuilder.build(PropertiesConfigurationBuilder.java:174)    at org.apache.logging.log4j.core.config.properties.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:56)    at org.apache.logging.log4j.core.config.properties.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:35)    at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:557)    at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:481)    at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:323)    at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:695)    at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:716)    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:270)    at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:155)    at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)    at org.apache.logging.log4j.LogManager.getContext(LogManager.java:196)    at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:137)    at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:55)    at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:47)    at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:33)    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:358)    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)    at org.apache.hbase.HBCK2.<clinit>(HBCK2.java:92) {code}
The Reason.
The current version of HBase uses version 2.17.2 of log4j2, which supports shorthand syntax for properties configuration ( LOG4J2-3341 :   https://issues.apache.org/jira/browse/LOG4J2-3341).
 
However, The current version of HBCK2 uses version 2.17.1 of log4j2, which does not support the feature.

So, we would get IndexOutOfBoundsException when HBCK2 uses as following log4j2 properties, and this is the default log configuration format for HBase
logger.http = INFO,NullAppender

In order to avoid this problem, I think we need bump log4j2 from 2.17.1 to 2.17.2 against HBCK2.
Bug: TestHFileCompressionZstd fails with IllegalArgumentException: Illegal bufferSize
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jun/24 13:47
Affects Version/s: 2.5.8

HADOOP-18810 added io.compression.codec.zstd.buffersize in core-default.xml with default value as 0.

So ZSTD buffer size will be returned as 0 based on core-default.xml,
{code:java}
  static int getBufferSize(Configuration conf) {
    return conf.getInt(ZSTD_BUFFER_SIZE_KEY,
      conf.getInt(CommonConfigurationKeys.IO_COMPRESSION_CODEC_ZSTD_BUFFER_SIZE_KEY,
        // IO_COMPRESSION_CODEC_ZSTD_BUFFER_SIZE_DEFAULT is 0! We can't allow that.
        ZSTD_BUFFER_SIZE_DEFAULT));
  }
{code}
HBASE-26259 added a value check, but got reverted in HBASE-26959.

 

This issue will also occur during region flush and abort the RegionServer.

 

TestHFileCompressionZstd and other zstd related test cases are are also failing,
{code:java}
java.lang.IllegalArgumentException: Illegal bufferSize
	at org.apache.hadoop.io.compress.CompressorStream.<init>(CompressorStream.java:42)
	at org.apache.hadoop.io.compress.BlockCompressorStream.<init>(BlockCompressorStream.java:56)
	at org.apache.hadoop.hbase.io.compress.aircompressor.ZstdCodec.createOutputStream(ZstdCodec.java:106)
	at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.createPlainCompressionStream(Compression.java:454)
	at org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.<init>(HFileBlockDefaultEncodingContext.java:99)
	at org.apache.hadoop.hbase.io.hfile.NoOpDataBlockEncoder.newDataBlockEncodingContext(NoOpDataBlockEncoder.java:85)
	at org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:846)
	at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.finishInit(HFileWriterImpl.java:304)
	at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.<init>(HFileWriterImpl.java:185)
	at org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:312)
	at org.apache.hadoop.hbase.io.compress.HFileTestBase.doTest(HFileTestBase.java:73)
	at org.apache.hadoop.hbase.io.compress.aircompressor.TestHFileCompressionZstd.test(TestHFileCompressionZstd.java:54)
{code}
Bug: Improvements in HFile prefetch
Status: Open | Priority: Major | Resolution: 
Created: 17/Apr/24 06:47

Currently hfile prefetch on open is configurable cluster wise. Would be better to make it table wise configurable. Also would be better to have region filters which can allow to specify which regions data can be prefetched. This will be useful when there are hot regions whose data prefetching can help for low latency requirements.
Bug: MapReduceBackupCopyJob should support custom DistCp options
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jun/24 14:57
Affects Version/s: 2.6.0

h4. Problem

The MapReduceBackupCopyJob class provides no means for updating DistCp job options. This means that you're stuck with defaults, which isn't always desirable. For example, my workplace would like the freedom to deviate from at least two DistCp defaults:
 # distcp.direct.write — we would like to set this to true, because writing and renaming tmp files is expensive in S3 (where we store our backups).
 # we would also like control over the number of mappers that DistCp will run

h4. Proposed Solution

It is not the prettiest solution, but I'm proposing that we support DistCp customizations via the given backup client configuration like [this.|https://github.com/HubSpot/hbase/compare/hubspot-2.6...HubSpot:hbase:backup-distcp-options] It's necessary to do this conf -> arg conversion because we still want to use [DistCp's run method|https://github.com/HubSpot/hadoop/blob/c4c25b0ea2be1c8bca31d86962597060b2630f62/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java#L134-L171], which expects args, so as to not change any error codes. Hadoop actually does something similar, but in the opposite direction — the DistCp job has logic to convert the args back to configurations (lol).

Further, the DistCp API is really unfortunately designed for programmatic use, so it doesn't leave us great alternatives. For example, it doesn't matter what you pass in as DistCpOptions to the constructor if you use the run method, your options will be overwritten based on the args that you pass in. Alternatively, if you pass in the DistCpOptions in the constructor and use DistCp#execute or DistCp#createAndSubmitJob, then you get none of the error specificity!
Bug: ExportSnapshot should verify checksums for the source file and the target file
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/May/24 01:38

In our cluster, we encountered cases where the target hfile was corrupted after executing ExportSnapshot. [HBASE-13588 |https://issues.apache.org/jira/browse/HBASE-13588] can only  checksum data transferred, but cannot solve our problem. Therefore, we believe it is necessary to verify checksums on the files exported by ExportSnapshot.
Bug: Remove CellWrapper and use ExtendedCell internally in client side data structure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jun/24 06:55

In general, all Cells in HBase are ExtendedCells, we introduce Cell interface is only for preventing user to call some methods which damage the system.

So I think we should have internal methods which can get ExtendedCell from the client side data structures so we do not need to cast everywhere.
Bug: hbase-rest client shading conflict with hbase-shaded-client in HBase 2.x
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Apr/24 14:12
Affects Version/s: 2.4.17

When trying to decode a protobuf encoded CellSet, I get 
{noformat}
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hbase.protobuf.ProtobufUtil.mergeFrom(Lcom/google/protobuf/Message$Builder;[B)V
	at org.apache.hadoop.hbase.rest.model.CellSetModel.getObjectFromMessage(CellSetModel.java:129)
	at RestClientExample.getMulti(RestClientExample.java:191)
	at RestClientExample.start(RestClientExample.java:138)
	at RestClientExample.main(RestClientExample.java:124)

{noformat}

Seems to be caused by relocating protobuf 2.5 in hbase-shaded-client.

It works fine with the unrelcoated client i.e. when using the 
{noformat}
export CLASSPATH=`hbase --internal-classpath classpath`:<a few extra jars>
{noformat}
command to set up the classpath for the client.
Bug: Ensure large batches are not indefinitely blocked by quotas
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jun/24 13:49
Affects Version/s: 2.6.0

At my day job we are trying to implement default quotas for a variety of access patterns. We began by introducing a default read IO limit per-user, per-machine — this has been very successful in reducing hotspots, even on clusters with thousands of distinct users.

While implementing a default writes/second throttle, I realized that doing so would put us in a precarious situation where large-enough batches may never succeed. If your batch size is greater than your TimeLimiter's max throughput, then you will always fail in the quota estimation stage. Meanwhile [IO estimates are more optimistic|https://github.com/apache/hbase/blob/bdb3f216e864e20eb2b09352707a751a5cf7460f/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/DefaultOperationQuota.java#L192-L193], deliberately, which can let large requests do targeted oversubscription of an IO quota:

 
{code:java}
// assume 1 block required for reads. this is probably a low estimate, which is okay
readConsumed = numReads > 0 ? blockSizeBytes : 0;{code}
 

This is okay because the Limiter's availability will go negative and force a longer backoff on subsequent requests. I believe this is preferable UX compared to a doomed throttling loop.

In my opinion, we should do something similar in batch request estimation, by estimating a batch request's workload at {{Math.min(batchSize, limiterMaxThroughput)}} rather than simply {{{}batchSize{}}}.
Bug: fix CVE-2024-29025 in netty package 
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 05/Apr/24 01:56

Following CVEs are observed in io.netty : netty-handler and io.netty : netty-codec-http : 4.1.100.Final


[CVE-2024-29025|http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2024-29025]

sonatype-2020-0026
Bug: BackupSystemTable#checkSystemTable should ensure that the backup system tables are enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jun/24 16:39
Affects Version/s: 2.6.0

If the backup system tables become disabled, then we enter a state which the backup client will not recover from. Without manual intervention, every subsequent backup attempt will fail on [BackupSystemTable's calls to waitForSystemTable|https://github.com/apache/hbase/blob/3a3dd66e21da3f85c72d75605857713716d579fb/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/BackupSystemTable.java#L214-L215].

This checkSystemTable method already ensures that the tables exist — it should also ensure that the tables are enabled before we await that condition.

Alternatively, we could fast-fail if the tables are disabled rather than awaiting an enabled state that will never occur.
Bug: Quotas should support hard limits
Status: Open | Priority: Major | Resolution: 
Created: 28/Jun/24 16:03
Affects Version/s: 2.6.0

Quotas only support soft limits. Previously, some limits have accidentally been overly strict in a way that caused difficult UX, and we made them more lenient in https://issues.apache.org/jira/browse/HBASE-28672

It would be nice if quotas allowed users to specify hard throughput limits that they would certainly not exceed. This would be simple for read/write/request number quotas, but would be more difficult for read size throttling where workload estimations are much less precise. It's possible that this can/should only be supported for some combination of read number, write number, request number, and write size throttles.
Bug: The failsafe snapshot should be deleted after rollback successfully
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jun/24 12:26
Affects Version/s: 2.4.13

When a table exists and is in a disabled state, hbase supports to restore by the snapshot of this table.

In this scenario, hbase would create a failsafe snapshot for this disabled table, and hbase would rollback by this failsafe snapshot when fail to restore.

However on hbase cluster, the failsafe snapshot still exists after rollback successfully by failsafe snapshot.

I think when restore failed by table snapshot but rollback successfully by failsafe snapshot, this failsafe snapshot should be deleted.
Bug: BackupLogCleaner causes HMaster WALs to pile up indefinitely
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jun/24 15:41
Affects Version/s: 2.6.0

We have been trying to setup daily incremental backups for hundreds of clusters at my day job. Recently we discovered that old WALs were piling up across many clusters inline with when we began running incremental backups.

This led to the realization that [the BackupLogCleaner will always skip archived HMaster WALs|https://github.com/HubSpot/hbase/blob/3a170578b04d5947c655319f1b8d8475ccb6bb4e/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/BackupUtils.java#L364-L367]. This is a problem because, if a cleaner is skipping a given file, then [the CleanerChore will never delete it|https://github.com/apache/hbase/blob/fb988373c1cd0184400fe17c4323fbf0de07c951/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java#L312-L332].

This seems like a misunderstanding of what it means to "skip" a WAL in a BaseLogCleanerDelegate, and, instead, we should always return these HMaster WALs as deletable from the perspective of the BackupLogCleaner. We could subject them to the same scrutiny as RegionServer WALs: are they older than the most recent successful  backup? But, if I understand correctly, HMaster WALs do not contain any data relevant to table backups, so that would be unnecessary.
Bug: Upgrade yetus to a newer version
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jun/24 14:48

Now we use the one in yetus for validating, where it uses jrunscript to execute a simple javascript script.

But the javascript engine has been removed from JDK starting from JDK 15, so in JDK 17 we can not use this way any more.

https://openjdk.java.net/jeps/372

We should find another way to validate xml files.

=========

In newer version, yetus has already changed to use xmllint, so let's see how to upgreade yetus.
Bug: Make -h arg respected by hbck2 and exit if unrecognized arguments are passed
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/May/24 18:59

The -h argument in hbck is not respected and instead of displaying the argument usage guide, the command continued to execute. Any unrecognized arguments should cause an exception and exit.

 

example: 
{code:java}
$ hbck2 addFsRegionsMissingInMeta -h
OpenJDK 64-Bit Server VM warning: Ignoring option --illegal-access=permit; support was removed in 17.0
19:09:08.831 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - hbase.client.pause.cqtbe is deprecated. Instead, use hbase.client.pause.server.overloaded
ERROR: Unrecognized option: -h
FOR USAGE, use the -h or --help option {code}
Bug: ITBLL and other MR-based integration tests should heartbeat often
Status: Open | Priority: Major | Resolution: 
Created: 20/Jun/24 08:52

We have this little note in our ITBLL harness,
{noformat}
      // If we cause enough chaos, RPC requests might get into long backoffs. During this
      // time, it won't send keep alives to the map/reduce context. So increase the timeout
      // a bunch
{noformat}

Investigating, the ITBLL Generator's persist method updates the MR context progress only every 100 puts. You'd think that would be enough, but given chaos, it really isn't. What if we update progress with every put? Digging through MR source code, it seems that calling the context.progress() method only sets an AtomicBoolean that a progress update needs sent, actual sending of progress reports is gated by {{mapreduce.task.progress-report.interval}}, or 1% of {{mapreduce.task.timeout}}, which defaults to 1% of 300_000ms, or 3 seconds. So yeah, we should probably update this AtomicBool much more often in chaotic jobs, as doing so is effectively free and will improve reliability.

But still, every put is perhaps excessive. What if we add a pre-flush hook to (Async)BufferedMutator so that a MR job can set this progress flag right before the client disappears down into a retry loop? I bet other applications would find such a hook useful as well.
Bug: Use Streams to unmarshall protobuf REST data
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jun/24 06:13

We've recently optimized REST marshalling by using streams directly.

We should do the same for unmarshalling.

The easy part is the server side, as that affects only a small set files.

However, we should also support streams on the client side, which requires duplicating each method the returns / expects a byte array to also work with streams.