av9ash/GitBugs huggingface dataset:
Bug: Zero length value would cause value compressor read nothing and not advance the position of the InputStream
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 13/Feb/23 15:13

This is a code sniff from the discussion of HBASE-27073

{code}
  public static void main(String[] args) throws Exception {
    CompressionContext ctx =
      new CompressionContext(LRUDictionary.class, false, false, true, Compression.Algorithm.GZ);
    ValueCompressor compressor = ctx.getValueCompressor();
    byte[] compressed = compressor.compress(new byte[0], 0, 0);
    System.out.println("compressed length: " + compressed.length);
    ByteArrayInputStream bis = new ByteArrayInputStream(compressed);
    int read = compressor.decompress(bis, compressed.length, new byte[0], 0, 0);
    System.out.println("read length: " + read);
    System.out.println("position: " + (compressed.length - bis.available()));
{code}

And the output is
{noformat}
compressed length: 20
read length: 0
position: 0
{noformat}

So it turns out that, when compressing, an empty array will still generate some output bytes but while reading, we will skip reading anything if we find the output length is zero, so next time when we read from the stream, we will start at a wrong position...
Bug: Also clear the Dictionary when resetting when reading compressed WAL file
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 08/Feb/23 09:53

After trying several times, now I can reproduce a critical problem when reading compressed WAL file in replication.

The problem is about how we construct the LRUDictionary when reset the WALEntryStream. In the current design, we will not reconstruct the LRUDictionary when reseting, but when reading again, we will call addEntry directly to add 'new' word into the dict, which will mess up the dict and cause data corruption.

I've implemented a UT to simulate reading partial WAL entry in replication, with the current code base, after reseting and reading again, we will stuck there for ever.

-The fix is to always use findEntry when constructing the dict when reading, so we will not mess things up.-
It turns out that the above solution does not work.
Another possible fix is to always reconstruct the dict after reseting, we will also clear the dict and reconstruct it again. But it is less efficient as we need to read from the beginning to the position we want to seek to, instead of seek to the position directly, especially when tailing the WAL file which is currently being written.

And notice that, the UT can only reproduce the problem in local file system, on HDFS, the available method is implemented so if there is not enough data, we will throw EOFException earlier before parsing cells with the compression decoder, so we will not add  duplicated word to dict. But in real world, it is possible that even if there are enough data to read, we could hit an IOException while reading and lead to the same problem described above.

And while fixing, I also found another problem that in TagConressionContext and CompressionContext, we use the result of InputStream incorrectly, as we just cast it to byte and test whether it is -1 to determine whether the field is in the dict. The return value of InputStream.read is an int, and it will return -1 if reaches EOF, but here we will consider it as not in dict... We should throw EOFException instead.

I'm not sure whether fix this can also fix HBASE-27073 but let's have a try first.
Bug: Incremental backups should re-use splits from last full backup
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Feb/23 22:49

All incremental backups require a previous full backup. Full backups use snapshots + ExportSnapshot, which includes exporting the SnapshotManifest. The SnapshotManifest includes all of the regions in the table during the snapshot.

Incremental backups use WALPlayer to turn new HLogs since last backup into HFiles. This uses HFileOutputFormat2, which writes HFiles along the split boundaries of the tables at the time that it runs.

Active clusters may have regions split and merge over time, so the split boundaries of incremental backup hfiles may not align to the original full backup. This means we need to use MapReduceHFileSplitterJob during restore in order to read all of the hfiles for all of the incremental backups and re-split them based on the restored table.
 * So let's say a cluster with regions A, B, C does a full backup. Data in that backup will be segmented into those 3 regions.
 * Over time the cluster splits and merges and we end up with totally different regions D, E, F. An incremental backup occurs, and the data will be segmented into those 3 regions.Later the cluster splits those 3 regions so we end up with new regions G, H, I, J, K, L.  Then next incremental backup goes with that

When we go to restore this cluster, it'll pull the full backup and the 2 incrementals. The full backup will get restored first, so the new table will have regions A, B, C.  Then all of the hfiles from the incrementals will be combined together and run through MapReduceHFileSplitterJob. This will cause all of those data files to get re-partitioned based on the A, B, C regions of the newly restored table (based on the full backup).

This splitting process is expensive on a large cluster. We could skip it entirely if incremental backups used the RegionInfos from the original full backup SnapshotManifest as the splits for WALPlayer. Therefore, all incremental backups will use the same splits as the original full backup. The resulting hfiles could be directly bulkloaded without any split process, reducing cost and time of restore.

One other benefit is that one could use the combination of a full backup + all incremental backups as an input to their own mapreduce job. This impossible now because all of the backups will have HFiles with different start/end keys which don't align to a common set of splits for combining into ClientSideRegionScanner.
Bug: Ignore invalid hostname when getNetworkInterfaces
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Feb/23 03:56

TestRegionServerHostname#testRegionServerHostname timeout in my CI pipeline.

!image-2023-02-23-11-54-42-036.png|width=503,height=98!

That's because NetworkInterface.getNetworkInterfaces get kube-ipvs0(k8s ipvs) with so many InetAddresses.

!image-2023-02-23-11-55-28-798.png|width=662,height=44!
Bug: hfile split occurs during bulkload, the new HFile file does not specify favored nodes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Mar/23 05:34

## BulkloadHFilesTool.class

/**
* Copy half of an HFile into a new HFile.
*/
private static void copyHFileHalf(Configuration conf, Path inFile, Path outFile,
Reference reference, ColumnFamilyDescriptor familyDescriptor) throws IOException {
FileSystem fs = inFile.getFileSystem(conf);
CacheConfig cacheConf = CacheConfig.DISABLED;
HalfStoreFileReader halfReader = null;
StoreFileWriter halfWriter = null;
try {
ReaderContext context = new ReaderContextBuilder().withFileSystemAndPath(fs, inFile).build();
StoreFileInfo storeFileInfo =
new StoreFileInfo(conf, fs, fs.getFileStatus(inFile), reference);
storeFileInfo.initHFileInfo(context);
halfReader = (HalfStoreFileReader) storeFileInfo.createReader(context, cacheConf);
storeFileInfo.getHFileInfo().initMetaAndIndex(halfReader.getHFileReader());
Map<byte[], byte[]> fileInfo = halfReader.loadFileInfo();

int blocksize = familyDescriptor.getBlocksize();
Algorithm compression = familyDescriptor.getCompressionType();
BloomType bloomFilterType = familyDescriptor.getBloomFilterType();
HFileContext hFileContext = new HFileContextBuilder().withCompression(compression)
.withChecksumType(StoreUtils.getChecksumType(conf))
.withBytesPerCheckSum(StoreUtils.getBytesPerChecksum(conf)).withBlockSize(blocksize)
.withDataBlockEncoding(familyDescriptor.getDataBlockEncoding()).withIncludesTags(true)
.withCreateTime(EnvironmentEdgeManager.currentTime()).build();
*halfWriter = new StoreFileWriter.Builder(conf, cacheConf, fs).withFilePath(outFile)*
 *.withBloomType(bloomFilterType).withFileContext(hFileContext).build();*
HFileScanner scanner = halfReader.getScanner(false, false, false);
scanner.seekTo();

...

 

When hfile splitting occurs during bulkload, the new HFile file does not specify favored nodes, which will affect the locality of data. Internally, we implemented a version of the code that allows us to specify the favored nodes of the split HFile in copyHFileHalf() to avoid compromising locality
Bug: Enhancements for the persistent cache
Status: Resolved | Priority: Major | Resolution: Delivered
Created: 22/Mar/23 18:06

Currently, we maintain a list of Hfiles on disk for which prefetch is complete and avoid prefetching those files after a restart. But we don't handle cases where blocks are evicted from the cache or when the region is moved and the blocks from that region are evicted (or not evicted). The Hfile is kept in the prefetch list forever and over time the prefetch list would have all the files of the cluster and the prefetch would never be executed. This change aims to update the prefetched list of HFiles upon block eviction.
Bug: Get slow/large log response that matched the ‘CLIENT_IP' without client port
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Feb/23 17:07
Affects Version/s: 2.4.14

'get_largelog_responses' and 'get_slowlog_responses' support filter the records with the given client by 'CLIENT_IP', but user has to provide the client ip and client port both. The client uses an ephemeral port mostly, there will be a better user experience if 'CLIENT_IP' without port is supported, then all the records matched the client ip will be returned.
Bug: Clean up ScanMetrics API
Status: Open | Priority: Major | Resolution: 
Created: 26/Mar/23 19:28

ScanMetrics object exposes public instance variables for all metrics. For example, ScanMetrics.countOfRPCcalls. This is not standard API design in java or in hbase, and requires suppressing VisibilityModifier checkstyle warnings. We should clean this up, but would require major version changes since it's part of public API
Bug: Support Ozone as a WAL backing storage
Status: Open | Priority: Major | Resolution: 
Created: 21/Mar/23 01:33

As discussed in the [thread|https://lists.apache.org/thread/tcrp8vxxs3z12y36mpzx35txhpp7tvxv], we'd like to make HBase workloads possible on Ozone.

This feature is to be built on top of 
# HDDS-7593 (support hsync and lease recovery in Ozone), and
# HADOOP-18671 (Add recoverLease(), setSafeMode(), isFileClosed() APIs to FileSystem).
Bug: Exclude the older versions of netty pulling from Hadoop dependencies
Status: Reopened | Priority: Major | Resolution: 
Created: 08/Mar/23 09:53

Currently the netty version of 3.10.6 is getting pulled from hdfs dependencies and sonatype kind of tools reporting the CVEs in HBase. To get rid of this better to exclude netty where hdfs or mapred client jars used.


 * org.apache.hbase : hbase-it : jar : tests : 2.5.2
 ** org.apache.hadoop : hadoop-mapreduce-client-core : 3.2.2
 *** io.netty : netty : 3.10.6.final
 ** org.apache.hbase : hbase-endpoint : 2.5.2
 *** org.apache.hadoop : hadoop-hdfs : jar : tests : 3.2.2
 **** io.netty : netty : 3.10.6.final
 *** org.apache.hadoop : hadoop-hdfs : 3.2.2
 **** io.netty : netty : 3.10.6.final
 * org.apache.hadoop : hadoop-mapreduce-client-jobclient : 3.2.2
 ** io.netty : netty : 3.10.6.final
 ** org.apache.hadoop : hadoop-mapreduce-client-common : 3.2.2
 *** io.netty : netty : 3.10.6.final
 * org.apache.hadoop : hadoop-mapreduce-client-jobclient : jar : tests : 3.2.2
 ** io.netty : netty : 3.10.6.final
 * org.apache.hadoop : hadoop-mapreduce-client-hs : 3.2.2
 ** io.netty : netty : 3.10.6.final
 ** org.apache.hadoop : hadoop-mapreduce-client-app : 3.2.2
 *** io.netty : netty : 3.10.6.final
 *** org.apache.hadoop : hadoop-mapreduce-client-shuffle : 3.2.2
 **** io.netty : netty : 3.10.6.final
Bug: Connection and Request Attributes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Feb/23 14:28

Currently we have the ability to set Operation attributes, via Get.setAttribute, etc. It would be useful to be able to set attributes at the request and connection level.

These levels can result in less duplication. For example, send some attributes once per connection instead of for every one of the millions of requests a connection might send. Or send once for the request, instead of duplicating on every operation in a multi request.

Additionally, the Connection and RequestHeader are more globally available on the server side. Both can be accessed via RpcServer.getCurrentCall(), which is useful in various integration points – coprocessors, custom queues, quotas, slow log, etc. Operation attributes are harder to access because you need to parse the raw Message into the appropriate type to get access to the getter.

I was thinking adding two new methods to Connection interface:

- setAttribute (and getAttribute/getAttributes)

- setRequestAttributeProvider

Any Connection attributes would be set onto the ConnectionHeader during initialization. The RequestAttributeProvider would be called when creating each RequestHeader.

An alternative to setRequestAttributeProvider would be to add this into HBaseRpcController, which can already be customized via site configuration.
Bug: Implement a tool to migrate replication peer data between different storage implementation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Mar/23 05:40

Replication peer data is usually stable unless you manually modify it, so it is OK to not migrate automatically. Instead, we could provide a tool to migrate the replication peer data.

And after migration, users can use online configuration change or via a cluster a restart to load the new configuration.
Bug: Shutdown zookeeper logs coming via ReadOnlyZKClient when hbase shell started
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 10/Feb/23 17:45

When hbase shell with HBase 2.5.2 started there is too much logging of zk connection realated, classpaths etc.  Even though we enabled ERROR log level for zookeeper package.
{noformat}
2023-02-10 17:34:25,211 INFO  [ReadOnlyZKClient-host1:2181,host2:2181,host3:2181@0x15c16f19] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.5.9-5-a433770fc7b303332f10174221799495a26bbca2, built on 02/07/2023 13:02 GMT
2023-02-10 17:34:25,212 INFO  [ReadOnlyZKClient-host1:2181,host2:2181,host3:2181@0x15c16f19] zookeeper.ZooKeeper: Client environment:host.name=host1
2023-02-10 17:34:25,212 INFO  [ReadOnlyZKClient-host1:2181,host2:2181,host3:2181:2181@0x15c16f19] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_352
2023-02-10 17:34:25,212 INFO  [ReadOnlyZKClient-host1:2181,host2:2181,host3:2181@0x15c16f19] zookeeper.ZooKeeper: Client environment:java.vendor=Red Hat, Inc.
2023-02-10 17:34:25,212 INFO  [ReadOnlyZKClient-host1:2181,host2:2181,host3:2181@0x15c16f19] zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.352.b08-2.el7_9.x86_64/jre

{noformat}
Better to change the  org.apache.hadoop.hbase.zookeeper package log level to error.
{noformat}
# Set logging level to avoid verboseness
org.apache.logging.log4j.core.config.Configurator.setAllLevels('org.apache.zookeeper', log_level)
org.apache.logging.log4j.core.config.Configurator.setAllLevels('org.apache.hadoop', log_level)

org.apache.logging.log4j.core.config.Configurator.setAllLevels('org.apache.hadoop.hbase.zookeeper', log_level)

{noformat}
Bug: User metrics for filtered and read rows are too expensive
Status: Open | Priority: Major | Resolution: 
Created: 09/Mar/23 19:22

MetricsUserAggregateImpl has a pattern like this:
{code:java}
String user = getActiveUser();
if (user != null) {
  MetricsUserSource userSource = getOrCreateMetricsUser(user);
  incrementFilteredReadRequests(userSource);
} {code}
So every update involves a getOrCreate call, which does a ConcurrentHashMap lookup. This overhead is not too bad for most requests, because it's just executed once per request (i.e. updatePut gets called once at the end, though multi's it happens for every action).

For updateFilteredReadRequests and updateReadRequestCount, these are currently called in RegionScannerImpl for every row scanned or filtered.  Doing the map lookup over and over adds up. Profiling the regionserver under load, I see over 5% of the time spent updating these metrics.

We should try to collect these metrics maybe in the RpcCallContext, and the translate into user metrics once at the end of the request. Or otherwise find a way to minimize querying the ConcurrentHashMap multiple times in the context of a request. Maybe we should actually stash the MetricsUserSource in the RpcCallContext so that all user metrics only need to do the lookup once, even for multi's.
Bug: Inconsistent synchronization in MetricsUserSourceImpl
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Mar/23 19:30

Picked up by spotbugs:

 
||Code||Warning||
|IS|Inconsistent synchronization of org.apache.hadoop.hbase.regionserver.MetricsUserSourceImpl.appendHisto; locked 50% of time|
| |[Bug type IS2_INCONSISTENT_SYNC (click for details)|https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5067/7/artifact/yetus-general-check/output/new-spotbugs-hbase-hadoop-compat.html#IS2_INCONSISTENT_SYNC]
In class org.apache.hadoop.hbase.regionserver.MetricsUserSourceImpl
Field org.apache.hadoop.hbase.regionserver.MetricsUserSourceImpl.appendHisto
Synchronized 50% of the time
Unsynchronized access at MetricsUserSourceImpl.java:[line 253]
Synchronized access at MetricsUserSourceImpl.java:[line 146]|
|IS|Inconsistent synchronization of org.apache.hadoop.hbase.regionserver.MetricsUserSourceImpl.getHisto; locked 50% of time|
| |[Bug type IS2_INCONSISTENT_SYNC (click for details)|https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5067/7/artifact/yetus-general-check/output/new-spotbugs-hbase-hadoop-compat.html#IS2_INCONSISTENT_SYNC]
In class org.apache.hadoop.hbase.regionserver.MetricsUserSourceImpl
Field org.apache.hadoop.hbase.regionserver.MetricsUserSourceImpl.getHisto
Synchronized 50% of the time
Unsynchronized access at MetricsUserSourceImpl.java:[line 241]
Synchronized access at MetricsUserSourceImpl.java:[line 141]|
|IS|Inconsistent synchronization of org.apache.hadoop.hbase.regionserver.MetricsUserSourceImpl.incrementHisto; locked 50% of time|
| |[Bug type IS2_INCONSISTENT_SYNC (click for details)|https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5067/7/artifact/yetus-general-check/output/new-spotbugs-hbase-hadoop-compat.html#IS2_INCONSISTENT_SYNC]
In class org.apache.hadoop.hbase.regionserver.MetricsUserSourceImpl
Field org.apache.hadoop.hbase.regionserver.MetricsUserSourceImpl.incrementHisto
Synchronized 50% of the time
Unsynchronized access at MetricsUserSourceImpl.java:[line 247]
Synchronized access at MetricsUserSourceImpl.java:[line 145]|
|IS|Inconsistent synchronization of org.apache.hadoop.hbase.regionserver.MetricsUserSourceImpl.scanTimeHisto; locked 50% of time|
| |[Bug type IS2_INCONSISTENT_SYNC (click for details)|https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5067/7/artifact/yetus-general-check/output/new-spotbugs-hbase-hadoop-compat.html#IS2_INCONSISTENT_SYNC]
In class org.apache.hadoop.hbase.regionserver.MetricsUserSourceImpl
Field org.apache.hadoop.hbase.regionserver.MetricsUserSourceImpl.scanTimeHisto
Synchronized 50% of the time
Unsynchronized access at MetricsUserSourceImpl.java:[line 264]
Synchronized access at MetricsUserSourceImpl.java:[line 142]|
Bug: Quotas can drastically overflow configured limit
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Mar/23 22:17

The original implementation did not allow exceeding quota. For example, you specify a limit of 10 resource/sec and consume 20 resources, it takes 1.1 seconds to be able submit another request. This was covered by the [testOverconsumption in TestRateLimiter|https://github.com/apache/hbase/blame/587b0b4f20bdc0415b6541023e611b69c87dba15/hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestRateLimiter.java#L97]. As an incidental part of HBASE-13686, that logic was changed. There is no mention of the reasoning behind the change in the issue comments or review board, I think it was missed. The goal of that issue was to add different refill strategies, but it also modified the over consumption. The testOverconsumption was [split out for both refill strategies|https://github.com/apache/hbase/blame/master/hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestRateLimiter.java#L104-L159], but the core reasoning was lost. The comment says:
{code:java}
// 10 resources are available, but we need to consume 20 resources109    
// Verify that we have to wait at least 1.1sec to have 1 resource available {code}
But the actual test was updated to only require a new resource after 100ms. This is incorrect. 

The problem is, when consuming if you go negative it sets to 0 [here|https://github.com/apache/hbase/blame/master/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/RateLimiter.java#L187-L191]. Additionally, when refilling the new logic does a Math.max(0, available + refillAmount): [here|https://github.com/apache/hbase/blame/master/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/RateLimiter.java#L159-L163]. So it's really impossible to get below 0, which is impractical for a rate limiter. 

With this setup it's very easy to drastically overconsume the rate limiter. See attached screenshot, which shows two humps. The first one has the current logic, the second hump has my fix which removes both of those problems. The rate limit was set to 500mb/s, but I was easily able to go over 700 mb/s without the fix.
Bug: CopyOnWriteArrayMap does not honor contract of ConcurrentMap.putIfAbsent
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Feb/23 14:32

I've been tracking down a meta cache issue and noticed this. I'm not sure if this is actually causing an issue, but if you enable TRACE logs on MetaCache it causes all calls to cacheLocation to go down the "Merged cached locations" path instead of the "Cached location" path for inserting new entries. This is just misleading, but also means going through the more expensive RegionLocations.mergeLocations. Reading the code, I don't think that would cause an actual issue, but hard to be sure.
Bug: Recovery of BucketCache and Prefetched data after RS Crash
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Mar/23 17:27

HBASE-27313 introduced the ability to persist a list of hfiles for which prefetch has already been completed, so the we can avoid prefetching those files again in the event of a graceful restart, but it doesn't cover crash scenarios, as if the RS is killed or abnormally stopped, the list wouldn't be saved. 

This change aims to persist the list of already prefetched from a background thread that periodically checks cache state and persists the list if updates have happened.