av9ash/GitBugs huggingface dataset:
Bug: PressureAwareCompactionThroughputController#tune log the opposite of the actual scenario
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 21/Apr/23 06:26
Affects Version/s: 1.5.0

In [https://github.com/apache/hbase/commit/07c71d630cd293ab55ac85e9bfa06033598134c4] ,    the condition of the if statement is incorrectly modified.

!image-2023-04-21-14-28-19-766.png!
Bug: Fix disallowed compatibility breaks on branch-2.5 and branch-2
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 01/Jun/22 02:11
Affects Version/s: 2.5.0

Change to Public interfaces without deprecation:

HTableDescriptor
Field DEFAULT_NORMALIZATION_ENABLED of type boolean has been removed from this class.

TableDescriptorBuilder
Field DEFAULT_NORMALIZATION_ENABLED of type boolean has been removed from this class.

RandomRowFilter
Field random of type java.util.Random has been removed from this class. 	A client program may be interrupted by NoSuchFieldError exception.

ReplicationLoadSink.ReplicationLoadSink ( long age, long timestamp, long timestampStarted, long totalOpsProcessed ) 
Access level has been changed from public to private. 	A client program may be interrupted by IllegalAccessError exception.

RSGroupInfo.getServers:()Ljava/util/Set;
Return value type has been changed from java.util.Set<org.apache.hadoop.hbase.net.Address> to java.util.SortedSet<org.apache.hadoop.hbase.net.Address>. 
This method has been removed because the return type is part of the method signature. A client program may be interrupted by NoSuchMethodError exception.

These changes are not allowed by policy for a minor release so I have committed fixes on my own authority as RM to branch-2.5 and branch-2. Further changes can be pursued on branch-2.
Bug: TestReplicationValueCompressedWAL.testMultiplePuts is flaky
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 29/May/22 19:34
Affects Version/s: 2.5.0

org.apache.hadoop.hbase.replication.regionserver.TestReplicationValueCompressedWAL.testMultiplePuts
   Run 1: TestReplicationValueCompressedWAL.testMultiplePuts:56 Waited too much time for replication
  Run 2: PASS
Bug: HMaster should exit gracefully, when stopped via hbase-daemon.sh
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 13/Apr/22 04:11
Affects Version/s: 3.0.0-alpha-2

Since ShutdownHook is not registered at startup, when HMaster is stopped through hbase-daemon.sh, HMaster cannot exit gracefully,  the threads and services of HMaster cannot be closed normally, and the master local region also cannot be closed normally, which may lead to some inexplicable problems.
Bug: Improve the comments and test coverage for compaction progress implementation
Status: Open | Priority: Major | Resolution: 
Created: 17/Apr/22 15:00

When fixing HBASE-26938, we also found the root cause that why sometimes we will get broken compaction progress value, i.e, a negative or more than 100%, it is because multiple compaction can happen at the same time but only use one compaction prorgress, which is not thread safe to track them all.

We have fixed this problem in HBASE-26938, but it is suggested to improve the test coverage of this ares so we will not make a mistake in the future again.
Bug: TestRetainAssignmentOnRestartSplitWithoutZk infinite loop
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Apr/22 09:55

{noformat}
2022-04-05T01:56:10,567 ERROR [Thread-350] server.NIOServerCnxnFactory$1(92): Thread Thread[Thread-350,5,FailOnTimeoutGroup] died
java.lang.IllegalThreadStateException: null
        at java.lang.Thread.start(Thread.java:794) ~[?:?]
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.startWorkers(ProcedureExecutor.java:616) ~[classes/:?]
        at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1666) ~[classes/:?]
        at org.apache.hadoop.hbase.master.TestRetainAssignmentOnRestart$HMasterForTest.access$101(TestRetainAssignmentOnRestart.java:57) ~[test-classes/:?]
        at org.apache.hadoop.hbase.master.TestRetainAssignmentOnRestart$HMasterForTest.lambda$startProcedureExecutor$0(TestRetainAssignmentOnRestart.java:70) ~[test-classes/:?]
        at java.lang.Thread.run(Thread.java:834) ~[?:?]
2022-04-05T01:56:10,570 INFO  [RS:2;4d77efa4cf73:37349] regionserver.HRegionServer(1179): stopping server 4d77efa4cf73,37349,1649123763933
2022-04-05T01:56:10,570 DEBUG [PEWorker-5] procedure2.AbstractProcedureScheduler(158): the scheduler is not running
2022-04-05T01:56:10,570 DEBUG [RS:2;4d77efa4cf73:37349] ipc.AbstractRpcClient(505): Stopping rpc client
2022-04-05T01:56:10,570 DEBUG [PEWorker-5] procedure2.AbstractProcedureScheduler(158): the scheduler is not running
2022-04-05T01:56:10,570 DEBUG [PEWorker-5] procedure2.AbstractProcedureScheduler(158): the scheduler is not running
2022-04-05T01:56:10,570 INFO  [RS:2;4d77efa4cf73:37349] regionserver.HRegionServer(1529): Waiting on 1 regions to close
2022-04-05T01:56:10,570 DEBUG [PEWorker-5] procedure2.AbstractProcedureScheduler(158): the scheduler is not running
2022-04-05T01:56:10,570 DEBUG [RS:2;4d77efa4cf73:37349] regionserver.HRegionServer(1533): Online Regions={3b6c92c681d8cf1316520e3a89546fe0=restartTableThree,,1649123769361.3b6c92c681d8cf1316520e3a89546fe0.}
2022-04-05T01:56:10,570 DEBUG [PEWorker-5] procedure2.AbstractProcedureScheduler(158): the scheduler is not running
2022-04-05T01:56:10,570 DEBUG [PEWorker-5] procedure2.AbstractProcedureScheduler(158): the scheduler is not running
2022-04-05T01:56:10,570 DEBUG [PEWorker-5] procedure2.AbstractProcedureScheduler(158): the scheduler is not running
2022-04-05T01:56:10,570 DEBUG [PEWorker-5] procedure2.AbstractProcedureScheduler(158): the scheduler is not running
{noformat}

After a strange exception then we enter an infinite loop and output a lot of log messages which eat all the spaces...
Bug: Update the doc for read replica configuration
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Apr/22 14:59
Affects Version/s: 2.4.11

The doc for read replica is somewhat outdated： there is no ‘hbase.region.replica.replication.memstore.enabled’ in the code now， which is replaced by per-table property `REGION_MEMSTORE_REPLICATION`, and if we set `REGION_MEMSTORE_REPLICATION` to false,`hbase.region.replica.wait.for.primary.flush` will be ignored.
Bug: Add security headers to Thrift/HTTP server 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jun/22 11:16
Affects Version/s: 2.0.6

In order to further improve HBase exposed service for vulnerability scanners, would be nice to implement additional security headers for the Thrift server when HTTP or HTTPS transport is enabled.

Similarly to REST and Web UIs, related tickets are attached.
Bug: Should use FileStatus to archive expired MOB files instead of construct HStoreFile object
Status: Open | Priority: Major | Resolution: 
Created: 16/Jun/22 12:17
Affects Version/s: 2.4.12

The MobUtils#removeMobFiles reused codes in HFileArchiver#archiveStoreFiles, which is used to archive compacted files on RSes under normal conditions. In HFileArchiver#archiveStoreFiles, store files are closed and removed, while MOB cleaner should only rename the expired files and no need to build new HStoreFile object before archive and close it after removed.
Bug: Config BucketCache as victim handler of LRUCache
Status: Open | Priority: Major | Resolution: 
Created: 06/May/22 06:35

Bucket cache is no longer as a victim handler for LRU by HBASE-19357.
{noformat}
When BC is used, data blocks will be strictly on BC only where as index/bloom blocks are on LRU L1 cache.
{noformat}
In this situation, the LRU cache's memory is totally on heap.

If the index and bloom filters on a region server are big, then a big LRU cache will introduce more GC cost.

So we should add a *configuration for user* to choose victim handler.

A small LRU cache with a big victim offheap bucket cache can reduce GC cost.

 

For example: a region server has 5GB index, 80 GB bloom filter, and 256GB DRAM, its configurations can be as follow:
{code:xml}
export HBASE_REGIONSERVER_OPTS="-Xms40g -Xmx40g -XX:MaxDirectMemorySize=180g"
{code}

{code:xml}
<property>
    <name>hbase.blockcache.victim.handler.enabled</name>
    <value>true</value>
</property>  

<property>
    <name>hfile.block.cache.size</name>
    <value>0.3</value>
</property> 

<property>
    <name>hbase.bucketcache.ioengine</name>
    <value>offheap</value>
</property>
<property>
    <name>hbase.bucketcache.size</name>
    <value>160000</value>
</property>
{code}
Bug: Move replication peer storage from zookeeper to other storage systems
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jun/22 13:00

This is a more specific issue based on the works which are already done in HBASE-15867.

There are several candidates for storing replication peer.

1. A new family of master local region
2. On FileSystem
3. A new family of hbase:meta

Either choice has pros and cons. We need to decide which way to go first.
Bug: HMaster's ShutdownHook does not take effect, if tablesOnMaster is false
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Apr/22 12:44
Affects Version/s: 1.7.1

Because we are going to upgrade the HBase kernel version, during the test, I found that the HMaster ShutdownHook of version 1.7.1 and 2.4.9 did not take effect if tablesOnMaster is false. After further confirmation, I found that all versions of HBase 1.x and 2.x have this problem.

I analyzed the cause of the problem：

Before HBase 3.x, HMaster registered ShutdownHook by calling the run() method of HRegionServer, but the run() method will be blocked on the waitForMasterActive() method of HMaster(before registering the shutdownhook) if tablesOnMaster is false. Eventually, the ShutdownHook of HMaster was not registered successfully.

1. The code path of  HRegionServer's run() method：

!image-2022-04-26-21-18-24-884.png!

 

2.  The preRegistrationInitialization() method will be called first:

!image-2022-04-26-21-12-51-843.png!

3. The HMaster's waitForMasterActive() method will be blocked if tablesOnMaster is false:

!image-2022-04-26-21-22-38-572.png!

Finally, because the default value of tablesOnMaster is false, the ShutdownHook of HMaster was not registered successfully.
Bug: Upgrade scalatest maven plugin for thread-safety
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jun/22 11:39
Affects Version/s: hbase-connectors-1.0.1

The {{master}} branch on the connectors repo warns when {{--threads}} is issued, the complaint being the scalatest-maven-plugin. Looks like the latest version resolves the complaint. Let's upgrade.
Bug: Exclude files which are not in source control as much as possible when running spotless
Status: In Progress | Priority: Major | Resolution: 
Created: 07/Jun/22 15:04

In HBASE-27084, we added spotless:check to mvn verify stage, but the general format section in spotless will include some files which are not in source control, such as .idea/* or .settings/*, and then fail the 'mvn install'.

Although usually it could be fixed by a spotless:apply call, but IDE may regenerate the file and fail the build again.

Let's add more exclude patterns to make it work better.
Bug: The length of compact queue grows too big when the compacting is slow
Status: Open | Priority: Major | Resolution: 
Created: 29/Apr/22 02:26

For some system compaction, we set the selectNow to false, so the file selecting will not be done until the compaction running, it brings side effect, if another compacting is slow, we may put lots of compaction to queue, because the filesCompacting of Hstore is empty in the meantime.

An example shows at attachments, there are 154 regions and about 2000 hfiles, but the length of compact queue grows to 1391, it cause confusion and may trigger unexpected alarm.

My approach is limit the compaction queue count, by compute the filesNotCompating and hbase.hstore.compaction.max.
Bug: Disable Style/FrozenStringLiteralComment for ruby
Status: Open | Priority: Major | Resolution: 
Created: 12/May/22 03:11

By default Style/FrozenStringLiteralComment is enabled in rubocop. If we update a ruby file, rubocop prompts `Missing frozen string literal comment` (see [https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/2/artifact/yetus-general-check/output/diff-patch-rubocop.txt)|https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/2/artifact/yetus-general-check/output/diff-patch-rubocop.txt].

To address this warn, we need to add `# frozen_string_literal: true` to the top of the ruby file(see https://github.com/rubocop/rubocop/blob/master/config/default.yml#L3637), which will be added to the top of the `Apache License` and will look strange.

I don't think this `FrozenStringLiteralComment` check is very necessary. We can disable it.
Bug: Replication Observability Framework
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Apr/22 13:59

In our production clusters, we have seen cases where data is present in source cluster but not in the sink cluster and 1 case where data is present in sink cluster but not in source cluster. 

We have internal tools where we take incremental backup every day on both source and sink clusters and we compare the hash of the data in both the backups. We have seen many cases where hash doesn't match which means data is not consistent between source and sink for that given day. The Mean Time To Detect (MTTD) these inconsistencies is atleast 2 days and requires lot of manual debugging.

We need some tool where we can reduce MTTD and requires less manual debugging.

I have attached design doc. Huge thanks to [~bharathv]  to come up with this design at my work place.
Bug: Server should close scanner if client times out before results are ready
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jun/22 18:18

When heartbeats are enabled, we try to return from a scan before {{clientTimeout / 2}} millis has passed. Previously, this did not account for queue times so could still easily timeout. That problem was handled in HBASE-27048. It's still possible to timeout if heartbeats are disabled, we have queued for longer than \{{ clientTimeout / 2 }} millis and scan is slow, or if the RegionScanner is otherwise delayed in returning.

How a scanner timeout is handled by the client depends on the point at which the timeout occurred:
 * In openScanner(), the call will be retried. We will not have received a scannerId, so cannot close that scanner that may have been open on the server side.
 * In next(), the timeout will bubble up and fail the scan. In this case we try to close the scanner, but that could be interrupted if server is overwhelmed (close call gets queued and then dropped) or client terminates.

Active scanners carry with them a non-trivial amount of memory and resource overhead on the server. In my experience, if a server becomes overwhelmed, client scanners can start to time out. Those scanners live on on the server, contributing to memory and resource pressure. That further slows down the server, etc. This is especially problematic when openScanner times out because of the inherent retries of that call, with a single scanner possibly contributing multiple "leaked" scanners on the server before finally failing.

We should attempt to close the scanner on the server side when a scan call takes longer than the client timeout to finish. I think this would be a matter of adding something like this to the end of RSRpcServices.scan:
{code:java}
if (EnvironmentEdgeManager.currentTime() > rpcCall.getDeadline()) {
  throw new TimeoutIOException("Client deadline exceeded, cannot return results");
}{code}
We already have a catch of IOException, wherein we close the scanner. The actual exception thrown shouldn't matter much since the client will not receive the response.
Bug: s390x: Skip tests on unsupported compression libs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jun/22 12:18

Since https://issues.apache.org/jira/browse/INFRA-21433, there are four s390x Jenkins build nodes available. Could the HBase CI be extended to build on s390x as well?

There is an old issue open about adding s390x support to the HBase CI but it has not received any update in quite some time. https://issues.apache.org/jira/browse/HBASE-25192. There was some interest in starting by adding s390x to the nightly builds. Would that still be the best way to start?

Let me know if there are any questions. I'm happy to help in any way I can.
Bug: Create WAL event tracker table to track all the WAL events.
Status: Resolved | Priority: Major | Resolution: Implemented
Created: 04/Apr/22 15:07

Design Doc: [https://docs.google.com/document/d/14oZ5ssY28hvJaQD_Jg9kWX7LfUKUyyU2PCA93PPzVko/edit#]

Create wal event tracker table to track WAL events. Whenever we roll the WAL, we will save the WAL name, WAL length, region server, timestamp in a table.
Bug: Investigate Netty resource usage limits
Status: Resolved | Priority: Major | Resolution: Done
Created: 12/Jun/22 17:55
Affects Version/s: 2.5.0

We leave Netty level resource limits unbounded. The number of threads to use for the event loop is default 0 (unbounded). The default for io.netty.eventLoop.maxPendingTasks is INT_MAX. 

We don't do that for our own RPC handlers. We have a notion of maximum handler pool size, with a default of 30, typically raised in production by the user. We constrain the depth of the request queue in multiple ways... limits on number of queued calls, limits on total size of calls data that can be queued (to avoid memory usage overrun, CoDel conditioning of the call queues if it is enabled, and so on.

Under load can we pile up a excess of pending request state, such as direct buffers containing request bytes, at the netty layer because of downstream resource limits? Those limits will act as a bottleneck, as intended, and before would have also applied backpressure through RPC too, because SimpleRpcServer had thread limits ("hbase.ipc.server.read.threadpool.size", default 10), but Netty may be able to queue up a lot more, in comparison, because Netty has been optimized to prefer concurrency.

Consider the hbase.netty.eventloop.rpcserver.thread.count default. It is 0 (unbounded). I don't know what it can actually get up to in production, because we lack the metric, but there are diminishing returns when threads > cores so a reasonable default here could be Runtime.getRuntime().availableProcessors() instead of unbounded?

maxPendingTasks probably should not be INT_MAX, but that may matter less.

The tasks here are:
- Instrument netty level resources to understand better actual resource allocations under load. Investigate what we need to plug in where to gain visibility. 
- Where instrumentation designed for this issue can be implemented as low overhead metrics, consider formally adding them as a metric. 
- Based on the findings from this instrumentation, consider and implement next steps. The goal would be to limit concurrency at the Netty layer in such a way that performance is still good, and under load we don't balloon resource usage at the Netty layer.

If the instrumentation and experimental results indicate no changes are necessary, we can close this as Not A Problem or WontFix.
Bug: Backport missing MOB related changes to branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jun/22 15:10
Affects Version/s: 2.6.0

While trying to backport https://issues.apache.org/jira/browse/HBASE-26969 to branch-2 I have found that multiple major MOB related changes are missing. 

This change is required for FileBased SFT correctness so the changes it depends on should be backported first. Also any improvement to MOB stability is usually welcomed.

The missing changes I have found so far:
https://issues.apache.org/jira/browse/HBASE-22749
https://issues.apache.org/jira/browse/HBASE-23723
https://issues.apache.org/jira/browse/HBASE-24163

There is also a docs change describing the new MOB functionality. But considering that the book is always generated based on master I think it is safe to skip backporting it.
https://issues.apache.org/jira/browse/HBASE-23198

I'm planning to backport these changes one by one until we reach a state where HBASE-26969  can be backported too.
Bug: Allow sharing connections between AggregationClient instances
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/May/22 20:47

AggregationClient only has a single constructor which takes a Configuration. The constructor uses the Configuration to create a Connection.

However, some of the AggregationClient methods take a Table argument. In those cases it doesn't use the created Connection at all.

We should add another constructor which does not create a Connection so that people can use AggregationClient with externally managed Connection.
Bug: Use AsyncConnection in ReplicationSink
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Apr/22 18:27
Affects Version/s: 2.4.11

We don't need to necessarily rewrite ReplicationSink to work fully async. I think it would simply benefit from ConnectionFactory.createAsyncConnection instead of ConnectionFactory.createConnection.

The reasons for this are:
 * AsyncConnection is the more modern implementation, the only implementation in master, and where most of the efforts will be going forward.
 * ReplicationSink only does batch calls, and batch calls are done with AsyncProcess. It's likely that the native AsyncTable is better than AsyncProcess for this.
 ** One specific example, AsyncProcess calls findAllLocationsOrFail sequentially for all actions in a batch. This can take quite a while with the default replication batch size of 5k, if actions are spread across many regions. In AsyncTable, these calls are done in parallel
Bug: Possible resource leak while creating new region scanner
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Apr/22 17:12
Affects Version/s: 2.4.11

We will miss to close the region scanner when RegionObserver#postScannerOpen impl throws exception.
https://github.com/apache/hbase/blob/1d8a5bf0e3246a1df11f0aa9215f29e6fbc4ecb1/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L3203

https://github.com/apache/hbase/blob/1d8a5bf0e3246a1df11f0aa9215f29e6fbc4ecb1/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L3207
Bug: Add snapshot scanner UT with SFT and some cleanups to TestTableSnapshotScanner
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Apr/22 21:28
Affects Version/s: 2.4.11

This is to replicate current TableSapshotScanner UTs to run over an SFT cluster. Just extending current TableSapshotScanner with overrides to setup method. Also applied some fix/cleanups to TableSapshotScanner.
Bug: MOB snapshot is broken when FileBased SFT is used
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/May/22 13:33
Affects Version/s: 2.5.0

During snapshot MOB regions are treated like any other region. When a snapshot is taken and hfile references are collected a StoreFileTracker is created to get the current active hfile list. But the MOB region stores are not tracked so an empty list is returned, resulting in a broken snapshot. When this snapshot is cloned the resulting table will have no MOB files or references.

The problematic code can be found here:

[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifest.java#L313]