av9ash/GitBugs huggingface dataset:
Bug: Bump up jruby to 9.3.9.0 and related joni and jcodings to 2.1.43 and 1.0.57 respectively
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jan/23 05:45

Bump up Jruby to 9.3.9.0 to ensure compliance which has multiple CVEs fixed related to openssl,snakeyaml etc.
 * rdoc has been updated to 6.3.3 to fix all known CVEs. ([#7396|https://github.com/jruby/jruby/issues/7396], [#7404|https://github.com/jruby/jruby/issues/7404])
 * rexml has been updated to 3.2.5 to fix all known CVEs. ([#7395|https://github.com/jruby/jruby/issues/7395], [#7405|https://github.com/jruby/jruby/issues/7405])
 * jruby-openssl has been updated to 0.14.0 to fix weak HMAC key hashing in bouncycastle, which itself is updated to 1.71. ([#7335|https://github.com/jruby/jruby/issues/7335], [#7385|https://github.com/jruby/jruby/issues/7385], [#7399|https://github.com/jruby/jruby/issues/7399])
 * psych has been updated to 3.3.4 to fix CVE-2022-38752 in the SnakeYAML library, which itself is updated to 1.33. ([#7386|https://github.com/jruby/jruby/issues/7386], [#7388|https://github.com/jruby/jruby/issues/7388], [#7400|https://github.com/jruby/jruby/issues/7400])
 * rubygems has been updated to 3.2.33 and bundler updated to 2.2.33 to address CVE-2021-43809. ([#7397|https://github.com/jruby/jruby/issues/7397], [#7401|https://github.com/jruby/jruby/issues/7401])
Bug: Region with one huge hfile should split
Status: Open | Priority: Major | Resolution: 
Created: 02/Feb/23 03:23
Affects Version/s: 2.4.14

I found some regions in our production cluster have one huge hfile and can not split automatically anymore, it causes the poor read performance.

If a region loaded lots of data via bulkload, it will split after compaction done. The reference files are cleared by 'CompactedHFilesDischarger' asynchronized, so the daughter regions will not split after compaction finished. If the daughter regions have no more write operation, they will never split although the rewrote hfile exceed the disired  max hfile size(ConstantSizeRegionSplitPolicy), 

!image-2023-02-02-11-25-16-499.png!
Bug: Scan quotas and limits should account for total block IO
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jan/23 17:03

Scan and Multi requests pull the byte throughput limit from Quotas.getReadAvailable(). Multis validate the result inline in RSRpcServices, by checking the accumulated {{RpcCallContext.getResponseCellSize}} and {{getResponseBlockSize}} against the read available after each action. Scans make use of {{{}ScannerContext{}}}, and only checks the total cell serialized size and {{{}cell.heapSize(){}}}.

The handling for Multis was added in HBASE-14978. The block size is checked because regardless of the actual cell size, the regionserver needs to retain entire blocks backing those cells for the lifetime of a request. If the retained blocks grows too large, a regionserver can OOM or experience heavy GC pressure.

So multis validate read available against the actual block size retained for the responses, but scans only account for cell sizes. We should extend the same block support to scans through ScannerContext tracking block bytes scanned.

Large scans can read over ranges of both returned and filtered rows. Despite what's returned the users, the server-side cost of the scan is just as impacted by filtered rows as non-filtered.

Both Scans and Multis take the Math.min of Quotas read available and hbase.server.scanner.max.result.size. Scans further take the min of that and Scan.setMaxResultSize.
Bug: The release script will fail if we run publish-dist step separately
Status: Open | Priority: Major | Resolution: 
Created: 30/Jan/23 14:34

Because of can not find CHANGES.md and RELEASENOTES.md in the parent directory. For 3.0.0+, we will not include CHANGES.md and RELEASENOTES.md in git repo so the solution is to mv it to parent directory and let later step to use it, but if we execute publish-dist separately, the solution can not work.
Bug: Bump up commons-codec to 1.15
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jan/23 05:50

commons-codec 1.15 has proper fix of few CVEs which may not effect in HBase but better to upgrade to ensure compliance.
Ex: While [a fix|https://github.com/apache/commons-codec/commit/48b615756d1d770091ea3322eefc08011ee8b113] was earlier made to {{commons-codec:commons-codec}} version 1.13, it was later found out to be incomplete. A [complete fix|https://github.com/apache/commons-codec/pull/29] exists in version 1.14 and that is the version users should upgrade to.
Bug: Remove -X option when building protoc check in nightly and pre commit job
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jan/23 09:15

We eat too many spaces on jenkins

{noformat}
root@jenkins-cm8 /home/jenkins/jenkins-home/jobs/HBase Nightly/branches # find . -type f -size +100G |xargs ls -l
-rw-r--r-- 1 jenkins jenkins 128604282880 Jan 21 06:45 ./branch-2-4.ib7ujg/builds/497/archive/output-general/patch-hbaseprotoc-root.txt
-rw-r--r-- 1 jenkins jenkins 249367112376 Jan 20 12:10 ./branch-2-5.lbpvbe/builds/282/archive/output-general/patch-hbaseprotoc-root.txt
-rw-r--r-- 1 jenkins jenkins 249367112376 Jan 20 12:10 ./branch-2-5.lbpvbe/builds/282/htmlreports/General_20Nightly_20Build_20Report/patch-hbaseprotoc-root.txt
-rw-r--r-- 1 jenkins jenkins 249572082864 Jan 20 21:50 ./branch-2/builds/723/archive/output-general/patch-hbaseprotoc-root.txt
-rw-r----- 1 jenkins jenkins 123016740864 Jan 21 06:45 ./branch-2/builds/723/htmlreports/General_20Nightly_20Build_20Report/patch-hbaseprotoc-root.txt
-rw-r--r-- 1 jenkins jenkins 247567066688 Jan 20 10:48 ./master/builds/759/archive/output-general/patch-hbaseprotoc-root.txt
-rw-r--r-- 1 jenkins jenkins 247567066688 Jan 20 10:48 ./master/builds/759/htmlreports/General_20Nightly_20Build_20Report/patch-hbaseprotoc-root.txt
{noformat}
Bug: Update hadoop netty version for hadoop-2.0 profile
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jan/23 14:51

In HBASE-27299 we updated our minimum hadoop 2 dependency in branch-2 to 2.10.2, but we left our hadoop.netty.version at 3.6.2.Final.  According to [https://github.com/apache/hadoop/blob/rel/release-2.10.2/hadoop-project/pom.xml#L721,] the correct version is 3.10.6.Final.

This is causing a NoSuchMethodError in our newly backported TestRemoteRestore tests. Bumping to 3.10.6.Final locally fixes the tests.
Bug: Add config options to delay assignment to retain last region location
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/23 12:56

HBASE-27313 introduced the ability to persist the list of files cached in a given RS, but temporary RSes loss or restarts would cause regions to be eagerly reassigned on other RSes, making the persisted cache useless. For some use cases, such as when using ObjectStores based persistence, performance degradation caused by cache misses have a worse impact than temporary region unavailability.  

This proposes and additional config property (disabled by default) to forcibly wait the TRSP for a configurable time while checking for the previous RS holding region to get back online, before proceeding with the region assignment.
Bug: Clear meta cache for full server when handling FailedServerException
Status: Open | Priority: Major | Resolution: 
Created: 25/Jan/23 16:08

Currently we prefer to clear meta cache for an individual region that fails. This is preferred in most cases, because clearing cache for an entire server is much more expensive. If a server hosts 100 regions, unnecessarily clearing the cache for the entire server would cause 100 meta requests per client.

However, when a client fails to connect to a regionserver, it gets added to the FailedServers list. Subsequent requests to that server are fast-failed, throwing a FailedServerException.

This is a pretty clear indicator that there's a problem with a specific server. In this case I think we should clear the cache for that full server.

We had a production incident recently where a server completely hung and we did see "Clear Region" calls, but the server hosted many regions and the meta clears continued for a while longer than necessary. Adding "Clear Server" call due to FailedServers would have mitigated this issue much quicker.
Bug: L1 cache leaks index blocks over time when under subscribed
Status: Open | Priority: Major | Resolution: 
Created: 24/Jan/23 22:09

Let's say you have CombinedBlockCache enabled. DATA goes to L2, INDEX/BLOOM go to L1.  Your regionserver has index size of 2gb and bloom size of 1gb. So you really only need around 3gb of L1 to fully hold all of the "L1 candidates".

When data set does not fit into cache, LRU will handle evictions to stay under max. But in the above scenario, if you configure 6gb for L1 (3 more than needed) over time you will end up filling that entire 6gb with old INDEX blocks. Once you reach max, LRU will handle evicting out the oldest ones.

Since the leak is contained to the configured max L1 size, this isn't a huge issue but it results in heap waste. Under high heap allocations, if you haven't left enough buffer outside memstore, L1, etc, you will start seeing GC pressure. This L1 leak then becomes a little more problematic, because you end up in a circumstance where longer lived regionservers (who've leaked closer to the max L1 size) have less extra buffer available than more newly restarted regionservers.

The best fix is to appropriately set your L1 size so there is not a lot of excess, but this can be painful to maintain over time as clusters shrink, grow, or data shape changes. It'd be a lot better if the L1 did not leak so you don't have to so finely tune the L1.

I haven't fully figured out where the leak comes from, but I think it's related to compactions. Perhaps the INDEX blocks are not being evicted as hfiles are compacted away. The leak is very linear over time in our experience.
Bug: Get supports RAW
Status: Open | Priority: Major | Resolution: 
Created: 17/Jan/23 07:39

[https://github.com/apache/hbase/blob/da261344cc55e7812dfe22d86d5fa88c93ed79b9/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java#L234]

 

I used the `Get` to query all put and delete in a column, but I got this error

*Cannot specify any column for a raw scan.*

Why add this restriction?
Bug: Make the initial corePoolSize configurable for ChoreService
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jan/23 17:14

The initial corePoolSize for ChoreService is set to 1. The pool size is increased when a scheduled task misses its start time.

On a cluster where the archive size is large, the HFileCleaner could run for a very long time and block the rest of the chores to run.

By making the initial pool size configurable we could solve the bottleneck of caused by a long-running HFileCleaner chore.
Bug: [hbase-connectors] org.apache.hadoop.hbase.spark.TestJavaHBaseContext failing with HBase 2.5.x
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 03/Jan/23 11:45

{noformat}

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.hbase.spark.TestJavaHBaseContext
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 5.157 s <<< FAILURE! - in org.apache.hadoop.hbase.spark.TestJavaHBaseContext
[ERROR] org.apache.hadoop.hbase.spark.TestJavaHBaseContext  Time elapsed: 5.142 s  <<< ERROR!
java.lang.NoClassDefFoundError: org/apache/logging/log4j/Level
    at org.apache.hadoop.hbase.spark.TestJavaHBaseContext.setUpBeforeClass(TestJavaHBaseContext.java:98)
Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.Level
    at org.apache.hadoop.hbase.spark.TestJavaHBaseContext.setUpBeforeClass(TestJavaHBaseContext.java:98)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]   TestJavaHBaseContext.setUpBeforeClass:98 » NoClassDefFound org/apache/logging/...
[INFO]
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO]

{noformat}
Bug: [JDK17] SecurityManager is deprecated since JDK17
Status: Open | Priority: Minor | Resolution: 
Created: 26/Jan/23 05:01

Our UTs will print out lots of WARNINGs if we run on JDK17:
{code:java}
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.apache.hadoop.hbase.SystemExitRule (file:/Users/tianhang.tang/workspaces/apache/hbase/hbase-common/target/test-classes/)
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.hbase.SystemExitRule
WARNING: System::setSecurityManager will be removed in a future release{code}
This is because [JEP 411|[https://openjdk.org/jeps/411],] Security Manager is deprecated since JDK17, and will removed in the future version after JDK18.

We introduced the SecurityManager in [HBASE-24185|https://issues.apache.org/jira/browse/HBASE-24185], mainly using ??checkExit?? to ensure that the JVM does not exit during testing. [JDK-8199704|https://bugs.openjdk.org/browse/JDK-8199704] may provide an API in the future, but at the moment we really don't seem to have an alternative available.

In short, this WARNING does not have any practical impact at present, what we need to do is to refactor HBASE-24185 in the future. But the large amount of logging is still annoying, and I haven't found a way to turn it off yet.
Bug: Upgrade commons-validator to version 1.7
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Mar/23 16:50

The current version of commons-validator (1.6) has a dependency on commons-beanutils-1.9.2.jar, this dependency comes with two CVEs:
- [CVE-2014-0114|https://nvd.nist.gov/vuln/detail/cve-2014-0114]
- [CVE-2019-10086|https://nvd.nist.gov/vuln/detail/cve-2019-10086]

With commons-validator version 1.7 these CVEs are no longer present.

I've also checked the master branch for usages. The only location where commons-validator is used is in org.apache.hadoop.hbase.zookeeper.ZKConfig for validating ipv6 addresses.
Bug: Add usage docs to HBase-Spark connector
Status: Open | Priority: Minor | Resolution: 
Created: 10/Feb/23 09:27

Documentation for the HBase Spark connectors is mostly non-existent.

We could start by adding most of the content from  [https://github.com/LucaCanali/Miscellaneous/blob/master/Spark_Notes/Spark_HBase_Connector.md]
to the Spark connector README.
Bug: Check if the file system supports storage policy before invoking setStoragePolicy()
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Mar/23 22:53

Found these messages on an Ozone cluster:

{noformat}
2023-03-20 12:27:09,185 WARN org.apache.hadoop.hbase.util.CommonFSUtils: Unable to set storagePolicy=HOT for path=ofs://ozone1/vol1/bucket1/hbase/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc. DEBUG log level might have more details.
java.lang.UnsupportedOperationException: RootedOzoneFileSystem doesn't support setStoragePolicy
        at org.apache.hadoop.fs.FileSystem.setStoragePolicy(FileSystem.java:3227)
        at org.apache.hadoop.hbase.util.CommonFSUtils.invokeSetStoragePolicy(CommonFSUtils.java:521)
        at org.apache.hadoop.hbase.util.CommonFSUtils.setStoragePolicy(CommonFSUtils.java:504)
        at org.apache.hadoop.hbase.util.CommonFSUtils.setStoragePolicy(CommonFSUtils.java:477)
        at org.apache.hadoop.hbase.regionserver.HRegionFileSystem.setStoragePolicy(HRegionFileSystem.java:225)
        at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:275)
        at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:6387)
        at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1115)
        at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1112)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
{noformat}

Ozone does not support storage policy. If we use FileSystem.hasPathCapability() API to check before invoking the API, these warning messages can be avoided.
Bug: hbase.master.port is ignored in processing of hbase.masters 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Jan/23 00:30
Affects Version/s: 2.5.2

Setting of hbase.master.port is ignored in default processing of hbase.masters. 

For example,

hbase.masters=localhost
hbase.master.port=8100

"Server at localhost:16000" is in the failed servers list"

hbase.masters=localhost:8100
hbase.master.port=8100

and all is well. This is confusing and a minor configuration handling bug.
Bug: Optimize writes of zero length values in compressed WALs
Status: Open | Priority: Minor | Resolution: 
Created: 14/Feb/23 21:16

If we unconditionally use the compressor, to "write" 0 bytes, then the compression codec will emit overheads... hadoop compressionstream header, compression bitstream header. All of that should be skipped so truly no compressed value data is written when the value is empty.
Bug: [JDK17] Backport to branch-2.4 HBASE-25058 HBase-27510
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 20/Mar/23 12:07

I use jdk17 on branch-2.4，encountered some WARN like:
{code:java}
WARN FilenoUtil : Native subprocess control requires open access to the JDK IO subsystem
Pass '--add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED' to enable.{code}
Backport HBASE-25058 and HBASE-27510 PR can resolve the warn problem.
Bug: Fall back to protoc osx-x86_64 on Apple Silicon
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Mar/23 14:54
Affects Version/s: 2.4.0

Building non-master branches on an Apple Silicon machine fails because there's no protoc binary available. Use a profile to fall back to the x86 version of the binary, as per

https://cwiki.apache.org/confluence/display/HADOOP/Develop+on+Apple+Silicon+%28M1%29+macOS
Bug: [HBCK2]  addFsRegionsMissingInMeta command should support dumping region list into a file which can be passed as input to assigns command
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Mar/23 19:54

_addFsRegionsMissingInMeta_ command currently outputs a command as last line of output which needs to be run with hbck2
{code:java}
assigns 22d30d9e332af3272302cf780da14c3c 43245731f82e5bb907a4433f688574c1 5a19939f4f219ab177dd5b376dcb882f 774514b1027846c4e3b6702e193ce03d 7f6ad3360e0a4811c4dace8c1a901f40 8cd363e4da1b95fd43166f451546ad63 90e3414947f9500ec01f6672103f29d0{code}
This is good, but the user has to copy and format the command, which can get really big depending on how many regions need to be assigned.

_addFsRegionsMissingInMeta_ should support a flag, say -f to facilitate dumping region list into a file, which can be passed onto as input to _assigns_ command via -i parameter.

Sample expected use-case:
{code:java}
# Dump output of command (in a formatted manner) to file
hbase hbck -j hbase-hbck2-version.jar addFsRegionsMissingInMeta -i table_list.txt -f regions_to_assign.txt

# Pass file as input to assigns
hbase hbck -j hbase-hbck2-version.jar assigns -i regions_to_assign.txt{code}
Bug: SlowLog does not include params for Mutations
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Jan/23 13:56

SlowLog params are extracted via [ProtobufUtil.getSlowLogParams|https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java#L2154]. This method has various if/else branches for each request type, but mutation (the line linked above) is incorrect. Currently it handles MutationProto, but it should be MutateRequest. A MutationProto is never passed into this method, only MutateRequests so any MutateRequests being passed in now will fall through to the default case which contains nothing useful about the request.

As part of fixing this, we should also ensure that we extract the region name from the MutateRequest to add into the SlowLogParams object like all the other requests.

While we are here, the CoprocessorServiceRequest (handled further down) has a getRegion() method, but that is not passed into the SlowLogParams either. We should add that too.
Bug: ChaosMonkey documentation enhancements
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Feb/23 05:30

Post refactoring the code to org.apache.hadoop.hbase.chaos package, new references need to be updated in the [https://hbase.apache.org/book.html#maven.build.commands.integration.tests.destructive] doc. The sample command given also is stale.
Bug: Read RPC threads may BLOCKED at the Configuration.get when using java compression
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Feb/23 11:23
Affects Version/s: 2.5.3

As in the jstack info, we can see some RPC threads or compaction threads BLOCKED,

!image-2023-02-27-19-22-52-704.png|width=976,height=355!
Bug: [JDK17] Fix failure TestHBaseTestingUtil#testResolvePortConflict
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Jan/23 11:17

{code:java}
Caused by: java.lang.IllegalAccessException: class net.bytebuddy.description.annotation.AnnotationDescription$ForLoadedAnnotation cannot access interface jdk.internal.util.random.RandomSupport$RandomGeneratorProperties (in module java.base) because module java.base does not export jdk.internal.util.random to unnamed module @20322d26
    at java.base/jdk.internal.reflect.Reflection.newIllegalAccessException(Reflection.java:392)
    at java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:674)
    at java.base/java.lang.reflect.Method.invoke(Method.java:560)
    at net.bytebuddy.description.annotation.AnnotationDescription$ForLoadedAnnotation.getValue(AnnotationDescription.java:719)
    ... 49 more{code}
!image-2023-01-25-14-56-07-904.png!
Bug: Fix update_all_config Table-level StoreHotnessProtector configuration was overwritten
Status: Open | Priority: Minor | Resolution: 
Created: 26/Mar/23 16:58
Affects Version/s: 2.3.4

When update_config was executed, the table level configuration was overwritten(hbase.region.store.parallel.put.limit), causing the table level personalized configuration to be restored to the global configuration unexpectedly.  Table level attributes should be minimum scoped and should not be affected by global configuration. 

1、alter table set hbase.region.store.parallel.put.limit to 100

   alter 'test',CONFIGURATION => {'hbase.region.store.parallel.put.limit' => 100}

2、update_all_config
   
   The table-level configuration will be overwrite
Bug: TestClusterRestartFailover is flakey due to metrics assertion
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Mar/23 14:48
Affects Version/s: 2.4.16

I'm seeing failures like this in PR,

{noformat}
[ERROR] Failures: 
[ERROR] org.apache.hadoop.hbase.master.TestClusterRestartFailoverSplitWithoutZk.test
[ERROR]   Run 1: TestClusterRestartFailoverSplitWithoutZk>TestClusterRestartFailover.test:143 serverNode should be deleted after SCP finished expected null, but was:<d09295c85a7e,33451,1678875566216/SPLITTING/regionCount=0/lock=java.util.concurrent.locks.ReentrantReadWriteLock@1823f707[Write locks = 0, Read locks = 0]>
[ERROR]   Run 2: TestClusterRestartFailoverSplitWithoutZk>TestClusterRestartFailover.test:147 serverCrashSubmittedCount(8) should be equal expected:<4> but was:<8>
[ERROR]   Run 3: TestClusterRestartFailoverSplitWithoutZk>TestClusterRestartFailover.test:147 serverCrashSubmittedCount(12) should be equal expected:<4> but was:<12>
{noformat}

Looks like subsequent runs would have passed, but for the firm metric count assertion.