av9ash/GitBugs huggingface dataset:
Bug: Metrics for tracking of active scanners
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Feb/22 13:07

Active scanners take up resources on the RegionServer. This can result in GC issues or even OOMs in cases of leaked scanners. It'd be helpful to have metrics for tracking scanners. I was thinking of:
 * activeScanners - a gauge, showing the number of active scanners at a point in time
 * scannerLeaseExpirations - a counter of lease expirations
Bug: TableInputFormatBase should cache RegionSizeCalculator
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Mar/22 13:56

TableInputFormatBase's getSplits() method instantiates a new RegionSizeCalculator every time. Instantiating a RegionSizeCalculator involves scanning for all regionlocations for a given table in meta. This can be costly for large tables, and we don't know how often a subclass will call getSplits().

When initializeTable is called, we already cache the RegionLocator and Admin that are used for passing into the RegionSizeCalculator. We should similarly cache the RegionSizeCalculator itself at that same time to avoid unnecessary meta scans on repeat getSplits() calls.
Bug: LICENSE handling should not allow non-aggregated "apache-2.0"
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Feb/22 19:11
Affects Version/s: 2.4.10

our non-aggregated license check currently allows the malformed name "apache-2.0". All ALv2 licenses should be aggregated.

- remove "apache-2.0" from the {{non_aggregate_fine}} list
- figure out what dependencies are referring to this license and correct them in {{supplemental-models.xml}}

nice to have: maybe add a precommit check that complains if the {{non_aggregate_fine}} set is changed?
Bug: Allow disable of region warmup before graceful move
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 31/Jan/22 22:17

We have encountered two issues with region warmup before the assignment manager gracefully moves a region from one regionserver to another. 

The first instance is HBASE-26722. Part of the failure chain is temporary double assignment like conditions where both the source and destination regionservers think they have exclusive rights to storefile management (as they should) but warmup opens a region before it is closed and then both regionservers take compaction related actions. While this can be remediated with more care to this case, it is unclear if warmup affords significant advantage. The motivation of the original commit in 2015 introducing this feature was avoidance of blockcache misses once region ownership transfer was advertised to clients. Depending on use case and additional default-false schema options (like preload) this could be valuable. Or not.

The second instance is in place upgrade from HBase 1 to HBase 2. In a scenario where regionservers have been replaced by HBase 2 versions, but the master is still HBase 1, the region warmup RPC request fails. This is not particularly harmful but indicates it will not be useful during the transition period. 

We think it would be good to allow for warmup before move to be optionally disabled by a site configuration setting. In particular there have been many unrelated changes committed since 2015 and expectations of invariants in the contribution of the warmup-on-move feature have been invalidated. HBASE-26722 may be the only case, or it might not.  This is a fairly trivial change.
Bug: Extend hbase shell 'status' command to support an option 'tasks'
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Feb/22 02:04

Extend status command to support an option 'tasks', which will iterate over live regionservers and pretty print the status of any active MonitoredTasks.

1. Expose monitored tasks state in ClusterStatus API via new option in ServerLoad

2. Add shell support for interrogating monitored tasks state in ServerLoad

Here is a prototype running on a small test cluster of a fork of branch-2.4: 

 !Screen Shot 2022-02-03 at 5.57.26 PM.png!
Bug: TestMobStoreScanner (testGetMassive) can OOME
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Feb/22 01:12
Affects Version/s: 2.4.10

Running TestMobStoreScanner, looks like it may need to go on a diet. The test case testGetMassive will reliably OOME for me. It might be because I am using an aarch64 Java 8 JVM and surefire.Xmx as currently defined in pom.xml at 2200m is not quite enough. 

{noformat}
2022-02-18T17:04:04,886 ERROR [RS:0;buildbox:45315] util.OOMEChecker(40): Run out of memory; RSRpcServices will abort itself immediate
ly
java.lang.OutOfMemoryError: Java heap space
{noformat}

Where the OOME is thrown is usually not germane so the rest of the stacktrace is not included. 
A repeatable failure for me.
Bug: Minor code cleanup in and around RpcExecutor
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Mar/22 17:08
Affects Version/s: 2.5.0

While working on tracing, I see some minor cleanup that can be done around the RpcExecutor class. The implementations of the {{dispatch}} methods don't actually throw the exceptions defined in their signatures, and there's some simplification to be done and checkstyle warnings to resolve while I'm in there.
Bug: Minor code cleanup in and around RpcScheduler
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Mar/22 16:46
Affects Version/s: 2.5.0

Following the coattails of HBASE-26782.

See detailed compatibility discussion on https://github.com/apache/hbase/pull/4173
See a related concern regarding {{IA.LimitedPrivate}} annotations on class hierarchies on HBASE-26817.
Bug: RpcRetryingCallerImpl translateException ignores return value of recursive call
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Mar/22 14:59
Affects Version/s: 2.4.11

RpcRetryingCallerImpl.translateException recurses if the translated exception is a ServiceException. The stated reason is because the ServiceException may be a RemoteException, which should be unwrapped. However, the return value of that recursion is ignored so the RemoteException is not being properly unwrapped (unless it's a DoNotRetryIOException in which case it will throw)

Looking at the similar translateException call in the AsyncRpcRetryingCaller, it looks like that bug was fixed there. We should similarly fix in the blocking client for branch-2.
Bug: Fix NPE in the retry of logroller
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Mar/22 02:39
Affects Version/s: 2.4.10

{code:java}
LOG.warn("Retry to roll log, nAttempts={}, waiting time={}ms, sleeping 1s to retry,"
+ " last excepiton= {}", nAttempts, waitingTime,
ioe.getCause().getClass().getSimpleName()); {code}
ioe.getCause() maybe null.
Bug: Normalizer exits without normalizing all the tables
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Feb/22 14:22
Affects Version/s: 1.7.1

Currently if there are multiple tables to normalize, normalizer is exiting even before iterating all the table if there is a table that doesnot require normalization

here is the offending code

[https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java#L1736]

 
{code:java}

List<NormalizationPlan> plans = this.normalizer.computePlansForTable(table);   
if (plans == null || plans.isEmpty()) {
   return true;
} {code}
this is running inside loop over tables
Bug: [hbase-thirdparty] hbase-shaded-jersey packages unshaded classes
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 27/Jan/22 23:58

Found artifact with unexpected contents: '/home/apurtell/src/hbase/hbase-shaded/hbase-shaded-mapreduce/target/hbase-shaded-mapreduce-2.5.0-SNAPSHOT.jar'

com/sun/activation/
javax/activation/
javax/xml/bind/
Bug: Bump checkstyle from 8.28 to 8.29
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 31/Mar/22 21:16
Affects Version/s: 2.4.11

Dependabot generated pull request. Bumps checkstyle from 8.28 to 8.29. The security fix is disabling the 'external-parameter-entities' feature by default.
Bug: Fix CME when closing region
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 18/Mar/22 10:27
Affects Version/s: 2.4.11

KeyValueHeap.scannersForDelayedClose is not thread safe, CME might occurs when closing a scanning region, who is adding scanners to the  delayed close list.
{code:java}
2022-03-18 17:24:17,435 INFO  [regionserver/hostname/ip:16020-splits-0] regionserver.SplitRequest: Running rollback/cleanup of failed split of hmart-waimaiups_poi_bitmap_theme_pool,,1645769651132.56363df04da85752aa67669989de19fd.; java.util.ConcurrentModificationException
java.io.IOException: java.util.ConcurrentModificationException
        at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1767)
        at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1593)
        at org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:394)
        at org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:276)
        at org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:567)
        at org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
        at org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:156)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.ConcurrentModificationException
        at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
        at java.util.ArrayList$Itr.next(ArrayList.java:851)
        at org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:221)
        at org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:548)
        at org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:520)
        at org.apache.hadoop.hbase.regionserver.HStore.closeChangedReadersObservers(HStore.java:1296)
        at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:1001)
        at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:131)
        at org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1745)
        at org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1741)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        ... 3 more {code}
Bug: Add unit test for custom RpcControllerFactory's ability to overwrite priorities
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Mar/22 18:32

We're planning on using the pluggable nature of RpcControllerFactory to send along custom priorities that are calculated based on some special logic in our stack. Unfortunately it's virtually impossible to test this from user code. On the other hand it's relatively easy to test within the hbase codebase. We should add a test class which verifies for users that this logic works. Then it's just up to them to test their own implementation of their HBaseRpcController rather than do the full integration test.
Bug: Make MetricsConnection scope configurable
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Mar/22 14:05

Currently MetricsConnection scope is just the connection.toString(), which is typically the default toString() implementation for an object. This is not very useful for identifying different connections in processes that might connect to multiple clusters or maintain different connections to the same cluster.

We can add a new config param "hbase.client.metrics.scope" which defaults to connection.toString().
Bug: Misspelling commands in hbase shell will crash the shell
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Mar/22 23:16
Affects Version/s: 2.4.10

Misspelling commands in hbase shell will crash the shell, e.g.

{noformat}
hbase:001:0> hlep
NameError: undefined local variable or method `hlep' for #<HBaseReceiver:0x1610c03c>
Did you mean?  help
                  evaluate at (hbase):1
                      eval at org/jruby/RubyKernel.java:1048
                  evaluate at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/workspace.rb:85
                  evaluate at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/context.rb:380
                eval_input at uri:classloader:/irb/hirb.rb:112
             signal_status at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:623
                eval_input at uri:classloader:/irb/hirb.rb:109
  each_top_level_statement at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/ruby-lex.rb:246
                      loop at org/jruby/RubyKernel.java:1442
  each_top_level_statement at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/ruby-lex.rb:232
                     catch at org/jruby/RubyKernel.java:1189
  each_top_level_statement at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/ruby-lex.rb:231
                eval_input at uri:classloader:/irb/hirb.rb:108
                       run at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:428
                     catch at org/jruby/RubyKernel.java:1189
                       run at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:427
                    <main> at classpath:/jar-bootstrap.rb:221
{noformat}
Bug: Bump httpclient from 4.5.3 to 4.5.13
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Mar/22 16:12

Dependabot auto-generated dependency upgrade: https://github.com/apache/hbase/pull/4291

We can't accept the dependabot PR as-is because it causes a unit test failure. Bump the dependency and fix the test by hand. 

There is a comment in our POM indicating this is a known issue:

{noformat}
    <!-- Updating the httpclient will break hbase-rest. It writes out URLs with '//' in it                                                                                          
      especially when writing out 'no column families'. Later httpclients collapse the '//'                                                                                         
      into single '/' as double-slash is not legal in an URL. Breaks #testDelete in                                                                                                 
      TestRemoteTable. -->  
{noformat}

Staying back on a version of httpclient with CVE listed vulnerabilities just for this isn't a good option.
Bug: [REST] Client must disable URI normalization
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 29/Mar/22 16:32

There is a comment in our POM indicating we cannot upgrade httpclient, which is problematic. 

{noformat}
    <!-- Updating the httpclient will break hbase-rest. It writes out URLs with '//' in it                                                                                          
      especially when writing out 'no column families'. Later httpclients collapse the '//'                                                                                         
      into single '/' as double-slash is not legal in an URL. Breaks #testDelete in                                                                                                 
      TestRemoteTable. -->  
{noformat}

This behavior in httpclient manifests when URI normalization is enabled, which is the default. The client must disable it, Otherwise some required URI constructions for performing various actions against the REST gateway are impossible.
Bug: Fix compatibility issues in 2.4.11RC0
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Mar/22 22:28

package org.apache.hadoop.hbase.http
SecurityHeadersFilter
LimitedPrivate(CONFIG)
Removed method: SecurityHeadersFilter.getDefaultParameters ( Configuration conf ) [static]  :  Map<String,String>

package org.apache.hadoop.hbase.filter
RandomRowFilter
Public
Field random of type java.util.Random has been removed from this class. A client program may be interrupted by NoSuchFieldError exception.
Bug: Move AsyncProcess configs to HConstants
Status: Open | Priority: Minor | Resolution: 
Created: 15/Mar/22 12:39

AsyncProcess has a few public static config constants, but the class itself is package-private so users cannot access them. Moving them to HConstants will make it easier for users to change these settings:
{code:java}
hbase.client.primaryCallTimeout.multiget
hbase.client.start.log.errors.counter
hbase.client.log.batcherrors.details
hbase.client.log.detail.period.ms
{code}
Bug: Move ConnectionImplementation public configs to HConstants
Status: Open | Priority: Minor | Resolution: 
Created: 15/Mar/22 12:43

ConnectionImplementation has a few public static config constants, but the class itself is package-private so users cannot access them. Moving them to HConstants will make it easier for users to change these settings:
{code:java}
hbase.client.retries.by.server
hbase.resolve.hostnames.on.failure {code}
Note: AsyncConnectionImpl also separately defines the second config, so we can unify them both in HConstants.
Bug: Expose meta lookup configs in HConstants
Status: Open | Priority: Minor | Resolution: 
Created: 15/Mar/22 12:37

This makes it easier for clients to set this config in an upgrade-compatible way. In other words, if a future release changes or removes these configs it will be clear to users because imports will break. Otherwise hbase may start silently ignoring these fields:
{code:java}
hbase.hconnection.meta.lookup.threads.max
hbase.hconnection.meta.lookup.threads.core{code}
Bug: MultiRowRangeFilter does not appear in hbase-client/src/main/java/org/apache/hadoop/hbase/filter/ParseFilter.java
Status: Resolved | Priority: Minor | Resolution: Works for Me
Created: 13/Mar/22 13:25
Affects Version/s: 2.4.4

You can instantiate a Filter using the REST API. For Instance, the following is the body of the POST request against the REST API  that instantiates a filter list comprised of a RowFilter and a ColumnRangeFilter:

<Scanner><filter>{"type": "FilterList", "op": "MUST_PASS_ALL", "filters": [\{"type": "RowFilter", "op": "EQUAL", "comparator": {"type": "BinaryComparator", "value": "Rm1vYy1BbGFuaW5lLU5lZ2F0aXZlLUNFLVJhbXAtRm1vYy1BbGFuaW5lIFtNLUhdLS0wMDAwMDAwMDA5NDk5"}}, \{"type": "ColumnRangeFilter", "minColumn": "MDAwNTAuMDI2OTA=", "minColumnInclusive": true, "maxColumn": "MDA1MDAuMDI2OTA=", "maxColumnInclusive": true}]}</filter></Scanner>

However, it seems impossible to do that with a MultiRowRangeFilter (using the REST API). Upon closer look, this class does not seem to appear in the ParseFilter class (hbase-client/src/main/java/org/apache/hadoop/hbase/filter/ParseFilter.java). It is completely missing.
Bug: Bump dependencies in /dev-support/git-jira-release-audit
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Mar/22 19:37

Bumps urllib3 from 1.25.8 to 1.26.5 to resolve two dependabot warnings

    - CRLF injection (Moderate) urllib3 (pip) · dev-support/git-jira-release-audit/requirements.txt

    - Catastrophic backtracking in URL authority parser when passed URL containing many @ characters (High) urllib3 (pip) · dev-support/git-jira-release-audit/requirements.txt

Bumps cryptography from 2.8 to 3.3.2 to resolve one dependabot warning

    - RSA decryption vulnerable to Bleichenbacher timing vulnerability (Moderate) cryptography (pip) · dev-support/git-jira-release-audit/requirements.txt
Bug: [HBase-Client] In some cases, no value is obtained using method Result.getValue()
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 21/Jan/22 08:09
Affects Version/s: 2.0.0

Results are not sorted during the query by Result.getValue(). As a result, existing values cannot be queried.

In normal cases, the results by get or scan are sorted and this problem does not occur. However, in some cases, such as when the HFile file is read by MR and the cells obtained are combined as result by rowkey, there is no sorting.  Result.getvalue (byte [] family, byte [] qualifier) is null.  

 
 * The test results are shown in the figures

!1.png!!2.png![^Test.java]
Bug: Fix flappy test TestSimpleRegionNormalizerOnCluster.java
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Feb/22 07:24
Affects Version/s: 1.7.1

TestSimpleRegionNormalizerOnCluster.java can hang after HBASE-26744
The assumption that order of HTable list is sorted is wrong so depending on that order can cause the test to hang or be inaccurate
Bug: Make ageOfLastShip, ageOfLastApplied  extend TimeHistogram instead of plain histogram.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Jan/22 14:21
Affects Version/s: 1.7.1

Currently age of last ship metric is an instance of an Histogram type. [Here|https://github.com/apache/hbase/blob/master/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSourceImpl.java#L58]
{quote}
   ageOfLastShippedOpHist = rms.getMetricsRegistry().getHistogram(SOURCE_AGE_OF_LAST_SHIPPED_OP);
{quote}

We can change it to TimeHistogram so that we get the range information also.
Bug: NPE caused by converting uppercase hostname to lowercase in RegionMover
Status: Open | Priority: Minor | Resolution: 
Created: 27/Jan/22 13:57
Affects Version/s: 2.4.9

In HBASE-19456, we introduced case-insensitivity feature in RegionMover and converted uppercase hostnames to lowercase hostnames. But this maybe causes that we can't get the rsgroup info of unloading server, because the addresses in hbase are case insensitive. This will make org.apache.hadoop.hbase.util.TestRegionMoverWithRSGroupEnable fail.
 
 
2022-01-27T20:53:31,948 INFO [Time-limited test] util.TestRegionMoverWithRSGroupEnable(127): Unloading {*}VM{*}-154-75-centos
2022-01-27T20:53:31,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=49232] master.MasterRpcServices(3011): rsGroupInfo of {*}vm{*}-154-75-centos:39126 is null
2022-01-27T20:53:31,961 INFO [pool-332-thread-1] util.RegionMover(419): rsgroup of {*}vm{*}-154-75-centos:39126 is null
2022-01-27T20:53:31,961 ERROR [pool-332-thread-1] util.RegionMover(471): Error while unloading regions
java.lang.NullPointerException: null
at org.apache.hadoop.hbase.util.RegionMover.lambda$unloadRegions$3(RegionMover.java:421) ~[classes/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_292]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
Bug: Backport HBASE-24443 Refactor TestCustomSaslAuthenticationProvider
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Jan/22 09:05

Test execution times for TestCustomSaslAuthenticationProvider on branch-2 takes 24 minutes. The refactored tests after HBASE-24443 on master succeed in just under 2 * 2 minutes.