av9ash/GitBugs huggingface dataset:
Bug: Add support for tcnative
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Aug/23 18:41

In investigating HBASE-27947, tcnative can make a big impact on throughput over built-in jdk SSL support. We need three things to make it work:
 # In X509Util, if Openssl.isAvailable() (meaning tcnative is on the classpath):
 ## Use SslProvider.OPENSSL_REFCNT
 ## Update default ciphers to remove CBC ciphers, which do not work with tcnative. We can either pull the ciphers from OpenSsl.availableJavaCipherSuites() or simply use the default GCM ciphers we already have defined.
 # Our netty is shaded, so one can't simply put the tcnative jar on the classpath. We might need to provide an hbase-shaded-netty-tcnative module which one can optionally include in their deployment. We will have to decide which of the many tcnative modules to provide a shaded version for.
Bug: Limit the maximum allocation when reading an HFileBlock
Status: Open | Priority: Major | Resolution: 
Created: 12/Sep/23 14:45

During PR discussion on HBASE-28065 we observe that the value of {{onDiskSizeWithoutHeader}} is read and used before its portion of an HFile has had its checksum validated. A method parameter is also provided which is used when the caller knows what size to expect based on some other source. While there are guards in place that limit the range of values this field can take, that range remains large, something like {{[33,Integer.MAX_VALUE)}}. We propose further limiting the range of this value to safeguard the region server from an excessively large allocation.

Conversation is in https://github.com/apache/hbase/pull/5384/files#r1322947549
Bug: [hbase-connectors] CVE-2020-15250 in org.glassfish:javax.el:jar:3.0.1-b08
Status: Open | Priority: Major | Resolution: 
Created: 12/Sep/23 03:07

This is a direct dependency in https://github.com/apache/hbase-connectors/blob/master/pom.xml 

*Version Affected*
[3.0-b01,3.0.1-b12]

*Issue*
org.glassfish:javax.el:jar:3.0.1-b08:compile cause vulnerability as mentioned in https://mvnrepository.com/artifact/org.glassfish/javax.el/3.0.1-b08
Bug: Add support for quotas on atomic operations
Status: Open | Priority: Major | Resolution: 
Created: 11/Sep/23 21:13

Atomic operations (CheckAndMutate, Increment, etc) are especially costly since they include both a random read and a write, and that occurs within a lock which can cause contention with other atomic reads. We find that HBase can only a serve a fraction of the atomic operations that it can of normal reads or writes. We should provide a separate quota for this so that one can tune it low while still allowing much higher normal read and write volume.
Bug: Hbase 2.4.13 vulnerable to CVE-2022-26612
Status: Open | Priority: Major | Resolution: 
Created: 07/Sep/23 13:26
Affects Version/s: 2.4.13

hbase 2.4.13 uses hadoop-common-2.10.0.jar which is vulnerable to CVE-2022-26612.

when replaced hadoop-common-2.10.0.jar with 3.2.3, getting version incompatible issue and as result hbase shell command failed.

is there any hbase version which is compatible with hadoop-common 3.2.3 or above?

or is there any hbase version available where the above CVE addressed?
Bug: [hbase-operator-tools] Add spotless for hbase-operator-tools
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jul/23 21:05

HBase code repo has spotless plugin to check and fix spotless issues seamlessly, making it easier for developers to fix issue in case the builds fails due to code formatting.

The goal of this Jira is to integrate spotless with hbase-operator-tools.
 * As a 1st step will try to add a plugin to run spotless check via maven
 * Next will fix all spotless issues as part of same task or another (as community suggests)
 * Finally will integrate the same to pre-commit build to not let PRs wit spotless issues get in. (Would need some support/direction on how to do this as I am not much familiar with the Jenkins and related code.)
Bug: Use correct units in RegionLoad#getStoreUncompressedSizeMB()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Aug/23 23:35
Affects Version/s: 2.5.5

When I run a snippet of code like this:
{code:java}
Map<byte[], RegionLoad> regionLoadMap = admin
  .getClusterStatus()
  .getLoad(
    ServerName.parseServerName(
      "my-server.my-company.net,60020,1693513660506"
    )
  )
  .getRegionsLoad();
for (byte[] startKey : regionLoadMap.keySet()) {
  RegionLoad regionLoad = regionLoadMap.get(startKey);
  LOG.info("Region {}: {}", Bytes.toStringBinary(startKey), regionLoad);
} {code}
I get logs like this:
{noformat}
Region <redacted-table-name>,<redacted row key>,1659484033280.2b89407a1223720344bed425bf3c29b0.: numberOfStores=1, numberOfStorefiles=3, storeRefCount=0, storefileUncompressedSizeMB=17998848, lastMajorCompactionTimestamp=1693211464712, storefileSizeMB=5895, compressionRatio=0.0003, memstoreSizeMB=1, readRequestsCount=118899553, writeRequestsCount=731192, rootIndexSizeKB=9, totalStaticIndexSizeKB=10413, totalStaticBloomSizeKB=6592, totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN, completeSequenceId=78093096, dataLocality=1.0{noformat}
The {{storefileUncompressedSizeMB}} is vastly larger than the {{{}storefileSizeMB{}}}, so much more that it's not believable. I checked the store files in question in this instance. Adding up the uncompressed size reported in the HFile trailers sums to 5895 MiB, exactly 1024 times less than the reported 17998848.

The reason for the misreporting is that {{RegionLoad#getStoreUncompressedSizeMB()}} converts the value from a {{Size}} object wrong.
Bug: An exception was thrown by org.apache.hadoop.hbase.io.asyncfs.FanOutBlockAsyncDFSOutputHelper$4.operationComplete()
Status: Open | Priority: Major | Resolution: 
Created: 04/Sep/23 14:43
Affects Version/s: 2.4.17

When I try running hbase using the code *./start-hbase.sh* , the terminal doesn't show an error or failure but logs do show it. Here I am dropping the link to the entire log file [log file|[http://justpaste.it/aqyjh|https://justpaste.it/aqyjh]] . I have tried some forums and reading documentation but nothing helped. Hadoop is compatible with Hbase version asHbase created a folder in Hadoop named */hbase* in the root directory of Hadoop and the directory does contain couple of directories (.hbck, .tmp, MasterData, WALs, archive, etc) and 2 files (hbase.id, hbase.version). So I don't think compatibility between Hadoop and Hbase is the issue here. Everything is running locally, literally everything as I am just testing some stuff. Also when I try doing *bin/hbase shell* , then the shell starts but anything I do in the shell for example if I type {*}status{*}, in the shell, I get the error which is same if I type any other code as "Error: KeeperErrorCode = ConnectionLoss for /hbase/master". I have uploaded two files of hbase which are the only two files I have altered to from the pre-built version of Apache. If you need any more information, please let me know.
Bug: Region (un)assignment should have a more direct timeout
Status: Open | Priority: Major | Resolution: 
Created: 14/Jul/23 19:37

h3. Problem

We've observed a few cases in which region (un)assignment can hang for significant, and sometimes seemingly indefinite, periods of time. This results in unpredictably long downtime which must be remediated via manually initiated ServerCrashProcedures.
h3. Example 1

If a RS is unable to communicate with the NameNode and it is asked to close a region then its RS_CLOSE_REGION thread will get stuck awaiting a NN failover. Due to several default configurations of options like:
 * hbase.hstore.flush.retries.number
 * hbase.server.pause
 * dfs.client.failover.max.attempts
 * dfs.client.failover.sleep.base.millis
 * dfs.client.failover.max.attempts

this region unassignment attempt will hang for approximately 30 minutes before it allows the failure to bubble up and automatically trigger a ServerCrashProcedure.

One can tune the aforementioned options to reduce the TTR here, but it's not a very obvious/direct solution.
h3. Example 2

In rare cases our public cloud provider may supply us with machines that have degraded hardware. If we're unable to catch this degradation prior to startup, then we've observed that the degraded RegionServer process may come online; as a result it will be assigned regions which can often never actually be successfully opened. If the RegionServer's assignment handling fails to intentionally fail, then there will never be outside intervention; the assignment will be stuck hanging indefinitely. I've written [a unit test|https://github.com/apache/hbase/compare/master...HubSpot:hbase:rsit-opening-repro] which reproduces this behavior. On this same branch is a unit test demonstrating that a timeout placed on the AssignRegionHandler helps to fast fail and reliably trigger the necessary ServerCrashProcedure.
h3. Proposal

I want to propose that we add optional and configurable timeouts to the AssignRegion and UnassignRegion event handlers.

This would allow us to much more intentionally & clearly prevent long running retries for these downtime inducing procedures and could consequently improve our reliability in both examples.
Bug: hbck2 bypass should provide an option to bypass existing top N procedures
Status: Open | Priority: Major | Resolution: 
Created: 22/Aug/23 22:46

For the degraded cluster state where several SCPs and underlying TRSPs are stuck due to network issues, it becomes difficult to resolve RITs and recover regions from SCPs.

In order to bypass stuck procedures, we need to extract and then provide list of proc ids from list_procedures or procedures.jsp page. If we could provide an option to bypass initial N procedures that are listed on the procedures.jsp page, that would be really helpful.

Implementation steps:
 # Similar to BypassProcedureRequest, provide BypassTopNProcedureRequest with only attribute value as N
 # MasterRpcServices to provide new API: 
 # 
{code:java}
bypassProcedure(RpcController controller,
  MasterProtos.BypassTopNProcedureRequest request) {code}

 # Hbck to provide utility to consume this master rpc
 # HBCK2 to use new hbck utility for bypassing top N requests

 

For this new option, top N procedures matter, hence they should follow the sorting order similar to the one followed by procedures.jsp.
Bug: Enhance ByteBufferUtils.findCommonPrefix to compare 8 bytes each time
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Aug/23 22:01
Affects Version/s: 2.5.4

Currently, the `ByteBufferUtils.findCommonPrefix family of methods compare two buffers a single byte at a time. In reviewing the patch for HBASE-28012, [~zhangduo] suggested that the ByteBufferUtils.findCommonPrefix methods could be enhanced to compare 8 bytes at a time like the ByteBufferUtils.compareToUnsafe family of methods already does (which was added in HBASE-12345)
Bug: Sync the hbck2 README page and hbck2 command help output
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jul/23 08:30

There are major differences in the hbck2 [README.md|https://github.com/apache/hbase-operator-tools/blob/master/hbase-hbck2/README.md] and the command help output, hence we should sync them across for all commands.

Ideally, it should be same as the output of hbck2 help command for ease of maintenance. 

Also few new commands like {{recoverUnknown}} and {{regionInfoMismatch}} are missing, making users unaware of existence of those.
Bug: Avoid CellUtil.cloneRow in BufferedEncodedSeeker
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Aug/23 02:06
Affects Version/s: 2.5.4

A RegionServer running a workload that consisted of mostly reverse (and some forward) scans of thousands of tall (36 byte rows) under moderate read pressure (based on block bytes scanned) saw near 100% CPU usage. We noticed that the high CPU usage tended to correlate with more reverse scan quantity and the throughput of the server running more reverse scans was vastly lower than expectation. 

 

A flame graph of the Region Server at 100% CPU usage. Analysis of the flamegraph reveals that each seek for a reverse scan (which has more seeks than the forward seek path) has 2 CellUtil.cloneRow rowkey copies per seek. These CellUtil#cloneRow calls can be removed from this code path for off-heap cells in a similar fashion to HBASE-27146
Bug: Make TestClusterScopeQuotaThrottle run faster
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/23 03:59

-The test always times out and it has several test methods.

Let's split the test into several smaller tests, so we can find out which one is the criminal.-

Finally I found that, the problem is we does not limit the operation timeout, just set the max retry number, but after some new improvements come in, sometimes we may get a 6 min sleep time and n doubt we will get a test timeout...

I changed the test a bit to set operation timeout on the Table instance we use, so it will fail immediately when we hit the quota throttling, and now the tests could finish very soon.

I think we could add another E2E tests to make sure that the refilling works as expected.
Bug: Downgrade or remove noisy log in AsyncRequestFutureImpl
Status: Open | Priority: Major | Resolution: 
Created: 18/Aug/23 13:21

In manageLocationError we construct a message which indicates that we failed to get a location for a particular action. We both log.error() that message, and also add it to an IOException which is set onto the final result array if retries are exhausted. This is redundant and very noisy, because those exceptions will often end up getting printed up the stack. We should remove the log, or at least downgrade to debug. One way it may be useful is if you are trying to inspect location failures for individual retries, which would not bubble up if they succeed. In that case you can enable debug loggin.
Bug: Adding unit test for FileNotFoundException when split a wal in SplitLogWorker
Status: Open | Priority: Major | Resolution: 
Created: 17/Aug/23 21:09
Affects Version/s: 3.0.0

In HBase-20583, it is proposed a buggy situation in which an FileException would not be handled in SplitLogWorker. However, in HBase-20583, the test patch is still missed. And in our reproduction, it needs sophisticated situation to trigger that and we would like to add a unit test path for it

The submitted patch containing the unit test and the explanation of the unit test are in the git pull: https://github.com/apache/hbase/pull/5358
Bug: DefaultMetricsSystemInitializer should be called during HMaster or HRegionServer creation
Status: Open | Priority: Major | Resolution: 
Created: 16/Aug/23 11:31

Concern is the jvm metrics somehow started populating with process name as IO instead of expected RegionServer/Master. See https://issues.apache.org/jira/browse/HBASE-27966 for RCA and a stop-gap fix. But this does not fix the problem, just hides it. We should be able to figure out why the metrics is getting initialized in BaseSourceImpl instead of during HMaster/RS process creation. We may need to refactor/move the code around to fix the same.

See review comments of [https://github.com/apache/hbase/pull/5323] for more details.

CC: [~zhangduo]
Bug: Remove netty 3 dependency in the pom file for hbase-endpoint
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Aug/23 03:44

We have this in the hadoop-3.0 profile, which is not necessary and will trigger  dependabot warnings about security problem. Hadoop mini cluster will pull in the necessary dependencies.
{code}
        <!-- Hadoop needs Netty 3.x at test scope for the minicluster -->
        <dependency>
          <groupId>io.netty</groupId>
          <artifactId>netty</artifactId>
          <version>3.6.2.Final</version>
          <scope>test</scope>
        </dependency>
{code}
Bug: Remove InterfaceStability.Stable from HTable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/23 15:31

HTable is InterfaceAudience.Private, but the InterfaceStability.Stable annotation was not removed when that was changed. Based on our compatibility guidelines, IS.Stable only applies to IA.Public classes. This annotation is confusing to new developers who might work on modifying HTable

Reference: https://hbase.apache.org/book.html#_interface_classifications
Bug: Document netty memory tunings
Status: Open | Priority: Major | Resolution: 
Created: 04/Aug/23 18:57

Netty can work pretty well out of the box, but you can protect your deployment from edge cases by tuning a few knobs:
{code:java}
-Dorg.apache.hbase.thirdparty.io.netty.maxDirectMemory
  By default, netty will use your -XX:MaxDirectMemorySize as the limit for its own off-heap memory usage. If you use any other off-heap memory features of hbase, this could cause a problem. For example, ByteBuffAllocator, BucketCache, or off-heap memstore. You should set netty's maxDirectMemory explicitly, and make sure to offset HBASE_OFFHEAPSIZE accordingly. Otherwise you risk OOM if netty comes under heavy load.

-Dorg.apache.hbase.thirdparty.io.netty.tryReflectionSetAccessible=true
  If you are running java9 or greater, the above maxDirectMemory setting will not take effect unless you also set this. Java9+ has extra security features which would typically cause a reflective setAccessible call to fail. But HBase uses a variety of --add-opens calls to enable access to Unsafe libraries, and these will also enable setAccessible to work for netty. Setting this to true will enable the feature which allows your custom maxDirectMemory to be be honored.{code}
 

We will be adding further protections in HBASE-27947 which we can also document here, but those have not been finalized yet.
Bug: Scan result wrong when using filterlist and specifing scan colums
Status: Open | Priority: Major | Resolution: 
Created: 13/Jul/23 09:30
Affects Version/s: 2.4.14

When I run these tests in hbase shell.

like this:

select cq1 from t1 where cq1<='2' and cq2>='5'

select cq1 from t1 where cq1<='2' or cq2>='5'

all these results is  is different from expectations...the results is different with MySQL and others DB...
{code:java}
// Test
create 't1',{NAME=>'cf1'}
put 't1','r1','cf1:cq1','1'
put 't1','r1','cf1:cq2','2'
put 't1','r2','cf1:cq1','2'
put 't1','r2','cf1:cq2','3'
put 't1','r3','cf1:cq1','3'
put 't1','r3','cf1:cq2','4'
put 't1','r4','cf1:cq1','4'
put 't1','r4','cf1:cq2','5'
java_import org.apache.hadoop.hbase.filter.SingleColumnValueFilter
java_import org.apache.hadoop.hbase.util.Bytes
java_import org.apache.hadoop.hbase.filter.CompareFilter
java_import org.apache.hadoop.hbase.filter.FilterList
filter1=SingleColumnValueFilter.new(Bytes::toBytes('cf1'),Bytes::toBytes('cq1'),CompareFilter::CompareOp::LESS_OR_EQUAL,Bytes::toBytes('2'))
filter2=SingleColumnValueFilter.new(Bytes::toBytes('cf1'),Bytes::toBytes('cq2'),CompareFilter::CompareOp::GREATER_OR_EQUAL,Bytes::toBytes('5'))
filterList1=FilterList.new(FilterList::Operator::MUST_PASS_ONE,filter1,filter2)
filterList2=FilterList.new(FilterList::Operator::MUST_PASS_ALL,filter1,filter2)
scan 't1',{COLUMNS=>['cf1:cq1'],FILTER =>filter1}
scan 't1',{COLUMNS=>['cf1:cq2'],FILTER =>filter2}
scan 't1',{COLUMNS=>['cf1:cq1'],FILTER =>filterList1}
scan 't1',{COLUMNS=>['cf1:cq1'],FILTER =>filterList2} {code}
!image-2023-07-13-17-34-03-488.png!

when setFilterIfMissing.... it also wrong

 
{code:java}
java_import org.apache.hadoop.hbase.filter.SingleColumnValueFilter
java_import org.apache.hadoop.hbase.util.Bytes
java_import org.apache.hadoop.hbase.filter.CompareFilter
java_import org.apache.hadoop.hbase.filter.FilterList
filter1=SingleColumnValueFilter.new(Bytes::toBytes('cf1'),Bytes::toBytes('cq1'),CompareFilter::CompareOp::LESS_OR_EQUAL,Bytes::toBytes('2'))
filter1::setFilterIfMissing(true)
filter2=SingleColumnValueFilter.new(Bytes::toBytes('cf1'),Bytes::toBytes('cq2'),CompareFilter::CompareOp::GREATER_OR_EQUAL,Bytes::toBytes('5'))
filter2::setFilterIfMissing(true)
filterList1=FilterList.new(FilterList::Operator::MUST_PASS_ONE,filter1,filter2)
filterList2=FilterList.new(FilterList::Operator::MUST_PASS_ALL,filter1,filter2)
scan 't1',{COLUMNS=>['cf1:cq1'],FILTER =>filter1}
scan 't1',{COLUMNS=>['cf1:cq2'],FILTER =>filter2}
scan 't1',{COLUMNS=>['cf1:cq1'],FILTER =>filterList1}
scan 't1',{COLUMNS=>['cf1:cq1'],FILTER =>filterList2}  {code}
 

!image-2023-08-04-17-54-04-346.png!

!image-2023-08-04-17-54-57-967.png!
Bug: Tunable hdfs readahead for STREAM reads
Status: Open | Priority: Major | Resolution: 
Created: 02/Aug/23 13:30

In HBASE-27896 we disabled HDFS readahead for ReadType.PREAD. In our environment this had a huge impact on disk IO (and network IO if low locality). The default HDFS readahead is 4mb, which is way larger than many scans might actually need. In our case, we have clusters serving millions of scans and those are almost all < 1mb.

I can think of two options, but open to others:
 # Set the readahead to whatever the scan.setMaxResultSize is, or some multiple of block size.
 # Simply make it configurable, and let operators configure it on a per table basis.

It may make sense to do both.
Bug: Add Get, Mutate, and Multi operations to slow log params
Status: Open | Priority: Major | Resolution: 
Created: 30/Jul/23 13:30

In https://issues.apache.org/jira/browse/HBASE-27536 we added the ability to include Scan operations in the slow log params. It would be useful to include more operations too. Beyond just showing the shape of the request to slow log readers, this would also ensure that operation attributes can be inferred.

There are a few complications to consider for some operation types:
 * Mutate:
 ** we should probably strip the columns from these puts. Otherwise we might produce unpredictably large slow log payloads, and there are potentially security concerns to consider
 * Multi
 ** we should also consider stripping columns from these requests
 ** (configurably?) limiting the number of operations that can be included. For example, maybe we only want to include 5 operations on a slow log payload for a 100 operation MultiRequest for the sake of brevity 
 ** we may want to deduplicate operation attributes. I'm not really sure how we'd do this without the output being misleading
Bug: Empty Increments in batch calls are too expensive
Status: Open | Priority: Major | Resolution: 
Created: 28/Jul/23 18:35

When an Increment is submitted via table.increment, we have a check which throws an exception if the Increment is empty. There is no such check when Increment is submitted through table.batch().

When this Increment makes its way to the server, it makes it all the way down past taking a row lock before the first check as to whether the Increment has any columns in it. Finally when we go to fetch the current cells from the table for atomic update, the fact that getFamilyMap() is empty starts to matter. Even still, beyond that I think we might still end up writing a somewhat useless WALEdit.

The big issue here is the row lock, as this causes contention under certain hotspot access patterns which greatly exacerbate the hotspot.

On the server side, we should check for whether the Increment is empty before even attempting to serve it. On the client side, we should probably also iterate the batch and check for emptiness.