av9ash/GitBugs huggingface dataset:
Bug: Update hbase_docker with an extra Dockerfile compatible with mac m1 platfrom
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Jun/23 14:45

When trying to use the current Dockerfile under "./dev-support/hbase_docker" on m1 macs, the docker build fails at the git clone & mvn build stage with below error:
{noformat}
 #0 8.214 qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory
{noformat}

It turns out for mac m1, we have to explicitly define the platform flag for the ubuntu image. I thought we could add a note in this readme, together with an "m1" subfolder containing a modified copy of this Dockerfile that works on mac m1s.
Bug: Enable cache control for logs endpoint and set max age as 0
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Apr/23 04:47

Not setting the proper header values may cause browsers to store pages within their respective caches. On public, shared, or any other non-private computers, a malicious person may search through the browser cache to locate sensitive information cached during another user's session.

/logs endpoint contains sensitive information that an attacker can exploit.

Any page with sensitive information needs to have the following headers in response:
Cache-Control: no-cache, no-store, max-age=0
Pragma: no-cache
Expires: -1
Bug: Tooling for parsing/reading the prefetch files list file
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Apr/23 10:18
Affects Version/s: 3.0.0-alpha-4

The content of the file defined by hbase.prefetch.file.list.path is encoded.  It would be nice to have some extra tool for properly parsing it and print the list in human readable format, for easy of troubleshooting.
Bug: Directly schedule procedures that do not need to acquire locks
Status: Open | Priority: Minor | Resolution: 
Created: 03/Jun/23 13:33
Affects Version/s: 3.0.0-alpha-3

Currently, in the procedure scheduler, we will not schedule any other procedures for a given queue if a procedure has held the exclusive lock, even if a procedure does not require any locks. For such procedures that do not require locks, we prefer that they can be executed directly without waiting until the procedure that held the exclusive lock is executed before starting to schedule execution. Otherwise, if the procedure holding the exclusive lock is stuck, the procedure that does not need the lock will also wait forever.
Bug: The sleep time in checkQuota of replication WAL reader should be controlled independently 
Status: Open | Priority: Minor | Resolution: 
Created: 24/May/23 02:45
Affects Version/s: 2.5.4

In theory the sleep time when checking quota failed in replication WAL reader should match the consume ability of the memory. But at the very least we should isolate the configuration here with the sleep time for common circumstances when replicating, e.g. sleep before reading from the head of the WAL, to avoid a little bit larger but reasonable sleep time(e.g. 3s) can make the consume speed always blocked by checking quota or cannot recover the consume speed in a very long time(except there exists long time of low source WAL production peak).
Bug: Increases the concurrency of off-peak compactions
Status: Open | Priority: Minor | Resolution: 
Created: 12/May/23 03:58

The current off-peak compactions are global comparisons.  There can only be one off-peak compact running at a time, which does not make better use of the system resources during the off-peak period.  This is not very friendly for clusters with significant peaks and valleys of business.

Code：
{code:java}
private static final AtomicBoolean offPeakCompactionTracker = new AtomicBoolean(); 


// Normal case - coprocessor is not overriding file selection.
if (!compaction.hasSelection()) {
 boolean isUserCompaction = priority == Store.PRIORITY_USER;
 boolean mayUseOffPeak =
   offPeakHours.isOffPeakHour() && offPeakCompactionTracker.compareAndSet(false, true);
 try {
   compaction.select(this.filesCompacting, isUserCompaction, mayUseOffPeak,
     forceMajor && filesCompacting.isEmpty());
 } catch (IOException e) {
   if (mayUseOffPeak) {
     offPeakCompactionTracker.set(false);
   }
   throw e;
 }
 assert compaction.hasSelection();
 if (mayUseOffPeak && !compaction.getRequest().isOffPeak()) {
   // Compaction policy doesn't want to take advantage of off-peak.
   offPeakCompactionTracker.set(false);
 }
}{code}
 

I think it could be optimized to allow the user to set the number of concurrent off-peak compactions.
Bug: Fix the javadoc for SyncFutureCache
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Jun/23 13:43

It does not have any html markers so spotless messed it up...

We should add html markers so it could keep the format after 'spotless:apply'

{code}
/**
 * A cache of {@link SyncFuture}s. This class supports two methods
 * {@link SyncFutureCache#getIfPresentOrNew()} and {@link SyncFutureCache#offer()}.
 * <p>
 * Usage pattern:
 * 
 * <pre>
 *   SyncFuture sf = syncFutureCache.getIfPresentOrNew();
 *   sf.reset(...);
 *   // Use the sync future
 *   finally: syncFutureCache.offer(sf);
 * </pre>
 * 
 * Offering the sync future back to the cache makes it eligible for reuse within the same thread
 * context. Cache keyed by the accessing thread instance and automatically invalidated if it remains
 * unused for {@link SyncFutureCache#SYNC_FUTURE_INVALIDATION_TIMEOUT_MINS} minutes.
 */
{code}
Bug: Distinguish the mutate types of rpc error in MetricsConnection
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/May/23 07:55

There are four mutate types in HBase, including {_}Append{_}, {_}Delete{_}, {_}Increment{_}, and _Put._ However, when rpc error occurs on the client, mutate types are not distinguished in MetricsConnection.
Bug: listTableNames has different criteria for judging the regular matching of tables in the default namespace
Status: Open | Priority: Minor | Resolution: 
Created: 14/Jun/23 07:16
Affects Version/s: 2.0.0

There are the following tables under default: order_info, test123 

When I execute the following logic (get the table that starts with test), test123 is returned
{code:java}
String tableNameRegex = "^(test).*";
TableName[] tns = admin.listTableNames(Pattern.compile(tableNameRegex));{code}
When I execute the following logic (get a table that doesn't start with test), the return is order_info, test123
{code:java}
String tableNameRegex = "^(?!test).*";
TableName[] tns = admin.listTableNames(Pattern.compile(tableNameRegex));{code}
As you can see, table test123 is returned both times.

 

When I look at the source code, I find that when I perform a regular match for the table in the default namespace, I perform one step more judgment, resulting in a table starting with test being returned both times.
{code:java}
boolean matched = pattern.matcher(tableName).matches();
if (!matched && htd.getTableName().getNamespaceAsString().equals(defaultNS)) {
    matched = pattern.matcher(defaultNS + TableName.NAMESPACE_DELIM + tableName).matches();
}{code}
 

I think there are problems with this regular matching logic
Bug: The description about hbase.hstore.comactionThreshold is not accurate
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Jun/23 14:24
Affects Version/s: 2.3.7

In document hbase-default.xml， The  description about [hbase.hstore.comactionThreshold|http://hbase.hstore.comactionthreshold/] is not accurate，as follow：

!2023-06-20_221700.png!

Actually，hbase will run compaction when the number of store files greater than or equal to [hbase.hstore.comactionThreshold|http://hbase.hstore.comactionthreshold/], as follow

!2023-06-20_221821.png!
Bug: ClientSideRegionScanner does not adhere to RegionScanner.nextRaw contract
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Jun/23 13:45

A follow-on for HBASE-26630 so that we can fix the actual logic in ClientSideRegionScanner so that it works for all ReadTypes and/or when used outside of the context of TableSnapshotInputFormat
Bug: Record readBlock message in log when it takes too long time
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/May/23 03:29
Affects Version/s: 2.5.3

After HBASE-15160， we can record the readBlock message by TRACE log in org.apache.hadoop.hbase.io.hfile.HFileBlock.FSReaderImpl#readBlockDataInternal， But it record all read block message in TRACE log， some times， we only focus the block read cost too much time..
Bug: Stable version outdated on https://hbase.apache.org/downloads.html
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Jun/23 08:36
Affects Version/s: 2.5.4

In HBASE-27849, the stable version of HBase was updated to 2.5.x. This updated the [https://downloads.apache.org/hbase/] page.

However, the download page ([https://hbase.apache.org/downloads.html)] still refers to 2.4.x as the stable version.
Bug: [hbase-thirdparty] Include netty-tcnative in hbase-shaded-netty
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 04/Jun/23 14:47

At my company, HubSpot, we are starting to use HBase's TLS support in our clusters. I've done some experiments with different TLS implementations to see which has the best performance.

I have tested Java 11 without {{{}UseAESCTRInstrinics{}}}, Java 11 with {{{}UseAESCTRInstrinsics{}}}, with and without {{netty-tcnative}} installed, on both {{x86_64}} and {{{}arm64{}}}. I will attach some charts.

-I have found that using the {{netty-tcnative}} library gives the best performance overall, and I would like to see it included in HBase by default.-

This ticket is for tracking the change to hbase-thirdparty and producing a new version of that. Another ticket will track the inclusion of that new version in HBase, which requires a few code changes.
Bug: Beautify the output information of the getStats method in ReplicationSource
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Jun/23 08:38
Affects Version/s: 3.0.0-alpha-3

When multiple replicationSourceShippers appear in our production environment, the output information of the getStats method is not wrapped and printed

  !image-2023-06-01-16-35-08-817.png!
Bug: xerial's snappy-java requires GLIBC >= 2.32
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/May/23 04:54
Affects Version/s: 2.5.4

We need to add a native library load check with a helpful error message if it fails to load due to a too old glibc or similar reason, and disable the unit test if the native library fails to load. 

{noformat}
TestHFileCompressionSnappy.test:54->HFileTestBase.doTest:72 »
UnsatisfiedLink /home/apurtell/src/hbase/hbase-compression/hbase-compression-snappy/target/snappy-1.1.9-6406bd03-44b4-4586-a91b-79d213c14062-libsnappyjava.so:
/lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.32'
{noformat}

In this case a Debian test VM has "Debian GLIBC 2.31-13+deb11u6" installed, which is too old.
Bug: Close the L1 victim handler race
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/May/23 21:14
Affects Version/s: 2.5.4

When we evict a block from L1 and move it to L2 there is a brief window of time where we have removed the block from the L1 map and yet the victim handler has not completed execution. Some read-your-write use cases can be significantly impacted even though the window is small. Imagine a use case where PRELOAD_DATA_ON_OPEN and CACHE_DATA_ON_WRITE are both enabled to warm cache ahead of access and any miss can be problematic. Perhaps a particularly latency sensitive case with HFiles backed by S3. 

Victim handling can be made atomic with respect to the unmapping operation with ConcurrentHashMap#computeIfPresent. The upside is there will be no L1+L2 misses during the transition. The downside is if the victim handler takes a long time to execute -- currently they are all very fast, so only a theoretical risk -- then other removals or insertions in L1 can block until it completes.
Bug: balance_rsgroup NullPointerException
Status: Open | Priority: Minor | Resolution: 
Created: 23/May/23 13:25

hbase(main):001:0> balance_rsgroup 'default'

ERROR: java.io.IOException: Cannot invoke "org.apache.hadoop.hbase.ServerName.getAddress()" because "currentHostServer" is null
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:466)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
Caused by: java.lang.NullPointerException: Cannot invoke "org.apache.hadoop.hbase.ServerName.getAddress()" because "currentHostServer" is null
        at org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.correctAssignments(RSGroupBasedLoadBalancer.java:320)
        at org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.balanceCluster(RSGroupBasedLoadBalancer.java:126)
        at org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.balanceRSGroup(RSGroupAdminServer.java:461)
        at org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl.balanceRSGroup(RSGroupAdminEndpoint.java:301)
        at org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService.callMethod(RSGroupAdminProtos.java:14948)
        at org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:921)
        at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:394)
        ... 3 more

For usage try 'help "balance_rsgroup"'
Bug: Provide option in HBase UI to disable stack trace for security
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Apr/23 05:08

Uncaught server exceptions occur when providing parameter values that the server or servlet does not understand.
Physical paths, versioning information, stack traces' content, and other data can be gathered and used to help further an attack when improper error handling is present.

Applications should always fail safe in their designs. If an application fails to an unknown state, it is likely that an attacker may be able to exploit this indeterminate state to access unauthorized functionality, or worse, create, modify or destroy data. Error messages may also aid in the identification of other attacks such as buffer overflows and SQL injection, and can generally contribute to an overall weaker security posture.

For example, if we use a HTTPS web server and explicitly provide Host header with a wrong value, say attackers.com, we get the following response in UI:
{code:java}
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 400 Host does not match SNI</title>
</head>
<body><h2>HTTP ERROR 400 Host does not match SNI</h2>
<table>
<tr><th>URI:</th><td>/tablesDetailed.jsp</td></tr>
<tr><th>STATUS:</th><td>400</td></tr>
<tr><th>MESSAGE:</th><td>Host does not match SNI</td></tr>
<tr><th>SERVLET:</th><td>-</td></tr>
<tr><th>CAUSED BY:</th><td>org.apache.hbase.thirdparty.org.eclipse.jetty.http.BadMessageException: 400: Host does not match SNI</td></tr>
</table>
<h3>Caused by:</h3><pre>org.apache.hbase.thirdparty.org.eclipse.jetty.http.BadMessageException: 400: Host does not match SNI
   at org.apache.hbase.thirdparty.org.eclipse.jetty.server.SecureRequestCustomizer.customize(SecureRequestCustomizer.java:279)
   at org.apache.hbase.thirdparty.org.eclipse.jetty.server.SecureRequestCustomizer.customize(SecureRequestCustomizer.java:210)
   at org.apache.hbase.thirdparty.org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:483)
   at org.apache.hbase.thirdparty.org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
   at org.apache.hbase.thirdparty.org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
   at org.apache.hbase.thirdparty.org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:439)
   at org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
   at org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
   at java.lang.Thread.run(Thread.java:750)
</pre>

</body>
</html> {code}
Bug: Update surefire version to 3.0.0 and use the SurefireForkNodeFactory
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/May/23 12:32

The final surefire version 3.0.0 is released so we can update from 3.0.0-M6 -> 3.0.0. The final version has several fixes and is more stable.
 
SurefireForkNodeFactory is a new strategy to control how the surefire forked nodes communicate with the main maven process. It uses a tcp channel instead of process pipes. It helps to fix some "corrupted channel" messages seen in the s390x build and should be more reliable on all platforms. I have been testing locally on amd64 and s390x and have not had any issues.
 
Ref: 
[https://maven.apache.org/surefire/maven-surefire-plugin/examples/process-communication.html]
Bug: HBaseClassTestRule: system exit not restored if test times out may cause test to hang
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/May/23 12:05

HBaseClassTestRule applies a timeout and a system exit rule to tests. The timeout rule throws an exception if it hits the timeout threshold. Since the timeout rule is applied after the system exit rule, the system exit rule does not see the exception and does not re-enable the default system exit behavior which can cause maven to hang on some tests. I saw the hang happen when certain tests timed out on s390x but it could happen on any platform.
 
If the org.apache.hadoop.hbase.TestTimeout.infiniteLoop test is enabled and run it will generate a *-jvmRun1.dump file which shows that the org.apache.hadoop.hbase.TestSecurityManager is still enabled:
 
{quote}# Created at 2023-04-27T15:51:58.947
org.apache.hadoop.hbase.SystemExitRule$SystemExitInTestException
        at org.apache.hadoop.hbase.TestSecurityManager.checkExit(TestSecurityManager.java:32)
        at java.base/java.lang.Runtime.exit(Runtime.java:114)
        at java.base/java.lang.System.exit(System.java:1752)
        at org.apache.maven.surefire.booter.ForkedBooter.acknowledgedExit(ForkedBooter.java:381)
        at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:178)
        at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
...{quote}
Bug: TestListTablesByState is silently failing due to a surefire bug
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/May/23 15:45

Surefire version 3.0.0-M6 has a bug where tests end up being removed from the test results if they fail with a long exception message. See:
 
https://issues.apache.org/jira/browse/SUREFIRE-2079
 
org.apache.hadoop.hbase.master.TestListTablesByState is currently failing in CI due to an error. However, it does not show up in the Test Results because of the surefire bug.
 
If you download the raw test_logs from the build artifacts, you will find the files:
/home/jenkins/jenkins-home/workspace/HBase_Nightly_master/output-jdk8-hadoop3/archiver/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.master.TestListTablesByState.txt
 
which contains:
{{-----------------------------------------------------------------------}}

{{Test set: org.apache.hadoop.hbase.master.TestListTablesByState}}

{{-----------------------------------------------------------------------}}

{{Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.929 s - in org.apache.hadoop.hbase.master.TestListTablesByState}}

 
and /home/jenkins/jenkins-home/workspace/HBase_Nightly_master/output-jdk8-hadoop3/archiver/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.master.TestListTablesByState-output.txt
 
which contains exceptions like:
{{...}}

{{2023-05-04T11:41:56,262 INFO  [RPCClient-NioEventLoopGroup-4-3 {}] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2603): Operation: CREATE, Table Name: default:test failed with org.apache.hadoop.hbase.DoNotRetryIOException: Table test should have at least one column family.}}

{{        at}}

{{...}}

 
I found this while testing the final surfire 3.0.0 version which fixes the bug and the test then shows up as failing.
Bug: Update zstd-jni from version 1.5.4-2 -> 1.5.5-2
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/May/23 13:47

A new version (1.5.5-2) of the zstd-jni dependency has been released that includes a fix for an s390x issue which was prevented it from working on common linux distros. The underlying zstd 1.5.5 library also includes a fix for a rare corruption bug.

 

Refs:

https://github.com/luben/zstd-jni/issues/257

https://github.com/facebook/zstd/releases/tag/v1.5.5
Bug: Guard Master/RS Dump Servlet behind admin walls
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Apr/23 10:36

Currently RSDumpServlet and MasterDumpServlet do not require any check for whether the user has privileges to access to instrumentation servlets.

This is unlike other servlets like ProfileServlet, ConfServlet, JMXJsonServlet etc. which are guarded by admin checks.

Goal of this JIRA is to add similar check for RS and Master Dump Servlet. Post this change only admins will be able to access RSDumpServlet and MasterDumpServlet, if _hadoop.security.instrumentation.requires.admin_ is enabled.
Bug: Backport "HBASE-24914 Remove duplicate code appearing continuously in method ReplicationPeerManager.updatePeerConfig" to branch-2
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Apr/23 03:24
Affects Version/s: 2.4.14

Branch-2/ Branch-2.4/Branch-2.5 also have duplicate code in ReplicationPeerManager#updatePeerConfig


newPeerConfigBuilder.putAllConfiguration(oldPeerConfig.getConfiguration());
newPeerConfigBuilder.putAllConfiguration(peerConfig.getConfiguration());
newPeerConfigBuilder.putAllConfiguration(oldPeerConfig.getConfiguration());
newPeerConfigBuilder.putAllConfiguration(peerConfig.getConfiguration());
Bug: Move Dockerfile and python-requirements.txt used only by flaky-tests job
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Apr/23 15:31
Affects Version/s: 3.0.0-alpha-4

We have a Dockerfile floating around in dev-support. It looks like it's only used by the flaky-test Jenkins job, so move it under that directory. I think that it used to be used in multiple places, as way to package up the dev-support/python-requirements.txt file. However, that too seems to only be used by the flaky-tests job, so move it as well. Update the job for the new invocation form, and remove mentions of python-requirements.txt from the book -- an old, out-dated section about how to format patches for attachment to JIRA.
Bug: Backport HBASE-27731 (Upgrade commons-validator to version 1.7) to branch-2.5
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 04/Apr/23 12:09

This is so we also fix these CVEs for 2.5
 - [CVE-2014-0114|https://nvd.nist.gov/vuln/detail/cve-2014-0114]
 - [CVE-2019-10086|https://nvd.nist.gov/vuln/detail/cve-2019-10086]
Bug: Migrate javax.el:3.0.1-b08 to jakarta.el-4.0.2
Status: Resolved | Priority: Trivial | Resolution: Won't Fix
Created: 26/Apr/23 13:08
Affects Version/s: 2.4.18

The javax.el artifact contains a CVE: [CVE-2021-28170. |https://nvd.nist.gov/vuln/detail/CVE-2021-28170]The CVE itself is not a big issue since we're pre-compiling our JSP pages when building HBase, no user input is parsed which reduces the risk considerably.

The org.glassfish:javax.el artifact was moved to org.glassfish:jakarta.el, which means a migration to get rid of the CVE.