av9ash/GitBugs huggingface dataset:
Bug: HCommon get local file system fails with a casting error
Status: Resolved | Priority: Critical | Resolution: Duplicate
Created: 20/Jul/23 08:49

h2. What happened

After setting {{{}fs.file.impl=org.apache.hadoop.fs.RawLocalFileSystem{}}}, trying to acquire local file system using the {{getLocal}} in {{org.apache.hadoop.fs}} fails with {{java.lang.ClassCastException}}
h2. Where's the bug

In the function {{getLocal}} of {{FileSystem}} in HCommon:
{code:java}
  public static LocalFileSystem getLocal(Configuration conf)
    throws IOException {
    return (LocalFileSystem)get(LocalFileSystem.NAME, conf);
  } {code}
the returned file system is directly casted to LocalFileSystem without checking. If the user set the implementation of the local filesystem to be Raw rather than Checksum, this type cast would fail.
h2. How to reproduce
 # Set {{fs.file.impl=org.apache.hadoop.fs.RawLocalFileSystem}}
 # Run {{org.apache.hadoop.hbase.TestHBaseTestingUtility#testMiniDFSCluster}}
and the following exception should be observed:

{code:java}
java.lang.ClassCastException: class org.apache.hadoop.fs.RawLocalFileSystem cannot be cast to class org.apache.hadoop.fs.LocalFileSystem (org.apache.hadoop.fs.RawLocalFileSystem and org.apache.hadoop.fs.LocalFileSystem are in unnamed module of loader 'app')	at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:441)	at org.apache.hadoop.hbase.HBaseTestingUtility.getNewDataTestDirOnTestFS(HBaseTestingUtility.java:550)	at org.apache.hadoop.hbase.HBaseTestingUtility.setupDataTestDirOnTestFS(HBaseTestingUtility.java:536)	at org.apache.hadoop.hbase.HBaseTestingUtility.getDataTestDirOnTestFS(HBaseTestingUtility.java:510)	at org.apache.hadoop.hbase.HBaseTestingUtility.getDataTestDirOnTestFS(HBaseTestingUtility.java:524)	at org.apache.hadoop.hbase.HBaseTestingUtility.createDirsAndSetProperties(HBaseTestingUtility.java:737)	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:682)	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:665)	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:653)	at org.apache.hadoop.hbase.TestHBaseTestingUtility.testMiniDFSCluster(TestHBaseTestingUtility.java:377) {code}
Bug:  Missing null check of rpcServer.scheduler.executor causes NPE with invalid value of hbase.client.default.rpc.codec 
Status: Resolved | Priority: Critical | Resolution: Incomplete
Created: 27/Jul/23 14:54

When {{hbase.client.default.rpc.codec}} is set to {{org.apache.hadoop.hbase.regionserver.wal.SecureWALCellCodec}} the method {{createRpcClientRTEDuringConnectionSetup}} gives a NoSuchMethodException for the provided default codec value.

Since there was an exception, the code inside try block is not executed which contained the below code line
{noformat}
try (AbstractRpcClient<?> client = createRpcClientRTEDuringConnectionSetup(clientConf)) {
      rpcServer.start();
      . . .
    }{noformat}
When finally block is executed which contains
{noformat}
finally {
      rpcServer.stop();
    }{noformat}
The method {{rpcServer.stop()}} eventually makes call to {{{}executor.shutdown(){}}}. A null pointer exception is returned as {{rpcServer.scheduler.executor}} was never instantiated, which happens inside {{rpcServer.start()}}
{noformat}
java.lang.NullPointerException
    at org.apache.hadoop.hbase.ipc.FifoRpcScheduler.stop(FifoRpcScheduler.java:71)
    at org.apache.hadoop.hbase.ipc.SimpleRpcServer.stop(SimpleRpcServer.java:441)
    at org.apache.hadoop.hbase.ipc.AbstractTestIPC.testRTEDuringConnectionSetup(AbstractTestIPC.java:194)
    . . .{noformat}
Bug: ITBLL's RollingBatchSuspendResumeRsAction runs the "suspendRs" method to perform the action, but it inadvertently uses the "waitForRegionServerToStop" method to check if it was executed successfully.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Aug/23 06:14
Affects Version/s: 2.7.0

When running ITBLL, a problem occurs that ultimately results in all region servers being suspended.

The following is the ITBLL running command：
{code:java}
hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList -DIntegrationTestBigLinkedList.table=itbll -m slowDeterministic Loop 10 10 10000000 /tmp/biglinkedlist 100 {code}
I have summarized the process as follows:
 # The Action RollingBatchSuspendResumeRsAction in ITBLL will execute the "sudo -u hbase ps ux | grep proc_regionserver | grep -v grep | tr -s ' ' | cut -d ' ' -f2 | xargs kill -s SIGSTOP" command to suspend the RegionServer process.
 # This command will pause the RegionServer process, rather than kill it.
 # The Action uses the waitForServiceToStop method to check if the execution was successful, using the "sudo -u hbase ps ux | grep proc_regionserver | grep -v grep | tr -s ' ' | cut -d ' ' -f2" command.
 # The waitForServiceToStop method used to check if the execution was successful does not match the suspendRs, causing ITBLL to not resume the RegionServer process and ultimately resulting in all RegionServer processes being suspended. Therefore, ITBLL fails to run.

{code:java}
2023-07-21 11:18:23,103 WARN  [ChaosMonkey-2] policies.Policy (DoActionsOncePolicy.java:runOneIteration(51)) - Exception occurred during performing action: java.io.IOException: Timed-out waiting for service to stop: core-1-3,16020,1689908619650
        at org.apache.hadoop.hbase.DistributedHBaseCluster.waitForServiceToStop(DistributedHBaseCluster.java:282)
        at org.apache.hadoop.hbase.DistributedHBaseCluster.waitForRegionServerToStop(DistributedHBaseCluster.java:131)
        at org.apache.hadoop.hbase.chaos.actions.Action.suspendRs(Action.java:200)
        at org.apache.hadoop.hbase.chaos.actions.RollingBatchSuspendResumeRsAction.perform(RollingBatchSuspendResumeRsAction.java:97)
        at org.apache.hadoop.hbase.chaos.policies.DoActionsOncePolicy.runOneIteration(DoActionsOncePolicy.java:48)
        at org.apache.hadoop.hbase.chaos.policies.PeriodicPolicy.run(PeriodicPolicy.java:41)
        at org.apache.hadoop.hbase.chaos.policies.CompositeSequentialPolicy.run(CompositeSequentialPolicy.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
 {code}
Bug: Reduce frequency of saving backing map in persistence cache
Status: Resolved | Priority: Major | Resolution: Abandoned
Created: 24/Aug/23 16:55

Currently we always write the whole cache mapping into the persistent map file. This is not a lightweight operation, on a full 1.6TB cache with ten millions of block, this can grow as much as 10GB. In the current persistent cache implementation, we flush it to disk every 1s. If we raise the "checkpoint" period, we risk lose more cache events in the event of a recovery. 

This proposes reduce the frequency needed to save the backing map as follows:
1) Save every block addition/eviction into a single file in disk;
2) Checkpoint at higher intervals, consolidating all transactions into the larger map file;
3) In the event of failure, recovery would consist of loading the latest map file, then applying all the transactions files sequentially;
Bug: Deadlock when opening mob files
Status: Reopened | Priority: Major | Resolution: 
Created: 28/Aug/23 04:28
Affects Version/s: 2.5.5

The hashcode of mob file name is used in MobFileCache to lock the cached mob files, but hashcode may be repeated and the IdLock is not reentrant. Then when opening a not cached file with evicting the opened by LRU, the repeated hashcode files will bring deadlock.

[^mobdeadlock.js]
Bug:  Replace javax.servlet.jsp dependency with tomcat-jasper
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Sep/23 02:40

*Problem Statement*

HBase has to explicitly depends on org.glassfish:javax.el:jar:3.0.1-b08 as this dependency is needed by javax.servlet.jsp. This direct dependency was added due to https://issues.apache.org/jira/browse/HBASE-18831

mvn dependency tree shows below
{code:java}
[INFO] |  +- org.glassfish.web:javax.servlet.jsp:jar:2.3.2:compile
[INFO] |  |  \- org.glassfish:javax.el:jar:3.0.1-b08:compile
{code}
org.glassfish:javax.el:jar:3.0.1-b08:compile has [https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-15250]

We have ton of Jiras and HBase around glass fish and issues caused due to it. With this Jira I plan to completely remove org.glassfish:javax.el:jar:3.0.1-b08 from our dependency tree. Also org.glassfish:javax.el is EOL and needs migration to jakarta-el which is not trivial. See [https://mvnrepository.com/artifact/org.glassfish/javax.el]

*Proposed Solution*

This Jira aims to replace javax.servlet.jsp dependency with tomcat-jasper (as javax.servlet.jsp strictly needs glassfish) and this requires minimal change wrt to migrating to jakarta-el.

Also, we use javax.servlet.jsp to generate/build JSP and same can be achieved via tomcat-jasper.

CC: [~zhangduo]
Bug: NullPointerException authorizing connection on regionserver startup
Status: Open | Priority: Major | Resolution: 
Created: 15/Sep/23 21:10

When a RegionServer starts up, the HRegionServer constructor calls createRpcServices() which creates RSRpcServices, which uses RpcServerFactory to create a NettyRpcServer. The NettyRpcServer constructor binds the process to the given regionserver port. At this point it is able to accept requests.

Later in the HRegionServer constructor, it calls rpcServices.start(zooKeeper). This calls NettyRpcServer.start(), which initializes the AuthManager.

Since there is a period of time where the server can accept requests but AuthManager is not initialized, if you have authentication enabled you will receive NullPointerExceptions (see below). These will clear up once the server starts.

We should fix the flow here to avoid these errors.
{code:java}
2023-09-15T20:47:35,300 [RS-EventLoopGroup-1-3] WARN org.apache.hadoop.hbase.ipc.NettyRpcServer: Connection xxx.xxx.xxx.xxx:49402; caught unexpected downstream exception.
java.lang.NullPointerException: Cannot invoke "org.apache.hadoop.security.authorize.ServiceAuthorizationManager.authorize(org.apache.hadoop.security.UserGroupInformation, java.lang.Class, org.apache.hadoop.conf.Configuration, java.net.InetAddress)" because "this.authManager" is null
        at org.apache.hadoop.hbase.ipc.RpcServer.authorize(RpcServer.java:654) ~[hbase-server-2.5-hubspot-20230913.131903-57.jar:2.5-hubspot-SNAPSHOT]
        at org.apache.hadoop.hbase.ipc.ServerRpcConnection.authorizeConnection(ServerRpcConnection.java:381) ~[hbase-server-2.5-hubspot-20230913.131903-57.jar:2.5-hubspot-SNAPSHOT]
        at org.apache.hadoop.hbase.ipc.ServerRpcConnection.processOneRpc(ServerRpcConnection.java:362) ~[hbase-server-2.5-hubspot-20230913.131903-57.jar:2.5-hubspot-SNAPSHOT]
        at org.apache.hadoop.hbase.ipc.NettyServerRpcConnection.process(NettyServerRpcConnection.java:89) ~[hbase-server-2.5-hubspot-20230913.131903-57.jar:2.5-hubspot-SNAPSHOT] {code}
Bug: Incremental backups should be forbidden after deleting backups
Status: Open | Priority: Major | Resolution: 
Created: 14/Sep/23 11:34

Imagine I have a set of backups (Full1, Incr2, Incr3), delete the last backup (Incr3), and then create a new incremental backup (Incr4).

This backup history will now show: Full1, Incr2, Incr4.

However, restoring Incr4 will not contain the data that was captured in Incr3, effectively leading to data loss. This will certainly surprise some users.

I suggest to add some internal bookkeeping to prevent incremental backups in case the most recent backup was deleted.
Bug: oldWALs naming can be incompatible with HBase backup
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/23 16:19

I am testing HBase backup functionality, and noticed following warning when running "hbase backup create incremental ...":

 
{noformat}
23/09/13 15:44:10 WARN org.apache.hadoop.hbase.backup.util.BackupUtils: Skip log file (can't parse): hdfs://hdfsns/hbase/hbase/oldWALs/hbase-region-0.hbase-region.minikube-shared.svc.cluster.local%2C16020%2C1694609964681.hbase-region-0.hbase-region.minikube-shared.svc.cluster.local%2C16020%2C1694609964681.regiongroup-0.1694609969312{noformat}
It appears in my setup, the oldWALs are indeed given names that seem to break "ServerName.valueOf(s)" in "BackupUtils#parseHostFromOldLog(Path p)":

 

 
{noformat}
user@hadoop-client-769bc9946-xqrt2:/$ hdfs dfs -ls hdfs:///hbase/hbase/oldWALs
Found 42 items
-rw-r--r--   1 hbase hbase     775421 2023-09-13 13:14 hdfs:///hbase/hbase/oldWALs/hbase-master-0.minikube-shared%2C16000%2C1694609954719.hbase-master-0.minikube-shared%2C16000%2C1694609954719.regiongroup-0.1694609957984$masterlocalwal$
-rw-r--r--   1 hbase hbase      26059 2023-09-13 13:29 hdfs:///hbase/hbase/oldWALs/hbase-master-0.minikube-shared%2C16000%2C1694609954719.hbase-master-0.minikube-shared%2C16000%2C1694609954719.regiongroup-0.1694610867894$masterlocalwal$
...
-rw-r--r--   1 hbase hbase     242479 2023-09-13 14:16 hdfs:///hbase/hbase/oldWALs/hbase-region-0.hbase-region.minikube-shared.svc.cluster.local%2C16020%2C1694609964681.hbase-region-0.hbase-region.minikube-shared.svc.cluster.local%2C16020%2C1694609964681.regiongroup-0.1694609969312
-rw-r--r--   1 hbase hbase       4364 2023-09-13 14:16 hdfs:///hbase/hbase/oldWALs/hbase-region-0.hbase-region.minikube-shared.svc.cluster.local%2C16020%2C1694609964681.hbase-region-0.hbase-region.minikube-shared.svc.cluster.local%2C16020%2C1694609964681.regiongroup-0.1694610188654
...
-rw-r--r--   1 hbase hbase      70802 2023-09-13 13:15 hdfs:///hbase/hbase/oldWALs/hbase-region-0.hbase-region.minikube-shared.svc.cluster.local%2C16020%2C1694609964681.meta.1694609970025.meta
-rw-r--r--   1 hbase hbase         93 2023-09-13 13:04 hdfs:///hbase/hbase/oldWALs/hbase-region-0.hbase-region.minikube-shared.svc.cluster.local%2C16020%2C1694609964681.meta.1694610188627.meta
...{noformat}
I'd say this is not a bug in the backup system, but rather in whatever gives the oldWAL files its name. I'm however not that familiar with HBase code to find where these files are created. Any pointers are appreciated.

Given that this causes some logs to be missed during backup, I guess this can lead to data loss in a backup restore?
Bug: RSProcedureDispatcher to fail-fast for krb auth failures
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Aug/23 17:18

As discussed on the parent Jira, let's mark the remote procedures fail when we encounter SaslException (GSS initiate failed) as this belongs to the category of known IOException where we are certain that the request has not yet reached to the target regionserver yet.

This should help release dispatcher threads for other ExecuteProceduresRemoteCall executions.

 

Example log:
{code:java}
2023-08-25 02:21:02,821 WARN [ispatcher-pool-40777] procedure.RSProcedureDispatcher - request to rs1,61020,1692930044498 failed due to java.io.IOException: Call to address=rs1:61020 failed on local exception: java.io.IOException: org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException: org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed, try=0, retrying...  {code}
Bug: Implement cache aware load balancer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jul/23 14:21

HBase uses ephemeral cache to cache the blocks by reading them from the slow storages and storing them to the bucket cache. This cache is warmed up everytime a region server is started. Depending on the data size and the configured cache size, the cache warm up can take anywhere between a few minutes to few hours. Doing this everytime the region server starts can be a very expensive process. To eliminate this, HBASE-27313 implemented the cache persistence feature where the region servers periodically persist the blocks cached in the bucket cache. This persisted information is then used to resurrect the cache in the event of a region server restart because of normal restart or crash.

This feature aims at enhancing this capability of HBase to enable the balancer implementation considers the cache allocation of each region on region servers when calculating a new assignment plan and uses the region/region server cache allocation info reported by region servers which takes into account to calculate the percentage of HFiles cached for each region on the hosting server, and then use that as another factor when deciding on an optimal, new assignment plan.

 

A design document describing the balancer can be found at https://docs.google.com/document/d/1A8-eVeRhZjwL0hzFw9wmXl8cGP4BFomSlohX2QcaFg4/edit?usp=sharing
Bug: Enhance prefetch executor to record region prefetch information along with the list of hfiles prefetched
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jul/23 11:03
Affects Version/s: 2.6.0

HBASE-27313 implemented the prefetch persistence feature where it persists the list of hFiles prefetched in the bucket cache. This information is used to reconstruct the cache in the event of a server restart/crash.

Currently, only the list of hFiles is persisted.

However, for the new PrefetchAwareLoadBalancer (work in progress) to work, we need the information about how much a region is prefetched on a region server.

This Jira introduces an additional map in the prefetch executor to maintain the information about how much a region has been prefetched on that region server. The size of region server prefetched is calculated as the total size of all hFiles prefetched for that region.
Bug: Corrupt HFile data is mishandled in several cases
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Sep/23 11:20
Affects Version/s: 2.5.2

While riding over a spat of HDFS data corruption issues, we've observed several places in the read path that do not fall back to HDFS checksum appropriately. These failures manifest during client reads and during compactions. Sometimes failure is detected by the fallback {{verifyOnDiskSizeMatchesHeader}}, sometimes we attempt to allocate a buffer with a negative size, and sometimes we read through to a failure from block decompression.

After code study, I think that all three cases arise from using a block header that was read without checksum validation.

Will post up the stack traces in the comments. Not sure if we'll want a single patch or multiple.
Bug: Implement truncate_region command to truncate region directly from FS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Sep/23 23:08
Affects Version/s: 2.5.6

One of our users has brought up a use-case where they need to truncate a region to delete data within a specific range. There are two scenarios to consider:

* In the first scenario, the region boundaries involve a time range defined through pre-splitting, and user is looking to efficiently clean old date data. If HBase can directly truncate the region from the file system and then the user can merge the empty region with adjacent regions to effectively eliminate it which will be more optimized compared to deleting the data using Delete API.

* In another case, if the HFile for that region becomes corrupted for some reason, user want to get rid of the HFile and reload the entire region to avoid consistency issues and ensure performance.

we can do this by taking the region offline and taking write lock to avoid the consideration of race conditions involving Region In Transition (RITs), region re-opening, and merge/split scenarios.
Bug: Performance improvement for scan over several stores. 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Aug/23 05:06
Affects Version/s: 2.5.5

During the fix of HBASE-19863, an additional check for fake cells that trigger reseek was added.  It comes that this check produces unnecessary reseeks because
matcher.compareKeyForNextColumn should be used only with indexed keys. Later  [~larsh] suggested doing a simple check for OLD_TIMESTAMP and it looks like a better solution.
Bug: Add TLS settings to ZooKeeper client
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Aug/23 16:36
Affects Version/s: 2.4.17

ZooKeeper supports TLS connection from its clients. Currently the only way to set up HBase for this is to add the following Java properties to the HBase process:
{noformat}
        -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty 
        -Dzookeeper.client.secure=true 
        -Dzookeeper.ssl.keyStore.location=/path/to/keystore.jks 
        -Dzookeeper.ssl.keyStore.password=password 
        -Dzookeeper.ssl.trustStore.location=/path/to/truststore.jks 
        -Dzookeeper.ssl.trustStore.password=password
{noformat}
KeyStore is only needed if ZooKeeper server wants client certificate to be provided.

I'd like to add these options to hbase-site.xml in the following way:
{noformat}
hbase.zookeeper.property.clientCnxnSocket
hbase.zookeeper.property.client.secure
hbase.zookeeper.property.ssl.keyStore.location
hbase.zookeeper.property.ssl.keyStore.password or hbase.zookeeper.property.ssl.keyStore.passwordPath
...{noformat}
It will follow the way that we already do for ZooKeeper clientPort and quorum settings. ("hbase.zookeeper.property.clientPort", "hbase.zookeeper.quorum")
Bug: Persistent cache map can get corrupt if crash happens midway through the write
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Aug/23 15:18
Affects Version/s: 2.6.0

HBASE-27686 added a background thread for periodically saving the cache index map, together with a list of completed cached files so that we can recover the cache state in case of crash or restart. Problem is that the cache index can become few GB large (a sample case with 1.6TB of used bucket cache would map to between 8GB to 10GB indexes), and these writes take few seconds to complete, causing any RS crash very likely to leave a corrupt index file that can't be recovered when the RS starts again. Worse, since we store the list of cached files on a separate file, this also leads to cache inconsistencies, with files in the list of cached files never cached once the RS is restarted, even though we have no cache index for those and every read ends up going to the FS.

This task aims to refactor the cache persistent as follows: 
1) Write both the list of completely cached files and the cache indexes in a single file, so that we can have this synced atomically;
2) When writing the persistent cache file, use a temp name first, then once the write is successfully finished, rename it to the actual name. This way, if crash happens whilst the persistent cache is still being written, the temp file would be corrupt, but we could still recover from the last successful sync, and we would only lose the caching ops since the last sync.
Bug: Limit max count of rows filtered per scan request.
Status: Open | Priority: Major | Resolution: 
Created: 22/Sep/23 02:37
Affects Version/s: 3.0.0

In some scenarios, when users do not understand the impact of the filters they set, it may lead to a large number of rows being filtered, causing very high CPU usage and system load, which may affect the requests of others. Therefore, we need to have the defensive capability to detect and kill such abnormal requests.
Bug: Netty SSL throughput improvement
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/23 12:15

Digging into HBASE-27947, I discovered an area for optimization in netty's SslHandler. I submitted that upstream to [https://github.com/netty/netty/issues/13549,] and submitted a PR for their review [https://github.com/netty/netty/pull/13551.] 

It's likely we will need changes in HBase to integrate this, including updating hbase-thirdparty once the change is released, and adding support for calling SslHandler.setWrapDataSize. This issue encapsulates that work.