av9ash/GitBugs huggingface dataset:
Bug: Rowkey pushdown does not work with complex conditions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Mar/22 16:19
Affects Version/s: connector-1.0.0

When using pushdown column filter feature of hbase-spark-connector, issuing complex query containing rowkey conditions does not get expected rowkey pushdown.
{code:java}
{
  "table":{"namespace":"default", "name":"t1"},
  "rowkey":"key",
  "columns":{
    "KEY_FIELD":{"cf":"rowkey", "col":"key", "type":"string"},
    "A_FIELD":{"cf":"c", "col":"a", "type":"string"},
    "B_FIELD":{"cf":"c", "col":"b", "type":"string"}
  }
}
{code}
For example, given the catalog, a query `spark.sql("SELECT * FROM table WHERE KEY_FIELD >= 'get1' AND KEY_FIELD <= 'get3' AND A_FIELD IS NOT NULL")` gets incomplete rowkey pushdown (ScanRange:(upperBound:get3,isUpperBoundEqualTo:true,lowerBound:,isLowerBoundEqualTo:true)).

If a query is `spark.sql("SELECT * FROM table WHERE KEY_FIELD >= 'get1' AND KEY_FIELD <= 'get3'")`, we get normal rowkey pushdown (ScanRange:(upperBound:get3,isUpperBoundEqualTo:true,lowerBound:,isLowerBoundEqualTo:true)).

I found that ScanRange#getOverlapScanRange and ScanRange#mergeIntersect return incorrect results if the range from the arguments is wider than the instance (or  scanRange.getOverlapScanRange(scanRange) where scanRange1⊂scanRange2). Depending on the order of the Filters that the Spark optimization results produce, the order of the scan ranges that these methods receive could be the one that causes such a problem.

I will create a PR later.
Bug: HBase 1.x clients will choke on rep_barrier rows when scanning hbase 2.x meta
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Mar/22 15:09
Affects Version/s: 1.7.1

In hbase 2.x, support for serial replication included adding a new CF to meta called rep_barrier. When regions are split or merged, these rep_barrier rows will not be cleaned up. Instead there's a ReplicationBarrierCleaner chore which runs every 12 hours. HBase 2.x clients will ignore these rep_barrier rows, per the [addFamily call in locateRegionInMeta|[https://github.com/apache/hbase/blob/branch-2/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java#L929].]

Encountering these orphan rep_barrier rows causes the hbase 1.x client to fail when it [tries to extract the region location from the meta row|[https://github.com/apache/hbase/blob/branch-1/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java#L1340-L1344].] This is a non-recoverable exception, so retries will fail and it will eventually bubble up.

The immediate fix when encountering this is to run {{{}hbck2 fixMeta{}}}, but we should fix the hbase 1.x client to similarly filter on the CATALOG_FAMILY to avoid these issues altogether.
Bug: Remove javax.ws.rs-api dependency
Status: Reopened | Priority: Major | Resolution: 
Created: 08/Mar/22 20:33
Affects Version/s: 2.5.0

I see that we still have {{javax.ws.rs-api}} as a dependency in our hbase-http pom. If, for example, the {{ClientBuilder}} from this jar is used, it'll instantiate whatever non-shaded jersey client is on the class path.
Bug: RSRpcServices.scan should deep clone cells when RpcCallContext is null
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Mar/22 11:41
Affects Version/s: 2.4.11

When inspect HBASE-26812, I find that if {{RpcCallContext}} is null,  {{RSRpcServices.scan}}  does not set {{ServerCall.rpcCallback}} and directly closes {{RegionScannerImpl}},  but it does not deep clone the result cells , so these cells may be returned to the {{ByteBuffAllocator}} and may be overwritten before the caller reads them, similar as HBASE-26036, and at the same time, if {{RpcCallContext}} is null, when {{RSRpcServices.scan}} return partial results, it does not invoke {{RegionScannerImpl.shipped}} to release the used resources such as {{HFileScanner}}, pooled {{ByteBuffer}} etc.

No matter {{ShortCircuitingClusterConnection}} should be removed or not, I think  this {{RSRpcServices.scan}} problem should fix for future maintainability.
Bug: Default StoreHotnessProtector to off, with logs to guide when to turn it on
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Mar/22 20:41

As discussed in HBASE-26522 and somewhat in HBASE-26575, StoreHotnessProtector is too conservative. One can easily trigger RegionTooBusyException on an otherwise idle regionserver with relatively small amount of traffic. The idea behind this protection is good, but it needs to take into account current load in order to warrant being enabled by default.

Until it's improved as such, we should change it to default off. One thing we can do to help drive usage would be to add warning logs when it would have triggered. Operators who are diagnosing performance issues will naturally look at the logs and see indication of a potential tool for mitigation.

cc [~andrew.purtell@gmail.com]
Bug: Shutdown WAL may abort region server
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Mar/22 02:06

https://nightlies.apache.org/hbase/HBase-Flaky-Tests/master/3140/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.replication.TestSyncReplicationActive-output.txt

TestSyncReplicationAcive is flaky because of we may abort the region server when shutting down WAL.
{noformat}
2022-03-18T04:50:37,205 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36877] master.MasterRpcServices(682): jenkins-hbase13.apache.org,33377,1647579008859 reported a fatal error:
***** ABORTING region server jenkins-hbase13.apache.org,33377,1647579008859: Log rolling failed *****
Cause:
java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL$$Lambda$681/1458648270@37209753 rejected from java.util.concurrent.ThreadPoolExecutor@69662eb7[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 0]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
	at java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.cleanOldLogs(AbstractFSWAL.java:773)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriterInternal(AbstractFSWAL.java:935)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.lambda$rollWriter$8(AbstractFSWAL.java:953)
	at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:196)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:953)
	at org.apache.hadoop.hbase.wal.AbstractWALRoller$RollController.rollWal(AbstractWALRoller.java:316)
	at org.apache.hadoop.hbase.wal.AbstractWALRoller.run(AbstractWALRoller.java:214)
{noformat}

The problem here is that, the removal of WAL is async, when shuttting down the WAL, we will close the thread pool so it will throw reject execution exception and cause region server abort.
Bug: Optimize the implementation of LRUCache in LRUDictionary
Status: Reopened | Priority: Major | Resolution: 
Created: 16/Mar/22 09:14
Affects Version/s: 1.7.1

*This issue is to unify the behavior of reading and writing links.*

 

During my research on HBASE-26849, I found that there seems to be something wrong with the implementation of LRUDictionary.

It uses array to save data, and uses hashMap to save array index.

For the write link:
{code:java}
CompressedKvEncoder#write
->
LRUDictionary#findEntry
->
LRUDictionary#addEntry
{code}
And for the read link:
{code:java}
CompressedKVDecoder#readIntoArray
->
LRUDirectory#addEntry
{code}
Then we could see the logic in {_}findEntry{_}:
{code:java}
  @Override
  public short findEntry(byte[] data, int offset, int length) {
    short ret = backingStore.findIdx(data, offset, length);
    if (ret == NOT_IN_DICTIONARY) {
      addEntry(data, offset, length);
    }
    return ret;
  }

    private short findIdx(byte[] array, int offset, int length) {
      Short s;
      final Node comparisonNode = new Node();
      comparisonNode.setContents(array, offset, length);
      if ((s = nodeToIndex.get(comparisonNode)) != null) {
        moveToHead(indexToNode[s]);
        return s;
      } else {
        return -1;
      }
    }
{code}
At first it will try to find if there is same node in the cache, If there is, use it directly.

The problem is, *if we have some same nodes, The behavior on the read and write links are inconsistent:*
On the write link we will reuse just one node, but on the read link multiple copies will be stored in the array.
but there is only one mapping from the node to the array index in the map. 
So, on the read link, if the first 'same' node is evict, we will remove it from the map, then we might met a NPE because when the second 'same' node evict we can not find it in the map.
Bug: [hbase-thirdparty] Upgrade Netty to 4.1.73.Final
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Feb/22 20:15

Netty 4.1.71.Final included a fix for [HTTP-request-smuggling|https://github.com/netty/netty/security/advisories/GHSA-wx5j-54mm-rqqq]. We are still on 4.1.70.Final. 

Latest is 4.1.73.Final, use that.
Bug: getAllRegionLocations can cache locations with null hostname
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Mar/22 15:53

RegionLocator methods typically delegate to ConnectionImplementation.locateRegion, which throws a NoServerForRegionException if the located region's serverName is null. RegionLocator.getAllRegionLocations does not go through that path, instead caching all returned region locations without any validation. This can result in a "dirty" meta cache, since clients do not expect to have null serverNames in the meta cache. We should add the same throwing of NoServerForRegionException to this method as used in the others. Or at least we should not cache the result if the serverName is null.
Bug: ExportSnapshot should validate the source snapshot before copying files
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jan/22 01:14
Affects Version/s: 0.99.0

Running {{ExportSnapshot}} with default parameters will copy the snapshot to a target location, and then use {{verifySnapshot}} to validate the integrity of the written snapshot. However, it is possible for the source snapshot to be invalid which leads to an invalid exported snapshot.

We can validate the source snapshot before export.

By default, we can validate the source snapshot unless the {{-no-target-verify}} parameter is set. We could also introduce a separate parameter for {{-no-source-verify}} if an operator wanted to validate the target but not validate the source for some reason, to provide some amount of backwards compatibility if that scenario is important.
Bug: RpcServer failure to SASL handshake always logs user "unknown" to audit log
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Feb/22 10:23
Affects Version/s: 2.4.10

Over on HBASE-26622, we noticed that the "attempting user" is always returned as "Unknown". This information is sent to the audit log when constructing the {{HBaseSaslRpcServer}}.
Bug: Upgrade Maven Enforcer Plugin
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/22 19:17
Affects Version/s: 2.5.0

I've noticed that the beanshell rule run by our configuration of the enforcer plugin will sometimes cause build failures when I run maven with {{-T1.0C}}. It seems that this is a known issue (MENFORCER-336) that has been fixed in a later version of the plugin. Let's upgrade.
Bug: Close and do not write trailer for the broken WAL writer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jan/22 09:56
Affects Version/s: 2.0.0

{code:java}
public static AsyncWriter createAsyncWriter(Configuration conf, FileSystem fs, Path path,
    boolean overwritable, long blocksize, EventLoopGroup eventLoopGroup,
    Class<? extends Channel> channelClass, StreamSlowMonitor monitor) throws IOException {
  // Configuration already does caching for the Class lookup.
  Class<? extends AsyncWriter> logWriterClass = conf.getClass(
    WRITER_IMPL, AsyncProtobufLogWriter.class, AsyncWriter.class);
  try {
    AsyncWriter writer = logWriterClass.getConstructor(EventLoopGroup.class, Class.class)
        .newInstance(eventLoopGroup, channelClass);
    writer.init(fs, path, conf, overwritable, blocksize, monitor);
    return writer;
  } catch (Exception e) {
    if (e instanceof CommonFSUtils.StreamLacksCapabilityException) {
      LOG.error("The RegionServer async write ahead log provider " +
        "relies on the ability to call " + e.getMessage() + " for proper operation during " +
        "component failures, but the current FileSystem does not support doing so. Please " +
        "check the config value of '" + CommonFSUtils.HBASE_WAL_DIR + "' and ensure " +
        "it points to a FileSystem mount that has suitable capabilities for output streams.");
    } else {
      LOG.debug("Error instantiating log writer.", e);
    }
    Throwables.propagateIfPossible(e, IOException.class);
    throw new IOException("cannot get log writer", e);
  }
} {code}
I think writer should be closed when encounters init exception here.

This can reduce the recoverLease time before log split, and will reduce side effects after HBASE-26552. 

Broken writers need to be closed, so that when doing log split, there will be no need to recover lease for those length 0 wals.
Bug: Increase the concurrency when running UTs in pre commit job
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Mar/22 16:13

The default forkCount is 0.25C. We used to run on a 16 cores machine so the concurrency is 4, but now our build nodes are all 8 cores, so the concurrency is 2. The build time is way more slower than before.

Since we only have 2 slots on the build node, I think it is OK to increase the forkCount to 0.5C. But I'm not sure whether we should change the default value in pom, or we just set it in the pre commit job.
Bug: Junit jar is not included in the hbase tar ball, causing issues for some  hbase tools that do rely on it
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Mar/22 22:02
Affects Version/s: 2.4.11

We used to include junit jar on the generated tar ball lib directory. After some sanitisation of unnecessary libs for most of hbase processes, junit got removed from the packing, so that it don't end in hbase classpath by default. 

Some testing tools, however do depend on junit at runtime, and would now fail with NoClassDefFoundError, like [IntegrationTestIngest:|https://hbase.apache.org/book.html#chaos.monkey.properties]
{noformat}
2022-03-14T21:54:50,483 INFO  [main] client.AsyncConnectionImpl: Connection has been closed by main.
Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
	at org.apache.hadoop.hbase.IntegrationTestIngest.initTable(IntegrationTestIngest.java:101)
	at org.apache.hadoop.hbase.IntegrationTestIngest.setUpCluster(IntegrationTestIngest.java:92)
	at org.apache.hadoop.hbase.IntegrationTestBase.setUp(IntegrationTestBase.java:170)
	at org.apache.hadoop.hbase.IntegrationTestBase.doWork(IntegrationTestBase.java:153)
	at org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:153)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.hadoop.hbase.IntegrationTestIngest.main(IntegrationTestIngest.java:259)
Caused by: java.lang.ClassNotFoundException: org.junit.Assert
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
	... 7 more {noformat}
Discussing with [~elserj] internally, we believe a reasonable solution would be to include junit jar back into the tarball, under "lib/test" dir so that it's not automatically added to hbase processes classpath, but still allow operators to manually define it in a convenient way, like below:
{noformat}
HBASE_CLASSPATH="$HBASE_HOME/lib/tests/*" hbase org.apache.hadoop.hbase.IntegrationTesngest -m slowDeterministic {noformat}
Bug: Use HIGH_QOS for ResultScanner.close requests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Mar/22 20:58

Active scanners take resources on the RegionServer, and scanner leaks can have a big impact on GC. Under high load, ResultScanner.close() calls might get rejected or timed out before executing. Exceptions thrown by close() are currently swallowed, so clients will not retry. This leads to a leak of active scanners on the server side.

These calls should be relatively free to execute, and executing them will free up the resources which can have an impact on overall RegionServer performance. As such it can be beneficial to execute all ResultScanner.close() requests with HIGH_QOS regardless of the priority of the underlying Scan.

In a scan-heavy stress test I ran, after a couple minutes the Full GC time (using G1GC) starts increasing and we start seeing 1s pauses, due to increasing number of active scanners (into the 10s of thousands). Re-running the stress test with HIGH_QOS for ResultScanner.close() eliminates the leak and GC time remains stable.
Bug: ScannerCallable doubly clears meta cache on retries
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Mar/22 18:50
Affects Version/s: 2.4.10

Way back in HBASE-15658 [~ghelmling] fixed RegionServerCallable to not clear meta in {{{}prepare(boolean reload){}}}. because it already would have cleared it in the try/catch when {{{}throwable(Throwable t, boolean retrying){}}}.

I have recently been doing some load tests where I am causing HBase RegionServers to throw many CallDroppedExceptions because they are overloaded by the test. While this is an extreme example, it does sometimes crop up in production when a bad actor executes a job without rate limiting, etc. What I noticed was that the RegionServer hosting meta was most affected by the load, way more than any other server in the cluster. Digging into the issue I realized that the extra meta load was coming mostly from the scans, originating from {{{}ScannerCallable.prepare(boolean reload){}}}.

I'm not sure why ScannerCallable was excluded from the original jira, maybe an oversight. But ScannerCallable is called in the same context of RetryingRpcCaller, which will handle clearing meta in the try/catch like other callables. We should similarly update ScannerCallable's prepare method not always pass useCache=true when getting region locations.
Bug: The TRSP should not go on when it get a bogus server name from AM
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Mar/22 11:23

Currently it will submit lots of unnecessary OpenRegionProcedure by retry.

Related log looks like below, 'localhost,1,1' is the bogus server:
{code:java}
2022-03-22 10:17:48,301 WARN  [PEWorker-8] assignment.RegionRemoteProcedureBase: Can not add remote operation pid=17952, ppid=17951, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure for region {ENCODED => 490391c232c7aa13f7e0d50bfe1f7235, NAME => 'TestTable1,0000000000000000002497747,1647568640784.490391c232c7aa13f7e0d50bfe1f7235.', STARTKEY => '0000000000000000002497747', ENDKEY => ''} to server localhost,1,1, this usually because the server is alread dead, give up and mark the procedure as complete, the parent procedure will take care of this.
org.apache.hadoop.hbase.procedure2.NoServerDispatchException: localhost,1,1; pid=17952, ppid=17951, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure
        at org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher.addOperationToNode(RemoteProcedureDispatcher.java:168)
        at org.apache.hadoop.hbase.master.assignment.RegionRemoteProcedureBase.execute(RegionRemoteProcedureBase.java:285)
        at org.apache.hadoop.hbase.master.assignment.RegionRemoteProcedureBase.execute(RegionRemoteProcedureBase.java:58)
        at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:962)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1648)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1395)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:78)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1965)
2022-03-22 10:17:48,301 DEBUG [PEWorker-8] procedure2.RootProcedureState: Add procedure pid=17952, ppid=17951, state=SUCCESS, locked=true; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure as the 8th rollback step {code}
Bug: User.createUserForTesting should not reset UserProvider.groups every time if hbase.group.service.for.test.only is true
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/22 16:05
Affects Version/s: 2.4.9

The _if check_ below will always unnecessarily reset static var _UserProvider.groups_ to a newly created instance of TestingGroups every time `User.createUserForTesting` is called.
{noformat}
        if (!(UserProvider.groups instanceof TestingGroups) ||
            conf.getBoolean(TestingGroups.TEST_CONF, false)) {
          UserProvider.groups = new TestingGroups(UserProvider.groups);
        }
{noformat}
For tests creating multiple {_}test users{_}, this causes the latest created user to reset _groups_ and all previously created users would now have to be available on the {_}User.underlyingImplementation{_}, which not always will be true.
Bug: Balancer encounters NPE in rare case
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jan/22 19:34
Affects Version/s: 2.4.9

 
{code:java}
ERROR [ster-1:60000.Chore.1] hbase.ScheduledChore - Caught error
java.lang.NullPointerException
    at org.apache.hadoop.hbase.master.assignment.AssignmentManager.balance(AssignmentManager.java:758)
    at org.apache.hadoop.hbase.master.HMaster.executeRegionPlansWithThrottling(HMaster.java:1834)
    at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1797)
    at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1707)
    at org.apache.hadoop.hbase.master.balancer.BalancerChore.chore(BalancerChore.java:49)
    at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:153)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 {code}
Let's fix this.