av9ash/GitBugs huggingface dataset:
Bug: hbase shell config override option does not work
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Nov/21 11:59
Affects Version/s: 3.0.0-alpha-2

According to the docs, we should be able to specify properties on the command line with the -D switch.

https://hbase.apache.org/book.html#_overriding_configuration_starting_the_hbase_shell

However:

{noformat}
./bin/hbase shell -Dhbase.zookeeper.quorum=ZK0.remote.cluster.example.org,ZK1.remote.cluster.example.org,ZK2.remote.cluster.example.org -Draining=false

Usage: shell [OPTIONS] [SCRIPTFILE [ARGUMENTS]]

 -d | --debug            Set DEBUG log levels.
 -h | --help             This help.
 -n | --noninteractive   Do not run within an IRB session and exit with non-zero
                         status on first error.
 --top-level-defs        Compatibility flag to export HBase shell commands onto
                         Ruby's main object
 -Dkey=value             Pass hbase-*.xml Configuration overrides. For example, to
                         use an alternate zookeeper ensemble, pass:
                           -Dhbase.zookeeper.quorum=zookeeper.example.org
                         For faster fail, pass the below and vary the values:
                           -Dhbase.client.retries.number=7
                           -Dhbase.ipc.client.connect.max.retries=3
classpath:/jar-bootstrap.rb: invalid option -- b
GetoptLong::InvalidOption: invalid option -- b
  set_error at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/getoptlong.rb:395
        get at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/getoptlong.rb:572
       each at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/getoptlong.rb:603
       loop at org/jruby/RubyKernel.java:1442
       each at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/getoptlong.rb:602
     <main> at classpath:/jar-bootstrap.rb:98
{noformat}

Something is broken in the command line parsing.
Bug: Use unique thread name for group WALs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Dec/21 08:22
Affects Version/s: 2.0.0

The consumer threads for each WAL group has the same name, since they only use the WAL root dir in the thread name.
{code:java}
new ThreadFactoryBuilder().setNameFormat("AsyncFSWAL-%d-" + rootDir.toString()).
  setDaemon(true).build()); {code}
For example, for BoundedGroupingStrategy, the consumer threads names are as follows,

!image-2021-12-01-16-20-18-912.png|width=1199,height=130!

We can use the log prefix instead, the consumer threads names will be changed to

!image-2021-12-02-17-38-21-959.png|width=1102,height=197!

So we can clearly see what happens from the log and the jstack info if something wrong with the WAL.
Bug: Clone snapshot doesn't load reference files into FILE SFT impl
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Oct/21 10:13

After cloning a snapshot from a FILE SFT enabled table, noticed that none of the cloned table files were added into the FILE SFT meta files, in fact, FILE SFT meta dir didn't even got created. Scanning this cloned table gives no results, as none of the files are tracked. I believe we need to call StoreFileTracker.add during the snapshot cloning.
Bug: CreateTableProcedure still relies on temp dir and renames when creating table FS layout
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Nov/21 16:29

As pointed by [~elserj] when reviewing some other issue, internally, CreateTableProcedure still creates the FS layout on temp folder first, then rename to actual dir. I think this is unnecessary, we could create it straight in the final dir, just like we did in [CloneSnapshotProcedure here|https://github.com/apache/hbase/pull/3749/files#r728424375].
Bug: Fix MiniMapReduceCluster failure under Java17
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Nov/21 11:52

MiniMapReduceCluster in HBaseTestingUtil.java will not work. Find the error message in the container log:

{code:java}
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make protected final java.lang.Class java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) throws java.lang.ClassFormatError accessible: module java.base does not "opens java.lang" to unnamed module @755c9148
	at java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354) ~[?:?]
	at java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297) ~[?:?]
	at java.lang.reflect.Method.checkCanSetAccessible(Method.java:199) ~[?:?]
	at java.lang.reflect.Method.setAccessible(Method.java:193) ~[?:?]
	at com.google.inject.internal.cglib.core.$ReflectUtils$2.run(ReflectUtils.java:56) ~[guice-4.0.jar:?]
	at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
	at com.google.inject.internal.cglib.core.$ReflectUtils.<clinit>(ReflectUtils.java:46) ~[guice-4.0.jar:?]
{code}
The container of MiniMapReduceCluster is running in an independent JVM. So that we need to pass the JVM opts to the container JVM.
Impacted UTs:
TestSecureExportSnapshot
TestMobSecureExportSnapshot
TestVerifyReplicationCrossDiffHdfs
Bug: Provide HBase upgrade guidelines from 1.6 to 2.4+ versions
Status: Resolved | Priority: Major | Resolution: Implemented
Created: 12/Oct/21 06:57

Provide some ref guide under section: [https://hbase.apache.org/book.html#upgrade2.0.rolling.upgrades] 

This should include experience of performing in-place rolling upgrade (without downtime) from 1.6/1.7 to 2.4+ release versions.
Bug: check-aggregate-license error related to javadns after HADOOP-17317
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Nov/21 21:37
Affects Version/s: 2.5.0

This impacts any HBase branch compiling vs. Hadoop trunk (current 3.4.0-SNAPSHOT), which includes HADOOP-17317. Build fails with
{noformat}
[INFO] --- maven-enforcer-plugin:3.0.0-M3:enforce (check-aggregate-license) @ hbase-shaded-client ---                                                                                         
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.EvaluateBeanshell failed with message:                                                                                                    
License errors detected, for more detail find ERROR in                                                                                                                                        
                    /Users/ndimiduk/repos/apache/hbase/hbase-shaded/hbase-shaded-client/target/maven-shared-archive-resources/META-INF/LICENSE
{noformat}
The aggregated license file contains this entry,
{noformat}
--
This product includes dnsjava licensed under the BSD-2-Clause.

ERROR: Please check ^^^^^^^^^^^^ this License for acceptability here:

https://www.apache.org/legal/resolved

If it is okay, then update the list named 'non_aggregate_fine' in the LICENSE.vm file.
If it isn't okay, then revert the change that added the dependency.

More info on the dependency:

<groupId>dnsjava</groupId>
<artifactId>dnsjava</artifactId>
<version>3.4.0</version>

maven central search
g:dnsjava AND a:dnsjava AND v:3.4.0

project website
https://github.com/dnsjava/dnsjava
project source
https://github.com/dnsjava/dnsjava
{noformat}
Bug: Introduce a SnapshotProcedure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Oct/21 13:39

Currently，snapshot in hbase uses zk as coordinator. It has some limitations, 
 a. Snapshot maybe fails when there are region server crashes.
 b. Snapshot maybe failed when master restarts.
 c. Only one snapshot per table can be taken in a time.
 d. Snapshot verify will be handled by master, which may take long time when our table has a large number of regions, for example 10000.

 

Since we have procedure v2 framework now, it is possible to solve the above problems. So here is a procedure2-based snapshot implementation. It has some goals,
 a. Snapshot can continue when there are region server crashes.
 b. Snapshot can continue when master restarts.
 c. More than one snapshot per table can be taken in a time.
 d. We can use region servers to verify snapshot to accelerate procedure.

 

Here are some details about implementation.
 *SnapshotProcedure*
 SnapshotProcedure is used to take snapshot on a table. It acquires shared table lock on the snapshot table and hold the shared lock during suspend and yield. 
 *SnapshotRegionProcedure*
 SnapshotRegionProcedure is used to take snapshot on a specific region of the snapshot table. It acquires exclusive region lock and releases lock during suspend and yield. Before dispatch remote snapshot operations to region server, it will check target region in RIT or not. If target region is in RIT, it will sleep some time and retry.
 *SnapshotVerifyProcedure*
 SnapshotVerifyProcedure is used to send snapshot verify request to region server. If snapshot is corrupted, it will notify parent snapshot to retry. When remote region server is crashed, it will choose another online server and retry.

 

I would be very grateful for any advice and guidance. Is anyone interested in taking a look?
Bug: Using RefCnt to fix the flawed MemStoreLABImpl
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Nov/21 04:25
Affects Version/s: 2.4.9

As  HBASE-26465 said,  reference count implementation in {{MemStoreLABImpl}} is flawed because its checking and increasing or decreasing is not done in atomicity and it ignores state checking when there is illegal state in reference count(eg. increasing or decreasing when the resource is already freed) , just as following {{incScannerCount}} and {{decScannerCount}} methods illustrated, and this flawed implementation has shield the bugs HBASE-26465 and HBASE-26488.
{code:java}
  public void incScannerCount() {
    this.openScannerCount.incrementAndGet();
  }

  public void decScannerCount() {
    int count = this.openScannerCount.decrementAndGet();
    if (this.closed.get() && count == 0) {
      recycleChunks();
    }
  }
{code}

We could Introduce {{RefCnt}} into {{MemStoreLABImpl}} to replace its flawed reference count implementation, so checking and increasing or decreasing is done in atomicity and if there is illegal state in reference count, it would throw exception rather than continue using the corrupt data to cause some subtle bugs, and meanwhile, the code is more simpler.
Bug: Hbase-client Meta lookup performance regression between hbase-1 and hbase-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Dec/21 20:09
Affects Version/s: 2.3.7

One of our users complained higher latency after application upgrades from hbase-1.2 client (CDH-5.16.2) to hbase-2.4.5 client with meta replica Load Balance mode during app restart. I reproduced the regression by a test for meta lookup. 

At my test cluster, there are 160k regions for the test table, so there are 160k entries in meta region. Used one thread to do 1 million meta lookup against the meta region server.

 
||Version ||Meta Replica Load Balance Enabled||Time               ||
||2.4.5-with-fixed||Yes||336458ms||
||2.4.5-with-fixed||No||333253ms||
||2.4.5||Yes||469980ms||
||2.4.5||No||470515ms||
|      *cdh-5.16.2*|                                *No* |  *323412ms*|
Bug: Introduce a timeout to shutdown of WAL
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Dec/21 09:00
Affects Version/s: 2.0.0

When the regionserver is aborted(server.isAborted() is true), it will just reject the newly coming requests, so it is important to keep a short time to stop to make a short MTTR. 

When the WAL is shutting down, we can wait some time to flush the unacked appends, but if that is a long time, I think we can just abort without flushing them. 

By introducing a timeout for WAL shutdown, we can control the shutdown time, especially for those sensitive businesses.
Bug: Separate Log Cleaner DirScanPool to prevent the OLDWALs from filling up the disk when archive is large
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Oct/21 22:50
Affects Version/s: 1.7.1

We currently share the DirScanPool (threadpool for scanning for files to delete in the OldLogs and archive directories) between the LogCleaner and HFileCleaner. This means that if the archive directory is large/has lots of files/directories, the threads can get stuck scanning through the archive directory, starving the LogCleaner. This is especially apparent on S3 where list can be slower than on HDFS.

This JIRA creates separate DirScanPools for the LogCleaner and HFileCleaner
Bug: maven-gpg-plugin failing with "Inappropriate ioctl for device"
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Dec/21 01:55

In some environments, if the Maven GPG plugin is too old, GPG signing will fail with "Inappropriate ioctl for device"

The fix is to specify the latest version of the GPG plugin in the {{pluginManagement}} section of the project's POM, currently 3.0.1. 

{noformat}
    ...
    <pluginManagement>
      <plugins>
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-gpg-plugin</artifactId>
          <version>3.0.1</version>
        </plugin>
    ...
    </pluginManagement>
{noformat}
Bug: FuzzyRowFilter backwards compatibility
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Dec/21 15:06
Affects Version/s: 1.1.13

HBASE-15676 introduced a backwards incompatible change which makes it impossible to upgrade in our designated upgraded order (server first, then client) without potential bad results. Worse, the failure mode is silent – a pre-HBASE-15676 client would incorrectly receive 0 results from a post-HBASE-15676 server.

I solved this internally as part of our upgrade from 1.2.0 to 2.4.6 by adding a new proto field to switch between the two implementations. I'm submitting this Jira to capture and potentially backport that fix for anyone else who encounters it.
Bug: Introduce retry to logroller to avoid abort
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Dec/21 07:18
Affects Version/s: 2.4.10

When calling RollController#rollWal in AbstractWALRoller, the regionserver may abort when encounters exception,
{code:java}
...
} catch (FailedLogCloseException | ConnectException e) {
  abort("Failed log close in log roller", e);
} catch (IOException ex) {
  // Abort if we get here. We probably won't recover an IOE. HBASE-1132
  abort("IOE in log roller",
    ex instanceof RemoteException ? ((RemoteException) ex).unwrapRemoteException() : ex);
} catch (Exception ex) {
  LOG.error("Log rolling failed", ex);
  abort("Log rolling failed", ex);
} {code}
I think we should support retry of rollWal here to avoid recovering the service by killing regionserver. The restart of regionserver is costly and very not friendly to the availability.

I find that when creating new writer for the WAL in FanOutOneBlockAsyncDFSOutputHelper#createOutput, it supports retry to addBlock by setting this config "hbase.fs.async.create.retries". The idea of retry to roll WAL is similar to it, they both try best to make roll WAL succeed. 

But the initialization of new WAL writer also includes flushing the write buffer flush and waiting until it is completed by AsyncProtobufLogWriter#writeMagicAndWALHeader, which can also fail by some hardware reasons. The regionserver connected to the datanodes after addBlock, but that not means the magic and header can be flushed successfully.
{code:java}
protected long writeMagicAndWALHeader(byte[] magic, WALHeader header) throws IOException {
  return write(future -> {
    output.write(magic);
    try {
      header.writeDelimitedTo(asyncOutputWrapper);
    } catch (IOException e) {
      // should not happen
      throw new AssertionError(e);
    }
    addListener(output.flush(false), (len, error) -> {
      if (error != null) {
        future.completeExceptionally(error);
      } else {
        future.complete(len);
      }
    });
  });
}{code}
We have found that in our production clusters, there exists aborting of regionservers that caused by "IOE in log roller". And the practice in our clusters is that just one more retry of rollWal can make the WAL roll complete and continue serving.
Bug: Refactor code related to "dump"ing ZK nodes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Dec/21 02:26
Affects Version/s: 2.5.0

The code starting at {{ZKUtil.dump(ZKWatcher)}} is a small mess -- it has cyclic dependencies woven through itself, {{ZKWatcher}} and {{RecoverableZooKeeper}}. It also initializes a static variable in {{ZKUtil}} through the factory for {{RecoverableZooKeeper}} instances. Let's decouple and clean it up.
Bug: Support custom paths of independent configuration and pool for hfile cleaner
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Oct/21 05:34
Affects Version/s: 1.7.1

With this, we can clean some paths more quickly.

We found in our cluster, when the very huge table with thousands of regions and high write throughputs and many snapshots tables on the same cluster, the speed of delete files in  archive path will lower than the speed of moved in files by compaction. Then archive may remains PB level data. 

The bottleneck is in cleaner but not in the thread pool size or queue size. It is because there is synchronized lock in SnapshotFileCache, and a batch of files need once SnapshotFileCache#refreshCache(), which look through all the snapshot dirs.

The speed of clear a path without the SnapshotHFileCleaner is thirty times faster.
Bug: hbase-shaded-client missing required thirdparty classes under hadoop 3.3.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Dec/21 17:44
Affects Version/s: 2.4.3

In HBASE-25792, the shaded thirdparty libraries from hadoop were removed from the hbase-shaded-client fat jar to satisfy invariant checks. Unfortunately this causes users of hbase-shaded-client to fail, because required classes are not available at runtime.

The specific failure I'm seeing is when trying to call new Configuration(), which results in:

 
 
{code:java}
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/thirdparty/com/google/common/base/Preconditions   
  at org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configuration.java:430)   
  at org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configuration.java:443)   
  at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:525){code}
 
 
If you take a look at the hbase-shaded-client fat jar, it contains the org.apache.hadoop.conf.Configuration class as you'd expect. If you decompile that class (or look at the 3.3.1 source), you'll see that there is an import for org.apache.hadoop.thirdparty.com.google.common.base.Preconditions but the fat jar does not provide it.
 
One way for clients to get around this is to add an explicit dependency on hadoop-shaded-guava, but this is problematic for a few reasons:
 
- it's best practice to use maven-dependency-plugin to disallow declared, unused dependencies (which this would be)
- it requires users to continually keep the version of hadoop-shaded-guava up-to-date over time.
- it only covers guava, but there is also protobuf and potentially other shaded libraries in the future.
 
I think we should remove the exclusion of {{org/apache/hadoop/thirdparty/**/*}} from the shading config and instead add that pattern to the allowlist so that hbase-shaded-client is all clients need to get started with hbase.
Bug: Refactor code related to ZooKeeper authentication
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Dec/21 23:14
Affects Version/s: 2.5.0

This refactor reduces the size and scope of the {{ZKUtil}} class. The core of this refactor is moving the {{login*}} methods from {{ZKUtil}} into their own class, {{ZKAuthentication}}. The class {{JaasConfiguration}} is also moved along with them.
Bug: hbaseprotoc plugin should initialize maven
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Dec/21 00:13
Affects Version/s: 3.0.0-alpha-1

My PR testing HBASE-26542 fails because the mvn incantation does not find other project module dependencies. Looking over our personality file and the Yetus documentation [precommit-advanced|https://yetus.apache.org/documentation/0.12.0/precommit-advanced/] says

{quote}
NOTE: If the plug-in has support for maven, the maven_add_install pluginname should be executed. See more information in Custom Maven Tests in the build tool documentation.
{quote}
Bug: Support detect and exclude slow DNs in fan-out of WAL
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Oct/21 12:06
Affects Version/s: 2.0.0

We all knows the WAL sync performance directly affects the RPC process time.

And we use self-designed FanOutOneBlockAsyncDFSOutput to sync WAL entries, which connect straightly to all the block located DNs. But when even one DN of the locations is slow, e.g. some disk hardware failures, the WAL syncs slow. And what's more, the hardware failure detected by the lower layer HDFS system is not so sensitive.

We can detect slow DNs by the ACK time of packets in FanOutOneBlockAsyncDFSOutput, and exclude them when add new blocks after log rolled(rolling log can also be triggered by slow syncs). And shows this info in UI. We can also invalid these excluded DN cache after a duration, to aware the recovery of those DNs. 

I think this idea can quickly reduce the influence of slow DNs, and improve the service availability.
Bug: Investigate mTLS in RPC layer
Status: Resolved | Priority: Major | Resolution: Done
Created: 08/Dec/21 17:38

Current authentication options are heavily based on SASL and Kerberos. For organizations that don't already deploy Kerberos or other token provider, this is a heavy lift. Another very common way of authenticating in the industry is mTLS, which makes use of SSL certifications and can solve both wire encryption and auth. For those already deploying trusted certificates in their infra, mTLS may be much easier to integrate.

It isn't necessarily easy to implement this, but I do think we could use existing Netty SSL support in the NettyRpcClient and NettyRpcServer. I know it's easy to add SSL to non-blocking IO through a hadoop.rpc.socket.factory.class.default which returns SSLSockets, but that doesn't touch on the certification verification at all.

Much more investigation is needed, but logging this due to some interest encountered on slack.

Slack thread: https://apache-hbase.slack.com/archives/C13K8NVAM/p1638980520110600
Bug: region_mover should gracefully ignore null response from RSGroupAdmin#getRSGroupOfServer
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 17/Dec/21 09:38
Affects Version/s: 1.7.1

If regionserver has any non-daemon thread running even after it's own shutdown, the running non-daemon thread can prevent clean JVM exit and regionserver could be stuck in the zombie state. We have recently provided a workaround for this in HBASE-26468 for regionserver exit hook to wait 30s for all non-daemon threads to get stopped before terminating JVM abnormally.

However, if regionserver is stuck in such state, region_mover unload fails with:
{code:java}
NoMethodError: undefined method `getName` for nil:NilClass
  getSameRSGroupServers at /bin/region_mover.rb:503
             __ensure__ at /bin/region_mover.rb:313 
          unloadRegions at /bin/region_mover.rb:310               
                 (root) at /bin/region_mover.rb:572               
 {code}
This happens if the cluster has RSGroup enabled and the given server is already stopped, hence RSGroupAdmin#getRSGroupOfServer would return null (as the server is not running anymore so it is not part of any RSGroup). region_mover should ride over this null response and gracefully exit from unloadRegions() call.

 

We should also check if the fix is applicable to branch-2 and above.