av9ash/GitBugs huggingface dataset:
Bug: HMaster should move non-meta region only if meta is ONLINE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Nov/21 12:01
Affects Version/s: 1.7.1

Any non-meta region movement has dependency over meta's availability, hence it is important for any non-meta region to wait for meta to be assigned and be available for scan before attempting to move non-meta region.

This concept has already been well managed by SCP (ServerCrashProcedure) on HBase 1.x and 2.x versions. However, for 1.x versions, HMaster#move API doesn't check for any prerequisite of meta being available before attempting to move any non-meta region.

On the other hand, 2.x versions already have TransitRegionStateProcedure (TRSP) in place that uses lock _LockState.LOCK_EVENT_WAIT_ in case if meta is not yet assigned and loaded in AssignmentManager's memory:
{code:java}
@Override
protected boolean waitInitialized(MasterProcedureEnv env) {
  if (TableName.isMetaTableName(getTableName())) {
    return false;
  }
  // First we need meta to be loaded, and second, if meta is not online then we will likely to
  // fail when updating meta so we wait until it is assigned.
  AssignmentManager am = env.getAssignmentManager();
  return am.waitMetaLoaded(this) || am.waitMetaAssigned(this, getRegion());
}
 {code}
For 1.x versions, it is recommended to introduce configurable wait time in master's region move API for non-meta region movement until meta region is available. If meta remains in transition after the wait time elapses, we should fail fast and avoid non-meta region move.
Bug: Make timestamp format configurable in HBase shell scan output
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Nov/21 12:16
Affects Version/s: 3.0.0-alpha-1

HBASE-23930 and HBASE-24937 has changed the timestamp format shown in scan results in HBase shells.

This may break existing use cases that use hbase shell as a client. (as opposed to the java, rest, or thrift APIs)

I propose adding a configuration option to make it configurable.
Bug: Region Server doesn't exit cleanly incase it crashes.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Nov/21 21:59
Affects Version/s: 1.6.0

Observed this in our production cluster running 1.6 version.
RS crashed due to some reason but the process was still running. On debugging more, found out there was 1 non-daemon thread running and that was not allowing RS to exit cleanly. Our clusters are managed by Ambari and have auto restart capability within them. But since the process was running and pid file was present, Ambari also couldn't do much. There will be some bug where we will miss to stop some non daemon thread. Shutdown hook will not be called unless one of the following 2 conditions are met:

# The Java virtual machine shuts down in response to two kinds of events:
The program exits normally, when the last non-daemon thread exits or when the exit (equivalently, System.exit) method is invoked, or
# The virtual machine is terminated in response to a user interrupt, such as typing ^C, or a system-wide event, such as user logoff or system shutdown.

Considering the first condition, when the last non-daemon thread exits or when the exit method is invoked.

Below is the code snippet from [HRegionServerCommandLine.java|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServerCommandLine.java#L51]

{code:java}
  private int start() throws Exception {
    try {
      if (LocalHBaseCluster.isLocal(conf)) {
         // Ignore this.
      } else {
        HRegionServer hrs = HRegionServer.constructRegionServer(regionServerClass, conf);
        hrs.start();
        hrs.join();
        if (hrs.isAborted()) {
          throw new RuntimeException("HRegionServer Aborted");
        }
      }
    } catch (Throwable t) {
      LOG.error("Region server exiting", t);
      return 1;
    }
    return 0;
  }
{code}

Within HRegionServer, there is a subtle difference between when a server is aborted v/s when it is stopped. If it is stopped, then isAborted will return false and it will exit with return code 0.

Below is the code from [ServerCommandLine.java|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerCommandLine.java#L147]


{code:java}
  public void doMain(String args[]) {
    try {
      int ret = ToolRunner.run(HBaseConfiguration.create(), this, args);
      if (ret != 0) {
        System.exit(ret);
      }
    } catch (Exception e) {
      LOG.error("Failed to run", e);
      System.exit(-1);
    }
  }
{code}

If return code is 0, then it won't call System.exit. This means JVM will wait to call ShutdownHook until all non daemon threads are stopped which means infinite wait if we don't close all non-daemon threads cleanly.
Bug: TestStochasticLoadBalancerRegionReplicaWithRacks fails consistently
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Nov/21 12:42

https://nightlies.apache.org/hbase/HBase/HBase-Find-Flaky-Tests/master/442/output/dashboard.html

[~claraxiong] Mind taking a look? I guess it is because some recent changes...

Thanks.
Bug: Make DefaultMemStore extensible for HStore.memstore 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Nov/21 13:05
Affects Version/s: 2.4.8

When  {{HStore.memstore}} is {{DefaultMemStore}} ,  I find that it is impossible to extend {{DefaultMemStore}} to add some custom logic when I was work on  HBASE-26465, because following code in {{HStore.getMemstore}} and {{HStore.getMemstore}} is a private method and is called in {{HStore}} ctor .We should make the {{DefaultMemStore}} extensible just as in branch-1.
{code:java}
    switch (inMemoryCompaction) {
      case NONE:
        ms = ReflectionUtils.newInstance(DefaultMemStore.class,
            new Object[] { conf, getComparator(),
                this.getHRegion().getRegionServicesForStores()});
        break;
      default:
        Class<? extends CompactingMemStore> clz = conf.getClass(MEMSTORE_CLASS_NAME,
            CompactingMemStore.class, CompactingMemStore.class);
        ms = ReflectionUtils.newInstance(clz, new Object[]{conf, getComparator(), this,
            this.getHRegion().getRegionServicesForStores(), inMemoryCompaction});
    }
    return ms;
  }
{code}
Bug: Hbase Export/Import via MapReduce job
Status: Resolved | Priority: Major | Resolution: Not A Bug
Created: 11/Nov/21 11:55
Affects Version/s: 2.0.1

Hi Hbase support team,

 

While using the MapReduce job with export/import commands we have the below behaviour.

 

{+}Step1{+}:  I have a hbase table(tsdb) with 3 rows. and export the table like below:

/opt/hbase/bin/hbase org.apache.hadoop.hbase.mapreduce.Export tsdb file:///opt/hbase/backup/tsdb

{+}Step2{+}: Then I delete 1 row.

{+}Step3{+}: Then I import tsdb table from the exported data from Step 1.

/opt/hbase/bin/hbase org.apache.hadoop.hbase.mapreduce.Import tsdb file:///opt/hbase/backup/tsdb

 

There are still only 2 rows in the table. Is this a valid behaviour?

 

Br,

Chris
Bug: [hboss] Delete self lock without orphaning znode
Status: Open | Priority: Major | Resolution: 
Created: 18/Nov/21 01:21

Fallout from HBASE-26437
{quote}Could do the {{removeInMemoryLocks}} separately in HBASE-26453, but I think then znodes would get created again when unlocking, failing this PR tests. So, once we fix {{{}removeInMemoryLocks{}}}, we need to make sure {{rename}} and {{delete}} would not recreate the path again when calling {{{}unlock{}}}.
{quote}
The changes from HBASE-26453 inadvertently passed their unit tests because we didn't remove the Mutex object like we intended to do (after deleting a file/dir or renaming a file/dir, we intend to remove the mutex and znode for that file/dir and all beneath it).

Right now, we only actually delete the children (znode and mutex objects) for that deleted/renamed path. Meaning, we are still orphaning resources. I implemented the fix in lockRename based on what we did in lockDelete, so we're making incremental progress.

The lock cleanup process and Mutex logic need to be reworked because we cannot do it in two-phases as we currently do. In order to get the mutex to release it (when we are holding it already), we currently will re-create znodes back in ZooKeeper.

The other solution, based on googling, appears to be to use a [Reaper|https://www.javadoc.io/doc/org.apache.curator/curator-recipes/2.4.1/org/apache/curator/framework/recipes/locks/Reaper.html]. This might also be an easier solution to the problem to do the rest of the cleanup.
Bug: Some BaseLoadBalancer log lines should be at DEBUG level
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Nov/21 22:10
Affects Version/s: 2.4.9

These are printed per chore run, per host:
{noformat}
[00]2021-11-10 22:07:32,998 INFO  [master/ip-172-31-58-47:8100.Chore.1] balancer.BaseLoadBalancer: server <N> is on rack <R>
{noformat}

One log line per server. On a large cluster, this is a lot of unnecessary logging. The 'server <N> is on rack <R>' log line arguably should be at TRACE. It is not interesting unless you are debugging the balancer.

This 'Start Generate Balance plan for cluster.' log line should be at DEBUG:
{noformat}
[00]2021-11-10 22:07:32,984 INFO  [master/ip-172-31-58-47:8100.Chore.1] balancer.BaseLoadBalancer: Start Generate Balance plan for cluster.
{noformat}

While we are in here, this should also be logged at DEBUG level:
{noformat}
[00]2021-11-10 22:14:33,005 INFO  [master/ip-172-31-58-47:8100.Chore.1] master.HMaster: Balancer is going into sleep until next period in 60000ms
{noformat}
Bug: BucketCacheWriter should log only the BucketAllocatorException message, not the full stack trace
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Nov/21 22:13
Affects Version/s: 2.4.8

We have recently improved the logging of BucketAllocatorException to not overwhelm the log if there are a lot of blocks that are too large, but when we do log it, we still log the full stack trace. That is wasteful. The messages always come from the same location in source code. The stack trace provides no value.
Bug: Rollback from ZK-less to ZK-based assignment could produce inconsistent state - doubly assigned regions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Nov/21 17:48
Affects Version/s: 1.7.1

By enabling config {_}hbase.assignment.usezk.migrating{_}, we initiate the transition of HBase 1.x cluster from default ZK-based region assignment to ZK-less region assignments. Once the migration is enabled, any subsequent region transition is going to add two additional CQs in meta: info:sn and info:state. The workflow that adds new CQs in meta should be the only workflow reading it (unless it requires coordination among multiple workflows), however that is not the case here. Reading info:sn and info:state to rebuild user region states in RegionStateStore data structure is a hidden bug because it doesn't restrict the usage for only ZK-less region assignment.

What are the effects?

After enabling ZK-less migration, if we revert it back, info:state and info:sn are not reverted. Moreover, new active master rebuilds the region states in memory and use this info. So if all regions have consistent info:sn values (i.e. consistent with info:server and info:serverstartcode), nothing goes wrong and this is likely going to happen when we revert the config with rolling restart of masters. However, after this config revert, if any region moves, only info:server and info:serverstartcode get updated but info:sn and info:state values stay the same. Because of the missing condition, subsequent active master restart would try to rebuild regions and assign regions as per info:sn, but those regions are already OPEN on info:server, hence we get doubly assigned regions.

We need two part fix for this:
 # Guard reading of info:sn and info:state with proper conditions.
 # Once active master init is complete, if ZK-based region assignment is enabled and redundant CQs are available in meta (info:sn and info:state), delete them all.
Bug: Replicas cohosted on a rack shouldn't keep triggering Balancer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Oct/21 22:23

Currently, Balancer has a shortcut check for cohosted replicas of the same region/host/rack and will keep triggering balancer if it is non-zero.

With the trending of kube and cloud solution for HBase, operators don't have full control of the topology or are not even aware of the topology. There are cases that it is not possible to satisfy or requires sacrificing other constraints such as region count balancing on RS. We want to keep there per RS/host check for availability of regions, especially for meta region. We haven't heard problem with rack so far. The cost functions will still be considered during balancing.
Bug: TestReplicationEndpoint#testInterClusterReplication fails in branch-1
Status: Open | Priority: Major | Resolution: 
Created: 10/Nov/21 19:53
Affects Version/s: 1.7.1

{noformat}
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ hbase-server ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.hbase.replication.TestReplicationEndpoint
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 20.978 s <<< FAILURE! - in org.apache.hadoop.hbase.replication.TestReplicationEndpoint
[ERROR] org.apache.hadoop.hbase.replication.TestReplicationEndpoint  Time elapsed: 3.921 s  <<< FAILURE!
java.lang.AssertionError
	at org.junit.Assert.fail(Assert.java:86)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.junit.Assert.assertTrue(Assert.java:52)
	at org.apache.hadoop.hbase.replication.TestReplicationEndpoint.tearDownAfterClass(TestReplicationEndpoint.java:88)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   TestReplicationEndpoint.tearDownAfterClass:88
[INFO] 
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0
{noformat}

It fails if testInterClusterReplication is run alone but succeeds (depending on the order of test it ran) as a test suite.
Bug: HeapMemoryManager fails memstore flushes with NPE if enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Nov/21 22:01
Affects Version/s: 2.4.8

Nonstandard configuration under test:

{quote}
<property>
  <name>hfile.block.cache.size.min.range</name>
  <value>0.1</value>
</property>
<property>
  <name>hfile.block.cache.size.max.range</name>
  <value>0.5</value>
</property>

<property>
  <name>hbase.regionserver.global.memstore.size.min.range</name>
  <value>0.2</value>
</property>
<property>
  <name>hbase.regionserver.global.memstore.size.max.range</name>
  <value>0.7</value>
</property>
{quote}

The NPE:
{noformat}
2021-11-05 21:52:13,828 ERROR [MemStoreFlusher.1] regionserver.MemStoreFlusher: Cache flusher failed for entry [flush region IntegrationTestLoadCommonCrawl,com.yeqingteng.www//gysy/28,1636143958130.4be7e8d968cc73cf729cb4713328f729.]
java.lang.NullPointerException
        at org.apache.hadoop.hbase.regionserver.HeapMemoryManager$HeapMemoryTunerChore.flushRequested(HeapMemoryManager.java:402)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.notifyFlushRequest(MemStoreFlusher.java:661)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:616)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:586)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$1000(MemStoreFlusher.java:68)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:360)
{noformat}

The reason for the NPE is obvious in MemStoreFlusher.java.
Bug: Limit min size of stripe and L0 files compaction
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 18/Oct/21 14:04
Affects Version/s: 2.0.0

In the StripeCompactPolicy#needsSingleStripeCompaction, it only checks if the file count is up to the min compaction limit.
{code:java}
protected boolean needsSingleStripeCompaction(StripeInformationProvider si) {
  int minFiles = this.config.getStripeCompactMinFiles();
  for (List<HStoreFile> stripe : si.getStripes()) {
    if (stripe.size() >= minFiles) return true;
  }
  return false;
}
{code}
When select files in stripe by ExploringCompactionPolicy#applyCompactionPolicy, there is no min size limit too.
{code:java}
// Sanity checks
if (potentialMatchFiles.size() < minFiles) {
  continue;
}
if (potentialMatchFiles.size() > maxFiles) {
  continue;
}

// Compute the total size of files that will
// have to be read if this set of files is compacted.
long size = getTotalStoreSize(potentialMatchFiles);

// Store the smallest set of files.  This stored set of files will be used
// if it looks like the algorithm is stuck.
if (mightBeStuck && size < smallestSize) {
  smallest = potentialMatchFiles;
  smallestSize = size;
}

if (size > comConf.getMaxCompactSize(mayUseOffPeak)) {
  continue;
}

++opts;
if (size >= comConf.getMinCompactSize()
    && !filesInRatio(potentialMatchFiles, currentRatio)) {
  continue;
}{code}
While we should also limit the min size of compaction for stripes, because small stripe compactions will occupy the compaction queue.
Bug: Aborting to preserve WAL as source of truth can abort in recoverable situations
Status: Open | Priority: Major | Resolution: 
Created: 29/Oct/21 19:25
Affects Version/s: 1.8.0

HBASE-26195 added an important feature to avoid data corruption by preserving the WAL as a source of truth when WAL sync fails. See that issue for background.

That issue's primary driver was a TimeoutIOException, but the solution was to catch and abort on Throwable. The idea here was that we can't anticipate all possible failures, so we should err on the side of data correctness. As pointed out by [~rushabh.shah] in his comments, this solution has the potential to lose HBase capacity quickly in "not very grave" situations. It would be good to add an escape hatch for those explicit known cases, of which I recently encountered:

I recently rolled this out to some of our test clusters, most of which are small. Afterward, doing a rolling restart of DataNodes caused the following IOException: "Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try..."

If you're familiar with HDFS pipeline recovery, this error will be familiar. Basically the restarted DataNodes caused pipeline failures, those datanodes were added to an internal exclude list that never gets cleared, and eventually there were no more nodes to choose from resulting in an error.

This error is pretty explicit, and at this point the DFSOutputStream for the WAL is dead. I think this error is a reasonable one to simply bubble up and not abort the RegionServer on, instead just failing and rolling back the writes.

What do people think about starting an allowlist of known good error messages for which we do not trigger an abort of the RS? Something like this:

{{} catch (Throwable t) {}}
 {{  // WAL sync failed. Aborting to avoid a mismatch between the memstore, WAL,}}
 {{  // and any replicated clusters.}}
 {{  if (!walSyncSuccess && !allowedException(t)) {}}
 {{  rsServices.abort("WAL sync failed, aborting to preserve WAL as source of truth", t);}}
 \{{ }}}

{{... snip ..}}

{{private boolean allowedException(Throwable t) {}}{\{  }}

{{  return t.getMessage().startsWith("Failed to replace a bad datanode");}}

{{}}}

We could of course make configurable if people like, or just add to it over time.
Bug: Update javadoc for CellUtil#createCell  with tags methods.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Oct/21 13:11
Affects Version/s: 2.4.8

We have the following methods in CellUtil class which are deprecated. We used to use this method within custom COPROC to create cells with custom tags. We deprecated them in 2.0.0 version and created a new class called RawCell which is LimitedPrivate with visibility set to COPROC. There is no reference to RawCell#createCell(Cell cell, List<Tag> tags) method in javadoc of CellUtil#createCell methods which are now deprecated. This is not user friendly. We should improve the javadoc within CellUtil#createCell(Cell, tags) method.

{noformat}
  /**
   * Note : Now only CPs can create cell with tags using the CP environment
   * @return A new cell which is having the extra tags also added to it.
   * @deprecated As of release 2.0.0, this will be removed in HBase 3.0.0.
   *
   */
  @Deprecated
  public static Cell createCell(Cell cell, List<Tag> tags) {
    return PrivateCellUtil.createCell(cell, tags);
  }
{noformat}

{noformat}
  public static Cell createCell(Cell cell, byte[] tags) 
  public static Cell createCell(Cell cell, byte[] value, byte[] tags) {
{noformat}

This improvement is only applicable to branch-2 since we have removed these methods in branch-3.
Bug: Cache in RSRpcServices.executeProcedures does not take effect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Oct/21 11:28
Affects Version/s: 2.4.8

In  {{RSRpcServices.executeProcedures}} , we use tdCache to  avoid reading from the {{TableDescritor}} every time(usually it will read from the file system), just as following line 3923 :
{code:java}
3914 public ExecuteProceduresResponse executeProcedures(RpcController controller,
3915     ExecuteProceduresRequest request) throws ServiceException {
3916   try {
3917     checkOpen();
3918     throwOnWrongStartCode(request);
3919      regionServer.getRegionServerCoprocessorHost().preExecuteProcedures();
3920      if (request.getOpenRegionCount() > 0) {
3921        // Avoid reading from the TableDescritor every time(usually it will read from the file
3922        // system)
3923        Map<TableName, TableDescriptor> tdCache = new HashMap<>();
3924        request.getOpenRegionList().forEach(req -> executeOpenRegionProcedures(req, tdCache));
3925      }
   ......
{code}

but for {{RSRpcServices.executeOpenRegionProcedures}} in above line 3924, there is no place to fill the {{tdCache}}, so the {{tdCache}} does not take effect in fact.