av9ash/GitBugs huggingface dataset:
Bug: Assignment manager should never give up assigning meta
Status: Open | Priority: Critical | Resolution: 
Created: 30/Apr/20 18:21
Affects Version/s: 2.3.0

Not yet sure how we got here, but,

{noformat}
2020-04-29 22:39:16,140 INFO org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure: pid=308, state=RUNNABLE:SERVER_CRASH_ASSIGN_META, locked=true; ServerCrashProcedure server= host-a.example.com,16020,1588033841562, splitWal=true, meta=true found a region state=OFFLINE, location=null, table=hbase:meta, region=1588230740 which is no longer on us host-a.example.com,16020,1588033841562, give up assigning...
{noformat}

Assignment manager gives up on this procedure and nothing can progress. Manual intervention is necessary.

From this [conditional block|https://github.com/apache/hbase/blob/1415a82d41a1e125440014a4b23364371b30d065/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java#L475], it seems the {{regionNode}} location is {{null}}.

{noformat}
        // This is possible, as when a server is dead, TRSP will fail to schedule a RemoteProcedure
        // to us and then try to assign the region to a new RS. And before it has updated the region
        // location to the new RS, we may have already called the am.getRegionsOnServer so we will
        // consider the region is still on us. And then before we arrive here, the TRSP could have
        // updated the region location, or even finished itself, so the region is no longer on us
        // any more, we should not try to assign it again. Please see HBASE-23594 for more details.
        if (!serverName.equals(regionNode.getRegionLocation())) {
          LOG.info("{} found a region {} which is no longer on us {}, give up assigning...", this,
            regionNode, serverName);
          continue;
        }
{noformat}
Bug: Correct snapshot handling
Status: Resolved | Priority: Critical | Resolution: Not A Problem
Created: 01/Apr/20 19:39


Reopening this umbrella to address correct snapshot handling. Particularly, the following scenario must be verified:
# load data to a table
# take snapshot
# major compact table
# run mob file cleaner chore
# load data to table
# restore table from snapshot into another table
# verify data integrity
# restore table from snapshot into original table
# verify data integrity
Bug: Create table is slow in large cluster when AccessController is enabled.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Apr/20 22:57
Affects Version/s: 1.3.6

*Problem:*

In HBase 1.3.x  large, performance test, cluster (100 RS, 60k tables, 600k regions) a simple table creation takes around 150 seconds. The time taken varies but still takes lot of time.

*Analysis:*

1. When HBase creates a table , it calls AssignmentManager#assign(final ServerName destination, final List<HRegionInfo> regions)
 In AssignmentManager#assign，it calls asyncSetOfflineInZooKeeper(state, cb, destination), and waits in below code loop for 2 minutes. 
{code:java}
 if (useZKForAssignment) {
          // Wait until all unassigned nodes have been put up and watchers set.
          int total = states.size();
          for (int oldCounter = 0; !server.isStopped();) {
            int count = counter.get();
            if (oldCounter != count) {
              LOG.debug(destination.toString() + " unassigned znodes=" + count +
                " of total=" + total + "; oldCounter=" + oldCounter);
              oldCounter = count;
            }
            if (count >= total) break;
            Thread.sleep(5);
          }
        }
{code}
2. asyncSetOfflineInZooKeeper creates a znode under /hbase/region-in-transition/ and calls exist to ensure that znode is created. This is simple operation should not take much time. Then where the time it taken!!!

3. ZooKeeper client API process watcher notification and async API response through a queue one by one.
 If there is a delay in any watcher/response processing by the client, in this case HBase, all other response processing is delayed. Then it appears as if API call has taken more time.
 Same thing happen in this issue.

Watcher processing for znode creation under /hbase/acl took most of the time and delayed /hbase/region-in-transition/region znode creation processing. This is why wait in loop was too long. 

4. Watcher processing for znode creation under hbase/acl/ calls ZKPermissionWatcher#nodeChildrenChanged, which internally calls ZKUtil.getChildDataAndWatchForNewChildren
 *which calls ZooKeeper's getData API, in this use case, 60k times which takes most of the time.*

*Solutions:*
 Move getChildDataAndWatchForNewChildren call into the async code block in ZKPermissionWatcher#nodeChildrenChanged.
Bug: Skip compaction request/check for replica regions at the early stage.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jun/20 21:13
Affects Version/s: 2.3.0

I found that in certain cases replica regions can trigger compaction, one example as follows, need to check all places to avoid compaction for replica regions.
{code:java}
@Override
public void postOpenDeployTasks(final PostOpenDeployContext context) throws IOException {
  HRegion r = context.getRegion();
  long openProcId = context.getOpenProcId();
  long masterSystemTime = context.getMasterSystemTime();
  rpcServices.checkOpen();
  LOG.info("Post open deploy tasks for {}, openProcId={}, masterSystemTime={}",
    r.getRegionInfo().getRegionNameAsString(), openProcId, masterSystemTime);
  // Do checks to see if we need to compact (references or too many files)
  // TODO: SHX, do not do this for replica regions? Otherwise, it is going to lost data locality for primary regions.
  for (HStore s : r.stores.values()) {
    if (s.hasReferences() || s.needsCompaction()) {
      this.compactSplitThread.requestSystemCompaction(r, s, "Opening Region");
    }
  }
 {code}
Bug: Hadoop2 and Hadoop3 profiles being activated simultaneously causing test failures
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Apr/20 00:05

[~ndimiduk] pointed out that, after this change went in, TestSecureRESTServer started failing with Hadoop3 on branch-2.3

https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/56/

Of course, I ran this with 1.8.0_241 and Maven 3.6.33 and it passed :) {{mvn clean package -Dtest=TestSecureRESTServer -Dhadoop.profile=3.0 -DfailIfNoTests=false}}

FYI [~stoty] in case you can repro a failure and want to dig in. Feel free to re-assign.

It looks like we didn't have a nightly run of branch-2.2 due to docker container build issues. Will be interesting to see if it fails there. It did not fail the master nightly.
Bug: Revisit onlineRegions and onlineRegionsLock in HRegionServer
Status: Patch Available | Priority: Major | Resolution: 
Created: 02/Jun/20 08:47

The onlineRegionsLock is only used in closeMetaTableRegions and closeUserRegions, and only writeLock is used so I wonder whether we can change it to a Lock instead of ReadWriteLock.

And also, in the several getRegions method, we have synchronized (this.onlineRegions), which is not necessary, since the onlineRegions is already a ConcurrentMap.
Bug: CatalogJanitor resubmits GCMultipleMergedRegionsProcedure for the same region
Status: Resolved | Priority: Major | Resolution: Implemented
Created: 24/Apr/20 04:40
Affects Version/s: 2.2.4

If a lot of regions were merged (due to change of region sizes, for example), there can be a long backlog of procedures to clean up the merged regions. If going through this backlog is slower than the CatalogJanitor's scan interval, it will end resubmitting GCMultipleMergedRegionsProcedure for the same regions over and over again.
Bug: GCRegionProcedure doesn't assign region from RegionServer leading to orphans
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 24/Apr/20 16:12
Affects Version/s: 2.2.4

We've found ourselves in a situation where parents of merged or split regions needed to be opened again on a regionserver due to having to recover from cluster meltdown (HBCK2's fixMeta kicks off GCMultipleMergedRegionsProcedure which requiters all regions to be merged to be open). Then, when a GCProcedure is kicked of to clean a parent region up by GCMultipleMergedRegionsProcedure, it ends up deleting it from hbase:meta, but doesn't unassign it from RegionServer leading for it to show up in "Orphan Regions on RegionServer" in hbck tab of HBase Master. Also, the hbase client doesn't detect that the region is closed either because it's still technically open on a regionserver (it doesn't reread hbase:meta all the time). The only way to recover from this is to restart regionserver which isn't idea as it can lead to other issues in clusters with region inconsistencies.
Bug: Change fork count from 0.5C to 0.25C
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Apr/20 04:04

The parent issue, not surprisingly, shows that 0.25C fork count is more likely to pass than the more aggressive 0.5C. Let me down branch-2.3/master branches to 0.25C (I've already done it on branch-2).
Bug: RSGroupInfo add configuration map to store something extra
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/May/20 03:28
Affects Version/s: 3.0.0-alpha-1

Maybe we should add a _Map<String, String> configuration_ into RSGroupInfo to store extra infomation.

For example, we can store the minimum number of machines the group needs, in order to move machine into this group automatically.
Bug: Support loading cluster level CPs from Hadoop file system
Status: Open | Priority: Major | Resolution: 
Created: 23/May/20 12:42

Right now we allow configuring CPs, which needs to be loaded from hadoop FS, at table level. (Via the Java API or shell)
> alter 't1', METHOD => 'table_att', 'coprocessor'=>'hdfs:///foo.jar|com.foo.FooRegionObserver|1001|arg1=1,arg2=2'
But for the cluster level CPs at Master/RS/WAL level, only way is to config at hbase-site.xml. But here we dont allow to specify any jar path. This jira suggest to add such a feature
Note: We already support config the priority of CP at xml level (FQCN|<priority>).  Same way how shell command works we can take the jar pathalso. <jar path>|<class name>|<priority>
If no '|' separator at all, consider that as FQCN in the classpath. If one '|' that will be FQCN and priority (Same as of today). If 2 '|' separators we consider the 1st part as path to the external jar.

This will help in cloud scenario specially with auto scaling. Or else customer should be executing some special scripts to make the CP jar available within the HBase classpath.
Bug: Backport fix for "Netty SASL implementation does not wait for challenge response" to branch-2.x
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Apr/20 15:08

From the parent:

* Need backports of the Netty impl to the other 2.x branches which contain HBASE-23347
* Need to make the same fix to the NIO impl to branches which contain HBASE-23347
Bug: Redo all of our github notification integrations on new ASF infra feature
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Apr/20 19:51

The new [ASF Infra feature for customizing how project gets notifications from github|https://cwiki.apache.org/confluence/display/INFRA/.asf.yaml+features+for+git+repositories#id-.asf.yamlfeaturesforgitrepositories-Notificationsettingsforrepositories] appears to have silently thrown away all the integration we already had set up.

I don't know that full set of things we need. We presumably need to do this for all of our repos.

* make sure all notifications on PRs is going to issues@
* make sure we get links on JIRA for related PRs
* make sure we do not get updates on JIRA for every PR comment
Bug: Move MetaTableAccessor out of hbase-client
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jun/20 12:47

On master branch we have AsyncMetaTableAccessor which is used at client side and MetaTableAccessor has lots of internal methods for implementing assignment, which is not part of our client code.

So let's move it to hbase-server, and in the future, maybe in hbase-balancer?
Bug: hbase spark connector doesn't support writing to table not in default namespace
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Apr/20 11:07
Affects Version/s: connector-1.0.0

Defining the following table catalog:

{code:java}
val catalog = """{
  |"table": {"namespace": "ns1", "name": "test1"},
  |"rowkey": "id",
  |"columns": {
    |"id": {"cf": "rowkey", "col": "id", "type": "string"}, 
    |"x": {"cf": "d", "col": "xxx", "type": "int"}
  |}
|}""".stripMargin
{code}

Try to write some test data to {{ns1:test1}} table using spark:

{code:java}
val df = Seq(("abc", 1), ("def", 2)).toDF("id", "x")
df.write.options(Map(HBaseTableCatalog.tableCatalog -> catalog, "hbase.spark.use.hbasecontext" -> "false", HBaseTableCatalog.newTable-> "5")).format("org.apache.hadoop.hbase.spark").save()
{code}

After executing the code above, I found out that the test data was written to {{default:test1}}, rather than {{ns1:test1}}.
Bug: Calculate FIXED_OVERHEAD automatically
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jun/20 01:57

Now the FIXED_OVERHEAD in some classes are maintained manually, an we have a method to TestHeapSizes to confirm that the value is correct.

But it is really hard for developers to count the fields in a complicated class like HRegion. Since we have the ability to calcuate the accurate size in UT, I think we it is also possible to calcuate it when loading the class, which is a one time operation so should not effect the performance too much.
Bug: HMaster won't become healthy after after cloning or creating a new cluster pointing at the same file system
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 29/Apr/20 17:20
Affects Version/s: 2.2.3

h1. How to reproduce:
 # user starts an HBase cluster on top of a file system
 # user performs some operations and shuts down the cluster, all the data are still persisted in the file system
 # user creates a new HBase cluster using a different set of servers on top of the same file system with the same root directory
 # HMaster cannot initialize

h1. Root cause:

During HMaster initialization phase, the following happens:
 # HMaster waits for namespace table online
 # AssignmentManager gets all namespace table regions info
 # region servers of namespace table are already dead, online check fails
 # HMaster waits for namespace regions online, keep retrying for 1000 times which means forever

Code waiting for namespace table to be online: https://github.com/apache/hbase/blob/rel/2.2.3/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java#L1102
h1. Stack trace (running on S3):

2020-04-23 08:15:57,185 WARN [master/ip-10-12-13-14:16000:becomeActiveMaster] master.HMaster: hbase:namespace,,1587628169070.d34b65b91a52644ed3e77c5fbb065c2b. is NOT online; state=\{d34b65b91a52644ed3e77c5fbb065c2b state=OPEN, ts=1587629742129, server=ip-10-12-13-14.ec2.internal,16020,1587628031614}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.

where ip-10-12-13-14.ec2.internal is the old region server hosting the region of hbase:namespace.
h1. Discussion for the fix

We see there is a fix for this at branch-3: https://issues.apache.org/jira/browse/HBASE-21154. Before we provide a patch, we would like to know from the community if we should backport this change to branch-2, or if we should just perform a fix with minimum code change.
Bug: Release scripts in docker mode should be able to use a named volume for maven repo
Status: In Progress | Priority: Major | Resolution: 
Created: 28/May/20 23:56

Allow optional volume name to use to mount the maven repo so we get reuse of third party artifacts between runs.


Forcefully remove artifacts we build from it (assuming we can't make a read-only one that's shared for third party dependencies and an isolated one that's for what we're building)
Bug: MOB compactor implementations should use format specifiers when calling String.format
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Apr/20 03:21
Affects Version/s: 3.0.0-alpha-1

HBASE-23723 includes a few IOException message constructions that rely on String.format but use slf4j parameters instead of format specifiers.

Discovered by nightly failing error-prone.
Bug: Failed SASL authentication does not result in an exception on client side
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jun/20 09:30
Affects Version/s: 2.2.5

When HBaseSaslRpcClient.saslConnect tries to authenticate it only reads the input stream if the process is not complete yet. However if the authentication failed and the process is completed the exception sent back in the stream never gets read.

We should always try to read the input stream even if the process is complete to make sure it was empty.
Bug: Document general framework to execute remote procedure on RegionServer
Status: Closed | Priority: Major | Resolution: Duplicate
Created: 19/Jun/20 17:12

Document -- probably best in javadoc -- the  system added by HBASE-19216 'Implement a general framework to execute remote procedure on RS ' for having procedures run tasks on remote regionservers. The system evolved a good while ago in HBASE-19216 to manage replication peers. It was then adopted by the new procedure-based distributed WAL splitting feature with some worry that the latter didn't use the framework properly (so far, all looks good).

This issue is about vetting the distributed WAL splitters usage and doc'ing the framework as I go. This framework is useful. Others will want to exploit it.
Bug: Failed start recovering crash in standalone mode if procedure-based distributed WAL split & hbase.wal.split.to.hfile=true
Status: Closed | Priority: Major | Resolution: Not A Problem
Created: 18/Jun/20 05:51

(This description got redone after I figured out what was going on. Previously it was just a litany of me banging around trying to learn procedure-based WAL splitting and hbase.wal.split.to.hfile; no one needs to read that; hence the refactor).

HBASE-24574 procedure-based distributed WAL splitting is enabled and split-to-hflie too. A force crash requires recovery with ServerCrashProcedure splitting old WALs on restart. The recovery fails because we get stuck. The Master can't assign meta because it is being recovered. The recovery can't make progress because it is asking for a table descriptor for meta -- needed by the hbase.wal.split.to.hfile feature -- and the master is not yet initialized.  After the default timeout, Master shuts down because it can't initialize.

{code}
 2020-06-18 19:53:54,175 ERROR [main] master.HMasterCommandLine: Master exiting
 java.lang.RuntimeException: Master not initialized after 200000ms
   at org.apache.hadoop.hbase.util.JVMClusterUtil.waitForEvent(JVMClusterUtil.java:232)
   at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:200)
   at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:430)
   at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:232)
   at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)
   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
   at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
   at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3059)
{code}

The abort of Master interrupts other ongoing actions so later in the log we'll see the WAL split show as interrupted
{code}
 2020-06-17 21:20:37,472 ERROR [RS_LOG_REPLAY_OPS-regionserver/localhost:16020-0] handler.RSProcedureHandler: Error when call RSProcedureCallable:
 java.io.IOException: Failed WAL split, status=RESIGNED, wal=file:/Users/stack/checkouts/hbase.apache.git/tmp/hbase/WALs/localhost,16020,1592440848604-splitting/localhost%2C16020%2C1592440848604.meta.1592440852959.meta
   at org.apache.hadoop.hbase.regionserver.SplitWALCallable.splitWal(SplitWALCallable.java:106)
   at org.apache.hadoop.hbase.regionserver.SplitWALCallable.call(SplitWALCallable.java:86)
   at org.apache.hadoop.hbase.regionserver.SplitWALCallable.call(SplitWALCallable.java:49)
   at org.apache.hadoop.hbase.regionserver.handler.RSProcedureHandler.process(RSProcedureHandler.java:49)
   at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)
{code}

This issue becomes how to make hbase.wal.split.to.hfile work in standalone mode.
Bug: Generate version.h include file during compilation
Status: Closed | Priority: Major | Resolution: Fixed
Created: 27/May/20 23:32
Affects Version/s: 3.0.0-alpha-1

Curious case of {{version.h}}

hbase-native-client relies on this include file to compile. However that is not generated by the HBase repo. I see no trace of it in the git log, wonder how it even worked until now.

I think it will be useful to generate it during compilation and package it with the release tar ball. This helps [~phrocker] in HBASE-24400.
Bug: Automatically download CMake Dependencies
Status: Closed | Priority: Major | Resolution: Fixed
Created: 19/May/20 19:04

To improve the ability to build we should download and link a local version of dependencies ( in the build folder )

 

This will help with skew of versions and the ability to build the project.

 

This will help the build process in docker and allow people to develop locally. this will also pave the way for future work to support
Bug: Lets get a clean test run for hbase-native-client
Status: Closed | Priority: Major | Resolution: Not A Problem
Created: 10/Jun/20 23:24
Affects Version/s: native-client-0.1.0

Now that the build system is cleaned up somewhat (via HBASE-24400), lets aim to get a clean test run on hbase-native-client. Upon quick inspection of code, there are tons of issues with tests that needs to be fixed. Lets make them as sub tasks of this jira so that its easy to track the whole effort.
Bug: [hbase-operator-tools] Tool to Report on row sizes and column counts
Status: Closed | Priority: Major | Resolution: Fixed
Created: 18/May/20 18:29

Tool to run against a table that will report on row sizes and column counts.

{code}
$   HBASE_CLASSPATH_PREFIX=./target/hbase-table-reporter-1.1.0-SNAPSHOT.jar ../../hbase.apache.git/bin/hbase org.apache.hbase.reporter.TableReporter
Usage: reporter <OPTIONS> TABLENAME
OPTIONS:
 -h,--help        Output this help message
 -l,--limit       Scan row limit (per thread): default none
 -f,--fraction    Fraction of table Regions to read; between 0 and 1: default 1.0 (all)
 -r,--region      Scan this Region only; pass encoded name; 'fraction' is ignored.
 -t,--threads     Concurrent thread count (thread per Region); default 1
 -Dproperty=value Properties such as the zookeeper to connect to; e.g:
                  -Dhbase.zookeeper.quorum=ZK0.remote.cluster.example.org
{code}

Primitive. Uses sketches to generate percentiles and histograms. Can be expanded upon.
Bug: Implement JvmMetrics in HBase instead of using the one in hadoop
Status: Closed | Priority: Major | Resolution: Later
Created: 14/May/20 08:23

The JvmMetrics from hadoop is hard coded to use log4j.

Although we do not make use of the ability, it still prevents us to ban the log4j dependencies completely.

So let's implement the JvmMetrics by our own, based on log4j2.