av9ash/GitBugs huggingface dataset:
Bug: Bootstrap should not delete the META table directory if it's not partial
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Aug/20 18:03
Affects Version/s: 2.3.0

this issues were discussed in [PR#2113|https://github.com/apache/hbase/pull/2113] as part of HBASE-24286, and it is a dependencies before we solve HBASE-24286.

The changes were introduced in [HBASE-24471 |https://github.com/apache/hbase/commit/4d5efec76718032a1e55024fd5133409e4be3cb8#diff-21659161b1393e6632730dcbea205fd8R70-R89] that partial meta was introduced and `partial` was defined as InitMetaProcedure did not succeed and INIT_META_ASSIGN_META was not completed.
{code:java}
  private static void writeFsLayout(Path rootDir, Configuration conf) throws IOException { 
   LOG.info("BOOTSTRAP: creating hbase:meta region"); 
   FileSystem fs = rootDir.getFileSystem(conf); 
   Path tableDir = CommonFSUtils.getTableDir(rootDir, TableName.META_TABLE_NAME); 
   if (fs.exists(tableDir) && !fs.delete(tableDir, true)) { 
     LOG.warn("Can not delete partial created meta table, continue..."); 
   }

{code}
however, in the cloud use case where HFiles store on S3, WALs store on HDFS, ZK data are stored within the cluster, this partial meta becomes a block when cluster recreate on existing HFiles; Here, Zk data and WALs cannot be retained (HDFS was associated with cloud instance and was terminated together) when cluster recreates on the flushed HFiles, and existing meta are always considered as partial and deleted in `INIT_META_WRITE_FS_LAYOUT` during bootstrap. As a result, the recreate cluster starts with a empty meta table, either the cluster hangs during the master initialization (branch-2) because table states of namespace table cannot be assigned, or starts as a fresh cluster without any region assigned and table opens (may need HBCK to rebuild the meta).

Potential solution suggested by Anoop
{quote}In case of HM start and the bootstrap we create the ClusterID and write to FS and then to zk and then create the META table FS layout. So in a cluster recreate, we will see clusterID is there in FS and also the META FS layout but no clusterID in zk. Ya seems we can use this as indication for cluster recreate over existing data. In HM start, this is some thing we need to check at 1st itself and track. If this mode is true, later when (if) we do INIT_META_WRITE_FS_LAYOUT , we should not delete the META dir. As part of the Bootstrap when we write that proc to MasterProcWal, we can include this mode (boolean) info also. This is a protobuf message anyways. So even if this HM got killed and restarted (at a point where the clusterId was written to zk but the Meta FS layout part was not reached) we can use the info added as part of the bootstrap wal entry and make sure NOT to delete the meta dir.
{quote}
In this JIRA, we're going to fix the `partial` definition when we found cluster ID was stored in HFiles but ZK were deleted or fresh on cluster creates.
Bug: KeyPrefix related split policies should extends SteppingSplitPolicy
Status: Resolved | Priority: Major | Resolution: Implemented
Created: 02/Jul/20 06:28

Currently the KeyPrefixRegionSplitPolicy and DelimitedKeyPrefixRegionSplitPolicy are extends IncreasingToUpperBoundRegionSplitPolicy, since we had changed the default split policy as SteppingSplitPolicy, I think it is better to change them to extends SteppingSplitPolicy too.
Bug: Correct the description of hbase.offpeak.end.hour
Status: Open | Priority: Major | Resolution: 
Created: 19/Aug/20 12:04

The inclusive in description should be exclusive, it will mislead users.

OffPeakHours.java
{code:java}
private static class OffPeakHoursImpl extends OffPeakHours {
  final int startHour;
  final int endHour;

  /**
   * @param startHour inclusive
   * @param endHour exclusive
   */
  OffPeakHoursImpl(int startHour, int endHour) {
    this.startHour = startHour;
    this.endHour = endHour;
  }

  @Override
  public boolean isOffPeakHour() {
    return isOffPeakHour(CurrentHourProvider.getCurrentHour());
  }

  @Override
  public boolean isOffPeakHour(int targetHour) {
    if (startHour <= endHour) {
      return startHour <= targetHour && targetHour < endHour;
    }
    return targetHour < endHour || startHour <= targetHour;
  }
}
{code}
Bug: incorrect value for AuthUtil.HBASE_CLIENT_KERBEROS_PRINCIPAL
Status: Open | Priority: Major | Resolution: 
Created: 26/Aug/20 12:34
Affects Version/s: 2.2.0

[HBASE-20886|https://issues.apache.org/jira/browse/HBASE-20886] introduced constants for HBASE_CLIENT_KEYTAB_FILE and HBASE_CLIENT_KERBEROS_PRINCIPAL, however the value for HBASE_CLIENT_KERBEROS_PRINCIPAL is incorrectly assigned as "hbase.client.keytab.principal". The correct value should be "hbase.client.kerberos.principal".

"hbase.client.keytab.principal" is inconsistent with the [previous code|https://github.com/apache/hbase/blob/rel/2.1.9/hbase-common/src/main/java/org/apache/hadoop/hbase/AuthUtil.java#L96], so clients migrating to 2.2.0 would need to update their configurations to match the incorrect value.
Bug: Add a config hbase.rsgroup.fallback.enable for RSGroup fallback feature
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jul/20 11:10
Affects Version/s: 3.0.0-alpha-1

In HBASE-22738 we allow tables fallback to specific rs groups, If there is no online servers in the table's rsgroup.

-But for system tables, if there is no specified fallback rsgroup or the servers in the fallback rsgroup all went down, It is necessary to allow system tables fallback to any rsgroup in order to keey available at all times.-

For Availability, refactor design of rsgroup fallback, finally only introduced one config property `hbase.rsgroup.fallback.enable`, allow all table, whether or not system tables, fallback to the default rsgroup first, then fallback to any group if no online servers in default rsgroup.
Bug: Add new procedure metrics for master UI only
Status: Open | Priority: Major | Resolution: 
Created: 29/Sep/20 03:55

We will clear procedure histogram metrics if getting master metrics from path /jmx.However, some metrics in master UI, such as open/close region procedure or TRSP, make  sense for a long time, and we should add new metrics, keep it and not clearing even when accessing /jmx.
Bug: Support CheckAndMutate in Region.batchMutate()
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 08/Sep/20 08:37

After HBASE-24602, Region.batchMutate() supports Put/Delete/Increment/Append operations, but it doesn't support CheckAndMutate. If we support CheckAndMutate Region.batchMutate(), we can perform Put/Delete/Increment/Append/CheckAndMutate operations atomically, which is very useful for some cases.
Bug: Do not assign regions to region server which has not called regionServerReport yet
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Sep/20 19:29

As part of RS start up, RS reports for duty to Master . Master acknowledges the request and adds it to the onlineServers list for further assigning any regions to the RS

Once Master acknowledges the reportForDuty and sends back the response, RS does a bunch of stuff like initializing replication sources etc before becoming online. However, sometimes there could be an issue with initializing replication sources when it is unable to connect to peer clusters because of some kerberos configuration and there would be a delay of around 20 mins in becoming online.

 

Since master considers it online, it tries to assign regions and which fails with ServerNotRunningYet exception, then the master tries to unassign which again fails with the same exception leading the region to FAILED_CLOSE state.

 

It would be good to have a check to see if the RS is ready to accept the assignment requests before adding it to online servers list which would account for any such delays as described above
Bug: Find a way to resolve WALFileLengthProvider#getLogFileSizeIfBeingWritten problem
Status: Resolved | Priority: Major | Resolution: Done
Created: 14/Jul/20 04:08

Now we use WALFileLengthProvider#getLogFileSizeIfBeingWritten to get the synced wal length and prevent replicating unacked log entries. But after offload ReplicationSource to new ReplicationServer, we need a new way to resolve this problem.
Bug: STUCK RIT by hbck2 assigns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Aug/20 03:08
Affects Version/s: 2.2.3

If a region has been assign to rs1 and then client assigns region again by "hbck2 assigns"

1、if  regionPlan is region to be assign to rs2，the region will be opened on rs1 and rs2.

master log:
{quote}WARN org.apache.hadoop.hbase.master.assignment.AssignmentManager: rit=OPEN, location=rs2, table=tableName, region=reionName reported OPEN on server=rs1 but state has otherwise
{quote}
2、if regionPlan is region to be assign to rs1， the TransitRegionStateProcedure and OpenRegionProcedure will stuck. because rs1 is not responding to master
 rslog:
{quote}Receiving OPEN for the region:{}, which we are already trying to OPEN - ignoring this new request for this region.
{quote}
Bug: Change default to false for HBASE-15519 per-user metrics
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jul/20 00:04
Affects Version/s: 2.3.0

HBASE-15519 added a nice feature to show per-user metrics. It needs a bit of work still – see base of HBASE-15519 for notes but in particular, it can spin up lots of threads, more than makes sense  – but currently it is enabled by default. Here we disable it by setting default for hbase.regionserver.user.metrics.enabled to false.
Bug: Dynamic master discovery
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jul/20 19:48
Affects Version/s: 2.4.0

[~stack]'s idea in the design doc for splittable-meta.

We can keep a live list of masters to query by fetching the list of available masters from any of the available masters configured in the seed list. User configured list of masters ("hbase.masters") would be used as a seed list.

The endpoints are refreshed every 5mins or if any of the registry RPCs hit an error (which ever happens first).
Bug: Backport HBASE-20417 to branch-1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Jul/20 14:55
Affects Version/s: 1.4.14

The wal reader shouldn't keep running with peer disabled. We need to backport HBASE-20417 to branch-1 or do something similar, if backport isn't possible due to differences in the code base.
Bug: TestLogLevel failing on hadoop-ci (branch-1)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/20 18:58

TestLogLevel is failing the branch-1 builds on hadoop-ci.

The test needs some improvement. The code seems to be doing the right thing but the error condition the test is expecting varies by JVM or JVM version:
{noformat}
Expected to find 'Unrecognized SSL message' but got unexpected exception:
javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
{noformat}
Bug: Index rebuilds should record scan timestamp to job counter
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 28/Jul/20 00:12

The index tool output tables (PHOENIX_INDEX_TOOL and PHOENIX_INDEX_TOOL_RESULT) are both keyed on the effective scan time of the index rebuild. This makes the table quite difficult to query if you know the index you're interested in but not the exact millisecond it was last rebuilt. 

Short term: record this timestamp in the Phoenix index rebuild job counters (this JIRA)
Longer term: we might want to consider reversing the PK to both cut down on hotspotting and make the table easier to query.
Bug: Offload the replication source/sink job to independent Replication Server
Status: Open | Priority: Major | Resolution: 
Created: 01/Jul/20 04:36

The basic idea is add a role "ReplicationServer" to take the replication source/sink job. HMaster is responsible for scheduling the replication job to different ReplicationServer.

[link Design doc|https://docs.google.com/document/d/16kRPVGctFSf__nC3yaVZmAm3GTxIbHefekKC_rMmTw8/edit?usp=sharing]

Suggestions are welcomed. Thanks.
Bug: SCP reduce unnecessary get requests
Status: Open | Priority: Major | Resolution: 
Created: 21/Aug/20 02:52
Affects Version/s: 2.2.3

SCP should reduce unnecessary Get request

[https://github.com/apache/hbase/blob/65d28da7c22382e040363c607840d5ab6e6b45da/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java#L520]

during startup， the tableNam2State of tableStateManager is not loading tableState data form metaTable yet.  if procThread num is 50 and hbase has 10K tables, in the worst case, the master needs to query meta table 500K times(50*10K. and the regions that all SCPs simultaneously check tableState belong to the same table )

 

i think master can reduce Get request, and AM#loadMeta can load regions and all tables through asynchronous threads.

[https://github.com/apache/hbase/blob/65d28da7c22382e040363c607840d5ab6e6b45da/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java#L1532]
Bug: Rename configs that support atypical DNS set ups to put them in hbase.unsafe
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jul/20 17:08

HBASE-18226 added a config for disabling reverse DNS checks {{hbase.regionserver.hostname.disable.master.reversedns}} and the release note calls out that the config is dangerous:
{quote}he following config is added by this JIRA:

hbase.regionserver.hostname.disable.master.reversedns

This config is for experts: don't set its value unless you really know what you are doing.
 When set to true, regionserver will use the current node hostname for the servername and HMaster will skip reverse DNS lookup and use the hostname sent by regionserver instead. Note that this config and hbase.regionserver.hostname are mutually exclusive. See https://issues.apache.org/jira/browse/HBASE-18226 for more details.

Caution: please make sure rolling upgrade succeeds before turning on this feature.
{quote}
We should make clear the risks of using this config by making sure the name starts with {{hbase.unsafe}}.

Rename {{hbase.regionserver.hostname.disable.master.reversedns}} to {{hbase.unsafe.regionserver.hostname.disable.master.reversedns}} but make sure the former is kept with a "deprecated config name" warning.
Bug: Provide an option for do-release-docker to ignore svn ci
Status: Patch Available | Priority: Major | Resolution: 
Created: 15/Sep/20 20:22

do-release-docker, when run with -f, tries to checkin CHANGES.md, RELEASENOTES.md and all release tarballs to svn, however if the user doesn't have svn checkin access, the script fails at svn ci step and rest of the steps in *publish-release* are left untouched. Although majority users of the script have access to checkin, we should definitely provide an option to make svn checkin step no-op (similar to dry_run mode) for users without access.
Bug: Make retain assignment configurable during SCP
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Aug/20 18:00
Affects Version/s: 2.1.9

HBASE-23035 change the "retain" assignment to round-robin assignment during SCP which will make the failover faster and surely improve the availability, but this will impact the scan performance in non-cloud scenario.

This jira will make this assignment plan configurable.
Bug: Unbounded org.glassfish:javax.el in hbase-server:org.glassfish.web.javax.servlet.jsp
Status: Open | Priority: Major | Resolution: 
Created: 23/Sep/20 17:35
Affects Version/s: 2.2.6

When I try to build in a fresh environment I see that pulling in hbase-server dependencies will fail. This is because the [javax.servlet.jsp-2.3.2 pom|https://repo1.maven.org/maven2/org/glassfish/web/javax.servlet.jsp/2.3.2/javax.servlet.jsp-2.3.2.pom] specifies a dependency with no upper bound for org.glassfish:javax.el which includes SNAPSHOT dependencies. One solution is to exclude that dependency and explicitly include the }}org.glassfish:javax.el{{ dep. What does the community think of that solution or should we be upgrading some other dependency?

{{Failed to execute goal on project hive-hbase-handler: Could not resolve dependencies for project org.apache.hive:hive-hbase-handler:jar:3.1.2-SNAPSHOT: Failed to collect dependencies at org.apache.hbase:hbase-server:jar:tests:2.2.6-SNAPSHOT -> org.glassfish.web:javax.servlet.jsp:jar:2.3.2 -> org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT: Failed to read artifact descriptor for org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT: Could not transfer artifact org.glassfish:javax.el:pom:3.0.1-b06-SNAPSHOT from/to}}
Bug: [hbase-operator-tools] hbck2 filesystem fails against hbase-2.3.x
Status: Open | Priority: Major | Resolution: 
Created: 29/Jul/20 15:33

{code:java}
15:23:15.323 [main] INFO  org.apache.hadoop.hbase.client.ConnectionImplementation - Closing master protocol: MasterService
Exception in thread "main" java.lang.NoSuchMethodError: 'org.apache.hadoop.fs.Path org.apache.hadoop.hbase.util.FSUtils.getRootDir(org.apache.hadoop.conf.Configuration)'
	at org.apache.hbase.FileSystemFsck.<init>(FileSystemFsck.java:49)
	at org.apache.hbase.HBCK2.doCommandLine(HBCK2.java:720)
	at org.apache.hbase.HBCK2.run(HBCK2.java:631)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
	at org.apache.hbase.HBCK2.main(HBCK2.java:865) {code}
 

If build hbase-operator-tools against hbase-2.3.x its fine again. Could try putting pack the method in hbase-2.3.x or its probably worth a new release of hbase-operator-tools (last was 2018 for 1.0.0) but then it wouldn't work against 2.1 (but 2.1 is EOL'd so ok)? Probably wouldn't work against 2.2 then so maybe put back the method for 2.3?
Bug: Meta region assignment is blocked when all RS in meta table group are restarted.
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 03/Jul/20 13:33
Affects Version/s: 2.2.3

This issue happened in a test cluster. The issue does not reproduce easily. But we can reproduce it with debug points in code.
Steps to reproduce:

# Install a HBase cluster with three RS(rs1,rs2 and rs3) and one Master
# Create two rsgroups r1 and r2 and move rs1 to r1 and rs2 to r2
{code}
add_rsgroup 'r1';add_rsgroup 'r2';move_servers_rsgroup 'r1',['rs1Host:16020'];move_servers_rsgroup 'r2',['rs2Host:16020']
{code}
# Create a table t1
{code}create 't1','f1','f2';put't1','r1','f1:c1','v1'{code}
# Start debugging master, put debug point in while loop of {code}org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl.ServerEventsListenerThread#run{code} method.
# Stop rs3
# When debug flow comes, wait around 30 seconds to let the meta be offline and then let the debug flow execute. By now meta will be offline as rs3 is stopped. HMaster UI will hang as meta is offline.
# Now start rs3, after start meta should be online and Master UI should open.
# No, still master UI hangs, then you have reproduced the issue.
Bug: [LOG][RSGroup]Error message is confusing while adding a offline RS to rsgroup
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jul/20 11:36
Affects Version/s: 2.2.3

Move a offine RS from default RSGroup to another RSGroup. The error messge says 'Source RSGroup for server xxx does not exist.'. It's confusing.

!image-2020-07-22-19-34-42-628.png!
Bug: RegexStringComparator in ParseFilter should support case-insensitive regexes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Sep/20 21:41
Affects Version/s: 1.6.0

I would like to be able to set CASE_INSENSITIVE to RegexStringComparator (using thrift.)  There doesn't seem (currently) to be a way to do that.
Bug: TransitionRegionStateProcedure of non-meta regions should yield when meta is unavailable
Status: Open | Priority: Major | Resolution: 
Created: 02/Jul/20 17:40
Affects Version/s: 2.3.0

One observation from HBASE-24526 is that while meta is unavailable, other region movement procedures are getting stuck on meta RPCs. Let's make it so that non-meta transitions check the state of meta before attempting any RPCs. If meta is known unavailable, release the thread back to the scheduler.
Bug: Add option to avoid aborting RS process upon uncaught exceptions happen on replication source
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Aug/20 11:31
Affects Version/s: 2.4.0

Currently, we abort entire RS process if any uncaught exceptions happens on ReplicationSource initialization. This may be too extreme on certain deployments, where custom replication endpoint implementations may choose to do so when remote peers are unavailable, but source cluster shouldn't be brought down entirely. Similarly, source reader and shipper threads would cause RS to abort on any runtime exception occurrence while running. 

This patch adds configuration option (false by default, to keep the original behaviour), to avoid aborting entire RS processes under these conditions. Instead, if ReplicationSource initialization fails with a RuntimeException, it keeps retrying the source startup. In the case of readers/shippers runtime errors, it refreshes the replication source, terminating current source and its readers/shippers and creating new ones.
Bug: Some changing of split region by overall region size rather than only one store size
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jul/20 02:38
Affects Version/s: 2.4.0

As a distributed cluster, HBase distribute loads in unit of region, so if region grows too big,
 it will bring some negative effects, such as:
 1. Harder to homogenize disk usage(consider locality)
 2. Might cost more time on region opening
 3. After split, the daughter region might lead to more io cost on compaction in a short time(if write evenly)

I tried to introduce a new SteppingAllStoresSizeSplitPolicy in HBASE-24530, but after discussed in comments and related [thread|https://lists.apache.org/thread.html/r08a8103e2532eb667a0fcb4efa8a4117b3f82e6251bc4bd0bc157c26%40%3Cdev.hbase.apache.org%3E], finally we decide to change the existing split policy with a new option that if it should count all store files, and for master it would be true, else false.
Bug: stuck InitMetaProcedure in finishActiveMasterInitialization
Status: Open | Priority: Major | Resolution: 
Created: 21/Aug/20 01:23
Affects Version/s: 2.2.3

if procWAL has InitMetaProcedure, and InitMetaProcedure state is finished, and meta znode does not exist， finishActiveMasterInitialization will stuck

because during startup，If InitMetaProcedure exists, InitMetaProcedure recreates a new CountDownLatch.

master jstack
 !image-2020-08-21-09-12-33-894.png!

master log 
 !masterLog.gif!
Bug: Backport HBASE-24350 and HBASE-24779 to branch-2.2 & branch-2.3
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Sep/20 09:20
Affects Version/s: 2.2.5

Backport HBASE-24350 and HBASE-24779 to branch-2.2 & branch-2.3. 

HBASE-24779 was written on top of HBASE-24350. Whilst it is possible to manually resolve the conflict to exclude the piece of code introduced by HBASE-24350, ideally we should also include HBASE-24350 in this package of fixes, as it deals with some inconsistencies about table level metrics for replication.