av9ash/GitBugs huggingface dataset:
Bug: 350+ lossy-count threads running
Status: Open | Priority: Major | Resolution: 
Created: 10/Jan/20 00:13
Affects Version/s: 2.3.0

Looking at a server under load (branch-2), I see 350 instances of lossy-count threads running. They look like this:

{code}
8611 "lossy-count-0" #11672 daemon prio=5 os_prio=0 cpu=0.09ms elapsed=281.33s tid=0x00007f1baee76800 nid=0x2411 waiting on condition  [0x00007f1b78793000]
8612    java.lang.Thread.State: WAITING (parking)
8613         at jdk.internal.misc.Unsafe.park(java.base@11.0.4/Native Method)
8614         - parking to wait for  <0x00000000910a91e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
8615         at java.util.concurrent.locks.LockSupport.park(java.base@11.0.4/LockSupport.java:194)
8616         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.4/AbstractQueuedSynchronizer.java:2081)
8617         at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.4/LinkedBlockingQueue.java:433)
8618         at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.4/ThreadPoolExecutor.java:1054)
8619         at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.4/ThreadPoolExecutor.java:1114)
8620         at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.4/ThreadPoolExecutor.java:628)
8621         at java.lang.Thread.run(java.base@11.0.4/Thread.java:834)
{code}

Why we need 350 threads?
Bug: Implement an adaptive hedging policy 
Status: Open | Priority: Major | Resolution: 
Created: 06/Jan/20 02:31
Affects Version/s: 3.0.0-alpha-1

HBASE-23305 implements a new rpc channel that can hedge requests to multiple target end points. As [~andrew.purtell@gmail.com] noted in the code review, it makes more sense to have a pluggable hedging policy and make it adaptive based on certain rpc performance/success metrics. 

By default, the implementation hedges the requests across 'n' target addresses. However in most cases, for smaller rpcs, the first 1 or 2 end points should return the results pretty quickly  and there is no point in hedging with a bigger fan out as that only increases the load on the target servers. The idea is to increase the fan out only when needed. 

A naive way is to double the fan out at each attempt (starting from 1) until we reach the configured max fan out. For example if the max fan out is 10, instead of bursting to all 10 in each attempt, we fan out with values 1, 2, 4, 8, 10, 10, 10...

Any suggestions welcome.
Bug: How to improve replication performance.
Status: Open | Priority: Major | Resolution: 
Created: 09/Jan/20 06:46

Inside the Replication function, when the sink node is selected as the source node, a node is randomly selected as the sink node. This presents a problem: the selected node has a relatively large replication traffic of its own, which increases the stress of node processing.

To solve this problem, as a sink node, you can set up the flow control, which of course is for Replication. At the node of the sink end of the source end, the node with high flow can be filtered out and the node with low pressure can be selected by virtue of the flow control.
Bug: Use ReplicationPeerConfig.needToReplicate in HBaseFsck
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jan/20 13:43
Affects Version/s: hbase-operator-tools-1.1.0

ReplicationUtils.contains method was removed in HBASE-23345. HBCK2 used this method so it needs to be changed to ReplicationPeerConfig.needToReplicate.
Bug: Reintroduce ReplicationUtils.contains as deprecated
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jan/20 14:32
Affects Version/s: 2.1.9

ReplicationUtils.contains was removed in HBASE-23345. This is an IA.Private class but HBCK2 relied on this method.

The plan is to add back the method as deprecated and also move HBCK2 off of this to the new ReplicationPeerConfig#needToReplicate(TableName) instead (HBASE-23641).
Bug: Remove deprecated isLegalFamilyName(byte[]) from HColumnDescriptor
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Feb/20 19:24
Affects Version/s: 3.0.0-alpha-1

{{isLegalFamilyName(byte[])}} in HColumnDescriptor was deprecated back in 2.0.0 and should be removed for 3.0.0.
Bug: [MERGETOOL] HBASE Support Merge region by pattern
Status: Patch Available | Priority: Minor | Resolution: 
Created: 07/Jan/20 13:40

Design Objective:
 # Merge empty region
 # Neat region
 # merge expired region

Usage: bin/hbase onlinemerge [--tableName=] [--startRegion=] [--stopRegion=] [--maxRegionSize=] [--maxRegionCreateTime=] [--numMaxMergePlans=] [--targetRegionCount=] [--printExecutionPlan=] [--configMergePauseTime=]

Options:
 --h or --h print help
 --tableName table name must be not null
 --startRegion start region
 --stopRegion stop region
 --maxRegionSize max region size Unit GB
 --maxRegionCreateTime max Region Create Time yyyy/MM/dd HH:mm:ss
 --numMaxMergePlans num MaxMerge Plans
 --targetRegionCount target Region Count
 --configMergePauseTime config Merge Pause Time In milliseconds
 --printExecutionPlan Value default is true print execution plans false is execution merge

Examples:
 bin/hbase onlinemerge --tableName=test:test1 --startRegion=test:test1,,1576835912332.01d0d6c2b41e204104524d9aec6074fb. --stopRegion=test:test1,bbbbbbbb,1573044786980.0c9b5bd93f3b19eb9bd1a1011ddff66f. --maxRegionSize=0 --maxRegionCreateTime=yyyy/MM/dd HH:mm:ss --numMaxMergePlans=2 --targetRegionCount=4 --printExecutionPlan=false
Bug: TestUsersOperationsWithSecureHadoop fails when an existing ticket is present
Status: Open | Priority: Minor | Resolution: 
Created: 11/Mar/20 21:54
Affects Version/s: 2.3.0

{{TestUsersOperationsWithSecureHadoop}} makes assumptions about the state of the machine environment. When i have an existing ticket issued and active (shown via {{klist}}), this test fails. Signing out of all active sessions allows the test to again pass.

{noformat}
Caused by: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to me@EXAMPLE.COM
        at org.apache.hadoop.hbase.security.TestUsersOperationsWithSecureHadoop.testAuthUtilLogin(TestUsersOperationsWithSecureHadoop.java:146)
{noformat}
Bug: Moving classes out of hbase-it /test for direct API use of chaos.
Status: Resolved | Priority: Minor | Resolution: Abandoned
Created: 03/Jan/20 11:56

In hbase-it package, Class like RESTApiClusterManager and HBaseClusterManager which has some part of implementation code for Chaos Action.

 

If any user wants to create private implementation which resembles HBaseClusterManager, then it has to be merged with hbase-it package as ClusterManager Interface and it's extended implementations are private. 

 

We should make them public so anyone can have their own implementation and need to be merged with hbase-it.
Bug: Many split related metrics were present RS side but after split is moved to Master, these metrics are lost.
Status: Patch Available | Priority: Minor | Resolution: 
Created: 16/Mar/20 20:50
Affects Version/s: 2.2.3

Many of the split related metrics like following are present RS side :

"SplitTime_num_ops": 0,
 "SplitTime_min": 0,
 "SplitTime_max": 0,
 "SplitTime_mean": 0,
 "SplitTime_25th_percentile": 0,
 "SplitTime_median": 0,
 "SplitTime_75th_percentile": 0,
 "SplitTime_90th_percentile": 0,
 "SplitTime_95th_percentile": 0,
 "SplitTime_98th_percentile": 0,
 "SplitTime_99th_percentile": 0,
 "SplitTime_99.9th_percentile": 0,

Value of these will always be 0 since Split is happening Master side and there is no Metrics update. 

 

 

Update 1: After the fix, JMX is like this :

{
 "name": "Hadoop:service=HBase,name=Master,sub=Procedure",
 "modelerType": "Master,sub=Procedure",
 "tag.Context": "master",
 "tag.Hostname": "blrphispra01257",
 "numMasterWALs": 0,
 "splitProcedure_RequestCount": 0,
 "splitProcedure_FailedCount": 0,
 "splitProcedure_SuccessCount": 0,
 "splitProcedureTime_num_ops": 0,
 "splitProcedureTime_min": 0,
 "splitProcedureTime_max": 0,
 "splitProcedureTime_mean": 0,
 "splitProcedureTime_25th_percentile": 0,
 "splitProcedureTime_median": 0,
 "splitProcedureTime_75th_percentile": 0,
 "splitProcedureTime_90th_percentile": 0,
 "splitProcedureTime_95th_percentile": 0,
 "splitProcedureTime_98th_percentile": 0,
 "splitProcedureTime_99th_percentile": 0,
 "splitProcedureTime_99.9th_percentile": 0
 },

 

,
Bug: Periodically check whether a system stop is requested in compaction by time.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Mar/20 10:14

The compaction check that the system is stopped when it reaches hbase.hstore.close.check.interval size.
If the size is not reached, the compaction does not stop even though the system is stopped.
If compaction is not terminated, the region is not closed.
As a result, region remains pending-close.
 
Thread A (regionserver/longCompactions):
State: RUNNABLE
Stack:
org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:319)
org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:111)
org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:119)
org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1232)
org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1896)
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:525)
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:562)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)

Thread B (RS_CLOSE_REGION):
State: WAITING
Waiting on org.apache.hadoop.hbase.regionserver.HRegion$WriteState
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
org.apache.hadoop.hbase.regionserver.HRegion.waitForFlushesAndCompactions(HRegion.java:1567)
org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1417)
org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1380)
org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:138)
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
Bug: hbase/bin/region_status.rb Not available in hbase2
Status: Reopened | Priority: Minor | Resolution: 
Created: 26/Mar/20 08:20
Affects Version/s: 2.0.1

A series of strange errors occurred while running this script in hbase2, which was corrected as prompted, and note that in hbase2's MetaTableAccessor, getHRegionInfo has been replaced by getRegionInfo
Bug: The message "Please make sure that backup is enabled on the cluster." is shown even when the backup feature is enabled
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Feb/20 20:39

The following message is shown even when the backup feature is enabled, which is confusing:
{code}
Please make sure that backup is enabled on the cluster. To enable backup, in hbase-site.xml, set:
 hbase.backup.enable=true
hbase.master.logcleaner.plugins=YOUR_PLUGINS,org.apache.hadoop.hbase.backup.master.BackupLogCleaner
hbase.procedure.master.classes=YOUR_CLASSES,org.apache.hadoop.hbase.backup.master.LogRollMasterProcedureManager
hbase.procedure.regionserver.classes=YOUR_CLASSES,org.apache.hadoop.hbase.backup.regionserver.LogRollRegionServerProcedureManager
hbase.coprocessor.region.classes=YOUR_CLASSES,org.apache.hadoop.hbase.backup.BackupObserver
and restart the cluster
{code}
Bug: Minor improvements to Region Normalizer
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Mar/20 00:24
Affects Version/s: 2.3.0

While hbck2 runs through fixes to meta, the Normalizer kicks in to "help". Mostly this is fine, however, things get confused when {{fixMeta}} is action on a region and then the normalizer decides to do something as well. Could be we advise operators disable normalizer while running hbck2, but would be better if the normalizer was less aggressive, so that there's fewer flags to fiddle.
Bug: Document that with split-to-hfile data over the MOB threshold will be treated as normal data
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Jan/20 13:36
Affects Version/s: 2.3.0

h3. documentation update

Update the troubleshooting section of the MOB chapter to include a note for "Why is there data over the MOB threshold in the normal {{/hbase/data}} directory rather than in {{/hbase/mobdir}}. List the split-to-hfile feature as one source and bulk loading as another.  Note that in both cases the next compaction to include those files will write the data out to MOB hfiles.

h3. original
Steps to reproduce this issue.
1. create a table with 1 region, and mob enabled, keep threshold value to 5.
2. Load data into the table, keep the value size should be  more than 5.
3. flush the table.
4. observe the mobdir and data dir, hfiles should be there.
5. load data again with different data set, keep the value size is greater than 5.
6. Kill -9 RS where table region is online
7. Start RS

check the mob dir and data dir, both should have 2 hfiles each.
But data dir only have 2 hfiles, that means mob threshold crossed data is considered as normal data.
Bug: Avoid high speed recursion trap in AsyncRequestFutureImpl.
Status: Resolved | Priority: Minor | Resolution: Incomplete
Created: 09/Mar/20 22:39
Affects Version/s: 2.3.0

While working on branch-2, I ran into an issue where a retryable error kept occurring and code in AsyncRequestFutureImpl would reduce the backoff wait to 0 and extremely rapidly eat up a of thread stack space with recursive retry calls. This little patch stops the backoff wait kill after 3 retries. Chosen kind of arbitrarily, perhaps 5 is the right number, but I find large retry counts tend to hide things and that has made me default to fairly conservative in all my arbitrary number picking.
Bug: Improve the accuracy of the method sizeToString
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Mar/20 10:08

In QuotaSettings，the method of sizeToString reserved integer. But, this is not very accurate. 


{code:java}
hbase(main):001:0> set_quota TYPE => SPACE, TABLE => 't1', LIMIT => '2000G', POLICY => NO_INSERTS

hbase(main):002:0> list_quotas
OWNER                                    QUOTAS                                                                                                              
 TABLE => t1                             TYPE => SPACE, TABLE => t1, LIMIT => 1T, VIOLATION_POLICY => NO_INSERTS                                             
1 row(s) in 0.0340 seconds
{code}

after patch：

{code:java}
hbase(main):001:0> set_quota TYPE => SPACE, TABLE => 't1', LIMIT => '2000G', POLICY => NO_INSERTS

hbase(main):002:0> list_quotas
OWNER                                    QUOTAS                                                                                                              
TABLE => t1                             TYPE => SPACE, TABLE => t1, LIMIT => 1.95T, VIOLATION_POLICY => NO_INSERTS                                       
1 row(s) in 0.0230 seconds

{code}
Bug:  Bookmark for github PR to jira redirection
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Jan/20 00:25

Following is a simple js snippet that redirects from any HBase PR to its corresponding jira. Without this, one has to copy the jira ID from the PR, construct a jira URL manually and paste it in the browser URL bar. Saves a bunch of clicks.
{code:javascript}
javascript:location.href='https://issues.apache.org/jira/browse/'document.getElementsByClassName("js-issue-title")[0].innerHTML.match(/HBASE-\d/)[0];{code}
Particularly helpful for reviewers who'd like to read the jira contents often when reviewing a PR.

For chrome:
 - Right Click on the bookmarks bar
 - Click on Add page. Fill in the following details:
 Name: HBase jira redirect (or any other that you prefer)
 URL: – {{snippet from above}}--
 - Click Save

Now you should see "HBase jira redirect" (or any other name you gave) bookmark on the bar.
 Go to any Github PR, click on this button and it redirects to the corresponding jira.
Bug: Convert logging code  to use Parameters when possible
Status: Open | Priority: Minor | Resolution: 
Created: 21/Mar/20 00:41
Affects Version/s: 2.4.0

Based on [~ndimiduk]'s comments in HBASE-23957 ,
{code:java}
nit: we make a habit of using the logger's format string capability. That way it only performs the computation necessary to build the log message if the logger is emitting messages at the target level.

LOG.debug("Starting the test {}", name.getMethodName()){code}
 and 

[http://blog.logscape.com/2017/03/concatenation-or-parameters-both-whats-the-top-method-of-java-logging/], 

I did a quick search of the code base and found 

[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java#L334]

 

This needs to be converted to use parameter instead of string concatenation.

.
Bug: Use enforcer plugin to print JVM info in maven output
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Mar/20 00:53

This is a little nice-to-have. We currently dump OS info using {{os-maven-plugin}}. Since we build on multiple JVM environments, would be nice to add that info too. Looking around, there wasn't an obvious source of this information, until I realized it's readily available via {{enforcer:display-info}}. Should add this to the top-level pom, if we can get it to run just once/build instead of with every module.

{noformat}
[INFO] --- maven-enforcer-plugin:3.0.0-M2:display-info (default-cli) @ hbase-archetype-builder ---
[INFO] Maven Version: 3.6.3
[INFO] JDK Version: 1.8.0_222 normalized as: 1.8.0-222
[INFO] OS Info: Arch: x86_64 Family: mac Name: mac os x Version: 10.14.6
[INFO] ------------------------------------------------------------------------
{noformat}

{noformat}
[INFO] --- maven-enforcer-plugin:3.0.0-M2:display-info (default-cli) @ hbase-archetype-builder ---
[INFO] Maven Version: 3.6.3
[INFO] JDK Version: 11.0.4 normalized as: 11.0.4
[INFO] OS Info: Arch: x86_64 Family: mac Name: mac os x Version: 10.14.6
[INFO] ------------------------------------------------------------------------
{noformat}
Bug: Correct exception messages for table null and namespace unavailable.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Mar/20 18:04

Exception message for following two scenarios should be corrected. 

1. Change message to  "The list of tables cannot be null." in below code

{code:java}
@Override
  public void moveTables(Set<TableName> tables, String targetGroup) throws IOException {
    if (tables == null) {
      throw new ConstraintException("The list of servers cannot be null.");
    }
{code}
2. Change the message to "Region server group "+group+" does not exist" in below code.

{code:java}
public void preCreateNamespace(ObserverContext<MasterCoprocessorEnvironment> ctx,
                                 NamespaceDescriptor ns) throws IOException {
    String group = ns.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP);
    if(group != null && groupAdminServer.getRSGroupInfo(group) == null) {
      throw new ConstraintException("Region server group "+group+" does not exit");
    }
  }
{code}
Bug: FastPathBalancedQueueRpcExecutor should enforce queue length of 0
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Jan/20 17:53

FastPathBalancedQueueRpcExecutor allows RPC requests to skip the RPC queue and get worked by an available handler under certain circumstances. 

Relatedly, the hbase.ipc.server.max.callqueue.length parameter can be set to 0, including dynamically. This can be useful to temporarily prevent writes on a cluster. 

When this is the case the executor is supposed to block all dispatching. However, the FastPathBalancedQueueRpcExecutor will still dispatch the request if one of the "fast path" handlers is available on its stack. This both isn't the desired effect, and also makes TestSimpleRpcScheduler.testSoftAndHardQueueLimits unstable when it checks the queue length 0 behavior. 

A simple fix is just to check max queue length > 0 before FastPathBalancedQueueRpcExecutor pops the fast handler off the stack.
Bug: MasterFileSystem should not create MasterProcWALs dir on fresh installs
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Jan/20 15:42
Affects Version/s: 3.0.0-alpha-1

With the new procedure store that uses internal table for persisting procedures state, there's no need for creating the old MasterProcWALs folder on fresh installs. We should remove t from the list of dirs created by _MasterFileSystem.createInitialFileSystemLayout,_ so that it does not trigger an unnecessary conversion of procedure stores.
Bug: Expose shaded clients through WebUI as Maven repository
Status: Resolved | Priority: Minor | Resolution: Incomplete
Created: 26/Mar/20 20:20

Props to [~busbey] for this idea.

We have a number of shaded jars which are (largely) sufficient for launching any Java application against HBase. However, if users have multiple versions of HBase in their organization, it might be confusing to know "which client" do I need to use? Can we expose our shaded clients from HBase in such a way that build tools can just ask HBase for the client jars they should use?

The idea here is that we can use embedded Jetty to "fake" out a Maven repository that users can put in their client applications. We have no extra burden from HBase because we already are packaging these jars. I'll link an example Maven application which uses this "feature".
Bug: Meta browser should show all `info` columns
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Mar/20 21:14
Affects Version/s: 2.3.0

The Meta table browser lists region states. There are other {{info}} columns in the table, which should be displayed. Looking through {{HConstants}}, it seems we need to add the following:
 * {{server}}
 * {{sn}}
 * {{splitA}}
 * {{splitB}}
 * {{merge}}
 * {{mergeA}}
 * {{mergeB}}

Are there others?
Bug: Provide a much faster and efficient alternate option to maven and surefire for running tests.
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 06/Feb/20 10:23

With HBASE-23795, the hope is to drive tests with maven and surefire much closer to their potential.

That will still leave a lot of room for improvement.

For those that have some nice hardware and a need for speed, we can blow right past maven+surefire.
Bug: Race condition in Master startup can ignore shutdown RPC
Status: Open | Priority: Minor | Resolution: 
Created: 13/Feb/20 18:41
Affects Version/s: 2.3.0

Identified as the root-cause of HBASE-23808. The master starts its RPC server first thing in its constructor, but does not initialize it's {{clusterTracker}} until later (both from the super constructor). Thus it is possible for a client to issue a {{shutdown}} RPC is received by a master before it has a clusterTracker instance. It is highly likely (guaranteed?) to also not be the active master at this point. In such a case, the {{shutdown}} is received and acknowledged back to the client, but is effectively ignored.

After chatting with [~bharathv] on the PR, we believe this race becomes more likely (though still rare) after HBASE-18095. This is something to consider should we get around to a startup sequence refactor.

Setting priority as "minor" because the condition is merely annoying; from what I can tell, a subsequent {{shutdown}} RPC to any master would result in the correct shutdown of the first.