av9ash/GitBugs huggingface dataset:
Bug: Found duplicate classes/resources in hbase-protocol-shaded 2.3.2, hbase-protocol 2.3.2
Status: Open | Priority: Major | Resolution: 
Created: 27/Sep/20 03:53
Affects Version/s: 2.3.2

i want to use hbase client to connector hbase, when i add dependency in pom.xml, 

<dependency>

 <groupId>org.apache.hbase</groupId>

<artifactId>hbase-client</artifactId>

</dependency>

It will report error when mvn clean install, 

[WARNING]Found duplicate and different resources in [org.apche.hbase:hbase-protocol-shaded:2.3.2, org.apche.hbase:hbase-protocol:2.3.2]:

[WARNING]AccessControl.proto

[WARNING]Admin.proto

……
Bug: PerformanceEvaluation with presplit randomWrite test has severe hotspotting
Status: Open | Priority: Major | Resolution: 
Created: 11/Sep/20 17:17
Affects Version/s: 2.3.1

I'm attempting to apply a load to a small distributed cluster (12 RS). I use {{-presplit}} and specify 3 regions per region server. {{randomWrite}} test with {{-nomapred}} and 30 client threads. The result is severe hot-spotting on a single region region (10's of thousands of reqs/sec) and minimal load (high 10's to low 100's reqs/sec) to the others. It seems the split algorithm and the load generator do not agree on an even data distribution.
Bug: Refactor Replication: move the default ReplicationSinkService implementation out
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Sep/20 03:41

Now the Replication implements both ReplicationSourceService and ReplicationSinkService interface. But most of code is only related to ReplicationSourceService. Meanwhile, there is a bug when HRegionServer#buildServerLoad, as it not consider the case: ReplicationSourceService implementation is different with ReplicationSinkService.
Bug: Should not use XXXService.Interface.class.getSimpleName as stub key prefix in AsyncConnectionImpl
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Sep/20 13:13

It will just return "Interface" for all services. This is not a bug for AsyncConnectionImpl as we have different maps for ClientService and AdminService, but it is still a bit confusing to developers.

I suggest either we just use empty string and add a comment to say that the service name is not important here, or change to use XXXService.class.getSimpleName() or XXXService.getDescriptor().getName(), so it is less confusing to developers.
Bug: Up the container nproc uplimit to 30000
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Sep/20 15:05

We (Apache Phoenix team) have recently switched our precommit tests to Dockerized Yetus (mostly adopted from the solution in Hbase)

We see 
java.lang.OutOfMemoryError: unable to create new native thread
errors , while Yetus shows
|Max. process+thread count|6833 (vs. ulimit of 12500)|

While I couldn't determine what was the job that we shared the Agent with at the time, statistically it was very likely HBase, and an HBase job probably failed with a similar error.

Some research has thrown up the official Docker docs: 

[https://docs.docker.com/engine/reference/commandline/run/#set-ulimits-in-container-ulimit]

According to which it is not possible to set container level nprocs ulimit with Docker.

All settings apply to the docker Daemon user instead, and the limit is shared between all containers.

Based on this, I think that it makes no sense to set a container (really docker user) nprocs ulimit any lower than the current hard limit of 30000.

I have already set PROC_LIMIT=30000 in the Phoenix Yetus personality, but it is only a half solution until some Docker users set lower values, as the later setting will apply as soon as the container is started.
Bug: fix typo in MasterRegion java doc
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Sep/20 10:42

The archived HFile will be moved to the global HFile archived directory with the \{@link MasterRegionParams#archivedWalSuffix()} suffix. The archived WAL will be moved to the global WAL archived directory with the \{@link MasterRegionParams#archivedHFileSuffix()} suffix
Bug: REST Server failes to start without any error message
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Sep/20 02:10
Affects Version/s: 2.2.1

When I enable security and start rest server, i  see the messages like these:
{code:java}
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.8.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hbase-2.2.1/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2020-09-03 02:10:57,995 INFO  [main] RESTServer: ***** STARTING service 'RESTServer' *****
2020-09-03 02:10:57,998 INFO  [main] util.VersionInfo: HBase 2.2.1
2020-09-03 02:10:57,998 INFO  [main] util.VersionInfo: Source code repository git://hao-OptiPlex-7050/home/hao/open_source/hbase revision=f93aaf770cce81caacbf22174dfee2860dbb4810
2020-09-03 02:10:57,998 INFO  [main] util.VersionInfo: Compiled by hao on 2019年 09月 10日 星期二 14:28:27 CST
2020-09-03 02:10:57,998 INFO  [main] util.VersionInfo: From source with checksum c0f358e344b10713f61037e16fd39f8euse{code}
Rest server doesn't start successfully, but it was bad that I can't find any error message indicating that server fails to start and why failure happens.

Afer debuging, I found that loginServerPrincipal can throw exception, like:
{code:java}
java.lang.IllegalArgumentException: hbase.rest.keytab.file should be set if security is enabled
        at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:142)
        at org.apache.hadoop.hbase.rest.RESTServer.loginServerPrincipal(RESTServer.java:149)
        at org.apache.hadoop.hbase.rest.RESTServer.run(RESTServer.java:237)
        at org.apache.hadoop.hbase.rest.RESTServer.main(RESTServer.java:424)
{code}
 But these exceptions are caught without any messages printed.
{code:java}
org.apache.hadoop.hbase.rest.RESTServer.main
423 try {
424    server.run();
425    server.join();
426 } catch (Exception e) {
427     System.exit(1);
428 }
{code}
We need print the failure message to help user config the system.
Bug: Region hole contains wrong regions pair when hole is created by first region deletion
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Aug/20 17:11
Affects Version/s: 2.2.3

HBCK2 Holes: Wrong region details in HBCK report ,if a region is missing in meta and start key is empty

for hole, it reports two region where hole is present.

Scenario where first region is missing for eg: in below scenario r1 is missing in meta

start key    end key   region

empty     10        r1

10           20        r2

20           30        r2

30          empty  r3

 

In our test hbck report contains two region where first region belongs to different table and second region is r2.
Bug: ConnectionManager#locateRegionInMeta waits for user region lock indefinitely.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Aug/20 21:09
Affects Version/s: 1.3.2

One of our customers experienced high latencies (in order of 3-4 minutes) for point lookup query (We use phoenix on top of hbase).

We have different threads sharing the same hconnection.  Looks like multiple threads are stuck at the same place. [https://github.com/apache/hbase/blob/branch-1.3/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java#L1282] 

We have set the following configuration parameters to ensure query fails with a reasonable SLAs:

1. hbase.client.meta.operation.timeout

2. hbase.client.operation.timeout

3. hbase.client.scanner.timeout.period

But since  userRegionLock can wait for lock indefinitely the call will not fail within SLA.
Bug: Make RegionInfo#UNDEFINED IA.Private
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Aug/20 18:25
Affects Version/s: 2.3.2

From review of parent PR, RI#UNDEFINED is for internal use only... note it w/ a IA.Private marking. Also deprecate it so we have option to remove as its loading implicated in the parent issue static load deadlock.
Bug: ScheduledChore is never triggered when initalDelay > 1.5*period
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Sep/20 11:54
Affects Version/s: 2.2.3

In our recent tests, ScheduledChore is never triggered when initalDelay > 1.5*period.

The cause of the bug is the following:

The trigger time for a ScheduleChore must be within an acceptable time window that is 1.5 * period. see [here|https://github.com/apache/hbase/blob/e5ca9adc54f9f580f85d21d38217afa97aa79d68/hbase-common/src/main/java/org/apache/hadoop/hbase/ScheduledChore.java#L234]

timeOfLastRun and timeOfThisRun are two variables that record two adjacent trigger time. [The first initialization of timeOfThisRun|https://github.com/apache/hbase/blob/e5ca9adc54f9f580f85d21d38217afa97aa79d68/hbase-common/src/main/java/org/apache/hadoop/hbase/ScheduledChore.java#L273] is when the ScheduleChore is created, it's not a real trigger time.

If we set initialDelay > 1.5 period , after initialDelay, the first time when chore is triggered has exceeded the allowed window. Then [cancel the chore and schedule it again|https://github.com/apache/hbase/blob/e5ca9adc54f9f580f85d21d38217afa97aa79d68/hbase-common/src/main/java/org/apache/hadoop/hbase/ChoreService.java#L176].

So it's stuck in loop when initialDelay > 1.5 period :

1.  init timeOfThisRun at a wrong time.

2. wait initalDelay

3. chore trigger, but exceeded the allowed window.

4. cancel chore and schedule it again

5. go step 1.
Bug: HBASE-24359 causes replication missed log of some RemoteException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Sep/20 06:27
Affects Version/s: 2.3.0

HBASE-24359 broken the logic of handling exception. In branch2, it even causes some RemoteException log missed.

[File changed|[https://github.com/apache/hbase/pull/1855/files#diff-1e3f171b19474698601a0752b618af0eL435]] in branch2.

!image-2020-09-11-14-30-27-898.png!
Bug: FastLongHistogram#getCountAtOrBelow method is broken.
Status: Open | Priority: Major | Resolution: 
Created: 16/Sep/20 18:46
Affects Version/s: 1.6.0

FastLongHistogram#getCountAtOrBelow method is broken.
If I revert HBASE-23245 then it works fine.
Wrote a small test case in TestHistogramImpl.java : 

{code:java}
  @Test
  public void testAdd1() {
    HistogramImpl histogram = new HistogramImpl();
    for (int i = 0; i < 100; i++) {
      histogram.update(i);
    }
    Snapshot snapshot = histogram.snapshot();
    // This should return count as 6 since we added 0, 1, 2, 3, 4, 5
    Assert.assertEquals(6, snapshot.getCountAtOrBelow(5));
{code}

It fails as below:

java.lang.AssertionError: 
Expected :6
Actual      :100
Bug: [HBOSS] TreeLockManager should not define itself as the default implementation for fs.hboss.sync.impl
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Sep/20 19:20
Affects Version/s: hbase-filesystem-1.0.0-alpha1

Currently, abstract class TreeLockManager defines itself as the default implementation of fs.hboss.sync.impl. This is wrong, as in the event of fs.hboss.sync.impl omission from the config, it will end up trying to instantiate itself, failing as its abstract:

{noformat}
 2020-09-16 13:29:48,877 ERROR org.apache.hadoop.hbase.regionserver.HRegion: There was IO error when checking if the bulk load is ok in region wordcount,w,1600262698081.84cb78e467e3c3b40e78c7f07d643d5c..
java.io.IOException: Class referred to by fs.hboss.sync.impl, org.apache.hadoop.hbase.oss.sync.TreeLockManager, is not a valid implementation of org.apache.hadoop.hbase.oss.sync.TreeLockManager
        at org.apache.hadoop.hbase.oss.sync.TreeLockManager.get(TreeLockManager.java:70)
        at org.apache.hadoop.hbase.oss.HBaseObjectStoreSemantics.initialize(HBaseObjectStoreSemantics.java:120)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3423)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3483)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3451)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:518)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
        at org.apache.hadoop.hbase.regionserver.HStore.assertBulkLoadHFileOk(HStore.java:818)
        at org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:6221)
        at org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$1.run(SecureBulkLoadManager.java:291)
        at org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$1.run(SecureBulkLoadManager.java:266)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:360)
{noformat}
Bug: All executor service should start using guava ThreadFactory
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jul/20 17:09

Currently, we have majority Executor services using guava's ThreadFactoryBuilder while creating fixed size thread pool. There are some executors using our internal hbase-common's Threads class which provides util methods for creating thread factory.

Although there is no perf impact, we should let all Executors start using our internal library for using ThreadFactory rather than having external guava dependency (which is nothing more than a builder class). We might have to add a couple more arguments to support full fledged ThreadFactory, but let's do it and stop using guava's builder class.

*Update:*

Based on the consensus, we should use only guava library and retire our internal code which maintains ThreadFactory creation.
Bug: Lots of thread pool are changed to non daemon after HBASE-24750 which causes trouble when shutting down
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/20 08:54

We replace Threads.newDaemonThreadFactory with guava ThreadFactoryBuilder but miss the setDaemon call, which causes our client integration test to fail because of hbase shell can never quit cleanly.

We should add the setDaemon back for those thread factories.
Bug: Log RegionTooBusyException details
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Sep/20 10:18
Affects Version/s: 1.3.6

When we throw RegionTooBusyException, it gets wrapped in RetriesExhaustedWithDetailsException at times and the original source doing batch mutation might not be able to log region name and root cause that comes with RegionTooBusyException as it is wrapped with outer RetriesExhaustedWithDetailsException and outer Exception has it's own way of logging only necessary info, which is quite acceptable. Attaching a sample screenshot.

We should at least log details of RegionTooBusyException at server side before we throw it.
Bug: Create simple pattern matching query for retrieving metrics matching the pattern
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Sep/20 05:05
Affects Version/s: 2.2.5

Currently we allow querying a metric with a specific metric name. But generally say under the MBean sub=Tables under a server we might have 'n' number of tables and we might be interested in a specific metric across all tables. 
So in such cases giving a simple pattern based query will help so that we can retrieve all metrics that follow that pattern.
The other side effect is that we can also reduce the size of the json we pull from the server to the querying client. Which are generally smaller scripts which might not be able to process the big sized response JSON.

Thanks [~anoopsamjohn] for the suggestion.
Bug: [hbase-operator-tools] NPE in RegionsMerger#mergeRegions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Sep/20 09:00
Affects Version/s: hbase-operator-tools-1.1.0

RegionsMerger#mergeRegions will call RegionsMerger#getOpenRegions to get all OPEN region for target table. When target table's REPLICATION_SCOPE is '1', RegionsMerger#getOpenRegions may return null.
Bug: MetaFixer fails to fix overlaps when multiple tables have overlaps
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Sep/20 05:43
Affects Version/s: 2.2.3

MetaFixer fails to fix overlaps when multiple tables have overlaps
*Steps to reproduce from UT.*
# Create table t1 and t2 with split keys, ["bbb", "ccc", "ddd", "eee"]
# Create extra region in both t1 and t2 with start key "bbb" and end key "ddd"
# Run catalog janitor, It will report total 4 overlaps, 2 from each table.
# Run MetaFixer, wait for merges to finish.
# Run the catalog janitor again and verify report, there should not be any overlap
# Overlap still exists. Reproduced!!!

*Analysis.*
* When I run the same scenario for just one table t1, overlaps are fixed successfully.
* Seems problem with MetaFixer#calculateMerges. 
* I think merges should be calculated within a table. Across the table merge does not have significance.
Bug: [HBOSS] UTs failing when running against hadoop 3.3. 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Aug/20 18:48
Affects Version/s: hbase-filesystem-1.0.0-alpha1

Current UTs fail when running against hadoop 3.3. This is because on 3.3, *AbstractAmazonS3* defines additional _doesBucketExistV2_ that's called upon initialization, but UT utility *EmbeddedS3* class is not implementing it.
Bug: Wrap ConnectionImplemetation#locateRegioninMeta under operation timeout.
Status: Open | Priority: Major | Resolution: 
Created: 05/Sep/20 18:35
Affects Version/s: 1.6.0

We have config property (hbase.client.operation.timeout and hbase.client.meta.operation.timeout).
Description of hbase.client.operation.timeout which is for non meta tables.
{noformat}
Operation timeout is a top-level restriction (millisecond) that makes sure a blocking operation in Table will not be blocked more than this. In each operation, if rpc request fails because of timeout or other reason, it will retry until success or throw RetriesExhaustedException. But if the total time being blocking reach the operation timeout before retries exhausted, it will break early and throw SocketTimeoutException.
{noformat}

Most of the operations like get, put, delete are wrapped under this timeout but scan operation is not wrapped in this timeout. We need to wrap scan operations also within operation timeout.
More discussion in this PR thread:  https://github.com/apache/hbase/pull/2322#discussion_r478687341
Bug: Add help output for shell "magic" commands
Status: Open | Priority: Major | Resolution: 
Created: 20/Aug/20 20:57

The shell has a handful of "magic" commands, things like {{help}} and {{debug}}, that exist as commands in the shell, but are not implementations of the command class and have no help documentation of their own.

Let's promote them to proper commands, add a new command group, call it "shell command group", for example. This makes them discoverable. Give each of them a help documentation string with examples, as we have for the others, i.e., {{help 'help'}}.
Bug: runCatalogJanitor() API should return -1 to indicate already running status
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Aug/20 06:17
Affects Version/s: 2.2.3

runCleanerChore() API returns false if hbck chore is already running. This is quite helpful in many cases.

runCatalogJanitor() API  is not indicating whether scan is already running. Also it is not doing anything in case scan is already running.

I think we should return -1 to indicate already running status.
Bug: Fix dead links in HBase book
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Sep/20 17:10
Affects Version/s: 2.3.0

1.  -[https://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/file/tfile/TFile.html|https://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/file/tfile/TFile.html-]- => [https://hadoop.apache.org/docs/current/api/org/apache/hadoop/io/file/tfile/TFile.html]

2. -[https://vimeo.com/26804675|https://vimeo.com/26804675-]- => [https://www.youtube.com/watch?v=DdGKAorSSZ0]

3. -[http://www.cloudera.com/videos/hw10_video_how_stumbleupon_built_and_advertising_platform_using_hbase_and_hadoop|http://www.cloudera.com/videos/hw10_video_how_stumbleupon_built_and_advertising_platform_using_hbase_and_hadoop-]- has been invalid and cannot be found in other website, just remove it.

4. -[https://hadoop.apache.org/core/docs/stable/api/org/apache/hadoop/metrics/package-summary.html]- => [https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/metrics2/package-summary.html]
Bug: Refactor TestJMXConnectorServer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Aug/20 11:43
Affects Version/s: 3.0.0-alpha-1

Two optimization points for TestJMXConnectorServer in this issue:
 # Just run cluster once, not once per test case.
 # Use random free port to run ConnectorServer, avoid specifying a fixed port.
Bug: Remove MetaTableAccessor.tableExists
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Aug/20 03:45

At master side, we could make use of TableDescriptors to determine whether a table exists, as it caches all the table descriptors.

For other places, we could just make use Admin.tableExists, which is exactly the same with calling MetaTableAccessor.tableExists but avoid exposing meta directly.
Bug: ZKTableStateClientSideReader#isDisabledTable doesn't check if table exists or not.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Aug/20 01:47
Affects Version/s: 1.3.6

The following bug exists only in branch-1 and below.

ZKTableStateClientSideReader#isDisabledTable returns false even if table doesn't exists.

Below is the code snippet:

 {code:title=ZKTableStateClientSideReader.java|borderStyle=solid}
  public static boolean isDisabledTable(final ZooKeeperWatcher zkw,
      final TableName tableName)
      throws KeeperException, InterruptedException {
    ZooKeeperProtos.Table.State state = getTableState(zkw, tableName);    ---> We should check here if state is null or not.
    return isTableState(ZooKeeperProtos.Table.State.DISABLED, state);
  }
}
{code}
Bug: One of static initializers of CellComparatorImpl referring to subclass MetaCellComparator
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Aug/20 18:17

Superclass CellComparatorImpl has static initializer META_COMPARATOR which initializes sub-class MetaCellComparator, which could lead to deadlock while loading class (similar in nature to HBASE-24896 , although not that complex). We can break the dependency now before it becomes more complex (Splittable meta work will most likely impact this code path).