av9ash/GitBugs huggingface dataset:
Bug: MutableRangeHistogram#updateSnapshotRangeMetrics doesn't calculate the distribution for last bucket.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jun/20 22:24
Affects Version/s: 1.3.7

We are not processing the distribution for last bucket. 

https://github.com/apache/hbase/blob/master/hbase-hadoop-compat/src/main/java/org/apache/hadoop/metrics2/lib/MutableRangeHistogram.java#L70

{code:java}
  public void updateSnapshotRangeMetrics(MetricsRecordBuilder metricsRecordBuilder,
                                         Snapshot snapshot) {
    long priorRange = 0;
    long cumNum = 0;

    final long[] ranges = getRanges();
    final String rangeType = getRangeType();
    for (int i = 0; i < ranges.length - 1; i++) {         -----> The bug lies here. We are not processing last bucket.
      long val = snapshot.getCountAtOrBelow(ranges[i]);
      if (val - cumNum > 0) {
        metricsRecordBuilder.addCounter(
            Interns.info(name + "_" + rangeType + "_" + priorRange + "-" + ranges[i], desc),
            val - cumNum);
      }
      priorRange = ranges[i];
      cumNum = val;
    }
    long val = snapshot.getCount();
    if (val - cumNum > 0) {
      metricsRecordBuilder.addCounter(
          Interns.info(name + "_" + rangeType + "_" + ranges[ranges.length - 1] + "-inf", desc),
          val - cumNum);
    }
  }
{code}
Bug: [Flakey Tests] TestSnapshotScannerHDFSAclController
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Apr/20 00:34
Affects Version/s: 2.3.0

Still see the following error from the test.
{code:java}
-------------------------------------------------------------------------------
Test set: org.apache.hadoop.hbase.security.access.TestSnapshotScannerHDFSAclController
-------------------------------------------------------------------------------
Tests run: 23, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 342.32 s <<< FAILURE! - in org.apache.hadoop.hbase.security.access.TestSnapshotScannerHDFSAclController
org.apache.hadoop.hbase.security.access.TestSnapshotScannerHDFSAclController.testGrantTable  Time elapsed: 6.82 s  <<< FAILURE!
java.lang.AssertionError: expected:<6> but was:<-1>
        at org.apache.hadoop.hbase.security.access.TestSnapshotScannerHDFSAclController.testGrantTable(TestSnapshotScannerHDFSAclController.java:349)

 {code}
Bug: Tweak the master registry docs for branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jun/20 20:54
Affects Version/s: 2.3.0

As Nick pointed out in https://github.com/apache/hbase/pull/1880, we need to re-word the content a bit so that it makes sense to branch-2 users. Specifically the following

- What version has the feature shipped
- A slight modification to the config keys after hedging moved from rpc layer to the registry code (HBASE-24265)

The latter applies to the master branch too.
Bug: TestHttpServer.testBindAddress failure with latest jetty
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Apr/20 14:23

The latest jetty version (tested with 9.4.28.v20200408 which is not included in HBase yet) wraps BindException into an IOException when it fails to bind to a port.

This breaks HttpServer's findPort functionality, which manifests in TestHttpServer.testBindAddress failing.

The proposed patch handles both the old and the new jetty behaviour correctly.
Bug: [Flakey Test] regionserver.TestRegionReplicas 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Apr/20 16:28
Affects Version/s: 2.3.0

There are 3 failed runs from flakey test board.
{code:java}
org.apache.hadoop.hbase.regionserver.TestRegionReplicas.testFlushAndCompactionsInPrimaryFailing for the past 1 build (Since #5954 )Took 45 sec.Error Messageexpected null, but was:<org.apache.hadoop.hbase.NotServingRegionException: TestRegionReplicas,,1585782621571_0001.0896d89824217690d8a4b391bb139962. is not online on asf905.gq1.ygridcore.net,37761,1585782616299>Stacktracejava.lang.AssertionError: expected null, but was:<org.apache.hadoop.hbase.NotServingRegionException: TestRegionReplicas,,1585782621571_0001.0896d89824217690d8a4b391bb139962. is not online on asf905.gq1.ygridcore.net,37761,1585782616299>
	at org.apache.hadoop.hbase.regionserver.TestRegionReplicas.testFlushAndCompactionsInPrimary(TestRegionReplicas.java:432)
Standard OutputFormatting using clusterid: testClusterID
java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10]
    Thread[HFileArchiver-3,5,PEWorkerGroup]
    Thread[HFileArchiver-4,5,PEWorkerGroup]
Standard Error {code}
Bug: Fix the classpath for the local mini-cluster
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jun/20 23:40
Affects Version/s: 3.0.0-alpha-1

The classpath is created assuming the native-client is a module of the parent hbase repo. Since that is not the case anymore and we rely on a dynamically pulled HBase version, we need to fix the classpath for the JVM that is spun using JNI. Otherwise mini-cluster won't start..

{noformat}
using hbase::MiniCluster;

JNIEnv *MiniCluster::CreateVM(JavaVM **jvm) {
  JavaVMInitArgs args;
  JavaVMOption jvm_options;
  args.version = JNI_VERSION_1_6;
  args.nOptions = 1;
  char *classpath = getenv("CLASSPATH");
  std::string clspath;
  if (classpath == NULL || strstr(classpath, "-tests.jar") == NULL) {
    std::string clsPathFilePath("../../../hbase-build-configuration/target/cached_classpath.txt");
    std::ifstream fd(clsPathFilePath);
    std::string prefix("");
    if (fd.is_open()) {
      if (classpath == NULL) {
        LOG(INFO) << "got empty classpath";
      } else {
        // prefix bootstrapper.jar
        prefix.assign(classpath);
      }
      std::string line;
{noformat}
Bug: Deadlock in mini-cluster-test teardown on setup() failure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jun/20 23:35
Affects Version/s: 3.0.0-alpha-1

Code seems to be weirdly strucutred. Cleanup the life cycle

{noformat}
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f4e2ce84023 in __GI___pthread_mutex_lock (mutex=0x55d922b810e8) at ../nptl/pthread_mutex_lock.c:78
#2  0x000055d9207b263a in __gthread_mutex_lock ()
#3  0x000055d9207b3bf4 in std::mutex::lock() ()
#4  0x000055d9207c2d88 in std::lock_guard<std::mutex>::lock_guard(std::mutex&) ()  <== Attempts to lock mutex again.
#5  0x000055d9208cbf86 in hbase::MiniCluster::Setup() ()
#6  0x000055d9208cc87c in hbase::MiniCluster::env() ()
#7  0x000055d9208cd33b in hbase::MiniCluster::StopCluster() ()
#8  0x000055d9208c8ace in hbase::TestUtil::StopMiniCluster() ()
#9  0x000055d9208c86d8 in hbase::TestUtil::~TestUtil() ()
#10 0x000055d92061bbb0 in std::default_delete<hbase::TestUtil>::operator()(hbase::TestUtil*) const ()
#11 0x000055d920616c17 in std::unique_ptr<hbase::TestUtil, std::default_delete<hbase::TestUtil> >::~unique_ptr() ()
#12 0x00007f4e29e96041 in __run_exit_handlers (status=-1, listp=0x7f4e2a23e718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108
#13 0x00007f4e29e9613a in __GI_exit (status=<optimized out>) at exit.c:139
#14 0x000055d9208cbb67 in hbase::MiniCluster::CreateVM(JavaVM_**) ()
#15 0x000055d9208cbfb1 in hbase::MiniCluster::Setup() ()  <== Locks mutex
#16 0x000055d9208cc87c in hbase::MiniCluster::env() ()
#17 0x000055d9208cd227 in hbase::MiniCluster::StartCluster(int) ()
#18 0x000055d9208c879b in hbase::TestUtil::StartMiniCluster(int) ()
#19 0x000055d920613076 in AsyncBatchRpcRetryTest::SetUpTestCase() ()
#20 0x000055d9208eaa48 in testing::TestCase::RunSetUpTestCase() ()
#21 0x000055d9208f7347 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::TestCase, void>(testing::TestCase*, void (testing::TestCase::*)(), char const*) ()
#22 0x000055d9208f1b19 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::TestCase, void>(testing::TestCase*, void (testing::TestCase::*)(), char const*) ()
#23 0x000055d9208d6647 in testing::TestCase::Run() ()
#24 0x000055d9208dd552 in testing::internal::UnitTestImpl::RunAllTests() ()
#25 0x000055d9208f7d29 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) ()
#26 0x000055d9208f21a5 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) ()
#27 0x000055d9208dc116 in testing::UnitTest::Run() ()
#28 0x000055d920904ce7 in RUN_ALL_TESTS() ()
#29 0x000055d920904c76 in main ()
{noformat}
Bug: Switch docker to use new build system
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jun/20 23:52
Affects Version/s: 3.0.0-alpha-1

Currently we download and build them manually [1], switch to use DOWNLOAD_DEPENDENCIES flag. 

[1] https://github.com/apache/hbase-native-client/blob/master/docker-files/Dockerfile
Bug: FsDelegationToken should cache Token
Status: Patch Available | Priority: Major | Resolution: 
Created: 20/May/20 05:18
Affects Version/s: 2.2.5

When doing Bulkload, we find that FsDelegationToken will acquire token of NameNode everytime for a single file, although, from the comment of acquireDelegationToken(), it claims that it is trying to find token in cache firstly, but the newly requested token is never put to cache and thus the cache is still empty for the following request;

In cases there are many files to do the bulk load, the token request will cause big short to NameNode.

 
{code:java}
public void acquireDelegationToken(final FileSystem fs)
 throws IOException {
 if (userProvider.isHadoopSecurityEnabled()) {
 this.fs = fs;

 userToken = userProvider.getCurrent().getToken("HDFS_DELEGATION_TOKEN",
 fs.getCanonicalServiceName());
 if (userToken == null) {
 hasForwardedToken = false;
 try {
 userToken = fs.getDelegationToken(renewer);
 } catch (NullPointerException npe) {
 // we need to handle NullPointerException in case HADOOP-10009 is missing
 LOG.error("Failed to get token for " + renewer);
 }
 } else {
 hasForwardedToken = true;
 LOG.info("Use the existing token: " + userToken);
 }
 }
}{code}
Bug: The way we bootstrap meta table is confusing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/May/20 10:19

I used to thought the work is only done in the InitMetaProcedure but in fact we have several pieces of code related to this which are placed in different classes.

In HRegionServer.initializeFileSystem, we will test whether we are HMaster. If so, we will call FSTableDescriptors.tryUpdateMetaTableDescriptor. And in MasterFileSystem.checkRootDir, we will check if meta region is available. If not, we will bootstrap it, and it will also call FSTableDescriptors.tryUpdateMetaTableDescriptor.

The problem here is that, HRegionServer.initializeFileSystem is called in the constructor of HRegionServer, which means it will be executed before MasterFileSystem.checkRootDir, as we will only create the MasterFileSystem for active master...

Ideally I prefer we move all the bootstrap code into InitMetaProcedure, which will be much cleaner. If this can not be done, at least we could move the code into MasterFileSystem.
Bug: Improve/stable read replica
Status: Open | Priority: Major | Resolution: 
Created: 12/Jun/20 23:27
Affects Version/s: 2.3.0

Tracing some read replica issues recently, this is the umbrella Jira to track this effort. A few observations so far:
 # balancer balances replica regions too often, need to spend time on it. Replica region does not serve write and rarely serve reads (unless the client specifically selects the replica region). So data locality should be a very minimum factor for replica regions. 
 # Need to study split/merge for regions with replica, need to make them more robust. With proc-v2, probably it is already robust.
Bug: Removal from streamReaders can be done in finally
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 05/May/20 06:00
Affects Version/s: 2.1.6

As part of the fix for https://issues.apache.org/jira/browse/HBASE-21551 we removed the storeReaders from the set inside the try block.
The code there seems to catch IOException. But if you drill down and see how the actual readers are closed - FSDataInputStreamWrapper#close() uses IOUtils.closeQuietly where we swallow the IOException (So ideally IOException will not be thrown). But there are cases we endup getting other type of RuntimeExceptions which may fail the close() and we endup not removing the storeReader from the Set. So it is safe to always remove it in finally.
Bug: release scripts should try to use a keyid when refering to GPG keys.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jun/20 16:53

Right now the template us substituting the release manager's email address in for the variable {{GPG_KEY}}. I think it doesn't hurt to make note of the email address, but what we really want here is the key's fingerprint, or some meaningly identifiable portion of it.
Bug: [Hadoop3.3] Add Hadoop-3.3.0-SNAPSHOT to hadoopcheck in our yetus personality
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 17/Apr/20 20:39

Since HBASE-23833 we're paying attention to our builds on Hadoop trunk, currently 3.3.0-SNAPSHOT. Let's add this version to the version lists in hadoopcheck so our CI will let us know when things break, at least compile-time anyway.
Bug: Upgrade to Apache ZooKeeper 3.5.7
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Apr/20 14:22
Affects Version/s: 2.2.3

Apache ZooKeeper 3.5.7 has been released, HDFS and other projects have updated the dependent zookeeper verison. Perhaps Hbase may update as well. Some of the interfaces are changed in this zookeeper version .
Bug: Try compact the recovered hfiles firstly after region online
Status: Open | Priority: Major | Resolution: 
Created: 23/Jun/20 06:06
Affects Version/s: 2.3.0

As discussed in HBASE-23739 and in HBASE-24632, there may have many recovered hfiles. Should find a better way to compact them firstly after region online.

 

For instance (quoting our [~anoop.hbase]):

"Assume there were some small files because of flush but never got compacted before the RS down happened. We will look for the possible candidate from oldest files and in all chance the very old files would get excluded because of the size math. But It is possible that new flushed files would get selected. And we have the max files to compact config also which is 10 by default. Even these small files count alone might be >10. If there are say 15 WAL files to split, for sure we will have at least 15 small HFiles.
My thinking was this. After the region open, we have to make sure these small files are compacted in one go and we should not even consider the max files limit for this compaction. Also to note that this files might not even have the DBE/compression etc being applied. Ya coding wise am not sure how clean it might come."

 

And from our [~pankaj2461]

 

"...concern is the compaction after region open, which impact MTTR due to heavy IO in large cluster with many outstanding WALs"
Bug: Flush partial stores of region filtered by seqId when archive wal due to too many wals
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/May/20 09:44
Affects Version/s: 2.4.0

When the logRoller archive the oldest wal due to too many wals, if a region should be flushed, we flush all stores of it, but it is not necessary, maybe we can use unflushedSeqId of store to filter them.
Bug: Make it so can make an hbase1 schema and hbase2 schema equate
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/May/20 23:51

Currently, NEW_VERSION_BEHAVIOR displays in hbase2 schemas with default value of 'false'. This default is added whenever we create a new column descriptor. Its not needed. We'll return default if empty or undefined. Meantime, since this is not defined in hbase1, it makes it so we can't make equivalent schemas so not possible to replicate from hbase1 to hbase2.
Bug: [WAL] Add a parameter to config RingBufferEventHandler's SyncFuture count
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jun/20 04:51
Affects Version/s: 1.4.13

The current value of RingBufferEventHandler's handler is the value of {{hbase.regionserver.handler.count}}, which works good in default wal provider --- one WAL per regionserver.

When trying to use WAL group provider, either by group or wal per region, the default value is bad. If rs has 100 regions and wal per region strategy is used, then rs will allocate 100 * SyncFuture[$hbase.regionserver.handler.count] array
{code}
int maxHandlersCount = conf.getInt(HConstants.REGION_SERVER_HANDLER_COUNT, 200);
    this.ringBufferEventHandler = new RingBufferEventHandler(
        conf.getInt("hbase.regionserver.hlog.syncer.count", 5), maxHandlersCount); 
...

----

RingBufferEventHandler(final int syncRunnerCount, final int maxHandlersCount) {
      this.syncFutures = new SyncFuture[maxHandlersCount];
      ...
 }
{code}
Bug: Relocate test-only REST "client" from src/ to test/ and mark Private
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Apr/20 00:43

Relocate test-only REST "client" from src/ to test/ and annotate as Private. The classes o.a.h.h.rest.Remote* were developed to facilitate REST unit tests and incorrectly committed to src/ . 

Although this "breaks" compatibility by moving public classes to test jar and marking them private, no attention has been paid to these classes with respect to performance, convenience, or security. Consensus from various discussions over the years is to move them to test/ as was intent of the original committer, but misplaced by the same individual.
Bug: Ensure StochasticLoadBalancer can log details of decision to not run balancer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Apr/20 12:38

Ran into a customer case where the StochasticLoadBalancer was consistently deciding not to balance when bringing new region servers on line. Even setting the class to TRACE logging would only log a summary statement like:

{code}
2020-04-03 00:29:55,133 TRACE org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Skipping load balancing because balanced cluster; total cost is 25.24853189705185, sum multiplier is 602.0 min cost which need balance is 0.05
{code}

Without any details about what went into that decision it's really hard to figure out what we need to tune to get the behavior we want.
Bug: Use a fair ReentrantReadWriteLock for the region close lock
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Apr/20 17:38

Consider creating the region's ReentrantReadWriteLock with the fair locking policy. We have had a couple of production incidents where a regionserver stalled in shutdown for a very very long time, leading to RIT (FAILED_CLOSE). The latest example is a 43 minute shutdown, ~40 minutes (2465280 ms) of that time was spent waiting to acquire the write lock on the region in order to finish closing it.

{quote}
...

Finished memstore flush of ~66.92 MB/70167112, currentsize=0 B/0 for region XXXX. in 927ms, sequenceid=6091133815, compaction requested=false at 1585175635349 (+60 ms)

Disabling writes for close at 1585178100629 (+2465280 ms)

{quote}

This time was spent in between the memstore flush and the task status change "Disabling writes for close at...". This is at HRegion.java:1481 in 1.3.6:

{code}
1480:   // block waiting for the lock for closing

1481:  lock.writeLock().lock(); // FindBugs: Complains UL_UNRELEASED_LOCK_EXCEPTION_PATH but seems fine
{code}
 
The close lock is operating in unfair mode. The table in question is under constant high query load. When the close request was received, there were active readers. After the close request there were more active readers, near-continuous contention. Although the clients would receive RegionServerStoppingException and other error notifications, because the region could not be reassigned, they kept coming, region (re-)location would find the region still hosted on the stuck server. Finally the closing thread waiting for the write lock became no longer starved (by chance) after 40 minutes.

The ReentrantReadWriteLock javadoc is clear about the possibility of starvation when continuously contended: "_When constructed as non-fair (the default), the order of entry to the read and write lock is unspecified, subject to reentrancy constraints. A nonfair lock that is continuously contended may indefinitely postpone one or more reader or writer threads, but will normally have higher throughput than a fair lock._"

We could try changing the acquisition semantics of this lock to fair. This is a one line change, where we call the RW lock constructor. Then:

 "_When constructed as fair, threads contend for entry using an approximately arrival-order policy. When the currently held lock is released, either the longest-waiting single writer thread will be assigned the write lock, or if there is a group of reader threads waiting longer than all waiting writer threads, that group will be assigned the read lock._" 

This could be better. The close process will have to wait until all readers and writers already waiting for acquisition either acquire and release or go away but won't be starved by future/incoming requests.

There could be a throughput loss in request handling, though, because this is the global reentrant RW lock for the region.
Bug: Enable CompactionTool executions on non-HDFS filesystems
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Apr/20 10:47

CompactionTool on non-HDFS filesystems does not work because the used filesystems are mixed up. To collect the StoreFiles the {{Filesystem.get(conf)}} method is used instead of relying on the root dir filesystem.

With YARN the delegation tokens are not handled correctly with a different filesystem and the mappers only get delegation token for the staging directory's filesystem. 

Another issue is in MapReduce mode when YARN is used. In this case the TMP directory ({{hbase.tmp.dir}}) is created under {{yarn.nodemanager.local-dirs}} but this directory is created (even on HDFS a regular user can't create this directory) on the Storefile's filesystem where a local filesystem temp directory shouldn't be used.

In this ticket I plan to clean up the used filesystems, remove the custom temp directory (use the regions' .tmp directory) and collect delegation tokens for staging dir + store dir paths' FileSystems.
Bug: Add an "ignoreTimestamps" option (defaulted to false) to HashTable/SyncTable tool
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/May/20 18:51
Affects Version/s: 2.2.5

Currently, when hashing and comparing values between a source and a target table, HashTable/SyncTable always consider cell timestamp values. However, cell timestamp values are not always relevant for client applications, so these use cases could benefit of a more flexible comparison logic where timestamps could be ignored.

For such scenarios, HashTable/SyncTable could have better performance, since cells with only timestamps diverging would not be copied. 

Another case that would benefit from this option is when bulk deletes are wrongly applied at target. At the moment, HashTable/SyncTable on it's own is not capable of syncing back the clusters, as the source Puts would have an older TS than the delete markers in the target. That would require target to complete major compaction on the whole table before HashTable/SyncTable could be run.