av9ash/GitBugs huggingface dataset:
Bug: shaded mapreduce and shaded byo-hadoop client artifacts contains no classes
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 20/Mar/22 14:07
Affects Version/s: 2.5.0

After fixing the logging problem in HBASE-26870, we could see the actual error.

{noformat}
/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hadoop-3/bin/hadoop --config /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/output-integration/hadoop-3/hbase-conf/ jar /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/lib/shaded-clients/hbase-shaded-mapreduce-2.6.0-SNAPSHOT.jar importtsv -Dimporttsv.columns=HBASE_ROW_KEY,family1:column1,family1:column4,family1:column3 test:example example/ -libjars /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/shaded-clients/hbase-shaded-mapreduce-2.6.0-SNAPSHOT.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/audience-annotations-0.5.0.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/commons-logging-1.2.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/htrace-core4-4.1.0-incubating.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/jcl-over-slf4j-1.7.33.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/jul-to-slf4j-1.7.33.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/opentelemetry-api-1.0.1.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/opentelemetry-context-1.0.1.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/opentelemetry-semconv-1.0.1-alpha.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/slf4j-api-1.7.33.jar
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
Exception in thread "main" java.lang.ClassNotFoundException: org.apache.hadoop.hbase.mapreduce.Driver
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:311)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
{noformat}
Bug: The way we bypass broken track file is not enough in StoreFileListFile
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 22/Jan/22 15:23

When implementing HBASE-26690, I found that a pb message can be parsed with partial bytes, if we stop at some special point.

So we can not rely on the pb parser to always throw InvalidProtocolBufferException for us, we need to detect the incomplete file by ourselves.
Bug: NPE caused by WAL Compression and Replication
Status: Resolved | Priority: Critical | Resolution: Won't Fix
Created: 16/Mar/22 07:40
Affects Version/s: 1.7.1

My cluster uses HBase 1.4.12, opened WAL compression and replication.

I could found replication sizeOfLogQueue backlog, and after some debugs, found the NPE throwed by [https://github.com/apache/hbase/blob/branch-1/hbase-common/src/main/java/org/apache/hadoop/hbase/io/util/LRUDictionary.java#L109:]

!image-2022-03-16-14-25-49-276.png!

 

The root cause for this problem is:
WALEntryStream#checkAllBytesParsed:

!image-2022-03-16-14-30-15-247.png!

resetReader does not create a new reader, the original CompressionContext and the dict in it will still be retained.
However, at this time, the position is reset to 0, which means that the HLog needs to be read from the beginning, but the cache that has not been cleared is still used, so there will be problems: the same data has already in the LRUCache, and it will be directly added to the cache again.
Recreate a new reader here, the problem is solved.

I will open a PR later. But, there are some other places in the current code to resetReader or seekOnFs. I guess these codes doesn't take into account the wal compression case at all...

 

In theory, as long as the file is read again, the LRUCache should also be rolled back, otherwise there will be inconsistent behavior of READ and WRITE links.
But the position can be roll back to any intermediate position at will, but LRUCache can't...
Bug: Snapshot is corrupted due to interaction between move, warmupRegion, compaction, and HFileArchiver
Status: Resolved | Priority: Critical | Resolution: Duplicate
Created: 29/Jan/22 01:21
Affects Version/s: 1.3.5

There is an interesting sequence of events which leads to split-brain, double-assignment type of behavior with management of store files.

The scenario is this:
# Take snapshot
# RegionX of snapshotted table is hosted on RegionServer1.
# Stop RegionServer1, using {{region_mover}}, gracefully moving all regions to other regionservers using {{move}} RPCs.
# RegionX is now opened on RegionServer2.
# RegionServer2 compacts RegionX after opening.
# RegionServer1 starts and uses {{region_mover}} to {{move}} all previously owned regions back to itself.
# The HMaster RPC to {{move}} calls {{warmupRegion}} on RegionServer1.
# As part of {{warmupRegion}}, RegionServer1 opens all store files of RegionX. CompactedHFilesDischarger chore has not yet archived the pre-compacted store file. RegionServer1 finds both the pre-compacted store file and post-compacted store file. It logs a warning and archives the pre-compacted file.
# RegionServer1 has warmed up the region, so now HMaster resumes the {{move}} and sends {{close}} RegionX to RegionServer2.
# RegionServer2 closes its store files. As part of this, it archives any compacted files which have not yet been archived by the {{CompactedHFilesDischarger}} chore.
# Since RegionServer1 already archived the file, RegionServer2's {{HFileArchiver}} finds the destination archive file already exists. (code link)
# RegionServer2 renames the archived file, to free up the desired destination filename.
With the archived file renamed, RegionServer2 attempts to archive the file as planned. But the source file doesn't exist because RegionServer1 already moved it... to the location RegionServer2 expected to use!
# RegionServer2 silently ignores this archival failure. (code link)
# HMaster {{HFileCleaner}} chore later deletes the renamed archive file, because there is no active reference to it. (The snapshot reference is to the original named file, not the "backup" timestamped version.) The snapshot data is irretrievably lost.

HBASE-26718 tracks a potential, specific change to the archival process to avoid this specific issue.

However, there is a more fundamental problem here that a region opened by {{warmupRegion}} can operate on that region's store files while the region is opened elsewhere, which must not be allowed.

This was seen on branch-1, and is a combination of HBASE-22330 and not having the fix for HBASE-22163.
Bug: ShortCircuitingClusterConnection fails to close RegionScanners when making short-circuited calls
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 08/Mar/22 06:34
Affects Version/s: 2.4.9

Just ran into this on the Phoenix side.
We retrieve a Connection via {{{}RegionCoprocessorEnvironment.createConnection... getTable(...){}}}. And then call get on that table. The Get's key happens to be local. Now each call to table.get() leaves an open StoreScanner around forever. (verified with a memory profiler).

There references are held via RegionScannerImpl.storeHeap.scannersForDelayedClose. Eventially the RegionServer goes into a GC of death and can only ended with kill -9.

The reason appears to be that in this case there is no currentCall context. Some time in 2.x the Rpc handler/call was made responsible for closing open region scanners, but we forgot to handle {{ShortCircuitingClusterConnection}}

It's not immediately clear how to fix this. But it does make ShortCircuitingClusterConnection useless and dangerous. If you use it, you *will* create a giant memory leak.
Bug: Wait on the future returned by FanOutOneBlockAsyncDFSOutput.flush would stuck
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 18/Jan/22 09:17
Affects Version/s: 2.4.9

Consider there are three dataNodes: dn1,dn2,and dn3, and we write some data to {{FanOutOneBlockAsyncDFSOutput}} and then flush it, there are one {{Callback}} in {{FanOutOneBlockAsyncDFSOutput.waitingAckQueue}}.  If the ack from dn1 arrives firstly and triggers Netty to invoke {{FanOutOneBlockAsyncDFSOutput.completed}} with dn1's channel, then in {{FanOutOneBlockAsyncDFSOutput.completed}}, dn1's channel is removed from {{Callback.unfinishedReplicas}}. 
But dn2 and dn3 respond slowly, before dn2 and dn3 sending ack , dn1 is shut down or have a exception, so {{FanOutOneBlockAsyncDFSOutput.failed}} is triggered by Netty with dn1's channel, and because the {{Callback.unfinishedReplicas}} does not contain dn1's channel,the {{Callback}} is skipped in {{FanOutOneBlockAsyncDFSOutput.failed}} method, just as following line250, and at line 245, {{FanOutOneBlockAsyncDFSOutput.state}} is set to {{State.BROKEN}}.
{code:java}
233  private synchronized void failed(Channel channel, Supplier<Throwable> errorSupplier) {
234     if (state == State.BROKEN || state == State.CLOSED) {
235         return;
236      }
     ....
244    // disable further write, and fail all pending ack.
245    state = State.BROKEN;
246    Throwable error = errorSupplier.get();
247    for (Iterator<Callback> iter = waitingAckQueue.iterator(); iter.hasNext();) {
248      Callback c = iter.next();
249      // find the first sync request which we have not acked yet and fail all the request after it.
250      if (!c.unfinishedReplicas.contains(channel.id())) {
251        continue;
252      }
253      for (;;) {
254        c.future.completeExceptionally(error);
255        if (!iter.hasNext()) {
256          break;
257        }
258        c = iter.next();
259      }
260    break;
261    }
262   datanodeInfoMap.keySet().forEach(ChannelOutboundInvoker::close);
263  }
{code}
At the end of above method in line 262, dn1,dn2 and dn3 are all closed, so the {{FanOutOneBlockAsyncDFSOutput.failed}} is triggered again by dn2 and dn3, but at the above line 234, because {{FanOutOneBlockAsyncDFSOutput.state}}  is already {{State.BROKEN}}, the whole  {{FanOutOneBlockAsyncDFSOutput.failed}}  is skipped. So wait on the future returned by {{FanOutOneBlockAsyncDFSOutput.flush}} would stuck for ever.

When we roll the wal, we would create a new {{FanOutOneBlockAsyncDFSOutput}} and a new {{AsyncProtobufLogWriter}}, in {{AsyncProtobufLogWriter.init}} we write wal header to {{FanOutOneBlockAsyncDFSOutput}} and wait it to complete. If we run into this situation, the roll would stuck forever.

I have simulate this case in the PR, and my fix is even through the  {{FanOutOneBlockAsyncDFSOutput.state}}  is already {{State.BROKEN}}, we would still try to trigger {{Callback.future}}
Bug: HFileLinkCleaner should be added even if snapshot is disabled
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 14/Jan/22 07:37

We encountered a problem in our cluster:
1. Cluster has many snapshots, cause the archive directory is too big.
2. We delete some snapshots, but the cleaner runs slowly because this is a race in synchronized method of SnapshotHFileCleaner.
3. We delete all snapshots, and disable snapshot feature(hbase.snapshot.enabled=false), so the cleaner will skip the synchronized method in SnapshotHFileCleaner.
4. After cleaner runs, some back reference and data files under archive directory are deleted, but they are still used by some restored tables. This does not meet expectations.

One solution is add HFileLinkCleaner even if snapshot is disabled.
Bug: Incorrect exception handling in shell
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 07/Feb/22 17:02
Affects Version/s: 2.4.10

The exception handling changed in the shell compared to 2.2.
{noformat}
➜  hbase-upstream git:(branch-2.4) ✗ cat commands.txt
scan 'foo'
exit
➜  hbase-upstream git:(branch-2.4) ✗ bin/hbase shell -n commands.txt
2022-02-07 16:21:33,654 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
hbase:001:0> scan 'foo'
ROW                                  COLUMN+CELL
Took 0.3890 seconds                                                                                                                         Traceback (most recent call last):
RuntimeError (Unknown table foo!)
hbase:002:0> exit
➜  hbase-upstream git:(branch-2.4) ✗ echo $?
0 {noformat}
The execution continues even after an exception is thrown. In 2.2.7 the execution stops when an exception is thrown and the shell exits with error code.
{noformat}
➜  hbase-2.2.7 bin/hbase shell -n commands.txt
2022-02-07 16:33:54,930 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
ROW                                  COLUMN+CELL
Took 0.3082 seconds                                                                                                                         RuntimeError: Unknown table foo!
  translate_hbase_exceptions at /Users/petersomogyi/tmp/hbase-2.2.7/lib/ruby/shell/commands.rb:130
                command_safe at /Users/petersomogyi/tmp/hbase-2.2.7/lib/ruby/shell/commands.rb:49
            internal_command at /Users/petersomogyi/tmp/hbase-2.2.7/lib/ruby/shell.rb:148
                     command at /Users/petersomogyi/tmp/hbase-2.2.7/lib/ruby/shell.rb:140
                        scan at (eval):2
                      <main> at commands.txt:1
                        load at org/jruby/RubyKernel.java:973
                      <main> at /Users/petersomogyi/tmp/hbase-2.2.7/bin/../bin/hirb.rb:186
➜  hbase-2.2.7 echo $?
1 {noformat}
Bug: LoadBalancer should not return empty map
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 03/Jan/22 15:52
Affects Version/s: 2.5.0

After HBASE-26641, these two tests are still flaky, they also fail for me locally sometimes.

After digging, I found that, while we see this in the log output after master failover

{noformat}
Wanted to do round robin assignment but no servers to assign to
{noformat}

The UTs will fail.

Need to dig more.
Bug: Automatically add default security headers to http/rest if SSL enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Mar/22 13:01
Affects Version/s: 2.0.6

In the previous ticket https://issues.apache.org/jira/browse/HBASE-23303 we implemented these security headers as optional which had to explicitly enabled in the config.

With this change the headers will automatically be added with meaningful default values if SSL is enabled.
Bug: Scan#setRowPrefixFilter incorrectly marked as deprecated
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Feb/22 14:03
Affects Version/s: 3.0.0-alpha-2

Issue HBASE-25299 is about the Scan#setRowPrefixFilter having not the expected effects when using also the setStartRow and/or setStopRow.

The setRowPrefixFilter is intended to make setting the right startRow and stopRow for prefix scans. So it was always intended of using it *instead* of doing separate setStartRow and/or setStopRow calls.

The reason for adding this to the Scan (~ 8 years ago, HBASE-11990 ) was that calculating the correct start and stop row for doing the very efficient prefix scan is very very hard to do right in the generic case.

This is still a current usecase and many of my applications rely on this function (I even prefer the HBase Client when connecting to Google BigTable for this method).

With issue HBASE-25299 it has now been marked as deprecated.
Quote from the changes made in HBASE-25299:

{code}
@deprecated since 3.0.0. The scan result might be unexpected in some cases.
{code}
Yes, if you use this very valuable method incorrectly it will yield incorrect results.
I do not consider this a valid reason to deprecate it.

So I do agree with the confusion of the effects as described in HBASE-25299 which should be fixed with additional documentation.

I disagree with the deprecation of this method.
 

I'll put up a pull request in which I improve the documentation and remove the deprecation.
Bug: VerifyReplication recompare async
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Mar/22 13:51

VerifyReplication includes an option "sleepMsBeforeReCompare". This is useful for helping work around replication lag. However, adding a sleep in a hadoop job can drastically slow that job down if there is anything more than a small number of invalid results.

We can mitigate this by doing the recompare in a separate thread. We can limit the thread pool and fallback to doing the recompare in the main thread if the thread pool is full. This way we offload some of the slowness but still retain the same validation guarantees. A configuration can be added to control how many threads per mapper.
Bug: Replace log4j with reload4j for hbase-connector
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 15/Mar/22 02:46
Affects Version/s: 1.0.0

As a result of the [discussion thread of replacing log4j with reload4j|https://lists.apache.org/thread/kfmjg6zmvdjqcwolj0oh634nzv42y806] and HBASE-26691, we have replaced log4j with reload4j in branch-2. 

Where hbase-connector as a part of the community, this JIRA propose to align with the active hbase release and replace log4j with reload4j before the next release of hbase-connector
Bug: Report client backoff time for server overloaded in ConnectionMetrics
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Mar/22 22:16

Servers can throw CallQueueTooBigException and CallDroppedException when overloaded. As of HBASE-26807, these can have a configurable extra backoff to allow the server to recover. Depending on the server side queue implementation, different callers may be more or less impacted by server load. It is very useful to measure in the client how much time is spent backing off so we can see which clients are most affected by server overload.
Bug: The implementation of TestMergesSplitsAddToTracker is problematic
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jan/22 16:23

As it will clear the trackedFiles map in TestStoreFileTracker for every test but we will also store the files for system tables, so it will cause NPE. Not sure why on master the it will not fail but on branch-2 at least testSplitLoadsFromTracker will fail...
Bug: Data race on Compactor.writer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jan/22 15:09

In Compactor.getCompactionTargets, we use synchronized on the writer instance but it does not help resolve the data race problem, because in other places we do not use synchronized...

And since it is not a big deal to the upper layer an old snapshot, we could just make the writer instance volatile and use it directly in the method, just make sure we always use the same instance in the method, i.e, assign it to a local var first.
Bug: Backport HBASE-25368 to branch-2 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Mar/22 10:41
Affects Version/s: 2.5.0

HBASE-26640 introduced two extra paths under master:store table: ".initializing" and ".initialized", in order to control when such store has been completed started for SFT.

Problem is that TestHFileProcedurePrettyPrinter uses RegionInfo.isEncodedRegionName to determine if a given child path in the table dir is a valid region dir. Current implementation for RegionInfo.isEncodedRegionName considers ".initializing" and ".initialized" as valid region encoded names, thus the test ends up picking one of the flag dirs to list hfiles that should had been modified when validating the test outcome.

Further improvements have been made to RegionInfo.isEncodedRegionName in HBASE-25368 to proper validate region names, but those weren't backported to branch-2.
Bug: [hbase-thirdparty] Introduce a hbase-unsafe module in hbase-thirdparty to remove the direct references of Unsafe in our main code base
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Feb/22 03:28

For java 9+, sun.misc.Unsafe is in the module jdk.unsupported. When compiling with java 11 --release 8, these symbols will not be exported and lead to compile errors.

There is a jdk issue

https://bugs.openjdk.java.net/browse/JDK-8214165

But seems the jdk team does not want to fix it. So if we want to move up to java 11 but still keep the java 8 compatibility, a possible choice is to remove the direct reference of sun.misc.Unsafe so at compile time there will be no problem, and at runtime we export the jdk.unsupported so there will be no problem too.

For more details, please see the discussion of this thread on the dev mailing list.

https://lists.apache.org/thread/w5lrxkhswlonj09xf9hcwgvck3nsjdfx
Bug: Increase the timeout for TestStochasticLoadBalancerRegionReplicaLargeCluster
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jan/22 08:26
Affects Version/s: 2.5.0

TestStochasticLoadBalancerRegionReplicaLargeCluster is on the flaky list for branch-2, it fails 50%+.

Looking at the output, sometimes it can not finish all the calculation in time, so let's see if increasing the timeout can help here.
Bug: HFileBlock.verifyOnDiskSizeMatchesHeader throw IOException: Passed in onDiskSizeWithHeader= A != B
Status: Open | Priority: Major | Resolution: 
Created: 01/Mar/22 14:51
Affects Version/s: 2.2.2

When I scan a region, HBase throw IOException: Passed in onDiskSizeWithHeader= A != B

The HFile mentioned Error message can be access normally.

it recover by command – move region. I guess that onDiskSizeWithHeader of HFileBlock has been changed. And RS get the correct BlockHeader Info after region reopened.
Bug: HBASE-26664 hbase-connector upgrades extra-enforcer-rules to 1.5.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/22 20:02
Affects Version/s: hbase-connectors-1.1.0

hbase-connector fails when building with -Prelease after maven-enforcer-plugin upgraded to 3.0.0 from 3.0.0-M3 in HBASE-26534. 

We upgraded maven-enforcer-plugin from 3.0.0-M3 to 3.0.0, but then we hit the problem of  `extra-enforcer-rules` has different as mentioned in https://github.com/mojohaus/extra-enforcer-rules/issues/131#issuecomment-890329715


{code}
$ mvn clean install -Prelease

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0:enforce (enforce-maven-version) on project hbase-connectors: Unable to parse configuration of mojo org.apache.maven.plugins:maven-enforcer-plugin:3.0.0:enforce for parameter enforceBytecodeVersion: Cannot create instance of class org.apache.maven.plugins.enforcer.EnforceBytecodeVersion: org/apache/maven/shared/dependency/tree/DependencyTreeBuilderException: org.apache.maven.shared.dependency.tree.DependencyTreeBuilderException -> [Help 1]
{code}

https://github.com/mojohaus/extra-enforcer-rules/commit/d380fe56a7317a237bbce5b94fbab7f4581d3d1b