av9ash/GitBugs huggingface dataset:
Bug: Added ignored test for async connection that runs retries just so can check how long it takes and that retrying is happening
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 08/Feb/21 20:04

There's an old @Ignored test in TestClientNoCluster that has been useful from time to time figuring how long N retries takes and reproducing certain production 'Connection rejected', etc., stack traces. This issue adds an @Ignored method for the async connection case so can do similar for async (I needed it just now because it looked like retries at a certain juncture were not happening -- turns out they are... this addition helped me prove this the case).
Bug: [Documentation] Fix format error in security.adoc
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 10/Mar/21 02:33
Affects Version/s: 3.0.0-alpha-1

h3. 60.6. Configure the Thrift Gateway to Use the {{doAs}} Feature

...

*To enable the {{doAs}} feature*, add the following to the _hbase-site.xml_ file for every Thrift gateway:

  !image-2021-03-10-10-36-30-765.png!

The symbol “<” is missing after "ture".
Bug: Increase timeout for HBase-Flaky-Tests on branch-2.2
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 23/Feb/21 17:07
Affects Version/s: 2.2.7

The flaky test list got horrible on branch-2.2 and the HBase-Flaky-Tests can't finish in the 2 hours timeout so the list keeps increasing.

Change the timeout on this branch to 4 hours so the job could finish.
Bug: May throw StringIndexOutOfBoundsException when construct illegal tablename error
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 15/Jan/21 07:52
Affects Version/s: 1.2.12

When call Method:
{code:java}
// code placeholder
TableName.isLegalTableQualifierName(final byte[] qualifierName, int start, int end)
{code}
We want to construct quelifier String to print pretty error message. So we call method: 
{code:java}
// code placeholder
Bytes.toString(final byte[] b, int off, int len)
Bytes.toString(qualifierName, start, end)
{code}
But the param is wrong, we shoud pass *${Length}* instead of *${end index}* to *Bytes.toString third param*.
Bug: Let CPEPs also use our shaded protobuf
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 05/Feb/20 01:22

See this discussion thread:

https://lists.apache.org/thread.html/abd60a8985a4898bae03b2c3c51d43a6b83d67c00caff82ba9ab2712%40%3Cdev.hbase.apache.org%3E
Bug: Data loss when WAL split to HFile enabled
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 27/Jan/20 06:30
Affects Version/s: 2.3.0

Very simple steps as below,
1. Create table with 1 region
2. Insert 1 record 
3. Flush the table 
4. Scan table and observe timestamp of the inserted row
5. Insert same row key with same timestamp as previously inserted but with different value
6. Kill -9 RS where table region is online
7. Start RS

Scan the table and check the result, latest cell must be returned.

Thanks [~sreenivasulureddy] for finding this issue.
Bug: TestQuotasShell is broken
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 30/Mar/20 01:58

{noformat}
===============================================================================
^[[41;37;1mFailure^[[0m: <false> is not true.
test_can_set_and_remove_quota(Hbase::SpaceQuotasTest)
src/test/ruby/hbase/quotas_test.rb:183:in `block in test_can_set_and_remove_quota'
     180:     define_test 'can set and remove quota' do
     181:       command(:set_quota, TYPE => SPACE, LIMIT => '1G', POLICY => NO_INSERTS, TABLE => @test_name)
     182:       output = capture_stdout{ command(:list_quotas) }
^[[41;37;1m  => 183:       assert(output.include?("LIMIT => 1G"))^[[0m
     184:       assert(output.include?("VIOLATION_POLICY => NO_INSERTS"))
     185:       assert(output.include?("TYPE => SPACE"))
     186:       assert(output.include?("TABLE => #{@test_name}"))
===============================================================================
{noformat}

At least on branch-2.1.
Bug: Increment proto conversion is broken
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 10/Feb/20 22:14
Affects Version/s: 1.1.11

While converting the request back to Increment using ProtobufUtil.toIncrement we incorrectly use the optimization to avoid copying the byte array(HBaseZeroCopyByteString#zeroCopyGetBytes) on a BoundedByteString. The optimization was only meant for LiteralByteString where it is safe to use the backing byte array, however it ends up being used to BoundedByteString which is a subclass of LiteralByteString. This essentially breaks increments since we end up creating wrong cells on the server side.
Bug: Switch central repo to https on branch-1
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 15/Jan/20 23:35
Affects Version/s: 1.6.0

Maven central started enforcing https [1] today and branch-1 builds are failing presumably due to an older maven version. 

Maven 3.2.3 [2] and later already switched the defaults, so not every one might run into this problem, depending on what mvn they are using. Jenkins jobs are hitting this, so they are likely using an older version.

[1] https://blog.sonatype.com/central-repository-moving-to-https
[2] https://issues.apache.org/jira/browse/MNG-5672
Bug: Move the unsupported procedure type check before migrating to RegionProcedureStore
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 06/Jan/20 17:03
Affects Version/s: 2.2.2

Per [discussion|https://lists.apache.org/thread.html/c5f960e6cf3c89ffbf8f8302afb121462a08fd9a89d384c6795a7b81%40%3Cdev.hbase.apache.org%3E] on the list, let's figure out how to make the upgrade from a v2.1 procedure store less error-prone. Could be a simple as documenting runbook steps to execute during the rolling upgrade, but it would be nice if the software could roll over the data versions gracefully.
Bug: Old config hbase.hstore.compactionThreshold is ignored
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 12/Feb/20 18:26
Affects Version/s: 2.0.0

In 2.x we added new name 'hbase.hstore.compaction.min' for this.  Still for compatibility we allow the old config name and honor that in code
{code}
minFilesToCompact = Math.max(2, conf.getInt(HBASE_HSTORE_COMPACTION_MIN_KEY,
          /*old name*/ conf.getInt("hbase.hstore.compactionThreshold", 3)));
{code}
But if hbase.hstore.compactionThreshold alone is configured by user, there is no impact of that.

This is because in hbase-default.xml we have the new config with a value of 3. So the call conf.getInt(HBASE_HSTORE_COMPACTION_MIN_KEY) always return a value 3 even if it is not explicitly configured by customer and instead used the old key.
Bug: HMaster does not restart when rsgroup is enabled and /hbase/WALs is moved
Status: Resolved | Priority: Critical | Resolution: Won't Fix
Created: 18/Mar/20 10:08
Affects Version/s: 2.2.3

HMaster does not restart when rsgroup is enabled and /hbase/WALs is moved

HMaster restarts properly if rsgroup is not enabled even if /hbase/WALs is moved.

Steps to reproduce:
 # start the cluster
 # create a table do some put, delete
 # kill all the region servers and master
 # move WALs directory for backup (-mv /hbase/WALs /hbase/WALs2)
 # start the cluster
 # Master start fails, initialization keep failing
{code:java}
2020-03-18 11:42:55,369 ERROR [ActiveMasterInitializationMonitor-1584511075369] master.HMaster: Master failed to complete initialization after 900000ms. Please consider submitting a bug report including a thread dump of this process.
{code}
Bug: HBCK2 feature negotiation to check what commands are supported
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 24/Mar/20 08:42

Each HBase version is shipped with a specific version of HBCK2 therefore we can check the HBCK2 version and decide if the given command is supported in that version or not.
 It is not that easy to decide if the command is supported or not when dealing with back-ported features.
 It the conclusion of this I would recommend using feature negotiation to check what commands are supported on the server side instead of comparing versions.
Bug: Find a way to handle the corrupt recovered hfiles
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 03/Jan/20 09:08
Affects Version/s: 2.3.0

Copy the comment from PR review.

 

If the file is a corrupt HFile, an exception will be thrown here, which will cause the region to fail to open.
Maybe we can add a new parameter to control whether to skip the exception, similar to recover edits which has a parameter "hbase.hregion.edits.replay.skip.errors";

 

Regions that can't be opened because of detached References or corrupt hfiles are a fact-of-life. We need work on this issue. This will be a new variant on the problem -- i.e. bad recovered hfiles.

On adding a config to ignore bad files and just open, thats a bit dangerous as per @infraio .... as it could mean silent data loss.
Bug: NPE HFilesOutputSink
Status: Resolved | Priority: Critical | Resolution: Duplicate
Created: 13/Jan/20 18:18
Affects Version/s: 2.3.0

Enabling the new split to hfiles feature, HBASE-23286, running branch-2 tip, I see this out on RegionServers:

{code}
 2020-01-13 17:37:08,204 INFO org.apache.hadoop.hbase.wal.OutputSink: 3 split writer threads finished
 2020-01-13 17:37:08,233 INFO org.apache.hadoop.hbase.wal.WALSplitter: Processed 1007 edits across 0 regions cost 284 ms; edits skipped=76; WAL=hdfs://nameservice1/hbase/genie/WALs/hbasedn101.example.org,16020,1578934806382-splitting/hbasedn101.example.org%2C16020%2C1578934806382.1578937008832, size=128.5 M, length=134708720, corrupted=false, progress         failed=true
 2020-01-13 17:37:08,234 WARN org.apache.hadoop.hbase.regionserver.SplitLogWorker: log splitting of WALs/hbasedn101.example.org,16020,1578934806382-splitting/hbasedn101.example.org%2C16020%2C1578934806382.1578937008832 failed, returning error
 java.io.IOException: java.lang.NullPointerException
         at org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.writeRemainingEntryBuffers(BoundedRecoveredHFilesOutputSink.java:173)
         at org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.close(BoundedRecoveredHFilesOutputSink.java:140)
         at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:339)
         at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:181)
         at org.apache.hadoop.hbase.regionserver.SplitLogWorker.splitLog(SplitLogWorker.java:105)
         at org.apache.hadoop.hbase.regionserver.SplitLogWorker.lambda$new$0(SplitLogWorker.java:84)
         at org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:70)
         at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
         at java.base/java.lang.Thread.run(Thread.java:834)
 Caused by: java.lang.NullPointerException
         at org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.configContextForNonMetaWriter(BoundedRecoveredHFilesOutputSink.java:225)
         at org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.createRecoveredHFileWriter(BoundedRecoveredHFilesOutputSink.java:213)
         at org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.append(BoundedRecoveredHFilesOutputSink.java:117)
         at org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.lambda$writeRemainingEntryBuffers$3(BoundedRecoveredHFilesOutputSink.java:155)
         at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
         at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
         at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
{code}

It is a bit odd because log says there were zero regions. Not sure what that was about.
Bug: Split failure may cause region hole and data loss when use zk assign
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 15/Jan/20 12:15
Affects Version/s: 1.4.8

to mock this case, I add a sleep code in SplitTransactionImpl.excute after the PONR and before openDaughters:
{code:java}
public PairOfSameType<Region> execute(final Server server,
      final RegionServerServices services, User user) throws IOException {
    this.server = server;
    this.rsServices = services;
    useZKForAssignment = server == null ? true :
      ConfigUtil.useZKForAssignment(server.getConfiguration());
    if (useCoordinatedStateManager(server)) {
      std =
          ((BaseCoordinatedStateManager) server.getCoordinatedStateManager())
              .getSplitTransactionCoordination().getDefaultDetails();
    }
    PairOfSameType<Region> regions = createDaughters(server, services, user);
    if (this.parent.getCoprocessorHost() != null) {
      if (user == null) {
        parent.getCoprocessorHost().preSplitAfterPONR();
      } else {
        try {
          user.getUGI().doAs(new PrivilegedExceptionAction<Void>() {
            @Override
            public Void run() throws Exception {
              parent.getCoprocessorHost().preSplitAfterPONR();
              return null;
            }
          });
        } catch (InterruptedException ie) {
          InterruptedIOException iioe = new InterruptedIOException();
          iioe.initCause(ie);
          throw iioe;
        }
      }
    }
    
    //sleep here!!!
    try {
      Thread.sleep(1000 * 60 * 60);
    } catch (InterruptedException e) {
      e.printStackTrace();
    }

    regions = stepsAfterPONR(server, services, regions, user);

    transition(SplitTransactionPhase.COMPLETED);

    return regions;
  }
{code}
so the split transaction will hang.

then i try to reproduce this problem:

1.Create a test table and move it into a test rsgroup, there is only 1 RS in the test group

2.Trigger a region split

3.The split transaction step after the PONR and sleep, regioninfo in meta has been updated

4.Kill the RS process to mock machine crash

5.ServerCrashProcedure cleanup SPLITING_NEW region, the daughter regions will be deleted

6.ServerCrashProcedure try to assign the parent region, because RS is down and assign fails, the region status is set to FAILED_OPEN and put back into regionsInTransition. But at this time, due to RS crash, the node of the region under ZK region-in-transition no longer exist

7.CatalogJanitor thread is blocked due to RIT

8.Switch active master

9.The CatalogJanitor thread on the new master executes normally and the parent region is cleaned up because split = true && offline = true in the meta table

10.We have a hole in the test table and loss data.

 

I modified the code when ServerCrashProcedure cleans up the child regions, it will update the parent regioninfo in the meta table, and this problem is no longer reproduced.


I will upload the patch later.
Bug: FileSystem instance leaks due to bulk loads with Kerberos enabled
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 10/Jan/20 19:52

Spent the better part of a week chasing an issue on HBase 2.x where the number of DistributedFileSystem instances on the heap of a RegionServer would grow unbounded. Looking at multiple heap-dumps, it was obvious to see that we had an immense number of DFS instances cached (in FileSystem$Cache) for the same user, with the unique number of Tokens contained in that DFS's UGI member (one hbase delegation token, and two HDFS delegation tokens – we only do this for bulk loads). For the user's clusters, they eventually experienced 10x perf degradation as RegionServers spent all of their time in JVM GC (they were unlucky to not have RegionServers crash outright, as this would've, albeit temporarily, fixed the issue).

The problem seems to be two-fold with changes by HBASE-15291 being largely the cause. This issue tried to close FileSystem instances which were being leaked – however, it did this by instrumenting the method {{SecureBulkLoadManager.cleanupBulkLoad(..)}}. Two big issues with this approach:
 # It relies on clients to call this method (client's hanging up will leak resources in RegionServers)
 # This method is only called on the RegionServer hosting the first Region of the table which was bulk-loaded into. For multiple RegionServers, they are left to leak resources.

HBASE-21342 later tried to fix an issue where FS objects were now being closed prematurely via reference-counting (which appears to work fine), but does not address the other two issues above. Point #2 makes debugging this issue harder than normal because it doesn't manifest on a single node instance :)

Through all of this, I (re)learned the dirty history of UGI and how its caching doesn't work so great HADOOP-6670. I see trying to continue to leverage the FileSystem$CACHE as a potentially dangerous thing (we've been back here multiple times already). My opinion at this point is that we should cleanly create a new FileSystem instance during the call to {{SecureBulkLoadManager#secureBulkLoadHFiles(..)}} and close it in a finally block in that same method. This both simplifies the lifecycle of a FileSystem instance in the bulk-load codepath but also helps us avoid future problems with UGI and FS caching. The one downside is that we pay the penalty to create a new FileSystem instance, but I'm of the opinion that we cross that bridge when we get there.

Thanks for [~jdcryans] and [~busbey] for their help along the way.
Bug: The "HBase Nightly" job may exhaust all executors - and create a deadlock
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Mar/20 11:03

What seems to happen is:
* the job needs more and more executors to run
* in case there are several of them launched in parallel (different branches) say 5
* they take up all available executors labeled "Hadoop" and then they wait for more indefinetly

I've killed a few builds...see my notes here:

https://the-asf.slack.com/archives/C7M1AFYLS/p1584442366000900
Bug: Region History Redo
Status: Open | Priority: Major | Resolution: 
Created: 30/Jan/20 20:56

My initial thought is mainly to extend the HBase shell(gradually extend to CLI and UI) to allow the use of 
{code:java}
where <regionid>{code}
to get the necessary information and allow passing history as an additional parameter to get the history. We can configure how many transitions we want to store so that anything (ZK or small data structure in the table region itself or maybe something else) that is used for state management is not exploded.
As pointed by
[~andrew.purtell@gmail.com] need to be watchful of mistakes done in the past: HBASE-533
Bug: Netty SASL implementation does not wait for challenge response causing TestShadeSaslAuthenticationProvider failures
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Feb/20 18:26
Affects Version/s: 2.3.0

TestShadeSaslAuthenticationProvider now fails deterministically with the following exception..

{noformat}
java.lang.Exception: Unexpected exception, expected<org.apache.hadoop.hbase.DoNotRetryIOException> but was<java.io.IOException>
	at org.apache.hadoop.hbase.security.provider.example.TestShadeSaslAuthenticationProvider.testNegativeAuthentication(TestShadeSaslAuthenticationProvider.java:233)
{noformat}

The test now fails a different place than before merging HBASE-18095 because the RPCs are also a part of connection setup. We might need to rewrite the test..
Bug: Spotbugs warning complain on master build
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Mar/20 00:14
Affects Version/s: 3.0.0-alpha-1

Spotbugs currently complains on every master build with an extant warning related:

{code}
Return value of putIfAbsent is ignored, but node is reused in org.apache.hadoop.hbase.master.assignment.RegionStates.createRegionStateNode(RegionInfo)
{code}
Bug: Backport HBASE-23146 (Support CheckAndMutate with multiple conditions) to branch-1
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 06/Mar/20 23:32

Backport HBASE-23146 (Support CheckAndMutate with multiple conditions) to branch-1, including updates to REST (HBASE-23924) and Thrift (HBASE-23925).
Bug: Use less resources running tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Mar/20 15:40

Our tests can create thousands of threads all up in the one JVM. Using less means less memory, less contention, likelier passes, and later, more possible parallelism.

I've been studying the likes of TestNamespaceReplicationWithBulkLoadedData to see what it does as it runs (this test puts up 4 clusters with replication between). It peaks at 2k threads. After some configuration and using less HDFS, its possible to get it down to ~800 threads and about 1/2 the memory-used. HDFS is a main offender. DataXceivers (Server and Client), jetty threads, Volume threads (async disk 'worker' then another for cleanup...), image savers, ipc clients -- new thread per incoming connection w/o bound (or reuse), block responder threads, anonymous threads, and so on. Many are not configurable or boundable or are hard-coded; e.g. each volume gets 4 workers regardless. Biggest impact was just downing the count of data nodes. TODO: a follow-on that turns down DN counts in all tests.

I've been using Java Flight Recorder during this study. Here is how you get a flight recorder for the a single test run: \{code:java} MAVEN_OPTS=" -XX:StartFlightRecording=disk=true,dumponexit=true,filename=recording.jfr,settings=profile,path-to-gc-roots=true,maxsize=1024m" mvn test -Dtest=TestNamespaceReplicationWithBulkLoadedData -Dsurefire.firstPartForkCount=0 -Dsurefire.secondPartForkCount=0 \{code} i.e. start recording on mvn launch, bound the size of the recording, and have the test run in the mvn context (DON'T fork). Useful is connecting to the running test at the same time from JDK Mission Control. We do the latter because the thread reporting screen is overwhelmed by the count of running threads and if you connect live, you can at least get a 'live threads' graph w/ count as the test progresses. Useful. When the test finishes, it dumps a .jfr file which can be opened in JDK MC.

I've been compiling w/ JDK8 and then running w/ JDK11 so I can use JDK MC Version 7, the non-commercial latest. Works pretty well. Let me put up a patch for tests that cuts down thread counts where we can.

Let me put up a patch that does first pass on curtailing resource usage.
Bug: [Documentation] Doc the dead servers multicast feature
Status: Closed | Priority: Major | Resolution: Not A Problem
Created: 15/Feb/20 17:31

Setting a config on master enables the multicast feature where master will udp broadcast to all listeners -- clients -- when it learns of dead servers. Clients that get notification react by redoing their region locations cache which helps them be more response when servers die. This feature has been around for years but probably goes unknown since no mention in documentation. This issue is about adding a basic paragraph on what it does and how to use it. The original issue has a nice release note that could serve as the basis for this paragraph. See HBASE-7590