av9ash/GitBugs huggingface dataset:
Bug: `hbase.hstore.flusher.count` setting to 0 breaks HMaster
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jan/20 22:43

Summary 

The configuration parameter `hbase.hstore.flusher.count` allows `0` (which turns out to be a bad value after I understand the source code).

When we set `hbase.hstore.flusher.count` to 0, HMaster will fail in 200,000 ms without any error message, but

```
 java.lang.RuntimeException: Master not initialized after 200000ms

```

 

There’s no messages pinpointing `hbase.hstore.flusher.count` being the root cause.

 

0 means `MemStoreFlusher` is not alive. During the initialization time of HMaster, `isHealthy()` will be called and it will try to check if `MemStoreFlusher` is alive or not.  However, since we set `hbase.hstore.flusher.count` to 0, there will be no flusher created at the first place, which cause`isHealthy()` to return false. As a consequence, HRegionServer does not enter its main loop.

 

*How To Reproduce* 

 

The version of HBase: 2.2.3

 
 # Set the value of configuration parameter `hbase.hstore.flusher.count` to `0` in “conf/hbase-site.xml”
 # Start hbase by running “bin/start-hbase.sh”
 # Wait for 3-4 minutes
 # Open the log file “logs/hbase-<username>- master -<pcname>.log”
Bug: Wrong combination of hlog reader and writer could crash regionserver
Status: Patch Available | Priority: Major | Resolution: 
Created: 10/Mar/20 20:47

*Problem*

I found out that there two configuration parameters: `hbase.regionserver.hlog.reader.impl` and `hbase.regionserver.hlog.writer.impl` which should be used as a pair.

There are two options for the reader: `ProtobufLogReader` and `SecureProtobufLogReader`.

And four options for the writer: `ProtobufLogWriter`, `SecureProtobufLogWriter`, `AsyncProtobufLogWriter` and `SecureAsyncProtobufLogWriter`.

Using the wrong combination of the reader and the writer will throw an exception. For example, if setting the writer to SecureProtobufLogWriter and setting the reader to ProtobufLogReader(default value), an exception will be thrown:
{code:java}
java.io.IOException: Got unknown writer class: SecureProtobufLogWriter
    at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initInternal(ProtobufLogReader.java:224)
    at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initReader(ProtobufLogReader.java:173)
    at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:65)
    at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:168)
    at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:321)
    at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:303)
    at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:291)
    at org.apache.hadoop.hbase.wal.LogRecoveredEditsOutputSink.deleteOneWithFewerEntries(LogRecoveredEditsOutputSink.java:109)
    at org.apache.hadoop.hbase.wal.LogRecoveredEditsOutputSink.closeWriter(LogRecoveredEditsOutputSink.java:240)
    at org.apache.hadoop.hbase.wal.LogRecoveredEditsOutputSink$2.call(LogRecoveredEditsOutputSink.java:196)
    at org.apache.hadoop.hbase.wal.LogRecoveredEditsOutputSink$2.call(LogRecoveredEditsOutputSink.java:192)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748){code}
The current description of these two parameters (in the `hbase-default.xml`) doesn’t tell the users anything about which combination is right or not.

 

*Improvement*

We can add some more information in the description of these two parameters in `hbase-default.xml` to tell the users about the correct combinations.

I upload a patch for improving the description.
Bug: Master async shutdown rpc call may not complete
Status: Open | Priority: Major | Resolution: 
Created: 27/Mar/20 19:22

Debugging the parent issue and discussion over in the adjacent test fix issue HBASE-24052 and in its PR exposes a problem calling shutdown asynchronously. The shutdown runs inline with the rpc call which should change (as suggested by [~bharathv]). Quoting from a Bharath comment in the the HBASE-24052 PR:

bq. I think internally it is masking the problem of "shutdown()" implementation being synchronous on the server side. Ideally, I'd think it should be async on the server side too, meaning the rpc call just sets a flag (startShuttingDown = true) which is read by multiple running threads and they begin the tear down rather than doing it in the context of the rpc thread. We already do this partially but the main driver of master shutdown still runs in the rpc context.
Bug: Track sensitive resources to ensure they are closed and assist devs in finding leaks.
Status: Resolved | Priority: Major | Resolution: Information Provided
Created: 01/Mar/20 20:27

Closing some objects is quite critical. Issues with leaks can be quite slippery and nasty and growy. Maintaining close integrity is an embarrassing sport for humans.

In the past, those 3 thoughts led me to start tracking objects in tests to alert of leaks. Even with an alert though, the job of tracking down all of the leaks just based on what leaked was beyond my skill. If it's beyond even one devs skill that is committing, that tends to end up trouble. So I added the stack trace for the origin of the object. Things can still get a bit tricky to track down in some cases, but now I had the start of a real solution to all of the whack-a-mole games I spent too much time playing.
Bug: [Flakey Test] TestMasterShutdown#testMasterShutdownBeforeStartingAnyRegionServer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Feb/20 01:21
Affects Version/s: 2.3.0

Reproduces locally from time to time. Not much to go on here. Looks like the test is trying to do some fancy HBase cluster initialization order on top of a mini-cluster. Failure seems related to trying to start the HBase master before HDFS is fully initialized.
Bug: Add hbck2 command to "completebulkload" a batch of orphaned regions
Status: Open | Priority: Major | Resolution: 
Created: 04/Mar/20 22:09

Cleaning up a cluster with a bunch of messed up region assignments and state, hbck2 gets us to the point where meta and hdfs agree on split points, but some regions are left orphaned in hdfs. The current solution is to manually loop through the list of orphans and invoke {{completebulkload}} one-by-one. Let's add this as an action that hbck2 and do on the operator's behalf.
Bug: [hbase-thirdparty] Add nexus-staging-maven-plugin to apache-release profile
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jan/20 05:34

Parent issue added use of nexus-staging-maven-plugin manipulating maven.apache.org repos. Add plugin reference over in hbase-thirdparty so when we make RCs using create-release from hbase core, the repo manipulation 'works' (failed when I made the 2.3.0RC0.
Bug: Get `-PrunSmallTests` passing on JDK11
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Feb/20 18:37

Start with the small tests, shaking out issues identified by the harness. So far it seems like {{-Dhadoop.profile=3.0}} and {{-Dhadoop-three.version=3.3.0-SNAPSHOT}} maybe be required.
Bug: Support metrics  for region traffic
Status: Patch Available | Priority: Major | Resolution: 
Created: 19/Jan/20 07:10
Affects Version/s: 3.0.0-alpha-1

I think we need some metrics to measure the amount of traffic being read and written to each region, which can help us more clearly analyze how much network resources are consumed by each region.

 putBytes/getBytes/scanBytes/deleteBytes/incrementBytes/appendBytes/bulkLoadBytes
Bug: Nightlies reporting OutOfMemoryError: unable to create new native thread
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Mar/20 19:36

Seeing this kind of thing in nightly...

{code}
java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new native thread
	at org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83)
{code}

Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing our docker container. It is set to 20G (The hosts are 48G). Fork count is 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add the maven 4G and we could be over the top.

Play w/ downing the fork size (in earlier study we didn't seem to need this much RAM when running a fat long test). Let me also take th ms off the mvn allocation to see if that helps.
Bug: Document new RegionProcedureStore operation and migration
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jan/20 22:34
Affects Version/s: 2.3.0

Add a few notes to the refguide on the new RegionProcedureStore, how it works (A 'Region' but buried in the Master with dedicated flushing/compacting threads and archivers for WAL and hfile), how it differs from WALPS, and note it auto-migrates and there should be new issue moving on to the new store.

Mention the configuration. Mention it is on WALFS even though it is a 'Region', etc.
Bug: HBase Thrift bindings generating broken code
Status: Open | Priority: Major | Resolution: 
Created: 17/Feb/20 13:46
Affects Version/s: 2.2.3

Generated perl thrift bindings are broken:
{code:java}
$ thrift --gen perl src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift

$ cd gen-perl

$ perl -I . -Tc Hbase/Hbase.pm
Subroutine new redefined at /Library/Perl/5.18/Thrift/Exception.pm line 38.
Bareword "Thrift::TMessageType::CALL" not allowed while "strict subs" in use at Hbase/Hbase.pm line 9897.
BEGIN not safe after errors--compilation aborted at Hbase/Hbase.pm line 12163.
{code}
 

Tested from build from source of both HBase 2.1.2 and HBase 2.2.3 using both Thrift 0.12 and 0.13.
Bug: Make MasterRegistry the default registry impl
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jan/20 01:55
Affects Version/s: 3.0.0-alpha-1

Now that we are close to getting the registry implementation patch in, the idea here is to make it the default implementation in 3.0.0 and this means

- No known bugs with the implementation
- No known performance issues
- Entire nightly test suite is green (and without flakes).
Bug: Up the default fork count to make builds complete faster; make count relative to CPU count
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Jan/20 00:39

Tests take a long time. Our fork count running all tests are conservative -- 1 (small) for first part and 5 for second part (medium and large). Rather than hardcoding we should set the fork count to be relative to machine size. Suggestion here is 0.75C where C is CPU count. This ups the CPU use on my box.

Looking up at jenkins, it seems like the boxes are 24 cores... at least going by my random survey. The load reported on a few seems low though this not representative (looking at machine/uptime).

More parallelism willl probably mean more test failure. Let me take a look see.
Bug: ConcurrentModificationException occurred in ReplicationSourceManager while refreshing the peer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Mar/20 10:14

Peer update failed with ConcurrentModificationException,
{noformat}
2020-03-28 11:36:49,770 | WARN  | RpcServer.default.FPBQ.Fifo.handler=49,queue=4,port=hm_port | Refresh peer peer1 for UPDATE_CONFIG on rs_host,rs_port,1585228936395 failed | org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure.complete(RefreshPeerProcedure.java:114)
java.util.ConcurrentModificationException via szvphispra08478,21302,1585228936395:java.util.ConcurrentModificationException: 
	at org.apache.hadoop.hbase.procedure2.RemoteProcedureException.fromProto(RemoteProcedureException.java:124)
	at org.apache.hadoop.hbase.master.MasterRpcServices.lambda$reportProcedureDone$4(MasterRpcServices.java:2576)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1082)
	at org.apache.hadoop.hbase.master.MasterRpcServices.reportProcedureDone(MasterRpcServices.java:2571)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15341)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
Caused by: java.util.ConcurrentModificationException: 
	at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
	at java.util.ArrayList$Itr.next(ArrayList.java:859)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.refreshSources(ReplicationSourceManager.java:399)
	at org.apache.hadoop.hbase.replication.regionserver.PeerProcedureHandlerImpl.updatePeerConfig(PeerProcedureHandlerImpl.java:123)
	at org.apache.hadoop.hbase.replication.regionserver.RefreshPeerCallable.call(RefreshPeerCallable.java:68)
	at org.apache.hadoop.hbase.replication.regionserver.RefreshPeerCallable.call(RefreshPeerCallable.java:34)
	at org.apache.hadoop.hbase.regionserver.handler.RSProcedureHandler.process(RSProcedureHandler.java:47)
	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

{noformat}
Bug: Update test-patch.sh path for git-checkout use-cases to account for the new module layout in Yetus
Status: Open | Priority: Major | Resolution: 
Created: 19/Feb/20 18:29

It was noticed over on HBASE-23779 that we support running Yetus pre-commit/test-patch.sh directly out of a git checkout/source tarball. However, not all usages of this form of Yetus have been upgraded to the new maven layout. It appears we make this assumption in:
 * The Jenkins job configuration on [PreCommit-HBASE-Build|https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/configure]
 * {{dev-support/Jenkinsfile_GitHub}}
 * {{dev-support/hbase_nightly_yetus.sh}}
Bug: Run maven with more than default single thread (--threads=0.5C)
Status: Resolved | Priority: Major | Resolution: Later
Created: 09/Feb/20 17:41

See tail of parent task where we made commits to hbase-personality trying to make maven run more furiously on nightly builds by upping the maven thread count (the --threads flag ... see https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3). The effort was abandoned because it unsettled the build.  This issue separates messing with the maven threads flag from the broader topic the parent covers.
Bug: shadedjars check does not propagate --hadoop-profile
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Mar/20 19:30
Affects Version/s: 2.3.0

After HBASE-23829, we see the {{shadedjars}} check fail on JDK11 stages. From the log

{noformat}
Tue Mar 17 00:14:24 UTC 2020
cd /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1296/yetus-jdk11-hadoop3-check/src
/opt/maven/bin/mvn --batch-mode -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1296/yetus-m2/hbase-branch-2-patch-1 clean verify -fae --batch-mode -pl hbase-shaded/hbase-shaded-check-invariants -am -Dtest=NoUnitTests -DHBasePatchProcess -Prelease -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dspotbugs.skip=true
...
[INFO] --- maven-enforcer-plugin:3.0.0-M2:enforce (hadoop3-profile-required) @ hbase ---
[INFO] Adding ignore: module-info
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.RequireProperty failed with message:
HBase with JDK11 requires Hadoop3. Activate the profile with `-Dhadoop.profile=3.0`.
{noformat}
Bug: [Flakey Tests]  TestSlowLogRecorder.testOnlieSlowLogConsumption:178->confirmPayloadParams:97 expected:<client_1[0]> but was:<client_1[4]>
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Mar/20 21:07
Affects Version/s: 2.3.0

I see this occasionally on a linux vm. [~vjasani] .. you might have an idea why this is going on (this is your test I believe and a recent addition). If you have stuff I can try, shout.

Here is fail:
{code}
org.junit.ComparisonFailure: expected:<client_1[0]> but was:<client_1[4]>
   at org.apache.hadoop.hbase.regionserver.slowlog.TestSlowLogRecorder.confirmPayloadParams(TestSlowLogRecorder.java:97)
   at org.apache.hadoop.hbase.regionserver.slowlog.TestSlowLogRecorder.testOnlieSlowLogConsumption(TestSlowLogRecorder.java:178)
{code}

Here is log:

{code}
2020-03-12 17:02:06,266 INFO  [Time-limited test] hbase.ResourceChecker(179): after: regionserver.slowlog.TestSlowLogRecorder#testOnlineSlowLogWithDisableConfig Thread=7 (was   7), OpenFileDescriptor=226 (was 226), MaxFileDescriptor=131072 (was 131072), SystemLoadAverage=353 (was 353), ProcessCount=195 (was 195), AvailableMemoryMB=7037 (was 7027) - AvailableMemoryMB LEAK? -
 2020-03-12 17:02:06,281 INFO  [Time-limited test] hbase.ResourceChecker(151): before: regionserver.slowlog.TestSlowLogRecorder#testOnlieSlowLogConsumption Thread=7, OpenFileDescriptor=226,              MaxFileDescriptor=131072, SystemLoadAverage=353, ProcessCount=195, AvailableMemoryMB=7036
 2020-03-12 17:02:06,317 DEBUG [Time-limited test] slowlog.TestSlowLogRecorder(111): Initially ringbuffer of Slow Log records is empty
 2020-03-12 17:02:06,326 INFO  [Time-limited test] hbase.Waiter(183): Waiting up to [3,000] milli-secs(wait.for.ratio=[1])
 2020-03-12 17:02:06,528 INFO  [Time-limited test] hbase.Waiter(183): Waiting up to [3,000] milli-secs(wait.for.ratio=[1])
 2020-03-12 17:02:06,630 INFO  [Time-limited test] hbase.Waiter(183): Waiting up to [3,000] milli-secs(wait.for.ratio=[1])
 2020-03-12 17:02:06,732 INFO  [Time-limited test] hbase.Waiter(183): Waiting up to [3,000] milli-secs(wait.for.ratio=[1])
{code}
Bug: Retrieve online large RPC logs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Mar/20 16:57
Affects Version/s: 2.3.0

Similar to online slow RPC response logs, we should also provide get and clear of large RPC logs which should be stored in the online RingBuffer. Many of slow response logs would also turn out to be large response logs for RegionServers.
Bug: Fix a couple more test failures from nightly run
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jan/20 20:59
Affects Version/s: 1.6.0

After merging the patches HBASE-23731 and HBASE-23647, nightly run found the following issues.

TestFromClientSideWithCoprocessor - This was a bug in the test itself, PR runs missed it somehow.

TestCustomSaslAuthenticationProvider - This was written pre master registry. Has to be updated. It was never run as a part of the test suite.

TestSnapshotScannerHDFSAclController: Cluster restart did not reset the cached connection state

 

[~ndimiduk] FYI because this is a blocker for 2.3.0 too, I tagged it accordingly. Will submit a PR shortly.
Bug: Fix default master addr hostname in master registry
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Feb/20 02:14
Affects Version/s: 1.6.0

Currently, master RPC server (*not* info server) always binds to the address endpoint to which the default hostname of the server resolves to. However, master registry picks the default end point to connect to as "localhost:16000" when "hbase.masters" are not configured. This is leading to a mismatch because the server may not be listening on the loopback address. This is a problem only in the scripts (single proc/pseudo distributed modes) because these are the cases in which "hbase.masters" is not populated by default.

The fix is to pick the service endpoint the same way the RPC server does it.
Bug: [Flakeys] TestRSGroupsBasics#testClearNotProcessedDeadServer fails most of the time
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jan/20 00:40

Asserts a dead server but the dead server can be cleaned up because the ServerCrashProcedure completes before the assert can run.

Fails for me locally nearly every time I run it.

In flakies list, it shows as failing 55% of the time: https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2/lastSuccessfulBuild/artifact/dashboard.html
Bug: [Flakey Tests] Caused by: org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException: hflush and hsync
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jan/20 20:01

Tests that use the new RegionProcedureStoreTestBase utility fail in branch-2 nightlies complaining...

{code}
Error Message

cannot get log writer

Stacktrace

java.io.IOException: cannot get log writer
Caused by: org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException: hflush and hsync

{code}

They are using local fs which doesn't have support for above. Only shows when you run the tests with hadoop3 profile.  Let me do for these tests what was done over in HBASE-19289

It has been going on for a while now, probably since the move to region procedure store (the old wal procedure store had turned off the checks for sync/hflush).

{code}
health checks / yetus jdk8 hadoop3 checks / org.apache.hadoop.hbase.procedure2.store.region.TestHFileProcedurePrettyPrinter.test
health checks / yetus jdk8 hadoop3 checks / org.apache.hadoop.hbase.procedure2.store.region.TestRegionProcedureStore.testLoad
health checks / yetus jdk8 hadoop3 checks / org.apache.hadoop.hbase.procedure2.store.region.TestRegionProcedureStore.testCleanup
health checks / yetus jdk8 hadoop3 checks / org.apache.hadoop.hbase.procedure2.store.region.TestRegionProcedureStoreCompaction.test
health checks / yetus jdk8 hadoop3 checks / org.apache.hadoop.hbase.procedure2.store.region.TestRegionProcedureStoreMigration.testMigrateWithUnsupportedProcedures
health checks / yetus jdk8 hadoop3 checks / org.apache.hadoop.hbase.procedure2.store.region.TestRegionProcedureStoreMigration.test
health checks / yetus jdk8 hadoop3 checks / org.apache.hadoop.hbase.procedure2.store.region.TestRegionProcedureStoreWALCleaner.test
health checks / yetus jdk8 hadoop3 checks / org.apache.hadoop.hbase.procedure2.store.region.TestWALProcedurePrettyPrinter.test
{code}