av9ash/GitBugs huggingface dataset:
Bug: Include dump files in the jenkins archive
Status: Open | Priority: Major | Resolution: 
Created: 26/Jun/20 19:09

Tracking down jenkins test failures in the face of {{org.apache.maven.surefire.booter.SurefireBooterForkException: ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?}} is difficult right now. Let's include the "dump" files in our archives so that there's something to look at, maybe find a usable stack trace.

The failures I want to track look like

{noformat}
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M4:test (default-test) on project hbase-common: There are test failures.
[ERROR] 
[ERROR] Please refer to /home/jenkins/jenkins-slave/workspace/HBase_Nightly_branch-2.3@2/component/hbase-common/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /home/jenkins/jenkins-slave/workspace/HBase_Nightly_branch-2.3@2/component/hbase-common && /usr/lib/jvm/jdk8u232-b09/jre/bin/java -enableassertions -Dhbase.build.id=2020-06-24T15:53:27Z -Xmx2200m -Djava.security.egd=file:/dev/./urandom -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djdk.net.URLClassPath.disableClassPathURLCheck=true -Dorg.apache.hbase.thirdparty.io.netty.leakDetection.level=advanced -Dio.netty.eventLoopThreads=3 -jar /home/jenkins/jenkins-slave/workspace/HBase_Nightly_branch-2.3@2/component/hbase-common/target/surefire/surefirebooter9093790277421313959.jar /home/jenkins/jenkins-slave/workspace/HBase_Nightly_branch-2.3@2/component/hbase-common/target/surefire 2020-06-24T15-53-31_857-jvmRun1 surefire8040542579565444758tmp surefire_783341445535876751188tmp
[ERROR] Process Exit Code: 0
{noformat}
Bug: Add linter checks for patches
Status: Open | Priority: Major | Resolution: 
Created: 26/Jun/20 16:59

I've worked with clang-tidy before and I think it works well. There is also a [helper script|https://clang.llvm.org/extra/doxygen/clang-tidy-diff_8py_source.html] that runs clang-tidy on a patch rather than the entire source tree.

- pull in clang as a dependency
- Define .clang-format
- Run clang-tidy-diff on each checkin

There are also other tools like cpplint from Google. I don't have any preference, so either works for me.
Bug: [HBCK2] Remove reference to hase I.A. private class CommonFsUtils from FsRegionsMetaRecoverer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jun/20 18:14
Affects Version/s: hbase-operator-tools-1.0.0

FsRegionsMetaRecoverer used to reference I.A. private targeted interface FSUtils, which changed on hbase 2.3, causing hbck2 fail to compile. HBASE-24482 fixed it by pointing to CommonFSUtils interface, where the methods it was relying upon was actually defined. Since this is also a IA private interface, there's no compatibility guarantees. This PR removes reference to CommonFSUtils on FsRegionsMetaRecoverer. Other classes in hbck2 still require similar work.
Bug: Empty RegionAction added to MultiRequest in case of RowMutations/CheckAndMutate batch
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jun/20 12:18

When a client sends RowMutations/CheckAndMutate batch requests, no Action objects are added to the *builder* (RegionAction.Builder), so empty RegionAction is added to MultiRequest at the following line:
https://github.com/apache/hbase/blob/3c319811799cb4c1f51fb5b43dd4743acd28052c/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/RequestConverter.java#L593

We need to check if the *builder* has any Action objects or not here.
Bug: testReplicationStatusSink last applied op ts comparison failure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jun/20 09:36

The failure to compare timestamp of last applied op with RS start is not a permanent failure, nor does it seem flaky but while running all tests as part of 2.3 RC0 validation, I did encounter the failure:

 

[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 29.212 s <<< FAILURE! - in org.apache.hadoop.hbase.replication.TestReplicationStatus[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 29.212 s <<< FAILURE! - in org.apache.hadoop.hbase.replication.TestReplicationStatus[ERROR] org.apache.hadoop.hbase.replication.TestReplicationStatus.testReplicationStatusSink  Time elapsed: 1.016 s  <<< FAILURE!java.lang.AssertionError: expected:<1592477704810> but was:<1592477719964> at org.apache.hadoop.hbase.replication.TestReplicationStatus.testReplicationStatusSink(TestReplicationStatus.java:134)

 

It seems we should allocate separate resources to testReplicationStatusSink: separate class, shutdown peer cluster (if running already), bring it up and check timestamp of lastAppliedOp, then proceed with remaining test i.e timestampsOfLastAppliedOp > timestampStarted within specific period of time soon after inserting some records.
Bug: RollingBatchRestartRsAction loses track of dead servers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/May/20 23:37
Affects Version/s: 2.3.0

{{RollingBatchRestartRsAction}} doesn't handle failure cases when tracking its list of dead servers. The original author believed that a failure to restart would result in a retry. However, by removing the dead server from the failed list prematurely, that state is lost, and retry of that server never occurs. Because this action doesn't ever look back to the current state of the cluster, relying only on its local state for the current action invocation, it never realizes the abandoned server is still dead. Instead, be more careful to only remove the dead server from the list when the {{startRs}} invocation claims to have been successful.
Bug: Enable injecting build start time value as part of build
Status: Open | Priority: Major | Resolution: 
Created: 23/Jun/20 01:00
Affects Version/s: 2.3.0

The HBase build's creation time is presented in the HBase UI, and made available through Java, via the {{org.apache.hadoop.hbase.Version}} class's {{date}} value, which is generated at build time by {{hbase-common/src/saveVersion.sh}}. The script just invokes the shell command {{date}} and captures its result as a string.

The problem is, this occurs every time hbase-common is built. And, for good and sufficient reason, when making a release via dev-support/create-release, the task for building and deploying hbase jars as maven libraries and the task for building binary release artifacts as tarballs, EACH do a {{clean}} build. Thus, the build time found in the libs is different from the build time found in the release tarballs.

There is value in keeping the two tasks independent, and able to run fully each by themselves. And there is value in doing a {{clean}} at the start of such processes, to make sure you're releasing binaries that exactly match the source code.

So to keep these benefits, but enable the start time to be determined once and used for a couple builds in a row in a given environment, I propose to allow injecting the desired value. Specifically, I want to change saveVersion.sh to look for an existing value of env var HBASE_BUILD_TIME, and if it exists use it instead of calling {{date}}. One would of course set it as part of the build process (in create-release) and clear this value by unsetting the environment variable when done with the build.
Bug: [flakey test] TestExportSnapshot family of tests failing due to timeout in AbstractDelegationTokenSecretManager$ExpiredTokenRemover
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/20 17:17
Affects Version/s: 2.3.0

I've observed another occurrence of this test timing out, over on https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1786/3/

The failure message is cryptic, but I think i found the issue. This isn't the {{HBaseClassTestRule}} invoking the timeout, it's something in the test, I think in the mini-cluster.

There appears to be a timeout set in some kind of secret manager, which is too aggressive for this mini-cluster test. The last component of the mini-cluster, MapReduce, is finally available at T+273501ms -- ~4.5 minutes after process launch. This is how I interpret the log line

{noformat}
2020-06-02 03:20:49,252 INFO  [Thread-223] server.Server(419): Started @273501ms
{noformat}

a scant 20ms later we get

{noformat}
2020-06-02 03:20:50,274 ERROR [Thread[Thread-224,5,FailOnTimeoutGroup]] delegation.AbstractDelegationTokenSecretManager$ExpiredTokenRemover(700): ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
2020-06-02 03:20:50,351 INFO  [Time-limited test] hbase.HBaseTestingUtility(1272): Shutting down minicluster
{noformat}

These thread group names have no meaning to me.
Bug: Bring back old constructor of SnapshotDescription
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jun/20 18:25

As part of HBASE-22648 (Snapshot TTL), one of SnapshotDescription constructor was modified with an additional argument and hence, this is raising source compatibility concerns for minor releases. We need to bring back old constructor, mark it deprecated and internally point to new constructor with null/empty snapshotProps.
Bug: o.a.h.h.tool.LoadIncrementalHFiles deprecation comment does not span an entire release
Status: Open | Priority: Major | Resolution: 
Created: 22/Jun/20 16:05
Affects Version/s: 2.2.0

While evaluating interface compatibilities for 2.3.0RC0, I noticed the deprecation comment for {{o.a.h.h.tool.LoadIncrementalHFiles}} says

{noformat}
 * @deprecated since 2.2.0, will be removed in 3.0.0. Use {@link BulkLoadHFiles} instead. {noformat}

This is contrary to an explicit example in our [book|https://hbase.apache.org/book.html#hbase.versioning], which states:

{quote}
An API needs to be deprecated for a whole major version before we will change/remove it.

An example: An API was deprecated in 2.0.1 and will be marked for deletion in 4.0.0. On the other hand, an API deprecated in 2.0.0 can be removed in 3.0.0.
{quote}

Maybe there are other comments like this, which need addressed. We should grep the codebase and make appropriate adjustments.
Bug: Doesn't work SplitPolicy attributs for table
Status: Open | Priority: Major | Resolution: 
Created: 22/Jun/20 15:08
Affects Version/s: 1.4.8

Doesn't work SplitPolicy settings at the table level. Regions are splits to very low size.

Table describe: 

 
{code:java}
test_split, {TABLE_ATTRIBUTES => {MAX_FILESIZE => '2147483648', METADATA => {'SPLIT_POLICY' => 'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy'}} COLUMN FAMILIES DESCRIPTION {NAME => 'data', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} {NAME => 'number', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} {NAME => 'uuid', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}{code}
 

Global settings that affect split:
 * Maximum Region File Size - 10GB
 * Memstore Flush Size - 128MB
 * hbase.regionserver.region.split.policy - org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy 

As a result, regions are splits to  400-600MB, but we expecting should region size 2GB. 

!image-2020-06-22-18-04-27-155.png!

Why it happens? This is a bug?
Bug: get_table_rsgroup ignored the existence of rsgroup config for namespace
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jun/20 02:14
Affects Version/s: 3.0.0-alpha-1

{code:java}
public GetRSGroupInfoOfTableResponse getRSGroupInfoOfTable(RpcController controller,
  GetRSGroupInfoOfTableRequest request) throws ServiceException {
  TableName tableName = ProtobufUtil.toTableName(request.getTableName());
  ...
  try {
    ...
    GetRSGroupInfoOfTableResponse resp;
    TableDescriptor td = master.getTableDescriptors().get(tableName);
    if (td == null) {
      resp = GetRSGroupInfoOfTableResponse.getDefaultInstance();
    } else {
      RSGroupInfo rsGroupInfo = null;
      if (td.getRegionServerGroup().isPresent()) {
        rsGroupInfo = master.getRSGroupInfoManager().getRSGroup(td.getRegionServerGroup().get());
      }
      if (rsGroupInfo == null) {
        rsGroupInfo = master.getRSGroupInfoManager().getRSGroup(RSGroupInfo.DEFAULT_GROUP);
      }
      resp = GetRSGroupInfoOfTableResponse.newBuilder()
        .setRSGroupInfo(ProtobufUtil.toProtoGroupInfo(rsGroupInfo)).build();
    }
    ...
    return resp;
  } catch (IOException e) {
    throw new ServiceException(e);
  }
}

{code}
In method MasterRpcServices#getRSGroupInfoOfTable, ignored namespace hbase.rsgroup.name config. 
 
It should be replaced by RSGroupUtil#getRSGroupInfo.
Bug: TestSecureExport is failing 
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 21/Jun/20 15:23

Error Message
Expected exception was not thrown for user 'rouser'
Stacktrace
java.lang.AssertionError: Expected exception was not thrown for user 'rouser'
	at org.apache.hadoop.hbase.coprocessor.TestSecureExport.testAccessCase(TestSecureExport.java:319)
Bug: The current implementation of assignMetaReplicas() may assign replica meta regions to the same server hosting primary meta region.
Status: Open | Priority: Major | Resolution: 
Created: 17/Jun/20 21:29
Affects Version/s: 2.3.0

We need to take the approach similar to SplitTableRegionProcedure, which uses round robin algo to assign replica regions and excludes the primary server.

 
{code:java}
return AssignmentManagerUtil.createAssignProceduresForOpeningNewRegions(env, hris,
 getRegionReplication(env), getParentRegionServerName(env));{code}
Bug: [hbase-operator-tools] build of hbck2 fails with HBase branch-2.3, due to missing dependencies
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jun/20 05:01
Affects Version/s: hbase-operator-tools-1.1.0

If we do a local build of the current HBase branch-2.3 as version 2.3.0-SNAPSHOT, and 'mvn install' it in the local maven repository, 

then attempt to build hbase-operator-tools master branch, with -Dhbase.version=2.3.0-SNAPSHOT, 

the HBCK2 class file HBCKMetaTableAccessor.java (line 25) fails to compile with error: "package edu.umd.cs.findbugs.annotations does not exist", followed by multiple instances of missing "symbol: class Nullable".  (A longer log extract is attached.)

It would appear the default HBase version 2.1.6 made this dependency package available in some manner, but the newer HBase does not.

*Later Edit:* After addressing this missing dependency (see comment of 5 June below), it unmasked a bunch more, in the hbck1 code path.  Please see second attachment.

If the @Nullable annotation is still needed, it may be sufficient to simply add it to the hbase-operator-tools pom.xml file, so maven will load it.  However, it is actually deprecated in the current documentation for edu.umd.cs.findbugs.annotations. Various commentary online suggest that we should be using the similar jsr305.jar (*{{javax.annotation)}}* instead; see eg
 * [https://stackoverflow.com/questions/4963300/which-notnull-java-annotation-should-i-use]

 Does anyone know if that was HBase's approach?
Bug: Set values in `conf/hbase-site.xml` that enable running on `LocalFileSystem` out of the box
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Apr/20 18:10
Affects Version/s: 1.7.0

This ticket is to implement the changes as described on the [discussion on dev|https://lists.apache.org/thread.html/r089de243a9bc9d923fa07c81e6bc825b82be68f567b892a342a0c61f%40%3Cdev.hbase.apache.org%3E]. It reverts and supersedes changes made on HBASE-24086 and HBASE-24106.

{quote}
The conclusion I understand from this thread looks something like this:

1. revert HBASE-24086, make it so that running on `LocalFileSystem` is a fatal condition with default configs.
2. ship a conf/hbase-site.xml that contains hbase.unsafe.stream.capability.enforce=false, along with a big comment saying this is not safe for production.
3. ship a conf/hbase-site.xml that contains hbase.tmp.dir=./tmp, along with a comment saying herein you'll find temporary and persistent data, reconfigure for production with hbase.rootdir pointed to a durable filesystem that supports our required stream capabilities (see above).
4. update HBASE-24106 as appropriate.

Neither 2 nor 3 are suitable for production deployments, thus the changes do not go into hbase-default.xml. Anyone standing up a production deploy must edit hbase-site.xml anyway, so this doesn't change anything. It also restores our "simple" first-time user experience of not needing to run anything besides `bin/start-hbase.sh` (or `bin/hbase master start`, or whatever it is we're telling people these days).

We can reassess this once more when a durable equivalent to LocalFileSystem comes along.
{quote}
Bug: HBase upgrade from 2.2 to 2.3 fails due to failure in ZooKeeper
Status: Open | Priority: Major | Resolution: 
Created: 13/Jun/20 20:13
Affects Version/s: 2.3.0

When we tested upgrading HBase from 2.2 to 2.3, the upgraded node failed to start due to a known zookeeper failure - ZOOKEEPER-3056.

The error message is shown below:

 
{code:java}
2020-06-12 23:15:11,432 ERROR [main] master.HMasterCommandLine: Master exiting2020-06-12 23:15:11,432 ERROR [main] master.HMasterCommandLine: Master exitingjava.io.IOException: No snapshot found, but there are log entries. Something is broken! at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:240) at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240) at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:290) at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:450) at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:764) at org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.java:98) at org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster.startup(MiniZooKeeperCluster.java:259) at org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster.startup(MiniZooKeeperCluster.java:189) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:210) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3035)
{code}
 

It can be reproduced through the following steps:
 * Start a single-node HBase 2.2 with the following configuration.
{code:java}
  <property>
    <name>hbase.rootdir</name>
    <value>/var/lib/hbase</value>
  </property>  <property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/var/lib/zookeeper</value>
  </property>{code}

 * Use HBase performance evaluation tool to generate data.
{code:java}
/hbase/bin/hbase pe --nomapred --oneCon=true --valueSize=10 --rows=100 sequentialWrite 1{code}

 * Upgrade the node to 2.3 with the same configuration. The new version node failed to start because of the zookeeper.

HBase 2.2 is using zookeeper-3.4.10.jar, and Kafka 2.3 are using zookeeper-3.5.7.jar.

The bug has been fixed by ZOOKEEPER-3056 in zookeeper-3.6.0. Should we upgrade the dependency of HBase 2.3 to use zookeeper-3.6.0.jar? or upgrade the configuration file of zookeeper-3.5.7.jar?
Bug: WALSplit recreates region dirs for deleted table with recovered edits data
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Apr/20 22:01
Affects Version/s: 2.2.4

Under the following scenario region directories in HDFS can be recreated with only recovered.edits in them:
 # Create table "test"
 # Put into "test"
 # Delete table "test"
 # Create table "test" again
 # Crash the regionserver to which the put has went to force the WAL replay
 # Region directory in old table is recreated in new table
 # hbase hbck returns inconsistency

This appears to happen due to the fact that WALs are not cleaned up once a table is deleted and they still contain the edits from old table. I've tried wal_roll command on the regionserver before crashing it, but it doesn't seem to help as under some circumstances there are still WAL files around. The only solution that works consistently is to restart regionserver before creating the table at step 4 because that triggers log cleanup on startup: [https://github.com/apache/hbase/blob/f3ee9b8aa37dd30d34ff54cd39fb9b4b6d22e683/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java#L508|https://github.com/apache/hbase/blob/f3ee9b8aa37dd30d34ff54cd39fb9b4b6d22e683/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java#L508)]

 

Truncating a table also would be a workaround by in our case it's a no-go as we create and delete tables in our tests which run back to back (create table in the beginning of the test and delete in the end of the test).

A nice option in our case would be to provide hbase shell utility to force clean up of log files manually as I realize that it's not really viable to clean all of those up every time some table is removed.
Bug: Add backoff to SCP check on WAL split completion
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jun/20 06:09

Crashed cluster. Lots of backed up WALs. Startup. Recover hundreds of servers; each has a running SCP. Taking a thread dump during recovery, I noticed that there were 160 threads each in SCP waiting on split WAL completion. Each thread was scanning zk splitWAL directory every 100ms. The dir had thousands of entries in it so each check was pulling down MB from zk... * 160 (max configured PE threads (16) * 10 for the KeepAlive factor that has us do 10 * configured PEs as max for PE worker pool).

If lots of remaining WALs to split, have the SCP backoff on its wait so it checks less frequently.
Bug: Cluster locked up due to REST gateway queries for non-existent tables
Status: Open | Priority: Major | Resolution: 
Created: 12/Jun/20 15:39

We had a case where a REST gateway instance was picked up by an automated security scanning tool. Many of the http requests it generated were interpreted requests against non-existent tables. The result was a flood of queries against meta, completely overrunning the priority queue of the region server hosting meta.

We need some kind of back pressure mechanism that can protect meta from a single bad actor.