av9ash/GitBugs huggingface dataset:
Bug: compatibility report in release process should use existing binaries
Status: Open | Priority: Major | Resolution: 
Created: 16/May/20 18:17

right now the compatibility report generation in the release scripts does an additional build of both the current RC and the comparison release. Instead, we should have an option to grab artifacts for the previous release from the mirroring / archive system. Additionally, we should always use the binary bits we just created for the binary distribution for the RC.
Bug: Make kerberos value of hbase.security.authentication property case insensitive
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Apr/20 00:56

In hbase-20586 (https://issues.apache.org/jira/browse/HBASE-20586)

（commit_sha: [https://github.com/apache/hbase/commit/cd61bcc0] ）

The code added ([SyncTable.java|https://github.com/apache/hbase/commit/cd61bcc0#diff-d1b79635f33483bf6226609e91fd1cc3]) for the use of *hbase.security.authentication* is case-sensitive. So users setting it to “KERBEROS” won’t take effect. 

 
{code:java}
 private void initCredentialsForHBase(String zookeeper, Job job) throws IOException {
   Configuration peerConf = HBaseConfiguration.createClusterConf(job.getConfiguration(), zookeeper);
   if(peerConf.get("hbase.security.authentication").equals("kerberos")){
     TableMapReduceUtil.initCredentialsForCluster(job, peerConf);    }
 }
{code}
 

However, in current code base, other uses of *hbase.security.authentication* are all case-insensitive. For example in *MasterFileSystem.java.* 

 
{code:java}
public MasterFileSystem(Configuration conf) throws IOException{   
  ...   
  this.isSecurityEnabled = "kerberos".equalsIgnoreCase(conf.get("hbase.security.authentication"));  
  ... 
}
{code}
 

The doc in GitHub repo is also misleading (Giving upper-case value).
{quote}As a distributed database, HBase must be able to authenticate users and HBase services across an untrusted network. Clients and HBase services are treated equivalently in terms of authentication (and this is the only time we will draw such a distinction).

There are currently three modes of authentication which are supported by HBase today via the configuration property {{hbase.security.authentication}}

{{1.SIMPLE}}

{{2.KERBROS}}

{{3.TOKEN}}
{quote}
Users may misconfigure the parameter because of the case-senstive problem.

*How To Fix*

Using *eqaulsIgnoreCase* API consistently in every place when using *hbase.security.authentication* or make it clear in Doc.
Bug: Allow hbase1 cluster to replicate to an hbase2 cluster
Status: Open | Priority: Major | Resolution: 
Created: 11/May/20 23:49

Umbrella issue to make it so an hbase1 cluster can replicate to an hbase2 cluster. I thought this worked but just tried it and it doesn't seem like it can. Replication compares table schemas before it starts. The hbase2 has a default that is not defined in hbase1 so we are unable to make the schemas equivalent. There are probably other issues in here too. Let me make an umbrella.
Bug: Fix CI builds on branch-2.2
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 16/Apr/20 18:52
Affects Version/s: 2.2.5

From a recent [PR build|https://builds.apache.org/blue/organizations/jenkins/HBase-PreCommit-GitHub-PR/detail/PR-1532/1/pipeline/]

{noformat}
[2020-04-16T18:43:21.548Z] Setting up ruby2.3 (2.3.3-1+deb9u7) ...
[2020-04-16T18:43:21.548Z] Setting up ruby2.3-dev:amd64 (2.3.3-1+deb9u7) ...
[2020-04-16T18:43:21.548Z] Setting up ruby-dev:amd64 (1:2.3.3) ...
[2020-04-16T18:43:21.548Z] Setting up ruby (1:2.3.3) ...
[2020-04-16T18:43:22.261Z] Processing triggers for libc-bin (2.24-11+deb9u3) ...
[2020-04-16T18:43:22.975Z] Successfully installed rake-13.0.1
[2020-04-16T18:43:22.975Z] Building native extensions.  This could take a while...
[2020-04-16T18:43:25.277Z] ERROR:  Error installing rubocop:
[2020-04-16T18:43:25.277Z] 	rubocop requires Ruby version >= 2.4.0.
{noformat}

Looks like the Dockerfile on branch-2.2 has bit-rot. I suspect package versions are partially pinned or not pinned at all: the rubocop version has incremented by ruby version has not.
Bug: Create-release scripts fixes and enhancements
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/May/20 17:14
Affects Version/s: 2.3.0

The create-release tools are a set of scripts that promote best practice in making Apache releases, by automating most of the steps and running them under Docker for reliability. However, the current state of the scripts has many bugs and ambiguities.  

The proposed PR cleans up the code and clarifies usage. It enables:
- Clear statement of the four steps, which are now called `tag`, `publish-dist`, `publish-snapshot`, and `publish-release` (the latter two being mutually exclusive alternatives).
- Ability to do the three tag-dist-release steps with a single command, or do any of the steps singly. (Running singly had bugs and unfulfilled dependencies before.)
- Ability to do a reliable and useful "dry run" of all steps or each step, and chain together the tag step with publish steps in a dry run.
- Ability to run any or all steps correctly in Docker or outside of Docker, on Linux and Mac.
- Cleaned up all `shellcheck` errors in the scripts, and removed ambiguities and redundancies in the many environment variables used.

In addition, the changes move the code toward being more general / less HBase-specific, so it can be run on any Apache project (while still accommodating HBase-specific features regarding how sub-projects are named and organized in Jira and release repos). In future I propose to take it further along that path, and move create-release into Yetus (recognizing that this create-release code has been passed between a couple other projects already).

These changes have NO IMPACT on HBase functionality.
Bug: Add writable MinVersions and read-only Scan to coproc ScanOptions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/May/20 19:11

Between HBase 1.x and 2.0, the RegionObserver pre*ScannerOpen coprocessors were significantly changed so that the coproc implementer no longer has access to the actual Scanner, just a ScanOptions object that can be changed in limited ways. This is safer and prevents resource leaks and other bugs. 

While ScanOptions provides support for changing TTL, KeepDeletedCells, and MaxVersions, a fourth column family config parameter, MinVersions, appears to have been missed. This prevents coproc implementers from changing MinVersions dynamically. An example of this is PHOENIX-5645, which in the forthcoming Phoenix 4.16 (based on HBase 1.x) will allow users to configure a moving window where all versions are kept, and thus point-in-time queries are safe. This cannot be put in the forthcoming Phoenix 5.1 (based on HBase 2.1 and 2.2) because of the coproc changes.

Relatedly, preStoreScannerOpen lacks access to the Scan in HBase 2.0 and up. This prevents coprocs from reading the Scan parameters to check if, for example, a Scan has set the max time to a point in the past, and thus needs to override KeepDeletedCells. This can lead to incorrect behavior when doing point-in-time queries or using transactional engines that treat physically committed HBase writes as logically uncommitted parts of a transaction. It's also a correctness problem for PHOENIX-5645. Please note that only _read-only_ access to the Scan from the store scanner coproc hook is in scope for this change.
Bug: [h3/jdk11] REST server won't start
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Apr/20 14:21

{code}
2020-04-29 07:06:31,610 INFO  [main] server.AbstractConnector: Started ServerConnector@1bdf8190{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
Exception in thread "main" java.lang.NoSuchMethodError: 'java.util.Map javax.ws.rs.core.Application.getProperties()'
	at org.glassfish.jersey.server.ApplicationHandler.<init>(ApplicationHandler.java:331)
	at org.glassfish.jersey.servlet.WebComponent.<init>(WebComponent.java:392)
	at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:177)
{code}

Seems fine for hadoop2/jdk8.

I get the above when I do: mvn install -Dhadoop.profile=3.0 building w/ jdk11.

Fix coming...
Bug: [Flakey Tests] krb server for secure thrift tests throws BindException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/May/20 00:11

Thought I'd put all the BindExceptions in thrift to sleep but just came across this one putting up the *krb* server used by the secure thrift http tests. There are two tests that use krb:

{code}
 -------------------------------------------------------------------------------
 Test set: org.apache.hadoop.hbase.thrift.TestThriftSpnegoHttpFallbackServer
 -------------------------------------------------------------------------------
 Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.042 s <<< FAILURE! - in org.apache.hadoop.hbase.thrift.TestThriftSpnegoHttpFallbackServer
 org.apache.hadoop.hbase.thrift.TestThriftSpnegoHttpFallbackServer  Time elapsed: 0.017 s  <<< ERROR!
 org.apache.kerby.kerberos.kerb.KrbException: Failed to start KDC-Server
   at org.apache.hadoop.hbase.thrift.TestThriftSpnegoHttpFallbackServer.setUpBeforeClass(TestThriftSpnegoHttpFallbackServer.java:137)
 Caused by: java.net.BindException: Address already in use (Bind failed)
   at org.apache.hadoop.hbase.thrift.TestThriftSpnegoHttpFallbackServer.setUpBeforeClass(TestThriftSpnegoHttpFallbackServer.java:137)
{code}
Bug: [flakey test] replication.TestAddToSerialReplicationPeer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Apr/20 07:16
Affects Version/s: 2.3.0

From both 2.3 and branch-2 flaky test board, it constantly runs into the following flaky:

 
{code:java}
org.apache.hadoop.hbase.replication.TestAddToSerialReplicationPeer.testAddToSerialPeerFailing for the past 1 build (Since #6069 )Took 15 sec.Error MessageSequence id go backwards from 122 to 24Stacktracejava.lang.AssertionError: Sequence id go backwards from 122 to 24
	at org.apache.hadoop.hbase.replication.TestAddToSerialReplicationPeer.testAddToSerialPeer(TestAddToSerialReplicationPeer.java:176)
Standard Output{code}
Bug: Undo core of parent TestSecureRESTServer change; use fix over in HBASE-24280 instead
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/May/20 21:01
Affects Version/s: 2.3.0

Parent issue puts in a place that makes TestSecureRESTServer pass up on jenkins by shoving into the dependency list the jersey1 ServletContainer. Root issue was change in how we specified profiles in nightlies; both hadoop3 and hadoop2 were mistakenly active (HBASE-24280). This issue is about undoing the dependency insertion after fix for HBASE-24280 goes in.

Don't want to revert the parent. It has cleanups that should stay.
Bug: Add more details in MultiVersionConcurrencyControl STUCK log message
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/May/20 04:46

Now the warn logs look like
{code}
STUCK: MultiVersionConcurrencyControl{readPoint=30944485, writePoint=30944498}
{code}
We don't have any details on which region the writes are stuck. It would be better to include the region name detail also in this log.
May be we can even log for how long we are waiting for the read point to catch up.
cc [~stack]
Bug: Avoid two parallel running of GCMulitpleMergedRegionsProcedure over the same region.
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 04/May/20 23:37
Affects Version/s: 2.3.0

As [~stack] suggested in one of PRs, "This method is called from GCMergedRegionsProcedure and GCMulitpleMergedRegionsProcedure. I was wondering if we took a lock on the Region being Merged would it stop two overlapping GCs happening?"
Bug: [fakey test] TestInfoServersACL testJmxAvailableForAdmin
Status: Open | Priority: Major | Resolution: 
Created: 05/May/20 15:34
Affects Version/s: 2.3.0

There was a failure on branch-2.3 last night, https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/65/testReport/

This test failed for JDK11 but not JDK8.

Before the failed assertion, I do see a log from the loop on line 312:

{noformat}
2020-05-05 10:57:45,323 INFO  [Time-limited test] http.TestInfoServersACL$11(312): Hadoop:service=HBase,name=StartupProgress
{noformat}

This indicates to me that the HBase portion of the process hadn't finished coming online yet, and so the precondition was not yet satisfied. Just speculation though; I don't understand why the same bean wouldn't be available to the admin user as well.

Furthermore, looks like the test does not clean up after itself correctly. On retry, the subsequent attempt flat-out failed with

{noformat}
Failed to load or create keytab /home/jenkins/jenkins-slave/workspace/HBase_Nightly_branch-2.3/component/hbase-server/target/test-data/f2e84dcf-0d1f-fe85-3519-4fcb37058d39/keytab

org.apache.kerby.kerberos.kerb.KrbException: Failed to load or create keytab /home/jenkins/jenkins-slave/workspace/HBase_Nightly_branch-2.3/component/hbase-server/target/test-data/f2e84dcf-0d1f-fe85-3519-4fcb37058d39/keytab
	at org.apache.hadoop.hbase.http.TestInfoServersACL.beforeClass(TestInfoServersACL.java:114)
Caused by: java.io.IOException: No such file or directory
	at org.apache.hadoop.hbase.http.TestInfoServersACL.beforeClass(TestInfoServersACL.java:114)
{noformat}
Bug: Clearly document how profiles for the sake of Hadoop compatibility work across all branches
Status: Open | Priority: Major | Resolution: 
Created: 04/May/20 17:42

In HBASE-24280, we investigated a test failure which was ultimately caused by the simultaneous activation of the (intended mutually exclusive) hadoop-2 and hadoop-3 profiles.

After master has moved to only supporting profile activation via the profile itself (rather than a system property) with the removal of the hadoop-2 profile, the build was inadvertently broken as all branches (or is it just 2.x branches and master?) use the one build/yetus scripts in dev-support.

To make sure that these scripts continue to work against all branches, we need to have a clear decision on how profile activation is expected to work in our HBase build. Otherwise, we'll come back to this problem where each branch does things ever-so-slightly different, requiring a bunch of {{if branch-2; else if branch-2.2; else if branch-2.3}} type changes to our yetus scripts.
Bug: check-aggregate-license on hbase-shaded-client-byo-hadoop is not compatible with `--threads`
Status: Open | Priority: Major | Resolution: 
Created: 04/May/20 18:09
Affects Version/s: 2.3.0

I see occasional failures in this enforcer rule when running maven with {{-T1.0C}}. Usually it succeeds when building again. Looks related to the bean shell script that loops over license files.

I have been using {{mvn -U -T1.0C clean install -DskipTests}} for both JDK8 and JDK11.
Bug: Configuring HBase to Use HDFS HA
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 01/May/20 12:20
Affects Version/s: 2.2.4

Problem: HMasters fail to start.

Solution: Check for this error in the HMaster log:

2020-05-01 19:30:13,125 ERROR [main] regionserver.HRegionServer: Failed construction RegionServer
java.lang.IllegalArgumentException: java.net.UnknownHostException: hdfscluster
 
 
Hadoop's hdfs-site.xml and core-site.xml files are in your hbase/conf directory
Bug: Move hbase-rsgroup code into hbase-server code
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 03/May/20 08:11
Affects Version/s: 2.2.3

Keeping rsgroup code into separate module is causing many problem. HBASE-22740 and HBASE-24212 are blocked because of this.

In master branch hbase-rsgroup code is already moved into hbase-server. It is better to be in sync with master branch so issue fixes can be applied on branch-2 easily.

This jira moves hbase-rsgroup code into hbase-server as it is, does not make change in protobuff etc.
Bug: TestZooKeeper is flaky
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Apr/20 16:01

After checking the code, the problem is that, when creating table during master shutdown, it is easy to hit MasterStoppedException or other strange exceptions which make the creation fail.

In general I think this should be a test issue, need to learn why we do not have the problem on branch-2.x.
Bug: [Flakey Tests] TestThriftHttpServer BindException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Apr/20 21:32

Came across this last night:
{code}
Expected: (an instance of org.apache.thrift.transport.TTransportException and exception with message a string containing "HTTP Response code: 431")
     but: an instance of org.apache.thrift.transport.TTransportException <java.lang.Exception: java.net.BindException: Address already in use> is a java.lang.Exception
Stacktrace was: java.lang.Exception: java.net.BindException: Address already in use
	at org.apache.hadoop.hbase.thrift.TestThriftHttpServer.stopHttpServerThread(TestThriftHttpServer.java:253)
	at org.apache.hadoop.hbase.thrift.TestThriftHttpServer.runThriftServer(TestThriftHttpServer.java:183)
	at org.apache.hadoop.hbase.thrift.TestThriftHttpServer.testRunThriftServerWithHeaderBufferLength(TestThriftHttpServer.java:147)
{code}
Bug: [Flakey Tests] [ERROR] TestSecureRESTServer  java.lang.NoClassDefFoundError: com/sun/jersey/core/spi/factory/AbstractRuntimeDelegate
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Apr/20 03:32

Happened in two builds... 2.3 and 2:

jdk8 hadoop3 nightly just now https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-2.3/44/
{code}
[ERROR] org.apache.hadoop.hbase.rest.TestSecureRESTServer  Time elapsed: 0.01 s  <<< ERROR!
java.lang.NoClassDefFoundError: com/sun/jersey/core/spi/factory/AbstractRuntimeDelegate
	at org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:202)
Caused by: java.lang.ClassNotFoundException: com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate
	at org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:202)
{code}

Then again in branch2 https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-2/2619/

{code}
[ERROR] org.apache.hadoop.hbase.rest.TestSecureRESTServer  Time elapsed: 0.017 s  <<< ERROR!
java.lang.NoClassDefFoundError: com/sun/jersey/core/spi/factory/AbstractRuntimeDelegate
	at org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:202)
Caused by: java.lang.ClassNotFoundException: com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate
	at org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:202)
{code}
Bug: `RESTApiClusterManager` attempts to deserialize response using serialization API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Apr/20 00:27
Affects Version/s: 2.2.4

I'm not sure if this class ever worked, or if Gson changed their API behavior and we never noticed. The fix is quite simple, to use the streaming {{JsonParser}} API instead of the higher-level object API. However, testing this means standing up a web service that mocks Cloudera Manager response bodies.
Bug: Remove some stale hadoop.version properties in the pom for sub modules
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Apr/20 07:01

For example in hbase-client/pom.xml, we have this in the hadoop-3.0 profile

{code}
      <properties>
        <hadoop.version>3.0-SNAPSHOT</hadoop.version>
      </properties>
{code}

Which is useless and confusing.

Let's  purge them.
Bug: Move code in FSHDFSUtils to FSUtils and mark related classes as final
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Apr/20 03:45

It is a bit strange that a 'util' class can be instantiated.

I think the motivation is to provide multiple recoverFileLease implementation based on different filesystem implementation. But until now, there is still only one implementation for HDFS in HBase, and all the related classes are marked as IA.Private so we do not expect user to extend the class.

Let's just move the code in FSHDFSUtils to FSUtils directly and make FSUtils a true 'util' class.
Bug: Exclude jsr311-api from classpath
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 24/Apr/20 17:05
Affects Version/s: 2.3.0

When building on Hadoop3, we get two incompatible version of java-ws rs on the class path, both 1.1.1 and 2.0.1. These cause conflicts when running chaos monkey and integration test tools.

{noformat}
[INFO] |  |  +- org.apache.hadoop:hadoop-yarn-server-timelineservice:jar:3.1.2:test
[INFO] |  |  |  +- org.apache.commons:commons-csv:jar:1.0:test
[INFO] |  |  |  \- javax.ws.rs:jsr311-api:jar:1.1.1:test
{noformat}

{noformat}
[INFO] org.apache.hbase:hbase-http:jar:2.3.0-SNAPSHOT
[INFO] +- javax.ws.rs:javax.ws.rs-api:jar:2.0.1:compile
{noformat}

Problem looks like

{noformat}
20/04/23 15:36:04 INFO hbase.RESTApiClusterManager: Executing GET against ...
Exception in thread "ChaosMonkey" java.lang.NoSuchMethodError: 'void javax.ws.rs.core.MultivaluedMap.addAll(java.lang.Object, java.lang.Object[])'
        at org.glassfish.jersey.client.ClientRequest.accept(ClientRequest.java:336)
        at org.glassfish.jersey.client.JerseyWebTarget.request(JerseyWebTarget.java:221)
        at org.glassfish.jersey.client.JerseyWebTarget.request(JerseyWebTarget.java:59)
        at org.apache.hadoop.hbase.RESTApiClusterManager.getJsonNodeFromURIGet(RESTApiClusterManager.java:244)
{noformat}
Bug: TestExportSnapshotWithTemporaryDirectory fails with "No such file or directory"
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 29/Apr/20 17:34
Affects Version/s: 2.3.0

Running tests locally last night, I get this failure.

{noformat}
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: 
org.apache.hadoop.service.ServiceStateException: ExitCodeException exitCode=1: chmod: /private/tmp/hadoop-yarn-ndimiduk/node-attribute/nodeattribute.mirror.writing: No such file or directory

	at org.apache.hadoop.hbase.snapshot.TestExportSnapshotWithTemporaryDirectory.setUpBeforeClass(TestExportSnapshotWithTemporaryDirectory.java:43)
Caused by: org.apache.hadoop.service.ServiceStateException: 
ExitCodeException exitCode=1: chmod: /private/tmp/hadoop-yarn-ndimiduk/node-attribute/nodeattribute.mirror.writing: No such file or directory

	at org.apache.hadoop.hbase.snapshot.TestExportSnapshotWithTemporaryDirectory.setUpBeforeClass(TestExportSnapshotWithTemporaryDirectory.java:43)
Caused by: org.apache.hadoop.util.Shell$ExitCodeException: 
chmod: /private/tmp/hadoop-yarn-ndimiduk/node-attribute/nodeattribute.mirror.writing: No such file or directory

	at org.apache.hadoop.hbase.snapshot.TestExportSnapshotWithTemporaryDirectory.setUpBeforeClass(TestExportSnapshotWithTemporaryDirectory.java:43)
{noformat}
Bug: The -Phadoop-3.0 option does not work on branch-2 and branch-2.3 with JDK11
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Apr/20 09:07

Because we have an enforcer rule which requires the hadoop.profile property to be set to 3.0.

I tried to use requireActiveProfile but it failed when building hbase-annotations as in the pom of the module we do not declare a hadoop-3.0 profile(since it does not depend on hadoop...).

So we need to find another trick solution.
Bug: [JDK11] Switch default garbage collector from CMS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Apr/20 18:36
Affects Version/s: 2.3.0

When running HBase tools on the cli, one of the warnings generated is

{noformat}
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
{noformat}

Java9+ use G1GC as the default collector. Maybe we simply omit GC configurations and use the default settings? Or someone has some suggested settings we can ship out of the box?