av9ash/GitBugs huggingface dataset:
Bug: UnsafeAvailChecker should also check that required methods are available
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/May/20 23:21

We had a weird test failure due to accidentally running tests with some Java > 8 (exact version I am unsure of), where Unsafe is available, but the method signatures were different or some methods were removed, leading to this:
{noformat}
020-05-02 14:57:15,145 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster
    at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:143)
    at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:237)
    at org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseCluster.java:163)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:225)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:138)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127)
    at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2911)
Caused by: java.lang.NoSuchMethodError: 'void sun.misc.Unsafe.putInt(java.lang.Object, int, int)'
    at org.apache.hadoop.hbase.util.UnsafeAccess.putInt(UnsafeAccess.java:233)
    at org.apache.hadoop.hbase.util.Bytes$ConverterHolder$UnsafeConverter.putInt(Bytes.java:1499)
    at org.apache.hadoop.hbase.util.Bytes.putInt(Bytes.java:1021)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.appendMetaData(RecoverableZooKeeper.java:850)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:640)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.createEphemeralNodeAndWatch(ZKUtil.java:1027)
    at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.setMasterAddress(MasterAddressTracker.java:211)
    at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2095)
    at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:520)
    at org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.<init>(HMasterCommandLine.java:315)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:138)
    ... 7 more
{noformat}

We should also check that all methods that will be invoked on Unsafe in UnsafeAccess.java are available when deciding in UnsafeAvailChecker if Unsafe is available (and usable).
Bug: Improve WAL splitting log lines to enable sessionization
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/May/20 20:19

Looking to reconstruct a timeline from write of recovered.edits file back to start of WAL file split, with a bunch of unrelated activity in the meantime, there isn't a consistent token that links split file write messages (which include store path including region hash) to beginning of WAL splitting activity. Sessonizing by host doesn't work because work can bounce around through retries. Thread context names in the logs vary and can be like [nds1-225-fra:60020-7] or [fb472085572ba72e96f1] (trailing digits of region hash) or [splits-1589016325868] . 

We could have WALSplitter get the current time when starting the split of a WAL file and have it log this timestamp in every line as a splitting session identifier.

Related, we should track the time of split task execution end to end and export a metric that captures it.

It might also be worthwhile to wire up more of WAL splitting to TaskMonitor status logging. If we do this we can also enable status journal logging, so when a WAL split has completed, a line will appear in the log that has the list of all status messages recorded during splitting and the time delta in milliseconds between them.
Bug: The regionInfo parameter for MasterProcedureScheduler#waitRegions and MasterProcedureScheduler#wakeRegions should be plural 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 31/May/20 13:44
Affects Version/s: 3.0.0-alpha-1

MasterProcedureScheduler#waitRegions and MasterProcedureScheduler#wakeRegions deal with a list of regions, so the variable name of region info should be plural
Bug: Use EnvironmentEdgeManager to compute clock skew in Master
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/May/20 16:29
Affects Version/s: 1.6.0

There are few cases where the Master is not able to complete the initialization as it waiting for the region server to report to it. The region server actually reported to the master but the master rejected the request because of clock skew issue though both of them are on  same JVM

The Region server uses EnvironmentEdgeManager.currentTime to report the current time and HMaster uses System.currentTimeMillis() to get the current time for computation against the reported time by RS.  We should also just use EnvironmentEdgeManager even in Master as we are expected not to use System.currentTime directly and instead go through EnvironmentEdgeManager
Bug: ScheduledChore logging is too chatty, replace with metrics
Status: Open | Priority: Minor | Resolution: 
Created: 11/Jun/20 23:48

ScheduledChore logs at DEBUG level the execution time of each chore. 

We used to log an average execution time across all chores every five minutes, which by consensus was judged to not be useful. Derived metrics like averages or histograms should be calculated per chore. So we modified the logging to dump the chore execution time each time it runs, to facilitate such calculations with the log aggregation and searching tool of choice. Per chore execution logging is more useful, in that sense, but may be too chatty. This is not unexpected but let me provide my observations so we can revisit this.

On the master, for example, this is logged every second:
{noformat}
2020-06-11 16:35:28,263 DEBUG [master/apurtell-ltm:8100.splitLogManager..Chore.1] hbase.ScheduledChore: SplitLogManager Timeout Monitor execution time: 0 ms.
{noformat}

Does the value of these lines outweigh the cost of 86,400 log lines per day per master instance? (At least.)

On the regionserver it is somewhat better, these are logged every 10 seconds:
{noformat}
2020-06-11 16:37:57,203 DEBUG [regionserver/apurtell-ltm:8120.Chore.1] hbase.ScheduledChore: CompactionChecker execution time: 0 ms.
2020-06-11 16:37:57,203 DEBUG [regionserver/apurtell-ltm:8120.Chore.1] hbase.ScheduledChore: MemstoreFlusherChore execution time: 0 ms.
{noformat}

So that will be 17,280 log lines per day per regionserver. (At least.)

These could be moved to TRACE level, perhaps. 

I propose we replace this logging with histogram metrics. There should be a separate metric for each distinct chore classname, allocated as needed.
Bug: In HBase UI, srcChecksum has bad content
Status: Open | Priority: Minor | Resolution: 
Created: 19/Jun/20 00:12
Affects Version/s: 2.3.0

The value of srcChecksum displayed in the HBase UI is generated at build time by https://github.com/apache/hbase/blob/master/hbase-common/src/saveVersion.sh#L53-L61

On a Linux build host,
if `openssl`(v 1.1.1) is available, the resulting value of srcChecksum is `(stdin)=`
else if `gpg` (v 2.2.19) is available, the value is only the first 8-character substring of the sha512 hash.

On a Mac build host,
if `openssl`(LibreSSL 2.8.3) is available, the resulting value of srcChecksum is correct
else if `gpg` (v 2.2.19) is available, the value is again the first 8-character substring of the sha512 hash.

Furthermore, while "sha512" is well defined and gives the same value from all four programs, the associated leader text for file names has a different format, and gpg generates uppercase hex divided into 8-character blocks while openssl generates lowercase hex in a single string.  Either of these differences will cause the sum-of-sums generated by openssl and gpg to be different!  Which is unacceptable if srcChecksum is to be used for validation.
Bug: Persist map of regions-to-last-flush-sequenceid across restarts
Status: Open | Priority: Minor | Resolution: 
Created: 17/Jun/20 18:58

A nice suggestion by our [~ndimiduk] was that we persist the last-successful-flush-sequenceid per region kept up by an in-memory map in Master. We could add it to our nice new general local master store. On restart, could rehydrate the map. Would be helpful on cold start in particular, especially if a big backlog of WALs to replay.
Bug: Changing "whitelist" and "blacklist" in our docs and project.
Status: Open | Priority: Minor | Resolution: 
Created: 16/Jun/20 21:54

Replace instances of “whitelist” and “blacklist” in our project, trails, documentation and UI text. 
We can replace  blacklist with blocklist, blocklisted, or block  and whitelist with allowlist, allowlisted, or allow.
In my current workplace they are suggesting us to make this change. Also Google has an exhaustive list to write inclusive documentation: https://developers.google.com/style/inclusive-documentation

There might be few issues while making the change.
1. If these words are part of config property name then all customers needs to make the change.
2. There might be some client-server compatibility if we change server side variables/method names.

Creating this jira just to start the conversation. Please chip in with ideas.
Bug: Expose a function in HBase WALKey to set the tablename
Status: Resolved | Priority: Minor | Resolution: Abandoned
Created: 21/May/20 07:52

Currently, table name in WALKey can not be changed once set. But exposing this function to change the table name can be very helpful for Customized WAL filters since they can flip the table name and make replication possible between different table names in source and sink clusters.
Bug: Delete reference off to Hadoop wiki's HBase FAQ
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Jun/20 19:09

Our `faq.adoc` has a link off to [a FAQ|https://cwiki.apache.org/confluence/display/HADOOP2/Hbase+FAQ] in the Hadoop wiki, which is empty other than a pointer back to our book. Let's just delete the reference to the hadoop wiki.
Bug: Change the IA for MutableSizeHistogram and MutableTimeHistogram
Status: Open | Priority: Minor | Resolution: 
Created: 08/Jun/20 16:55

Currently the IA for MutableSizeHistogram and MutableTimeHistogram is private. We want to use these classes in phoenix project and I thought we can leverage the existing implementation from hbase histo implementation. IIUC the private IA can't be used in other projects. Proposing to make it LimitedPrivate and mark HBaseInterfaceAudience.PHOENIX. Please suggest.
Bug: Backport HBASE-23896 to branch-1: Snapshot owner cannot delete snapshot when ACL is enabled and Kerberos is not enabled
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Apr/20 10:24

For the owner of snapshots(not global admin user), currently list_snapshots returns empty if i just use simple acls for authorization but not use authentication.

The code in AccessController.preListSnapshot:
{code:java}
if (SnapshotDescriptionUtils.isSnapshotOwner(snapshot, user)) {
// list it, if user is the owner of snapshot
AuthResult result = AuthResult.allow("listSnapshot " + snapshot.getName(),
"Snapshot owner check allowed", user, null, null, null);
accessChecker.logResult(result);
}{code}
And SnapshotManager.takeSnapshotInternal:
{code:java}
if (User.isHBaseSecurityEnabled(master.getConfiguration()) && user != null) {
  builder.setOwner(user.getShortName());
}
{code}
User.isHBaseSecurityEnabled：
{code:java}
public static boolean isHBaseSecurityEnabled(Configuration conf) {
  return "kerberos".equalsIgnoreCase(conf.get(HBASE_SECURITY_CONF_KEY));
}
{code}
So i think the logic of setOwner is used for authorization, not authentication, SnapshotManager should not only setOwner when hbase.security.authentication = kerberos, which cause listSnapshots returns empty when i just use simple acls.
Bug: Migrate Seconding Indexing documentation from Hadoop wiki to our book
Status: Open | Priority: Minor | Resolution: 
Created: 10/Jun/20 19:04

I noticed we have a link in `cp.adoc` over to [a page|https://cwiki.apache.org/confluence/display/HADOOP2/Hbase+SecondaryIndexing] in the Hadoop wiki that pertains to secondary indexing. Hadoop Wiki has been deprecated as a reference point for HBase since HBASE-14481, but seems new content snuck in since then. We should take ownership over this advice.
Bug: ScheduledChore log elapsed timespan in a human-friendly format
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/May/20 22:30
Affects Version/s: 2.3.0

I noticed this in a log line,

{noformat}
2020-04-23 18:31:14,183 INFO org.apache.hadoop.hbase.ScheduledChore: host-a.example.com,16000,1587577999888-ClusterStatusChore average execution time: 68488258 ns.
{noformat}

I'm not sure if there's a case when elapsed time in nanoseconds is meaningful for these background chores, but we could do a little work before printing the number and time unit to truncate precision down to something a little more intuitive for operators. This number purports to be an average, so a high level of precision isn't necessarily meaningful.

Separately, or while we're here, if we think an operator really cares about the performance of this chore, we should print a histogram of elapsed times, rather than an opaque average.
Bug: PerformanceEvaluation options should not mandate any specific order
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/May/20 17:07
Affects Version/s: 2.1.0

During parsing of options, there are some validations.  One such is checking whether autoFlush = false AND multiPut > 0.  This validation code mandates an order that autoFlush=true should be specified before adding multiPut = x in PE command.
{code}
final String multiPut = "--multiPut=";
      if (cmd.startsWith(multiPut)) {
        opts.multiPut = Integer.parseInt(cmd.substring(multiPut.length()));
        if (!opts.autoFlush && opts.multiPut > 0) {
          throw new IllegalArgumentException("autoFlush must be true when multiPut is more than 0");
        }
        continue;
      }
{code}
'autoFlush ' default value is false. If multiPut is specified prior to autoFlush in the PE command, we will end up throwing IllegalArgumentException.
Checking other validations, seems not having such issue.  Still better to move all the validations together into a private method and call that once the parse is over.
Bug: Add repeated prefix logging for MultipleColumnPrefixFilter
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Jun/20 08:42

Current it is not easy to find which one is repeated if the input prefixes are not a few.

 
{code:java}
 java.lang.IllegalArgumentException: prefixes must be distinct
        at org.apache.hadoop.hbase.filter.MultipleColumnPrefixFilter.<init>(MultipleColumnPrefixFilter.java:50)
 

{code}
Bug: ITBLL, ChaosMonkey log message sheer/interleave
Status: Open | Priority: Minor | Resolution: 
Created: 05/Jun/20 22:01
Affects Version/s: 2.3.0

Running {{IntegrationTestBigLinkedList}} with {{ServerKillingChaosMonkey}}, I've noticed that some log messages are sheered/interwoven. This is particularly visible via the output of the {{DumpClusterStatusAction}}. I suspect we are running two logging instances simultaneously, rather than all loggers going through the same instance, and they're not coordinating on the output stream.

I'm running ITBLL against a distributed/external cluster, launched via the {{bin/hbase}} script.
Bug: Backport to branch-1 HBASE-17738 BucketCache startup is slow
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Jun/20 10:02

I'd like to backport [HBASE-17738|https://issues.apache.org/jira/browse/HBASE-17738] to branch-1.

Also removed some Unnecessary locks related to [HBASE-15785|https://issues.apache.org/jira/browse/HBASE-15785]
Bug: ExportSnapshot which does not honor the case that working dir/root dir are from different filesystem
Status: Open | Priority: Minor | Resolution: 
Created: 02/Jun/20 23:58
Affects Version/s: 2.3.0

HBASE-21098 introduce a config "hbase.snapshot.working.dir", which allows tmp directory for snapshot manifest files to be on a different filesystem than rootdir's. ExportSnapshot does not seem honor this config and assumes they are the same filesystem.
Bug: Remove the unused method online in admin.rb
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Jun/20 14:43

Skiming the todo list, seems this method not used any more.

 
{code:java}
# Enables/disables a region by name
def online(region_name, on_off)
  # Open meta table
  meta = @connection.getTable(org.apache.hadoop.hbase.TableName::META_TABLE_NAME)

  # Read region info
  # FIXME: fail gracefully if can't find the region
  region_bytes = region_name.to_java_bytes
  g = org.apache.hadoop.hbase.client.Get.new(region_bytes)
  g.addColumn(org.apache.hadoop.hbase.HConstants::CATALOG_FAMILY, org.apache.hadoop.hbase.HConstants::REGIONINFO_QUALIFIER)
  hri_bytes = meta.get(g).value

  # Change region status
  hri = org.apache.hadoop.hbase.util.Writables.getWritable(hri_bytes, org.apache.hadoop.hbase.HRegionInfo.new)
  hri.setOffline(on_off)

  # Write it back
  put = org.apache.hadoop.hbase.client.Put.new(region_bytes)
  put.addColumn(org.apache.hadoop.hbase.HConstants::CATALOG_FAMILY,
                org.apache.hadoop.hbase.HConstants::REGIONINFO_QUALIFIER,
                org.apache.hadoop.hbase.util.Writables.getBytes(hri))
  meta.put(put)
end
{code}
Bug: Flaky tests for region server in branch-1
Status: Resolved | Priority: Minor | Resolution: Not A Bug
Created: 27/May/20 22:47
Affects Version/s: 1.7.0

[https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1781/4/testReport/]

Tests failed after unrelated code style changes for metrics for branch-1. Rerun passed.
Bug: RetryCounter#sleepUntilNextRetry should use uninterrupted sleep
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 18/May/20 15:15

RetryCounter utility which is used at multiple places like MetaRegionLocationCache, RecoverableZookeeper etc provides a method to sleep for a back off time as supplied by the backoff policy. However, sleep used by sleepUntilNextRetry() makes thread sleep and throws InterruptedException which clients have to deal with. Clients using sleep for any backoff policy should not worry about handling InterruptedException. Dealing with InterruptedException is something for sleepUntilNextRetry() to take care of.
Bug: [Flakey Tests] Some UTs about RSGroup should wait RSGroupInfoManager to be online
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/May/20 12:54
Affects Version/s: 2.3.0

We will access table hbase:rsgroup when call addRSGroup, so we should ensure RSGroupInfoManagerImpl is online before testing in the UTs about RSGroup.

Otherwise, the following exceptions may be saw.
{code:java}
java.io.IOException: java.io.IOException: Only servers in default group can be updated during offline modejava.io.IOException: java.io.IOException: Only servers in default group can be updated during offline mode at org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl.flushConfig(RSGroupInfoManagerImpl.java:602) at org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl.addRSGroup(RSGroupInfoManagerImpl.java:217) at org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.addRSGroup(RSGroupAdminServer.java:391)
{code}
Bug: [flakey test] locking.TestEntityLocks.testEntityLockTimeout
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Apr/20 23:24
Affects Version/s: 2.3.0

From 2.3 flakey board, [https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-2.3/199/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.locking.TestEntityLocks.txt]

 
{code:java}
-------------------------------------------------------------------------------
Test set: org.apache.hadoop.hbase.client.locking.TestEntityLocks
-------------------------------------------------------------------------------
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.917 s <<< FAILURE! - in org.apache.hadoop.hbase.client.locking.TestEntityLocks
org.apache.hadoop.hbase.client.locking.TestEntityLocks.testEntityLockTimeout  Time elapsed: 2.022 s  <<< FAILURE!
java.lang.AssertionError
	at org.apache.hadoop.hbase.client.locking.TestEntityLocks.testEntityLockTimeout(TestEntityLocks.java:178)

Mapping to the code, it complains 
assertFalse(lock.getWorker().isAlive());


 {code}
Bug: release scripts should allow for creation of initial CHANGES/RELEASE on new major version.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/May/20 00:54

Right now if you try to run the release scripts on a new major version, e.g. one without a RELEASENOTES.md and CHANGES.md things fail:

{code}
/opt/hbase-rm/output
+ sed -i -e '/^## Release 3.0.0-alpha-1/,/^## Release/ {//!d; /^## Release 3.0.0-alpha-1/d;}' /opt/hbase-rm/output/hbase/CHANGES.md
sed: can't read /opt/hbase-rm/output/hbase/CHANGES.md: No such file or directory
+ true
+ sed -i -e '/^# HBASE  3.0.0-alpha-1 Release Notes/,/^# HBASE/{//!d; /^# HBASE  3.0.0-alpha-1 Release Notes/d;}' /opt/hbase-rm/output/hbase/RELEASENOTES.md
sed: can't read /opt/hbase-rm/output/hbase/RELEASENOTES.md: No such file or directory
+ true
+ sed -i -e '/## Release/,$!d' CHANGELOG.3.0.0-alpha-1.md
+ sed -i -e '/DO NOT REMOVE/r CHANGELOG.3.0.0-alpha-1.md' /opt/hbase-rm/output/hbase/CHANGES.md
sed: can't read /opt/hbase-rm/output/hbase/CHANGES.md: No such file or directory
+ cleanup
{code}

instead we should just create the new files.
Bug: MetricsTableWrapperAggregateImpl runnable fails due to exception and never runs
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/May/20 14:45
Affects Version/s: 2.1.9

MetricsTableWrapperAggregateImpl has a thread that periodically updates the values.
It seems that once the region server is online and the meta is assigned if there are no store files the Optional values  here
{code}
            mt.maxStoreFileAge = Math.max(mt.maxStoreFileAge, store.getMaxStoreFileAge().getAsLong());
            mt.minStoreFileAge = Math.min(mt.minStoreFileAge, store.getMinStoreFileAge().getAsLong());
{code}
is not available and thus throwing NoSuchElementException and the thread dies. It never updates the values till the RS is alive.
Bug: TableStateNotFoundException happends when table creation if rsgroup is enable
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Apr/20 19:18
Affects Version/s: 2.2.3

IF RS group is enabled then Unable to get table state error is thrown while creating a new table

 

Stpes:

1: Make sure RS group feature is enabled

2: Create a table say usertable2000

3: Check master log and observe below exception is thrown  org.apache.hadoop.hbase.master.TableStateManager$TableStateNotFoundException: usertable2000
Bug: Procedure related metrics is not consumed in the JMX metric
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Apr/20 11:49

We have ProcedureMetrics and that is being tracked for every procedure that we create for all the ops in the system. 
But when we check the UI, the UI does not display those information at all. It may be useful to know atleast in the case of ServerCrashProcedure exactly to know how much it has taken for the procedure to complete. Similarly other procedures also can be added to the UI. 
Currently checked in master code - but think it will apply to other branches also.