av9ash/GitBugs huggingface dataset:
Bug: Region normalizer chore thread is getting killed
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jan/21 15:21
Affects Version/s: 1.6.0

Region normalizer chore thread is getting killed when the region is not found on any server.

As per the method
{code:java}
// code placeholder
/**
 * @param serverName
 * @return ServerLoad if serverName is known else null
 */
public ServerLoad getLoad(final ServerName serverName) {
  return this.onlineServers.get(serverName);
}
{code}
 So ideally we should check for the returned null in SimpleRegionNormalizer
Bug: ServerManager affects MTTR of HMaster
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jan/21 09:46
Affects Version/s: 3.0.0-alpha-1

https://github.com/apache/hbase/blob/3488c44a21612aae1835fc3e91a4a12ed2abb8b7/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java#L925

If a cluster has N+W regions, this removeDeletedRegionFromLoadedFlushedSequenceIds takes a long time...
 !image-2021-01-14-17-44-16-091.png! 
 !image-2021-01-14-17-44-42-181.png!
Bug: [hbase-operator-tools]Fix OOME "unable to create new native thread" on UTs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jan/21 11:44
Affects Version/s: hbase-operator-tools-1.0.0

Have noticed some UTs sporadically failing with OOME "unable to create new native thread". This does happen when running locally. Talked to [~psomogyi] and he was suggesting some environment changes to be set on docker personality script. Opening this one to try having this fixed.
Bug: Backup not using parameters such as bandwidth, workers, etc while exporting snapshot
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/21 07:00
Affects Version/s: 3.0.0-alpha-1

 
{code:java}
// code placeholder
$ sudo /etc/init.d/yak master hbase backup create 
Please make sure that backup is enabled on the cluster. To enable backup, in hbase-site.xml, set:
 hbase.backup.enable=true
hbase.master.logcleaner.plugins=YOUR_PLUGINS,org.apache.hadoop.hbase.backup.master.BackupLogCleaner
hbase.procedure.master.classes=YOUR_CLASSES,org.apache.hadoop.hbase.backup.master.LogRollMasterProcedureManager
hbase.procedure.regionserver.classes=YOUR_CLASSES,org.apache.hadoop.hbase.backup.regionserver.LogRollRegionServerProcedureManager
hbase.coprocessor.region.classes=YOUR_CLASSES,org.apache.hadoop.hbase.backup.BackupObserver
and restart the clusterUsage: hbase backup create <type> <backup_path> [options]
  type           "full" to create a full backup image
                 "incremental" to create an incremental backup image
  backup_path     Full path to store the backup imageOptions:
  -b <arg>        Bandwidth per task (MapReduce task) in MB/s
  -d              Enable debug loggings
  -q <arg>        Yarn queue name to run backup create command on
  -s <arg>        Backup set to backup, mutually exclusive with -t (table list)
  -t <arg>        Table name list, comma-separated.
  -w <arg>        Number of parallel MapReduce tasks to execute {code}
Parameters -b, -q, -w are not being used when creating a export snapshot request 
{code:java}
// code placeholder

for (TableName table : backupInfo.getTables()) {
  // Currently we simply set the sub copy tasks by counting the table snapshot number, we can
  // calculate the real files' size for the percentage in the future.
  // backupCopier.setSubTaskPercntgInWholeTask(1f / numOfSnapshots);
  int res;
  String[] args = new String[4];
  args[0] = "-snapshot";
  args[1] = backupInfo.getSnapshotName(table);
  args[2] = "-copy-to";
  args[3] = backupInfo.getTableBackupDir(table);

  String jobname = "Full-Backup_" + backupInfo.getBackupId() + "_" + table.getNameAsString();
  if (LOG.isDebugEnabled()) {
    LOG.debug("Setting snapshot copy job name to : " + jobname);
  }
  conf.set(JOB_NAME_CONF_KEY, jobname);

  LOG.debug("Copy snapshot " + args[1] + " to " + args[3]);
  res = copyService.copy(backupInfo, backupManager, conf, BackupType.FULL, args);

  // if one snapshot export failed, do not continue for remained snapshots
  if (res != 0) {
    LOG.error("Exporting Snapshot " + args[1] + " failed with return code: " + res + ".");

    throw new IOException("Failed of exporting snapshot " + args[1] + " to " + args[3]
        + " with reason code " + res);
  }

  conf.unset(JOB_NAME_CONF_KEY);
  LOG.info("Snapshot copy " + args[1] + " finished.");
} {code}
Bug: WALKey Extended Attributes don't serialize to ReplicationSink
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jan/21 23:29
Affects Version/s: 1.5.0

HBASE-22622 introduced extended attributes on the WALKey object and protobuf, and HBASE-22623 created a coprocessor hook, preWALAppend, so that coprocessors can create and insert their own extended attributes. 

These attributes are readable on the source-side, such as in a custom ReplicationEndpoint. However, in branch-1 ReplicationProtbufUtil doesn't correctly populate the extended attributes on the WALKey protobuf. (In 2.1+, HBASE-20625 incidentally fixes this as part of a larger refactoring of WALCellCodec logic.)

This means that a custom ReplicationSink can't read extended attributes on a WALKey.
Bug: At a glance git branch release history tool
Status: Open | Priority: Major | Resolution: 
Created: 15/Jan/21 18:39

We do not have a single source of truth as to what are active branches and what was the most recent release by version number made from each branch. If we had such a resource, then committers could refer to it whenever merging PRs or cherry picking commits backward. When merging a PR into a branch or committing a cherry pick it is important that the committer, at that time, go to JIRA and update fix versions on the corresponding issue, so that at the next release the change will be correctly included in the change log. Perhaps something can be created that takes the union of git branch listing and reporter.a.o release version history.
Bug: [branch-2.4] Restore method removed by HBASE-25277 to LP(CONFIG) coprocessors
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/21 17:54
Affects Version/s: 2.4.1

We can't do this in a patch release on a LimitedPrivate interface:
{code}
-  @Override
-  public boolean postScannerFilterRow(final ObserverContext<RegionCoprocessorEnvironment> e,
-      final InternalScanner s, final Cell curRowCell, final boolean hasMore) throws IOException {
-    // 'default' in RegionObserver might do unnecessary copy for Off heap backed Cells.
-    return hasMore;
-  }
{code}
Bug: IntegrationTestMTTR fails with TableNotFoundException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/21 10:18

Admin.getDescriptor() fails beacuse table doesn't exist.

 
{noformat}
org.apache.hadoop.hbase.TableNotFoundException: IntegrationTestMTTR

	at java.lang.Thread.getStackTrace(Thread.java:1559)
	at org.apache.hadoop.hbase.util.FutureUtils.setStackTrace(FutureUtils.java:130)
	at org.apache.hadoop.hbase.util.FutureUtils.rethrow(FutureUtils.java:149)
	at org.apache.hadoop.hbase.util.FutureUtils.get(FutureUtils.java:172)
	at org.apache.hadoop.hbase.client.AdminOverAsyncAdmin.getDescriptor(AdminOverAsyncAdmin.java:163)
	at org.apache.hadoop.hbase.mttr.IntegrationTestMTTR.setupTables(IntegrationTestMTTR.java:236)
	at org.apache.hadoop.hbase.mttr.IntegrationTestMTTR.setUp(IntegrationTestMTTR.java:179)
	
{noformat}
Bug: HBase code download is failing on windows with invalid path error
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/21 15:47

git pull command is failing with "error: invalid path"
{noformat}
Host1 MINGW64 /d/hbase (master)
$ git pull
error: invalid path 'dev-support/design-docs/HBASE-18070-ROOT_hbase:meta_Region_Replicas.pdf'
Updating 2e96a5b2d3..dfefff7e59{noformat}
This problem is coming only on windows machines. Tried on windows 7 and windows 10 both have problem. Searched on net, seems no any easy solution.

Problem is because file HBASE-18070-ROOT_hbase:meta_Region_Replicas.pdf has colon in its name.

To fix the problem we should remove the colon : from file name
Bug: Enable error prone check in pre commit
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jan/21 13:58

This has been a pain in the past as the output is not stable, but in HBASE-25333, we found that some checks in error prone is really useful.

So here I think we need to find a way to try to enable it again in pre commit.

Maybe we could only check for ERROR level problems as it will cause compilation error.
Bug: False-positive findbugs issue on branch-2.2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/21 09:15
Affects Version/s: 2.2.7

Findbugs reports a false-positive issue.

new org.apache.hadoop.hbase.rest.client.Client(Cluster, String, Optional, Optional) may fail to clean up java.io.InputStream

Try-with-resource block is used so the resource is cleaned up but findbugs thinks it otherwise. I tried to reproduce the problem locally and using the same docker container nightly test is using but findbugs did not report the problem for me.

Since branch-2.2 will be EOL soon I'd like to just add SuppressWarning to ignore the false-positive findbugs warning.
Bug: set the loadMeta log level to debug.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jan/21 06:39
Affects Version/s: 2.2.3

https://github.com/apache/hbase/blob/2444d268901644d90def3fca39505627ff956b40/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java#L167
test 100w Regions, the log level is info, it takes more than 250 seconds to load metadata. The log is debug. It takes more than 100 seconds to load metadata.
Bug: [git-jira-audit] improve performance of --parse-release-tags
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jan/21 21:29

The {{--parse-release-tags}} flag is necessary for the more useful reports. With this mode enabled, execution time of building the audit database skyrockets. I found a python profiler, which points to {{apply_git_tag}} function being used in a tight loop.
Bug: [create-release] changes.md update broken
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jan/21 18:24

Adding logging broke the changes/releasenotes in create-release. A run last night failed w/ the below.

{code}
 2021-01-08T07:23:23Z update_releasenotes start
 JIRA returns HTTP error 400: 400. Aborting.
 - Please ensure that specified authentication, projects, fixVersions etc. are correct.
 - The value '2021-01-08T07:23:23Z HBASE' does not exist for the field 'project'.
 2021-01-08T07:24:01Z update_releasenotes stop (38 seconds)
{code}
Bug: [connectors] Purge use of VisibleForTesting
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jan/21 18:31
Affects Version/s: hbase-connectors-1.0.1

HBASE-24640 purged VisibleForTesting from the main repository but hbase-connector still uses this annotation in KafkaBridgeConnection which is marked as IA.Private.
Bug: Fix assembly of the hbase-shaded-testing-util*.jar; can't unzip on case-insensitive fs
Status: Open | Priority: Major | Resolution: 
Created: 08/Jan/21 17:41

See parent issue. We add exclusion of hbase-shaded-testing-utility.* because unzip fails on case-insensitive fs because the jar has a META-INF/LICENSE and then a directory to hold aggregated licenses at META-INF/license.

This issue is about reorg'ing the jar; e.g. name the license dir as 'licenses' instead.
Bug: [filesystem] Purge use of VisibleForTesting
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jan/21 16:48
Affects Version/s: hbase-filesystem-1.0.0-alpha1

HBASE-24640 purged VisibleForTesting from the main repository but hbase-filesystem still uses this annotation in a few classes. After replacing these we can remove the hbase-thirdparty dependency.
Bug: Add unitest for HBASE-25445 - SplitWALRemoteProcedure failed to archive split WAL
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 07/Jan/21 00:50
Affects Version/s: 2.2.6

If 'hbase.wal.dir' and 'hbase.rootdir' are configured to different filesystem, SplitWALRemoteProcedure archived split WAL failed since SplitWALManager using wrong fs instance. SplitWALManager should use WAL corresponding fs instance.

Steps to Reproduce:
 * Configure 'hbase.wal.dir' and 'hbase.rootdir' so that they point to different fs instances.
 * Start HBase with multiple RS. 
 * Create a couple of tables and some rows in them so that the RSs get assigned with some regions. 
 * Take any RS with non-zero number of regions offline. 
 * Check master logs for "Wrong FS" error as shown in the screenshot attached.
Bug: hbase-thriftpy2.transport.base.TTransportException
Status: Open | Priority: Major | Resolution: 
Created: 05/Jan/21 08:43

 

fname, mtype, rseqid = self._iprot.read_message_begin()
 File "thriftpy2/protocol/cybin/cybin.pyx", line 429, in cybin.TCyBinaryProtocol.read_message_begin
 File "thriftpy2/protocol/cybin/cybin.pyx", line 60, in cybin.read_i32
 File "thriftpy2/transport/buffered/cybuffered.pyx", line 65, in thriftpy2.transport.buffered.cybuffered.TCyBufferedTransport.c_read
 File "thriftpy2/transport/buffered/cybuffered.pyx", line 69, in thriftpy2.transport.buffered.cybuffered.TCyBufferedTransport.read_trans
 File "thriftpy2/transport/cybase.pyx", line 61, in thriftpy2.transport.cybase.TCyBuffer.read_trans
 File "/Users/daniel/opt/miniconda3/lib/python3.8/site-packages/thriftpy2/transport/socket.py", line 131, in read
 raise TTransportException(type=TTransportException.END_OF_FILE,
thriftpy2.transport.base.TTransportException: TTransportException(type=4, message='TSocket read 0 bytes')
Bug: ArrayIndexOutOfBoundsException thrown when table CF name is "#"
Status: Patch Available | Priority: Minor | Resolution: 
Created: 01/Feb/21 07:40

ArrayIndexOutOfBoundsException will be thrown when CF name is "#",

https://github.com/apache/hbase/blob/a04ea7ea4493f5bc583b4d08a2a6a88e7c6b8c54/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsTableSourceImpl.java#L340


{noformat}
2021-01-30 00:11:14,172 | ERROR | HBase-Metrics2-1 | Error getting metrics from source RegionServer,sub=Tables | org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:202)
java.lang.ArrayIndexOutOfBoundsException: 1
        at org.apache.hadoop.hbase.regionserver.MetricsTableSourceImpl.addGauge(MetricsTableSourceImpl.java:336)
        at org.apache.hadoop.hbase.regionserver.MetricsTableSourceImpl.snapshot(MetricsTableSourceImpl.java:321)
        at org.apache.hadoop.hbase.regionserver.MetricsTableAggregateSourceImpl.getMetrics(MetricsTableAggregateSourceImpl.java:98)
        at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:200)
        at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:183)
        at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:156)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassName(DefaultMBeanServerInterceptor.java:333)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:319)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
        at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:67)
        at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:222)
        at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:101)
        at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:268)
        at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(MetricsSystemImpl.java:239)
        at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsSystemImpl.java:324)
        at com.sun.proxy.$Proxy7.postStart(Unknown Source)
        at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:193)
        at org.apache.hadoop.metrics2.impl.JmxCacheBuster$JmxCacheBusterRunnable.run(JmxCacheBuster.java:109)
{noformat}
Bug: Support multiple tables as input in generateMissingTableDescriptorFile command in HBCK2
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Feb/21 13:33

Currently 'generateMissingTableDescriptorFile' command in HBCK2 supports only 1 TableName as input. HBCK's _fixOrphanTables()_ had a support for fixing the missing .tableinfo files for a list of tables and it also supported fixing the missing descriptor for all tables if no tables were specified.

It looks like a convinient  enhancement to have in generateMissingTableDescriptorFile command of HBCK2.
Bug: Use javac --release option for supporting cross version compilation
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Jan/21 20:22
Affects Version/s: 3.0.0-alpha-3

See https://www.morling.dev/blog/bytebuffer-and-the-dreaded-nosuchmethoderror/

{quote}
 the Java compiler’s --release parameter, which was introduced via JEP 247 ("Compile for Older Platform Versions"), added to the platform also in JDK 9. In contrast to the more widely known pair of --source and --target, the --release switch will ensure that only byte code is produced which actually will be usable with the specified Java version. For this purpose, the JDK contains the signature data for all supported Java versions (stored in the $JAVA_HOME/lib/ct.sym file).
{quote}

Using one JDK (i.e. Java 11) to build Java 8-and-up and Java 11-and-up compatible release artifacts would reduce some sources of accidental complexity, assuming the --release parameter actually works as advertised. To produce Java 8-and-up compatible artifacts, supply --release=8. To produce Java 11-and-up compatible release artifacts, supply --release=11. Maven activations based on JDK version and command line defined profiles can control what --release parameter, if any, should be passed to the compiler.
Bug: Thread pools should release unused resources
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Feb/21 19:06
Affects Version/s: 1.7.0

On this [PR|https://github.com/apache/hbase/pull/2909] [~stack] rightly pointed out that the thread pool just spins up threads that remain idle for 99.999% of time. Java's [ThreadPoolExecutor|https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/concurrent/ThreadPoolExecutor.html] lets us tune some configurations like the following for releasing the unused resources

* corePoolSize
* keepAliveTimeMs
* allowCoreThreadTimeOut

Current code makes it difficult to pass along these configurations to the executor service. Refactor and fix the defaults for low-priority thread pools to release resources quicker.
Bug: There is a problem with the SPLITS_FILE in the HBase shell statement
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Feb/21 02:06
Affects Version/s: 2.0.1

The Ruby script here has a problem passing in a null value:
 hbase-shell/src/main/ruby/hbase/admin.rb
{code:java}
if arg.key?(SPLITS_FILE)
 splits_file = arg.delete(SPLITS_FILE)
 unless File.exist?(splits_file)
 raise(ArgumentError, "Splits file #{splits_file} doesn't exist")
 end
 arg[SPLITS] = []
 File.foreach(splits_file) do |line|
 arg[SPLITS].push(line.chomp)
 end
 tdb.setValue(SPLITS_FILE, arg[SPLITS_FILE]){code}
Bug: Excessive logging of "hbase.zookeeper.useMulti is deprecated. Default to true always."
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Feb/21 19:12
Affects Version/s: 2.4.1

Log line printed frequently, can be as often as once per second:
{noformat}
21:56:46.381 WARN [RS_REFRESH_PEER-regionserver/ip-172-31-51-4:8120-0.replicationSource,
1.replicationSource.shipperip-172-31-51-4.us-west-2.compute.internal%2C8120%2C1613596855628.ip-172-31-51-4.us-west-2.compute.internal%2C8120%2C1613596855628.default,1]
   org.apache.hadoop.hbase.zookeeper.ZKUtil - hbase.zookeeper.useMulti is deprecated. Default to true always.
{noformat}

The old configuration in place had hbase.zookeeper.useMulti  set to true. This is the default and while I understand this setting is deprecated, constantly warning about it, especially when it conforms to the default, is log spam. 

Warn about this once over the lifetime of the replication source.
Bug: Non-idempotent test in TestReplicationHFileCleaner
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Mar/21 02:57

The test *{{org.apache.hadoop.hbase.master.cleaner.TestReplicationHFileCleaner.testIsFileDeletable}}* is not idempotent and fail if run twice in the same JVM, because it pollutes some states shared among tests. It may be good to clean this state pollution so that some other tests do not fail in the future due to the shared state polluted by this test.
h3. Detail

Running {{TestReplicationHFileCleaner.testIsFileDeletable}} twice would result in the second run failing due to the following assertion error:
{noformat}
java.lang.AssertionError: Cleaner should allow to delete this file as there is 

no hfile reference node for it in the queue.
{noformat}
The root cause is that the a hfile reference is added during the first test run, which doesn't get removed upon test exits. Therefore, in the second test run , {{cleaner.isFileDeletable(fs.getFileStatus(file)))}} would return {{false}}, resulting in the assertion error.

 

PR link: https://github.com/apache/hbase/pull/2984
Bug: Add detailed RIT info in JSON format for consumption as metrics
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Jan/21 00:43
Affects Version/s: 2.3.7

In HBase 2.1+, there is a RIT jsp page that was added as part of HBASE-21410.

There are some additional RIT details that would be helpful to have in one place:
 * RIT Start Time
 * RIT Duration (ms)
 * Server
 * Procedure Type

This info can be added to the table under the {{/rit.jsp}} page, and we can also add a button on that page to view info as JSON, for easy parsing into metrics, etc. This JSON dump can be served as a servlet.

We may also consider different ways of grouping the JSON results, such as by state, table, or server name, and/or adding counts of RIT by state or server name.

!Screen Shot 2021-07-27 at 10.34.45.png!

!Screen Shot 2021-07-27 at 10.34.53.png!