av9ash/GitBugs huggingface dataset:
Bug: Fixing hacky test in TestCoprocessorInterface
Status: Open | Priority: Major | Resolution: 
Created: 05/Nov/20 20:13
Affects Version/s: 2.3.2

The coprocessor was not being initialized in the region/store as the regionServices was set as null. The test was using an explicit setter in the region for populating the coprocessor after region initialization. This code path was exercised only by tests. This commit removes the back door entry for the test to set the coprocessor host explicitly and now the same is being set through the regular store initialization code path. This also allows the use of the storecontext otherwise the test would fail with the storecontext having a null coprocessor
Bug: RegionInfo.isOffline is now a duplication with RegionInfo.isSplit
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Oct/20 03:55

The only place, where we set it to true is in splitRegion, and at the same time we will set split to true.

So in general, I suggest that we deprecated isOffline and isSplitParent in RegionInfo, only leave the isSplit method. And in RegionInfoBuilder, we deprecated setOffline and only leave the setSplit method.

This could make our code base cleaner.

And for serialization compatibility, we'd better still keep the split and offline fields in the actual RegionInfo datastructure for a while.
Bug: hbase_generate_website is failing due to incorrect jdk and maven syntax
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Nov/20 10:16

While waiting for HBase download page to reflect new release and during an offline syncup with [~ndimiduk] , realized that generate website is failing for quite some time now, e.g https://ci-hadoop.apache.org/job/HBase/job/hbase_generate_website/80/
{code:java}
Obtained dev-support/jenkins-scripts/generate-hbase-website.Jenkinsfile from git https://gitbox.apache.org/repos/asf/hbase.git
Running in Durability level: PERFORMANCE_OPTIMIZED
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: 40: Tool type "maven" does not have an install of "Maven (latest)" configured - did you mean "maven_latest"? @ line 40, column 15.
           maven 'Maven (latest)'
                 ^

WorkflowScript: 42: Tool type "jdk" does not have an install of "JDK 1.8 (latest)" configured - did you mean "jdk_1.8_latest"? @ line 42, column 13.
           jdk "JDK 1.8 (latest)"
               ^

{code}
We might have to apply fix similar to HBASE-25204 to generate website specific Jenkinsfile.
Bug: Release 2.3.3
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Oct/20 06:29

Sub-tasks involved:
 # Spin RCs
 # "Release" staged nexus repository
 # Release version 2.3.3 in Jira
 # Promote 2.3.3 RC artifacts in svn
 # Update reporter tool with new release
 # Push signed release tag
 # Add 2.3.3 to the downloads page
 # Send announce email
Bug: Support lazy loading operation for editing  tableSchema config
Status: Open | Priority: Major | Resolution: 
Created: 04/Nov/20 10:00

Currently, when “alter” is used to modify the tableSchema, the region will be reopened immediately to load the meta information in .tabledesc. Now add a lazy loading modification command, there is no need to reopen the region immediately, and then load it when a region move or other region behavior occurs.
Bug: Try enlarge the flaky test timeout for branch-2.2
Status: Open | Priority: Major | Resolution: 
Created: 19/Oct/20 01:05

Now there are too many flaky tests to run. And the flaky test job cannot finished in 2 hours. Then these tests will be marked flaky again.

 

See [https://ci-hadoop.apache.org/job/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.2/124/artifact/dashboard.html]
Bug: Remove the usage of MetaTableAccessor in hbase-balancer
Status: Open | Priority: Major | Resolution: 
Created: 14/Oct/20 03:06

Let's try to find a way to move MTA to hbase-server. Putting it in hbase-balancer is a bit weird (Why is the MetaTableAccessor in the balancer module?).

MetaTableAccessor (MTA) is in hbase-balancer because the favored nodes (FN) feature wants to write to meta.

The FN feature (HBASE-7932 which arrived in hbase-0.96.0) I believe is unused. The idea of maintaining locality across a crash is a good one but having to enable a different balancer to default to enable FN operation has made it so the feature has been let rot.  HBASE-21014 makes an appeal for integrating FN into the default stochastic balancer. HBASE-15531 is an old effort that actually tried to integrate FN into stochastic balancer but stalled out.
Bug: Create an hbase-tool module and move here standalone tools bundled with core
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 15/Oct/20 20:17

HBASE-23933 is about moving balancer classes to a new hbase-balancer module as an effort at shrinking the size of the hbase-server module -- it is too big.

This issue is about creating a hbase-tool module into which we'd move standalone tools that ship with hbase core such as the bulkloadtool and the canary and all related supporting and test classes.

I do not see this making much of a dent in the size of hbase-server. This issue is offered as a small improvement.

(Aside, I'd move some favored node 'tool' classes that are currently in limbo here at least until we rule on whether to finish or remove if this module was put in place -- see HBASE-25185).
Bug: Auto-fix 'Unknown Server'
Status: Open | Priority: Major | Resolution: 
Created: 01/Oct/20 18:52

Addressing reports of 'Unknown Server' has come up in various conversations lately. This issue is about fixing instances of 'Unknown Server' automatically as part of the tasks undertaken by CatalogJanitor when it runs.

First though, would like to figure a definition for 'Unknown Server' and a list of ways in which they arise. We need this to figure how to do safe auto-fixing.

Currently an 'Unknown Server' is a server found in hbase:meta that is not online (no recent heartbeat) and that is not mentioned in the dead servers list.

In outline, I'd think CatalogJanitor could schedule an expiration of the RS znode in zk (if exists) and then an SCP if it finds an 'Unknown Server'. Perhaps it waits for 2x or 10x the heartbeat interval just-in-case (or not). The SCP would clean up any references in hbase:meta by reassigning Regions assigned the 'Unknown Server' after replaying any WALs found in hdfs attributed to the dead server.

As to how they arise:

 * A contrived illustration would be a large online cluster crashes down with a massive backlog of WAL files – zk went down for some reason say. The replay of the WALs look like it could take a very long time  (lets say the cluster was badly configured and a bug and misconfig made it so each RS was carrying hundreds of WALs and there are hundreds of servers). To get the service back online, the procedure store and WALs are moved aside (for later replay with WALPlayer). The cluster comes up. meta is onlined but refers to server instances that are no longer around. Can schedule an SCP per server mentioned in the 'HBCK Report' by scraping and scripting hbck2 or, better, catalogjanitor could just do it.

 * HBASE-24286 HMaster won't become healthy after after cloning... describes starting a cluster over data that is hfile-content only. In this case the original servers used manufacture the hfile cluster data are long dead yet meta still refers to the old servers. They will not make the 'dead servers' list.

Let this issue stew awhile. Meantime collect how 'Unknown Server' gets created and best way to fix.
Bug: Maximize sleep for checking meta and namespace regions availability
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Oct/20 09:20

The isRegionOnline method in HMaster is used on Master startup to check the availability of hbase:meta and hbase:namespace tables.
I've run into an issue when namespace was not online and Master was just waiting there. I've used HBCK2 to fix the cluster but the initialization was not completed because RetryCounterFactory was already waiting for 10+ hours.

Since Master is waiting in an idle state it makes no harm to check the region availability more frequently and limit the maximum sleep time.
Bug: Update book section "Submitting Patches" to exclusively describe a Github workflow
Status: Open | Priority: Major | Resolution: 
Created: 30/Oct/20 22:55
Affects Version/s: 3.0.0-alpha-1

Per this [discuss thread|https://lists.apache.org/thread.html/r091e883c4075eb13a8df22a3298029fac66b97f1b6b0e2c8debc39e2%40%3Cdev.hbase.apache.org%3E], we have dropped support for receiving patches via Jenkins. Our section on [Submitting Patches|https://hbase.apache.org/book.html#submitting.patches] could use an update. Remove old content, add instructions for creating a Github PR against our repository (or link to Github's own documentation).
Bug: Failure in refguide is causing build failures
Status: Open | Priority: Major | Resolution: 
Created: 30/Oct/20 21:56
Affects Version/s: 2.3.3

Recent runs on branch-2.3 Nightly have been failing. Looks like this is due to an issue in the refguide.

https://ci-hadoop.apache.org/view/HBase/job/HBase/job/HBase%20Nightly/job/branch-2.3/
Bug: Optimize in-memory representation for HBase map reduce table splits for MultiTableInputFormat
Status: Open | Priority: Major | Resolution: 
Created: 28/Oct/20 17:45

It has been observed that when the table has too many regions, MR jobs consume a lot of memory in the client. This is because we keep the region level information in memory and the memory heavy object is TableSplit because of the Scan object as a part of it.

There is a jira [HBASE-24859|https://issues.apache.org/jira/projects/HBASE/issues/HBASE-24859] which fix this single table TableInputFormat because we do not use the scan object from TableSplit in this case. 
However, it looks like we can do some optimization in case of MultiTableInputFormat as well since each split is not required to have memory heavy scan object.
Bug: Nightly job failed as  the name of jdk and maven changed
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Oct/20 05:29

See 
[https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/85/console]
[https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/103/console]
 
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: WorkflowScript: 508: Tool type "maven" does not have an install of "Maven (latest)" configured - did you mean "maven_latest"? @ line 508, column 19. maven 'Maven (latest)' ^ WorkflowScript: 510: Tool type "jdk" does not have an install of "JDK 1.8 (latest)" configured - did you mean "jdk_1.8_latest"? @ line 510, column 17. jdk "JDK 1.8 (latest)"
Bug: Do not publish workspace in flaky find job
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Oct/20 06:15

As said in the parent issue.

And I tried to use the publishHTML target on jenkins job configuration page, the help message for reportDir is

{noformat}
The path to the HTML report directory relative to the workspace.
{noformat}

For the reportFiles is

{noformat}
The file(s) to provide links inside the report directory
{noformat}

I think this clearly means that the plugin will publish all the files under reportDir, the reportFiles is just the index page.
Bug: TestMasterFailover.testSimpleMasterFailover is flaky 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Oct/20 21:28
Affects Version/s: 3.0.0-alpha-1

{noformat}
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 32.996 s <<< FAILURE! - in org.apache.hadoop.hbase.master.TestMasterFailover
[ERROR] org.apache.hadoop.hbase.master.TestMasterFailover.testSimpleMasterFailover  Time elapsed: 12.317 s  <<< FAILURE!
java.lang.AssertionError: expected:<1> but was:<2>
	at org.apache.hadoop.hbase.master.TestMasterFailover.testSimpleMasterFailover(TestMasterFailover.java:133)
{noformat}

Looks like this test depends on metrics to be updated as a source of side-effect used to verify the test. Seems like it should retry the check a few times, or maybe we need a last-updated monotonic value that the test can check before and after it expects a change to be visible.
Bug: JVMMetrics tag.processName regression between hbase-1.3 and hbase-2.x+ versions
Status: Open | Priority: Major | Resolution: 
Created: 16/Oct/20 05:44
Affects Version/s: 2.0.0

The regression is caused by 
https://issues.apache.org/jira/browse/HBASE-15160.
In order to monitor the FS latencies and pread latencies we have added the MetricsIO as part of metrics. Since we account this at the HFileBlock layer, we have created a static MetricIO variable at HFile.java so that we can use that metrics in a static way.
Internally the MEtricIO creates a MetricsIOWrapperImpl that in turns registers the metrics with the BaseSource. The flow is as follows,
{code}
this(CompatibilitySingletonFactory.getInstance(MetricsRegionServerSourceFactory.class)
            .createIO(wrapper), wrapper);
{code}
The createIO -> inturn creates a MetricsIOSourceImpl where the Metrics_Name is 'IO'.
The BaseSourceImpl registers a singleton JVMMetrics
{code}
synchronized void init(String name) {
      .......
      DefaultMetricsSystem.initialize(HBASE_METRICS_SYSTEM_NAME);
      JvmMetrics.initSingleton(name, "");
       ........
    }
{code}
The name passed here is 'IO'.  This is where the processName gets set with 'IO'.
All other metrics that we create in the HRS and master is not static level metrics whereas all are instance level metrics. So the very first time we create either Master metrics or Region server metrics so then the metrics would have had the processName as either RegionServer or master.
But pls note am note very sure on this now like if at all are we creating a  metric based on the actual process name like it was during the hbase-1.x time. In other words my doubt is even if we solve this 'IO' process case do we really get back the processName as 'Master' or 'RegionServer' as in https://issues.apache.org/jira/browse/HBASE-12328.
Bug: Cleanup the /tmp/std*deferred files generated by surefire
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 04/Oct/20 14:34

It is very easy to eat a lot of spaces and make the new jenkins node full.

Surefire plugin should clean these files but obviously, if the process does not quit normally,it has no way to delete the files.

So we have to do it by our own. Either an extra jenkins pipeline stage, or just a cron job?
Bug: Try create table with 100 regions for branch-2.2 nightly job's hadoop integration test
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 12/Oct/20 03:21

It still failed now.

[https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/88/execution/node/171/log/]

 

[https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/88//artifact/output-integration/hadoop-2.log]

 

It failed when create table with 1000 regions. And not import the example TSV to HDFS.
Bug: Validate earlier coprocessors against HBase 3.0+
Status: Open | Priority: Major | Resolution: 
Created: 13/Oct/20 21:37
Affects Version/s: 3.0.0-alpha-1

we should do something similar to HBASE-20656 but for the coprocessor APIs folks may be using from hbase 2. Probably a good idea to make sure the check against HBase 1 coprocessors still works, presuming we still support any HBase 1 by the time we expect HBase 3.
Bug: Remove the hadoop 3.3.0 personality hadoopcheck for branch-2.2/branch-2.3
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 12/Oct/20 04:19
Affects Version/s: 2.2.6

For branch-2.2, see [https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/88/execution/node/163/log/]

 It will fail when run "mvn clean install -DskipTests -DHBasePatchProcess -Dhadoop-three.version=3.3.0 -Dhadoop.profile=3.0".

 

For branch-2.3, see HBASE-23834. HBase failed to start on hadoop 3.3.0 because the jetty problem.
Bug: No need timelineservice for branch-2.2 nightly job's hadoop integration test
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Oct/20 08:16

[https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/86/execution/node/171/log/]

 
 /home/jenkins/jenkins-home/workspace/HBase_HBase_Nightly_branch-2.2/component/dev-support/hbase_nightly_pseudo-distributed-test.sh --single-process --working-dir output-integration/hadoop-2 --hbase-client-install hbase-client hbase-install hadoop-2/bin/hadoop {color:#ff0000}hadoop-2/share/hadoop/yarn/timelineservice{color} hadoop-2/share/hadoop/yarn/test/hadoop-yarn-server-tests-2.8.5-tests.jar hadoop-2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.5-tests.jar hadoop-2/bin/mapred
  

branch-2.2 still use hadoop 2.8.5 and hadoop 2.8.5 doesn't have timelineservice. The dev-support/hbase_nightly_pseudo-distributed-test.sh not consider this timelineservice and only consider 5 paramerters. But branch-2.3+ use 2.10.x hadoop, so they consider 6 parameters.

 

And for hadoop-3, the timelineservice is not used, too. See [https://github.com/apache/hbase/blob/master/dev-support/hbase_nightly_pseudo-distributed-test.sh#L286]
Bug: supporting hbase on aarch64 platform
Status: Open | Priority: Major | Resolution: 
Created: 10/Oct/20 02:51

HBase release 2.3.2 on September 21, 2020.   I download a new version and test on my machine. At first, I found that the fire-flame on x86 platform and aarch64 platform were inconsistent. On X86 platform netty use epoll, but on aarch64 platform netty use nio. Further I find out the code judge if use epoll make the difference. It do not consider aarch64 platform. Beside is the code:

{color:#4c9aff}return epollEnabled && JVM.isLinux() && JVM.isAmd64();{color}

When i add the aarch64 judge branch, i found regionserver can not start because of native lib libnetty-transport-native-epoll.so could not be loaded.

 

I check the third party jar package about HBase-2.3.2, and get the following table.
|third party jar info of HBase-2.3.2|latest jar package info|
|third party jar package|release time|lib in jar package do not support aarch64 platform|latest version|release time|Supporting aarch64？|
|commons-crypto-1.0.0.jar|2016/7/26|libcommons-crypto.so|commons-crypto-1.1.0.jar|2020/8/28|Yes|
|jline-2.11.jar|2013/5/19|libjansi.so|jline-2.14.6.jar|2018/3/26|No|
|leveldbjni-all-1.8.jar|2013/10/17|libleveldbjni.so|leveldbjni-all-1.8.jar|2013/10/17|No|
|netty-all-4.0.23.Final.jar|2014/8/15|libnetty-transport-native-epoll.so|netty-all-4.1.52.Final.jar|2020/9/8|Yes|
|snappy-java-1.0.5.jar|2013/5/17|libsnappyjava.so|snappy-java-1.1.7.7.jar|2020/8/26|Yes|

 

According to the table above, we can get the following conclusions:

1、third party jar package is now supporting aarch64 platform(apache hadoop is now release 3.3.0 to support aarch64, more and more bigdata component is now supporting aarch64 platform);

2、we now have only leveldbjni-all and jline packages do not support aarch64 platform.

 

I wish HBase community can support aarch64 platform on hbase in not far future.
Bug: Increase the timeout value for nightly jobs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Oct/20 12:41

The new hbase node is m5.2xlarge, which means it has 8 cores and 32 GB memory, while the old H node has 16 or 24 cores, 100GB memory, so it will spend more time to finish the nightly jobs. Since the new nodes are exclusive for HBase only, we are free to increase the timeout value.
Bug: Add extra logging at info level to HFileCorruptionChecker in order to report progress
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Oct/20 11:02

Currently there's no progress reporting in HFileCorruptionChecker: neither in the logs nor in stdout. It only creates a report about the entire operation at the end of the process and emits some warning messages is corruption is found.

Adding some logging about the progress would be beneficial for long running checks indicating that the process is running healthy.
Bug: Should store the regionNames field in state data for ReopenTableRegionsProcedure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Oct/20 13:22

We add this field in HBASE-22460 to make use of ReopenTableRegionsProcedure for reopening user specified regions, but we do not serialize it in the state data, so once master restart and we are still in the first state, we will miss the regionNames and reopen all the regions.
Bug: Flaky Test Dashboard scripts are blocked
Status: Open | Priority: Major | Resolution: 
Created: 05/Oct/20 21:23

The Flaky Test Dashboard hosted on ci-hadoop is no longer usable by humans because the browser blocks its scripts from executing. No images are loaded and none of the javascript functions work.

https://ci-hadoop.apache.org/view/HBase/job/HBase/job/HBase-Find-Flaky-Tests/job/master/lastSuccessfulBuild/artifact/dashboard.html
Bug: TestMasterRegionCompaction is flaky
Status: Open | Priority: Major | Resolution: 
Created: 05/Oct/20 21:34
Affects Version/s: 3.0.0-alpha-1

{noformat}
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.298 s <<< FAILURE! - in org.apache.hadoop.hbase.master.region.TestMasterRegionCompaction
[ERROR] org.apache.hadoop.hbase.master.region.TestMasterRegionCompaction.test  Time elapsed: 10.269 s  <<< FAILURE!
java.lang.AssertionError: expected:<1> but was:<4>
	at org.apache.hadoop.hbase.master.region.TestMasterRegionCompaction.test(TestMasterRegionCompaction.java:136)
{noformat}

This test uses calls to {{Thread.sleep}} of fixed durations instead of the {{waitFor}} pattern of retries. It's looking for side-effects of a background thread that may not be scheduled on precisely the cadence that the test expects, especially in an overloaded test environment.
Bug: Scan metrics are not reported to Hadoop MapReduce counters
Status: Open | Priority: Major | Resolution: 
Created: 02/Oct/20 13:23
Affects Version/s: 1.2.0

I setup a MapReduce job that scans HBase table with scan metrics enabled via Java API:
{code:java}
Scan scan = ...;
scan.setScanMetricsEnabled(true);
// Or via attribute
scan.setAttribute(                                               
        Scan.SCAN_ATTRIBUTES_METRICS_ENABLE,                             
        Bytes.toBytes(true));
TableMapReduceUtil.initTableMapperJob(...);                                                                 
job.waitForCompletion(true)
{code}
I expect HBase scan related counters to appear with current values in Hadoop web UI after job starts or in CLI after job ends alongside other counters such as CPU_MILLISECONDS. But no metrics were displayed. I can clearly see that scan is performed and mappers got non-zero inputs.

The JAR version is  1.2.0-cdh5.16.1.
Bug: Improve the experience of using the Master webpage by change the loading process of snapshot list to asynchronous
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Dec/20 04:51
Affects Version/s: 3.0.0-alpha-1

Background: When there are many snapshots, loading the master webpage is very slow, which affects the experience. (Our cluster has more than 3000  snapshots, and it takes about 10 seconds to load the master webpage each time)

 

1. The snapshot list is not in the master memory, hdfs need to be scanned when loading

2. Changing the process of loading  snapshots to asynchronous can improve the experience
Bug: HBase UI: sorting order by start key or end key is not considering empty start key/end key
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Dec/20 14:35
Affects Version/s: 2.2.3

HBase UI: sorting order by start key or end key is not considering empty start key/end key.

Steps:

1: Open any table page having multiple regions

2:  click on sort button of start key/end key

3: Observed that sorting doesn't consider start and end key.