av9ash/GitBugs huggingface dataset:
Bug: TestStochasticLoadBalancerBalanceCluster.testBalanceCluster is flaky
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 16/Apr/21 12:31

Seems after HBASE-25739.

Shows in [https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3176/1/testReport/org.apache.hadoop.hbase.master.balancer/TestStochasticLoadBalancerBalanceCluster/testBalanceCluster/]

Local test can reproduce the failure,
{code:java}
2021-04-16T20:11:21,425 INFO  [Time-limited test] balancer.TestStochasticLoadBalancerBalanceCluster(61): Mock Cluster : { srv1241949559:1 , srv609693614:5 , srv1125287745:6 , srv1143442391:6 , srv1165735784:6 , srv1221998538:6 , srv394489737:6 , srv593165442:6 , srv736809440:6 , srv741384165:6 } [srvr=10 rgns=54 avg=5.4 max=6 min=5]2021-04-16T20:11:21,425 INFO  [Time-limited test] balancer.TestStochasticLoadBalancerBalanceCluster(61): Mock Cluster : { srv1241949559:1 , srv609693614:5 , srv1125287745:6 , srv1143442391:6 , srv1165735784:6 , srv1221998538:6 , srv394489737:6 , srv593165442:6 , srv736809440:6 , srv741384165:6 } [srvr=10 rgns=54 avg=5.4 max=6 min=5]2021-04-16T20:11:21,425 INFO  [Time-limited test] balancer.BaseLoadBalancer(1791): Start Generate Balance plan for cluster.2021-04-16T20:11:21,425 DEBUG [Time-limited test] balancer.StochasticLoadBalancer$RegionCountSkewCostFunction(925): RegionCountSkewCostFunction sees a total of 10 servers and 54 regions.2021-04-16T20:11:21,425 DEBUG [Time-limited test] balancer.StochasticLoadBalancer(361): Skipping load balancing because balanced cluster; total cost=25.97402597402596, sum multiplier=582.0; cost/multiplier to need a balance is 0.052021-04-16T20:11:21,425 INFO  [Time-limited test] balancer.TestStochasticLoadBalancerBalanceCluster(67): Mock Balance : { srv1241949559:1 , srv609693614:5 , srv1125287745:6 , srv1143442391:6 , srv1165735784:6 , srv1221998538:6 , srv394489737:6 , srv593165442:6 , srv736809440:6 , srv741384165:6 } java.lang.AssertionError: All servers should have load no less than 5. server=srv1241949559,13844,-2719393974186553415 , load=1 at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.assertTrue(Assert.java:42) at org.apache.hadoop.hbase.master.balancer.BalancerTestBase.assertClusterAsBalanced(BalancerTestBase.java:207) at org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerBalanceCluster.testBalanceCluster(TestStochasticLoadBalancerBalanceCluster.java:68) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
{code}
@[~claraxiong] Could you help to fix this issue?
Bug: Filter out o.a.hadoop.thirdparty building shaded jars
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Apr/21 00:40
Affects Version/s: 2.4.3

Hadoop 3.3.1 (unreleased currently) shades guava. The shaded guava then trips the check in our shading that tries to exclude hadoop bits from the fat jars we build.

For the issue to trigger, need to build against tip of hadoop branch-3.3. You then get this complaint:

{code}
[INFO] --- exec-maven-plugin:1.6.0:exec (check-jar-contents) @ hbase-shaded-check-invariants ---
[ERROR] Found artifact with unexpected contents: '/Users/stack/.m2/repository/org/apache/hbase/hbase-shaded-mapreduce/2.3.6-SNAPSHOT/hbase-shaded-mapreduce-2.3.6-SNAPSHOT.jar'
    Please check the following and either correct the build or update
    the allowed list with reasoning.

    org/apache/hadoop/thirdparty/
    org/apache/hadoop/thirdparty/com/
    org/apache/hadoop/thirdparty/com/google/
    org/apache/hadoop/thirdparty/com/google/common/
    org/apache/hadoop/thirdparty/com/google/common/annotations/
    org/apache/hadoop/thirdparty/com/google/common/annotations/Beta.class
    org/apache/hadoop/thirdparty/com/google/common/annotations/GwtCompatible.class
    org/apache/hadoop/thirdparty/com/google/common/annotations/GwtIncompatible.class
    org/apache/hadoop/thirdparty/com/google/common/annotations/VisibleForTesting.class
    org/apache/hadoop/thirdparty/com/google/common/base/
    org/apache/hadoop/thirdparty/com/google/common/base/Absent.class
    org/apache/hadoop/thirdparty/com/google/common/base/AbstractIterator$1.class
    org/apache/hadoop/thirdparty/com/google/common/base/AbstractIterator$State.class
    org/apache/hadoop/thirdparty/com/google/common/base/AbstractIterator.class
    org/apache/hadoop/thirdparty/com/google/common/base/Ascii.class
    org/apache/hadoop/thirdparty/com/google/common/base/CaseFormat$1.class
    org/apache/hadoop/thirdparty/com/google/common/base/CaseFormat$2.class
    org/apache/hadoop/thirdparty/com/google/common/base/CaseFormat$3.class
    org/apache/hadoop/thirdparty/com/google/common/base/CaseFormat$4.class

....
{code}
Bug: [hbase-thirdparty] Shade jackson-jaxrs-json-provider for use with shaded jersey
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/May/21 22:18

The only pluggable feature of the this JSR spec that we take advantage of is using jackson-jaxrs-json-provider to get Jackson entity serialization. Rewrite this provider as well so that its use of the JSR API classes is against our shaded packages.
Bug: Balancer should explain progress in a better way in log
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jun/21 00:47
Affects Version/s: 3.0.0-alpha-1

In the log, balancer logs at info level at the beginning of run:

 {code}
balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer, initCost=277.3479243125063, functionCost=RegionCountSkewCostFunction : (500.0, 0.3749771215224234); ServerLocalityCostFunction : (25.0, 0.5807483226644186); RackLocalityCostFunction : (15.0, 0.0); TableSkewCostFunction : (1000.0, 0.0019704142954972883); StoreFileCostFunction : (200.0, 0.3668512059459341);  computedMaxSteps: 42270438200
{code}
the cost is reported without context, it is hard for operator to understand how unbalanced the cluster is for balancer and how much progress we are making.

For a large cluster, the calculation can take a long time, we also need to let operator understand that it will take up to the max time to complete the calculation. 

At the end of computation:
{code}
balancer.StochasticLoadBalancer: Finished computing new load balance plan. Computation took PT40M0.006S to try 1036409 different iterations. Found a solution that moves 161926 regions; Going from a computed cost of 118.75715593924485 to a new cost of 1.5509126920967042
{code}
The time to compute the plan is also printed in a  format that is not human readable. we also need to let operator understand that balancer is just submitting the plan and it be up to execution to complete the move.
Bug: Create a chore to update age related metrics.
Status: In Progress | Priority: Major | Resolution: 
Created: 11/May/21 19:06

We had a case where logRoller and ReplicationShipper thread were stuck for a day since some other thread was holding the lock.

We were not rolling the wal for 1 day and we were not shipping any edits for 1 day.
Still the oldestWalAge and age of last ship metric were not spiking as they should.

The way we calculate any age related metric is we calculate the diff between current time and the time at which any event happens and we add that to metrics Framework. We lose the event timestamp at that point.

If the thread populating the metric is stuck then we will always carry forward the same value forever. This will make it look like there is no problem in the system. In this case the oldestWalAge metric was stuck at 809 value and age of last ship metric was 0 the whole time and no alert was fired.

From Andrew Purtell,
We have the Chore/ScheduledChore framework. We could be making more use of it. Much of this is legacy, before Chore was formalized as it is today.
Bug: Cleanup netty-all transitive includes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jun/21 05:57

Our releases include lib/netty-all.jar as a transitive include from hadoop. -Purge.-

... looks like I can't purge the transitive netty-all includes just yet, not w/o moving MR out of hbase core. The transitively included netty-all w/ the old version is needed to run the tests that put up a MR cluster.
Bug: [hbase-thirdparty] Shade javax.ws.rs package for use with shaded Jersey
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/May/21 18:09

From a comment on the parent ticket,

bq. Jersey is an implementation of the specifications that define that package and its subcomponents. Thus, to use a shaded version of Jersey, one must also shade usage of this package as well.
Bug: Build includes unshaded netty .so; clashes w/ downstreamers who would use a different version of netty
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 02/Jun/21 04:19

A coworker was trying to use hbase client in a fat application that uses a different netty version to what hbase uses internally. Their app would fail to launch because it kept bumping into an incompatible netty .so lib. Here are the unshaded netty .so's we bundle looking at hbase-2.4.1...:

./lib/hbase-shaded-netty-3.4.1.jar has:

{code}
META-INF/native/libnetty_transport_native_epoll_aarch_64.so
META-INF/native/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_x86_64.so
META-INF/native/libnetty_transport_native_epoll_x86_64.so
{code}

(HBASE-25959 should fix the non-relocation of libnetty_transport_native_epoll_aarch_64).

./lib/shaded-clients/hbase-shaded-client-byo-hadoop-2.4.1.1-apple.jar has the same three .sos as does ./lib/shaded-clients/hbase-shaded-mapreduce-2.4.1.1-apple.jar
and ./lib/shaded-clients/hbase-shaded-client-2.4.1.1-apple.jar

We even bundle ./lib/netty-all-4.1.17.Final.jar which unsurprisingly has the netty .sos in it.

Looking at published builds of hbase-thirdparty, I see that these too include the above trio of .sos... The hbase-shaded-netty includes them in 3.4.1 https://repo1.maven.org/maven2/org/apache/hbase/thirdparty/hbase-shaded-netty/3.4.1/ as does 3.5.0.

I just tried running a build of hbase-thirdparty and it does NOT include the extras....

META-INF/native/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_aarch_64.so
META-INF/native/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_x86_64.so

(it has the fix for aarch included... when I built)

Here is link to the snapshot I made:

https://repository.apache.org/content/repositories/orgapachehbase-1451/org/apache/hbase/thirdparty/hbase-shaded-netty/3.5.1-stack4/
Bug: WAL value compression
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/May/21 01:06

WAL storage can be expensive, especially if the cell values represented in the edits are large, consisting of blobs or significant lengths of text. Such WALs might need to be kept around for a fairly long time to satisfy replication constraints on a space limited (or space-contended) filesystem.

We have a custom dictionary compression scheme for cell metadata that is engaged when WAL compression is enabled in site configuration. This is fine for that application, where we can expect the universe of values and their lengths in the custom dictionaries to be constrained. For arbitrary cell values it is better to use one of the available compression codecs, which are suitable for arbitrary albeit compressible data.
Bug: Duplicate cost calculation by TableSkewCostFunction when balancing by table
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 12/May/21 08:34
Affects Version/s: 2.0.0

When using balance by table, the StochasticLoadBalancer will create the cluster state according to the regions distribution of only one table. As a result,  the TableSkewCostFunction should be replaced by the RegionCountSkewCostFunction when the table count of the cluster state is less than 2.

The most important problem is that,  TableSkewCostFunction will cause unnecessary calculation steps when there is only one table.

For example，we have 40000+ regions for one table in one group, balance this table may calculate lots of steps, but we can avoid duplicate computing costs of TableSkewCostFunction while computing cost of RegionCountSkewCostFunction and balancing by table.

 

  !image-2021-08-17-20-20-12-768.png|width=907,height=164!
Bug: Release 1.7.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Apr/21 16:28

Discussion is [here|https://lists.apache.org/thread.html/r34d3fb86d667f8b3e58cbba78655733ac76e10f5883650f4910adc5c%40%3Cuser.hbase.apache.org%3E], this may or may not be the last release of branch-1.

But please allow me to go through the whole process before making any words. 

I'll spend the upcoming next whole week to do this release, having said to my employer.

cc [~andrew.purtell@gmail.com]
Bug: TestSnapshotScannerHDFSAclController.setupBeforeClass is flaky
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Apr/21 16:18

[https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3140/2/testReport/org.apache.hadoop.hbase.security.access/TestSnapshotScannerHDFSAclController/precommit_checks___yetus_jdk8_Hadoop3_checks______/]

SnapshotScannerHDFSAclController.postStartMaster alters hbase:acl to add a new cf "m", but `TestSnapshotScannerHDFSAclController.setupBeforeClass(TestSnapshotScannerHDFSAclController.java:101)`

 fails before the disable and enable hbase:acl complete.
Bug: Remove dependence on storing WAL filenames for backup
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/May/21 16:52
Affects Version/s: 3.0.0-alpha-1

Context:

Currently WAL logs are stored in `backup:system` meta table 
{code:java}
// code placeholder
wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:backupId, timestamp=1622003479895, value=backup_1622003358258 wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:file, timestamp=1622003479895, value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621996160175 wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:root, timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1 wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:backupId, timestamp=1622003479895, value=backup_1622003358258 wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:file, timestamp=1622003479895, 
value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621999760280 wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:root, timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1 
{code}
Also, Every backup (Incremental and Full) performs a log roll just before taking backup and stores what was the timestamp at which log roll was performed per regionserver per backup using following format.  
{code:java}
// code placeholder
rslogts:hdfs://xx.xx.xx.xx:8020/tmp/backup_yaktest\x00preprod-dn-2:16020 column=meta:rs-log-ts, timestamp=1622887363301,value=\x00\x00\x01y\xDB\x81ar
rslogts:hdfs://xx.xx.xx.xx:8020/tmp/backup_yaktest\x00preprod-dn-3:16020 column=meta:rs-log-ts, timestamp=1622887363294, value=\x00\x00\x01y\xDB\x81aP
rslogts:hdfs://xx.xx.xx.xx:8020/tmp/backup_yaktest\x00preprod-dn-1:16020 column=meta:rs-log-ts, timestamp=1622887363275, value=\x00\x00\x01y\xDB\x81\x85
{code}
 

There are 2 cases for which WAL log refrences stored in `backup:system` and are being used. 

*Use Case 1.*

*Existing Design:* To cleanup WAL's for which backup is already taken using `BackupLogCleaner`. Which uses this references to clean up backed up logs.

*New Design:*

Since log roll timestamp is stored as part of backup per regionserver. We can check all previous successfull backup's and then identify which logs are to be retained and which ones are to be cleaned up as follows
 * Identify which are the latest successful backups performed per table.
 * Per backup identified above, identify what is the oldest log rolled timestamp perfomed per regionserver per table. 
 * All those WAL's which are older than oldest log rolled timestamp perfomed for any table backed can be removed by `BackupLogCleaner` 

 

*Use Case 2.* 

*Existing Design:* During incremental backup, to check system table if there are any duplicate WAL's for which backup is taken again. 

*New Design:*
 * Incremental backup already identifies which all WAL's to be backed up using `rslogts:` mentioned above.
 * Additionally it checks `wals:` to ensure no logs are backuped for second time. And this is redundant and not seen any extra benefit.
Bug: Introduce RegionSplitRestriction that restricts the pattern of the split point
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Apr/21 12:34

As discussed in HBASE-25706, we can introduce RegionSplitRestriction that restricts the pattern of the split point.

See the following comment for the details:
https://issues.apache.org/jira/browse/HBASE-25706?focusedCommentId=17310190&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17310190

CC: [~zhangduo] [~stack]
Bug: Support for Parallel Backups enabling multi tenancy with rsgroups
Status: In Progress | Priority: Major | Resolution: 
Created: 17/Apr/21 06:40

*Existing Design*

  !existing_design.png|width=851,height=1667!

*Problem 1:* 
 With this design, Incremental and Full backup can't be run in parallel and leading to degraded RPO's in case Full backup is of longer duration esp for large tables.
  
 Example: 
 Expectation: Say you have a big table with 10 TB and your RPO is 60 minutes and you are allowed to ship the remote backup with 800 Mbps. And you are allowed to take Full Backups once in a week and rest of them should be incremental backups
  
 Shortcoming: With the above design, one can't run parallel backups and whenever there is a full backup running (which takes roughly 25 hours) you are not allowed to take incremental backups and that would be a breach in your RPO. 
  
 *Proposed Solution:* Barring some critical sections such as modifying state of the backup on meta tables, others can happen parallelly. Leaving incremental backups to be able to run based on older successful full / incremental backups and completion time of backup should be used instead of start time of backup for ordering. I have not worked on the full redesign, and will be doing so if this proposal seems acceptable for the community.
  
 *Problem 2:*
 With one backup at a time, it fails easily for a multi-tenant system. This poses following problems
 * Admins will not be able to achieve required RPO's for their tables because of dependence on other tenants present in the system. As one tenant doesn't have control over other tenants' table sizes and hence the duration of the backup
 * Management overhead of setting up a right sequence to achieve required RPO's for different tenants could be very hard.

*Proposed Solution:* Same as previous proposal
  
 *Problem 3:* 
 Incremental backup works on WAL's and org.apache.hadoop.hbase.backup.master.BackupLogCleaner ensures that WAL's are never cleaned up until the next backup (Full / Incremental) is taken. This poses following problem
 * WAL's can grow unbounded in case there are transient problems like backup site facing issues or anything else until next backup scheduled goes successful
 *Proposed Solution:* I can't think of anything better, but I see this can be a potential problem. Also, one can force full backup if required WAL files are missing for whatever other reasons not necessarily mentioned above. 
  

*Proposed Design.*

!proposed_design.png|width=865,height=1766!
Bug: HBoss tests compile/failure against Hadoop 3.3.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/May/21 13:50
Affects Version/s: hbase-filesystem-1.0.0-alpha1

Changes in Hadoop 3.3.x stop the tests compiling/working. 

* changes in signature of nominally private classes (HADOOP-17497): fix, update
* HADOOP-16721  -s3a rename throwing more exceptions, but no longer failing if the dest parent doesn't exist. Fix: change s3a.xml
* HADOOP-17531/HADOOP-17620 distcp moving to listIterator; test failures. 
* HADOOP-13327: tests on syncable which expect files being written to to be visible. Fix: skip that test

The fix for HADOOP-17497 stops this compiling against Hadoop < 3.3.1. This is unfortunate but I can't see an easy fix. The new signature takes a parameters class, so we can (and already are) adding new config options without breaking this signature again. And I've tagged it as LimitedPrivate so that future developers will know it's used here
Bug: Perf improvement in L1 cache
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 21/Jun/21 10:43
Affects Version/s: 2.3.5

After HBASE-25698 is in, all L1 caching strategies perform buffer.retain() in order to maintain refCount atomically while retrieving cached blocks (CHM#computeIfPresent). Retaining refCount is coming up bit expensive in terms of performance. Using computeIfPresent API, CHM uses coarse grained segment locking and even if our computation is not so complex (we just call block retain API), it will block other update APIs for the keys within bucket that is locked. computeIfPresent keeps showing up on flame graphs as well (attached one of them). Specifically when we see aggressive cache hits for meta blocks (with major blocks in cache), we would want to get away from coarse grained locking.

One of the suggestions came up while reviewing PR#3215 is to treat cache read API as optimistic read and deal with block retain based refCount issues by catching the respective Exception and let it treat as cache miss. This should allow us to go ahead with lockless get API.
Bug: Update default weight of cost functions
Status: Open | Priority: Major | Resolution: 
Created: 12/Apr/21 21:30

In production, we have seen some critical big tables that handle majority of the load. Table Skew is becoming more important. With the update of table skew function, balancer finally works for large table distribution on large cluster. We should increase the weight from 35 to a level comparable to region count skew: 500. We can even push further to replace region count skew by table skew since the latter works in the same way and account for region distribution per node.

Another weight we found helpful to increase is for store file size cost function. Ideally if normalizer works perfectly, we don't need to worry about it since region count skew would have accounted for it. But we are often in a situation it doesn't. Store file distribution needs to be given more way as accommodation. we tested changing it from 5 to 200 and it works fine.
Bug: Temporarily remove the trace support for RegionScanner.next
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Apr/21 16:06

Please see the test result in HBASE-25658, tracing RegionScanner.next could lead to a 6% performance downgrading even if we do not enable tracing.

I think the problem here is that, a single scan rpc call could lead to a bunch of RegionScanner.next. So if trace is enabled, there will be much more performance impact as we need to store a bunch of spans(maybe this could be limited by some parameters?).

Anyway, let's remove it for now to land the basic trace framework first. Can improve later.
Bug: Remove DfsBuilderUtility
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jun/21 07:08
Affects Version/s: 2.5.0

DfsBuilderUtility was created to reflectively access DistributedFileSystem$HdfsDataOutputStreamBuilder, which was added by 
HDFS-11170 and available since Hadoop 2.9.0.

We can remove this class and access the HDFS builder class directly in HBase 3 and 2.5.0.