av9ash/GitBugs huggingface dataset:
Bug: StochasticBalancer CostFunctions needs a better way to evaluate region count distribution
Status: Open | Priority: Major | Resolution: 
Created: 02/Mar/21 19:57

Currently CostFunctions including RegionCountSkewCostFunctions, PrimaryRegionCountSkewCostFunctions and all load cost functions calculate the unevenness of the distribution by getting the sum of deviation per region server. This simple implementation works when the cluster is small. But when the cluster get larger with more region servers and regions, it doesn't work well with hot spots or a small number of unbalanced servers. The proposal is to use the standard deviation of the count per region server to capture the existence of a small portion of region servers with overwhelming load/allocation.

TableSkewCostFunction uses the sum of the max deviation region per server for all tables as the measure of unevenness. It doesn't work in a very common scenario in operations. Say we have 100 regions on 50 nodes, two on each. We add 50 new nodes and they have 0 each. The max deviation from the mean is 1, compared to 99 in the worst case scenario of 100 regions on a single server. The normalized cost is 1/99 = 0.011 < default threshold of 0.05. Balancer wouldn't move.  The proposal is to use the standard deviation of the count per region server to detect this scenario, generating a cost of 3.1/31 = 0.1 in this case.

Patch is in test and will follow shortly.
Bug: StochasticBalancer improvement for large scale clusters
Status: Open | Priority: Major | Resolution: 
Created: 25/Mar/21 17:10

h2. Findings on a large scale cluster (100,000 regions on 300 nodes)
 * Balancer starts and stops before getting a plan
 * Adding new racks doesn’t trigger balancer
 * Balancer stops leaving some racks at 50% lower region counts
 * Regions for large tables don’t get evenly distributed
 * Observability is poor
 * Too many knobs makes tuning empirical and takes many experiments

h2. Improvements made and being made
 * Cost function enhancement to capture outliers especially table skew. https://issues.apache.org/jira/browse/HBASE-25625?filter=-2 
 * Explain why balancer stops https://issues.apache.org/jira/browse/HBASE-25666 will back port too https://issues.apache.org/jira/browse/HBASE-24528

h2. More proposals
 * minCostNeedBalance for each cost function instead of weights. We want to trigger balancing if any factor is out of balancer instead of trying to combine the factors in arbitrary weights. This makes operation and configuration much easier.
 * Simulated annealing to lower minCostNeedBalance periodically to unstuck the balancer from sub-optimum then gradually increase to keep the system stable. Also add cost of move as a counter measure for the decision [https://opensourcelibs.com/lib/tempest]
 * Orchestrated scheduling of compaction, normalizer and balancer
 * PID approach [https://www.amazon.com/dp/1449361692/ref=rdr_ext_tmb]
Bug: Adding replication peer should handle the undeleted queue exception
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 14/Mar/21 19:06

Currently, if we try to add a peer and there is a replication queue existing for that peer, it doesn't let the replication peer created. 

Instead, we should delete the queue and proceed with the creating of replication peer. Any queue without no corresponding replication peer is useless anyway. So, we shouldn't wait for cleaner to come and clean it before creating the peer. 

 
{code:java}
org.apache.hadoop.hbase.replication.ReplicationException: undeleted queue for peerId: xyz_peerid, replicator: hostname.fakeaddress.com,60020,1607576586258, queueId: xyz_peerid
java.lang.RuntimeException: org.apache.hadoop.hbase.replication.ReplicationException: undeleted queue for peerId: xyz_peerid, replicator: hostname.fakeaddress.com,60020,1607576586258, queueId: xyz_peerid

{code}
Bug: The configured loggers in log4j2.xml will always be created
Status: Closed | Priority: Major | Resolution: Fixed
Created: 30/Mar/21 07:53

No matter whether they are referenced.

So we will always create RFA log files even if we are using console appender, and since the default location for http request log is /var/log/hbase, we do not always have permission to create directory at this place so it may cause failure for UTs.

We need to find a way to fix this problem.
Bug: Need to initialize SLF4JBridgeHandler in jul-to-slf4j for redirecting jul to slf4j
Status: Closed | Priority: Major | Resolution: Fixed
Created: 25/Mar/21 08:58

When testing HBASE-22120 with logging exporter, the trace log messages were all redirected to console...

They use jul to log the spans and we have jul-to-slf4j on classpath.

Then I found out that we need to call SLF4JBridgeHandler.install...

Let me find a way to do this.
Bug: Bound LoadBalancer's RegionLocationFinder cache
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 02/Mar/21 00:29
Affects Version/s: 1.6.0

We have a large table in production that causes the balancer's RegionLocationFinder cache to consume 4 GB of heap, which, among other factors, triggered OOMEs, and made us aware of this problem. 

RegionLocationFinder embeds a cache backed by Guava's CacheLoader.  The RegionLocationFinder cache comes to consume heap for RegionInfos for all table regions and all HDFS block locations of all store files for all regions of all tables. 

The only limit we pass to the CacheBuilder is an expiration time of 14400000 milliseconds for individual cache entries. That's 4 hours. That's much too long; however, the cache also periodically refreshes itself, where the need for a refresh is checked whenever BaseLoadBalancer calls RegionLocationFinder's setClusterMetrics() method, which defeats the expiration based limit anyway. 

We should be bounding this cache with effective resource controls. Time based expiry is fine but the periodic refresh logic must be removed to make it effective. Implement size based limits too. CacheBuilder#maximumSize will limit by number cache entries. This might be fine but CacheBuilder#maximumWeight would be better, where weight is something determined by the API user. In this case it can be an estimate of the heap size of the hash map entries kept in the cache. 

Default should remain unbounded. Specific bounds should be supported by new site configuration options.
Bug: ZK watcher threads are daemonized; reconsider
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 14/Jan/21 02:09

On HBASE-25279 there was some discussion and difference of opinion about having ZK watcher pool threads be daemonized. This is not necessarily a problem but should be reconsidered. 

Daemon threads are subject to abrupt termination during JVM shutdown and therefore may be interrupted before state changes are complete or resources are released. 

As long as ZK watchers are properly closed by shutdown logic the pool threads will be terminated in a controlled manner and the JVM will exit.
Bug: when the cluster has many tables, UI can't be opened quickly
Status: Open | Priority: Major | Resolution: 
Created: 05/Jan/21 01:53
Affects Version/s: 3.0.0-alpha-1

my cluster has 60+K tables, and UI is opened slowly.
From the following code, we can reduce steps of class conversion

rsgroup.jsp
https://github.com/apache/hbase/blob/600be60a4bd4d3b3e9652027a0cb8bdd32016c6b/hbase-server/src/main/resources/hbase-webapps/master/rsgroup.jsp#L439
snapshot.jsp
https://github.com/apache/hbase/blob/600be60a4bd4d3b3e9652027a0cb8bdd32016c6b/hbase-server/src/main/resources/hbase-webapps/master/snapshot.jsp#L42
RegionStates.java
https://github.com/apache/hbase/blob/600be60a4bd4d3b3e9652027a0cb8bdd32016c6b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java#L538
RSStatusTmpl.jamon
https://github.com/apache/hbase/blob/600be60a4bd4d3b3e9652027a0cb8bdd32016c6b/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon#L47
Bug: [hbase-thirdparty] Update jetty version to fix CVE-2020-27218
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Feb/21 06:07

CVE-2020-27218 obseved in 9.4.0.RC0 to 9.4.34.v20201102, 10.0.0.alpha0 to 10.0.0.beta2, and 11.0.0.alpha0 to 11.0.0.beta2.

Currently we are using 9.4.34.v20201102,
https://github.com/apache/hbase-thirdparty/blob/a8e3629618642fd50daa31cc027bc674ab659711/pom.xml#L139
Bug: hbase thirdparty jdk bytecode enforcement shouldn't check test dependencies
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Feb/21 04:11
Affects Version/s: hbase-thirdparty-3.5.0

while staging a 3.5.0 RC the build failed when deploying a staged nexus repo for the hbase-noop-htrace module because the test dependency on hbase 2 fails the maven enforcer check for jdk7 compatibility. since the jdk compatibility for the artifacts we generate in hbase-thirdparty isn't impacted by test scope dependency or optional dependencies, we shouldn't check them in the enforcer rule.
Bug: Support hbase rpc compression for remote rpc only
Status: Open | Priority: Major | Resolution: 
Created: 05/Mar/21 19:38
Affects Version/s: 2.3.4

The purpose of Rpc compression is to save network bandwidth. For local communication (both hbase client and RS are on the same node), rpc compression is unnecessary as local communication is memory copy only and does not go through nic. Rpc compression for local communication will be a waste of cpu computation power as compress/decompress is cpu intensive.
Bug: Shell commands should accept "hbase:meta" as an alias for the encoded region name
Status: In Progress | Priority: Major | Resolution: 
Created: 01/Mar/21 22:10
Affects Version/s: 2.3.0

Simple one: the shell should accept the alias "meta" in place of the encoded region name. It's a little silly to have to say {{> move '1588230740', 'host, port,startcode'}}.
Bug: Utility to identify git commit / Jira fixVersion discrepancies for RC preparation
Status: Open | Priority: Major | Resolution: 
Created: 15/Jan/21 12:29

As part of RC preparation,  we need to identify all git commits that landed on release branch, however their corresponding Jira is either not resolved yet or does not contain expected fixVersions. Only when we have git commits and corresponding Jiras with expected fixVersion resolved, we get all such Jiras included in auto-generated CHANGES.md as per Yetus changelog generator.

Proposal of this Jira is to provide such script that can be useful for all upcoming RC preparations and list down all Jiras where we need manual intervention. This utility script should use Jira API to retrieve individual fields and use git log to loop through commit history.

The script should identify these issues:
 # commit is reverted as per commit message
 # commit does not contain Jira number format HBASE-XXXX in message
 # Jira does not have expected fixVersion
 # Jira has expected fixVersion, but it is not yet resolved

It can take inputs as:
 # First commit hash to start excluding commits from history
 # Fix Version
 # JIRA Project Name
 # Path of project's working dir
 # Jira server url

The script should also identify all Jiras that are marked resolved with given fixVersion but have no commit present with Jira number included.

RC preparation also needs to find this diff because we don't want to provide extra Jira in CHANGELOG without any corresponding commits present on git (unless it is svn checkin and fixVersion is relevant).
Bug: [Branch-2 and Master]Handle the NoNode exception in remove log replication in a better way then just log WARN
Status: Open | Priority: Major | Resolution: 
Created: 26/Feb/21 21:39
Affects Version/s: 2.5.0

Currently, when we see the NoNodeException in the replication source while removing a log from ZK, we swallow that exception and log WARN. 

 

In certain cases, we might have the peer removed and corresponding logs removing as well but the replication source continuous to run because of an RPC failure or anything. 

In stead of just log WARN we should check if the peer is removed, if it is the case, we should terminate the source or try to execute the removePeer workflow again.

 

This would prevent the orphaned source execution infinitely.
Bug: Regionserver cannot exit completely 
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 04/Mar/21 11:35
Affects Version/s: 1.4.7

When regionserver abort by some errors, the process cannot exit completely. Error logs and jstack of regionserver process are as follows,  

!1614857532776-image.png|width=823,height=364!
 * !1614845335795-image.png|width=532,height=330!
Bug: Support parallel balance RSGroups
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 17/Jan/21 14:36

When we call balanceRSGroup(), it waits for the lock of balancer, which may be occupied by the overall cluster balance(). Sometimes the balance of cluster will take a long time because it need to compute the costs or act the balance region plans. 

In RSGroup, we don't need to block a balance of one group due to other group balances. 

All group are mini clusters, and the balance of their regions should support concurrently execution and reduce the race conditions between them.
Bug: RegionServer JVM crash when append wal entry
Status: Open | Priority: Major | Resolution: 
Created: 27/Mar/21 03:57
Affects Version/s: 2.2.6

Region Server JVM crash when append wal entry,  JVM crash log:

#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00000000027af93f, pid=17992, tid=0x0000000000002d54
#
# JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build 1.8.0_202-b08)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# J 10214 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getFamilyLength()B (9 bytes) @ 0x00000000027af93f [0x00000000027af860+0xdf]
Bug: Fix or stop warning about already cached block
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Mar/21 22:37
Affects Version/s: 2.3.0

Our logs have as a fairly common occurrence:
{noformat}
2021-03-05 22:24:31,034 WARN  [StoreFileOpener-foo-1] hfile.BlockCacheUtil: Caching an already cached block: blah.bub. This is harmless and can happen in rare cases (see HBASE-8547)
{noformat}

If this is really harmless, why do we log? If it's not actually harmless, let's take another pass at fixing this situation.
Bug: Explain why balancer is skipping runs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Mar/21 18:54

Balancer needs to find the balance of keeping cluster balanced and stable. There is a configuration minCostNeedBalance as the threshold of imbalance to try to balance. Since we use a single score to combine all the factors we consider, it is hard for operation to understand why balancer is "stuck". We should add to master-status UI to show balancer is skipping runs and explain all factors considered, such as the weight and cost of all cost functions.
Bug: HBase Connectors pom should include nexus staging repo management
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Feb/21 22:32

The current RC creation scripts rely on project poms having a plugin configured to handle nexus staging repository closing / dropping. The hbase-connectors project is currently missing this plugin, which makes it so that a RM has to go manually close the staging repository and then copy the location into the generated vote text.

the needed plugin config is:

{code}
    <!-- this profile should match the name of the release profile in the root asf pom -->
    <profile>
      <id>apache-release</id>
      <build>
        <plugins>
          <!-- This should insert itself in place of the normal deploy plugin and then
               handle either closing or dropping the staging repository for us depending
               on if the build succeeds.
            -->
          <plugin>
            <groupId>org.sonatype.plugins</groupId>
            <artifactId>nexus-staging-maven-plugin</artifactId>
            <version>1.6.8</version>
            <extensions>true</extensions>
            <configuration>
              <nexusUrl>https://repository.apache.org/</nexusUrl>
              <serverId>apache.releases.https</serverId>
            </configuration>
          </plugin>
        </plugins>
      </build>
    </profile>
{code}
Bug: NORMALIZER_TARGET_REGION_SIZE needs a unit in its name
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Mar/21 22:58
Affects Version/s: 2.3.0

On a per-table basis, the normalizer can be configured with a specific region size target via {{NORMALIZER_TARGET_REGION_SIZE}}. This value is used in a context where size values are in the megabyte scale. However, this configuration name does not mention "megabyte" or "mb" anywhere, so it's very easy for an operator to get the configured value wrong by many orders of magnitude.
Bug: Link to the filter on hbase:meta from user tables panel on master page
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Mar/21 04:44

This is follow-on to the parent issue that added nice filtering mechanism on hbase:meta table. Parent allows finding all Regions in Table XYZ with state OPENING or FAILED_CLOSED.

The user table panel on the master home page has counts of Regions in each state. The opening and closing counts actually have links under them but they are useless currently as they only show RITs that are CLOSING or OPENING; good but not comprehensive enough.

This PR adds links under all counts so you can see all CLOSING Regions whether RIT or not; useful when doing fixup on a corrupt cluster.  Adds a bit of help text that tells users about the filter-on-meta feature too.

Here is how the panel currently looks:

 !image-2021-03-24-21-41-11-393.png! 

Here is what it looks like now with the bit of help text

 !image-2021-03-24-21-42-16-355.png! 


When you click on the CLOSED number -- '1' in this case -- this where you go to:

 !image-2021-03-24-21-43-24-426.png! 

i..e. lists all Regions in the TestTable that are in the CLOSED state (not very pretty with the 'Table Stats' and 'Table Regions' preamble but better than what was there before).
Bug: Specify and document the criteria for moving the 'stable' pointer
Status: Open | Priority: Major | Resolution: 
Created: 22/Mar/21 17:09

We can consider reported experience with new code lines released ahead of where the stable pointer currently points, when and if it happens but this should not be a formal criteria because it has nothing directly to do with our PMC or project. We need a criteria for deciding when new code lines are sufficiently stable as to warrant moving the 'stable' pointer that we as project and PMC can achieve and implement effectively.

IMHO "one of our project devs has it running" does not meet that requirement, because this depends on third party organizations (a dev's employer, and such) and idiosyncratic criteria.

Collect and document requirements, test scenarios, and success (and failure) criteria.
Bug: Misleading Range metrcis 
Status: Open | Priority: Major | Resolution: 
Created: 29/Jan/21 05:52
Affects Version/s: 2.3.0

Found some cases that max value is included in a smaller range, which is confusing. Please see the attach file. The max is 7032, however, it cannot be found in the timeRange report. The issue is that it is included in the 1000~3000 range. In this case, the time range should be 1000 - infinite.
Bug: Add metric for age of oldest wal in region server replication queue.
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 12/Jan/21 15:59

We have seen cases recently where we have wal from 2018 timestamp in our recovered replication queue. We came across this un replicated wal while debugging something else. We need to have metrics for the oldest wal in the replication queue and have alerts if it exceeds some threshold. Clearly 2 years old wal is not desirable.
Bug: [create-release] checkcompatibility.py failing with "KeyError: 'binary'"
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jan/21 06:46

Below is excerpt from log of RC0 2.3.4 build this afternoon when we try to do the compatibility check (inside docker create-release build). It had failed for me previously on another run of different versions. The failure is unfortunately cryptic.

{code}
2021-01-07T01:08:34Z generate_api_report start
INFO:root:Source revision: rel/2.3.3
INFO:root:Destination revision: 2.3.4RC0
INFO:root:Filtering classes using 1 annotation(s):
INFO:root:      org.apache.yetus.audience.InterfaceAudience.Public
INFO:root:Downloading Java ACC...
INFO:root:Java ACC version: 2.4

INFO:root:Removing scratch dir /home/vagrant/hbase-rm/output/hbase/target/compat-check
INFO:root:Creating empty scratch dir /home/vagrant/hbase-rm/output/hbase/target/compat-check
INFO:root:Checking out 5f7fe66efbdfc1c0b3262c0677f17e4055ba94cd in /home/vagrant/hbase-rm/output/hbase/target/compat-check/src
INFO:root:Checking out ec79612b9ac6afd73fa8fc7f7f5c8d3bac277679 in /home/vagrant/hbase-rm/output/hbase/target/compat-check/dst
INFO:root:Building in /home/vagrant/hbase-rm/output/hbase/target/compat-check/src
INFO:root:Building in /home/vagrant/hbase-rm/output/hbase/target/compat-check/dst
INFO:root:Will check compatibility between original jars:
        /home/vagrant/hbase-rm/output/hbase/target/compat-check/src/hbase-shaded/hbase-shaded-client-byo-hadoop/target/original-hbase-shaded-client-byo-hadoop-2.3.3.jar
        /home/vagrant/hbase-rm/output/hbase/target/compat-check/src/hbase-external-blockcache/target/hbase-external-blockcache-2.3.3.jar
        /home/vagrant/hbase-rm/output/hbase/target/compat-check/src/hbase-shaded/hbase-shaded-client/target/original-hbase-shaded-client-2.3.3.jar
        /home/vagrant/hbase-rm/output/hbase/target/compat-check/src/hbase-zookeeper/target/hbase-zookeeper-2.3.3.jar
        /home/vagrant/hbase-rm/output/hbase/target/compat-check/src/hbase-common/target/hbase-common-2.3.3.jar

....
        /home/vagrant/hbase-rm/output/hbase/target/compat-check/dst/hbase-shaded/hbase-shaded-testing-util-tester/target/hbase-shaded-testing-util-tester-2.3.4.jar
        /home/vagrant/hbase-rm/output/hbase/target/compat-check/dst/hbase-archetypes/hbase-shaded-client-project/target/hbase-shaded-client-project-2.3.4.jar
        /home/vagrant/hbase-rm/output/hbase/target/compat-check/dst/hbase-protocol/target/hbase-protocol-2.3.4.jar
INFO:root:Annotations are: ['org.apache.yetus.audience.InterfaceAudience.Public']
INFO:root:Annotations path: /home/vagrant/hbase-rm/output/hbase/target/compat-check/annotations.txt
INFO:root:Results: {}
Traceback (most recent call last):
  File "./hbase/dev-support/checkcompatibility.py", line 530, in <module>
    main()
  File "./hbase/dev-support/checkcompatibility.py", line 526, in main
    args.compare_warnings))
  File "./hbase/dev-support/checkcompatibility.py", line 232, in compare_results
    if tool_results[check][issue_type] > known_count]
{code}
Bug: Exclude CHANGES and RELEASENOTES files from source control
Status: Open | Priority: Major | Resolution: 
Created: 19/Jan/21 16:18

Over on the thread [Project management (JIRA fix version tracking) is in crisis |https://lists.apache.org/thread.html/r8db075fd974de32a90174be2db106ef710cf38cfde9ea9071fda4152%40%3Cdev.hbase.apache.org%3E], one of the suggested improvements to our process is to remove the CHANGES and RELEASENOTES files from version control. To accomplish this, we'll need to
* delete existing files from existing branches
* modify the release automation to not commit this files
* find someplace to deliver these files on the web