av9ash/GitBugs huggingface dataset:
Bug: Downgrading is complicated by refusal to assign system tables to lower version
Status: Open | Priority: Minor | Resolution: 
Created: 24/Sep/21 19:46

I was doing some rolling downgrades of test clusters and keep getting into a state where my automation gets stuck trying to drain the final RegionServer in the cluster. At this point that RegionServer hosts 3 regions: meta, quota, namespace. The HMaster is outputting logs like: "Passed destination servername is null/empty so choosing a server at random".

I's very hard to understand what's happening based on that log, so you really have to look at the code. Tracking down that log line, it becomes somewhat clear that you are getting trapped by AssignmentManager.getExcludedServersForSystemTable().

Looking at the code, you can see comments related to "hbase.min.version.move.system.tables" config, but the comments are very unclear. What should I set this to?

This setting was added in https://issues.apache.org/jira/browse/HBASE-22923 which focuses mostly on RSGroup, but this issue is affecting clusters that do not use RSGroup. The release note also is not super clear.

It would be great to clarify the docs to help the operator know what to change this to, or perhaps make the config itself more intuitive. For example, could we just make it an allowlist of versions that can hold system tables? At that point my path is clear: add the version I'm downgrading to to the allowlist.

This issue is also exacerbated by the fact that by the time you've realized this you're in a somewhat tricky situation where there's only 1 RegionServer left and your only way around it is to force stop it or to push a new config and rolling restart your HMasters. It would be great if this setting were able to be updated via Admin or at the very least reloadable with ConfigurationObserver.
Bug: Move NavigableSet add operation to writer thread in BucketCache
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Sep/21 02:29

We currently use a ConcurrentSkipList to store blocks by HFile in bucket cache. The average time complexity of the add function is O(logn). We can move this time costly to the writer threads to reduce the response latency of read requests. I have tested the time cost of function cacheBlock in BucketCache and attached the metrics screenshots.
Bug: Fix TestMaster#testMoveRegionWhenNotInitialized with hbase.min.version.move.system.tables
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Aug/21 14:43

In order to protect meta region movement unexpectedly during upgrade with rsGroup enabled, it is good practice to keep hbase.min.version.move.system.tables in hbase-default for specific branch so that the use-case for the specific version of HBase is well under control. However, TestMaster#testMoveRegionWhenNotInitialized would fail because it would not find server to move meta to. We should fix this.

 
{code:java}
INFO  [Time-limited test] master.HMaster(2029): Passed destination servername is null/empty so choosing a server at random

java.lang.UnsupportedOperationExceptionjava.lang.UnsupportedOperationException at java.util.AbstractList.add(AbstractList.java:148) at java.util.AbstractList.add(AbstractList.java:108) at org.apache.hadoop.hbase.master.HMaster.move(HMaster.java:2031) at org.apache.hadoop.hbase.master.TestMaster.testMoveRegionWhenNotInitialized(TestMaster.java:181) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{code}
Bug: StochasticLoadBalancer metrics should update even if balancer doesn't run
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Sep/21 20:01

Currently we only update StochasticLoadBalancerMetrics at the very end of a balancer run, once a plan has been found. In fact, we update the metrics based on that plan even if we don't end up executing the plan, which seems incorrect. 

Regardless of whether the balancer decides to run or not, cluster costs are changing all the time. Since we don't update these metrics any other time, operators miss out on important information about the balance of their cluster over time. 

I briefly looked into it and it would be relatively trivial to add another call to updateStochasticCosts at the beginning of the balanceTable method, before we determine if the cluster is in need of balancing. This would be an improvement but would still miss cases where the balancer is disabled or unable to run due to regions in transition, etc.

It would be good if we could make it so updateStochasticCosts get called periodically regardless of whether the balancer is enabled or can run.
Bug: Store configuration loss when use update_config
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Sep/21 10:09

When use update_config shell command, some store configuration is loss.

When initialize store, the conf is set by:
{code:java}
this.conf = new CompoundConfiguration()
  .add(confParam)
  .addBytesMap(region.getTableDescriptor().getValues())
  .addStringMap(family.getConfiguration())
  .addBytesMap(family.getValues());
{code}
when change configuration, the conf is set by:
{code:java}
this.conf = new CompoundConfiguration()
        .add(conf)
        .addBytesMap(getColumnFamilyDescriptor().getValues());
{code}
Bug: RegionServer ignored a procedure dute to repeatedly opening which lead region stuck in RIT
Status: Open | Priority: Minor | Resolution: 
Created: 14/Sep/21 03:44
Affects Version/s: 2.3.5

       when hbase cluster shut down Suddenlly,  there are some repeatedly opening regions and cause them to stuck in RTI。we found org.apache.hadoop.hbase.regionserver.handler.AssignRegionHandler#process() directly return when handle repeatedly opening regions， and don't report to master,  master unable to  process next status for this procedure， and regions stuck in RIT。so can we  do like this:
 #  AssignRegionHandler report the region is online if regionserver found the region is online,
 # or AssignRegionHandler report the region is failed online if regionserver found the region is online, let Hmater to retry。

please give some suggestions, thanks !
{code:java}
// code placeholder
Region onlineRegion = rs.getRegion(encodedName); if (onlineRegion != null) { 
LOG.warn("Received OPEN for the region:{}, which is already online", regionName);
 // Just follow the old behavior, do we need to call reportRegionStateTransition? Maybe not? 
// For normal case, it could happen that the rpc call to schedule this handler is succeeded,
 // but before returning to master the connection is broken. And when master tries again, we 
// have already finished the opening. For this case we do not need to call 
// reportRegionStateTransition any more.
 return; }
{code}
Bug: Add Admin and shell method to dynamically update master configurations
Status: Open | Priority: Minor | Resolution: 
Created: 17/Sep/21 22:09

It's possible to update master configuration by using update_config '<master servername>', or use update_all_config to update all regionservers and masters. The former is annoying because you need to find the master's full servername. Unlike regionservers, this is sort of hard to find. The latter is not great because it has to iterate ALL servers in the cluster, which might take a while for large clusters and not be what you want generally.

We should add an option to reload all master configurations. I could imagine this as a update_master_config, or maybe update_all_config 'master'.

Shell used for illustration here, but this all applies to Admin interface as well.
Bug: Improve developer docs for InterfaceAudience of public experimental APIs
Status: Open | Priority: Minor | Resolution: 
Created: 08/Sep/21 15:42

Per [https://lists.apache.org/thread.html/re7736008a7619029bf34ed81f5931c537281d22788719f2dcf37f0c0%40%3Cdev.hbase.apache.org%3E,] our InterfaceAudience.Public restrictions intentionally deviate from the default yetus docs, and we would like to keep it that way. It was acknowledged that there may be a reason to expose early access to certain APIs that may need to evolve in breaking ways. The suggested approach is to annotate these as InterfaceAudience.LimitedPrivate.

We should update developer docs to clarify both points.
Bug: Allow AuthUtil automatic renewal to be disabled
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Aug/21 17:09

Talking with [~bbende] who was looking at some "spam" in the NiFi log where AuthUtil was complaining that it couldn't renew the UGI. This is did not cause him problems (NiFi could always read/write to HBase), but it generated a lot of noise in the NiFi log.

NiFi is special in that it's managing renewals on its own (for all services it can communicate with), rather than letting each client do it on its own. Specifically, one way they do this is by doing a keytab-based login via JAAS, constructing a UGI object from that JAAS login, and then invoking HBase in a normal UGI.doAs().

The problem comes in that AuthUtil _thinks_ that it is capable of renewing this UGI instance on its own. AuthUtil can determine that the current UGI came from a keytab, and thus thinks that it can renew it. However, this actually fails because the LoginContext inside UGI *isn't* actually something that UGI can renew (remember: because NiFI did it directly via JAAS and not via UGI)
{noformat}
2021-08-19 17:32:19,438 ERROR [Relogin service.Chore.1] org.apache.hadoop.hbase.AuthUtil Got exception while trying to refresh credentials: loginUserFromKeyTab must be done first
java.io.IOException: loginUserFromKeyTab must be done first
	at org.apache.hadoop.security.UserGroupInformation.reloginFromKeytab(UserGroupInformation.java:1194)
	at org.apache.hadoop.security.UserGroupInformation.checkTGTAndReloginFromKeytab(UserGroupInformation.java:1125)
	at org.apache.hadoop.hbase.AuthUtil$1.chore(AuthUtil.java:206) {noformat}
After talking with Bryan about this: we don't see a good way for HBase to detect this specific "A UGI instance, but not created by UGI" case because the LoginContext inside UGI is private. It is great that AuthUtil will automatically try to renew keytab logins, even if not using {{hbase.client.keytab.file}} and {{hbase.client.keytab.principal}}, so I don't want to break that functionality.

NiFi is unique in this case that it is fully managing the renewals, so I think the best path forward is to add an option which lets NiFi disable AuthUtil since it knows it can safely do this. This should affect any others users (but also give us an option if AuthUtil ever does cause problems).
Bug: TestTableSnapshotScanner.testMergeRegion error message is misleading.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Aug/21 13:06
Affects Version/s: 2.4.5

In case of any exception thrown, this test wrongly reports a FNFE:
{noformat}
... 
    } catch (Exception e) {
       LOG.error("scan snapshot error", e);
       Assert.fail("Should not throw FileNotFoundException"); 
       Assert.assertTrue(e.getCause() != null);
       Assert.assertTrue(e.getCause().getCause() instanceof FileNotFoundException);
     } finally {
...{noformat}
Bug: config hbase.regions.slop is used for stochastic load balancer.
Status: Open | Priority: Minor | Resolution: 
Created: 18/Aug/21 23:40

The config is used to initialize StochasiticLoadBalancer for the field slop. But the field is not used anywhere as of now.

There is a related Jira that removed it from Stochastic Balancer https://issues.apache.org/jira/browse/HBASE-9310 

 

We can clean up the code.
Bug: Correct regex in hbase-personality.sh
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Aug/21 15:57

When activating the Hadoop profile in hbase-personality.sh the regex also accepts branch-1 while it should only set the Hadoop profile for branch-2.

https://github.com/apache/hbase/blob/master/dev-support/hbase-personality.sh#L151
Bug: Configurable disallowlist for live editing of loglevels
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Aug/21 13:33

We currently use log4j/slf4j for audit logging in AccessController. This is convenient but presents a security/compliance risk because we allow live-editing of logLevels via the UI. One can simply set the logger to OFF and then perform actions un-audited.

We should add a configuration for setting certain log levels to read-only
Bug: tablelist show pages in masterPage
Status: Open | Priority: Minor | Resolution: 
Created: 30/Jul/21 09:34

When the number of tables is large, the display length of the master page is too long, and it needs to be displayed in pages.

 

Table list supports paging and retrieval：

!image-2021-07-30-20-27-56-468.png!

!image-2021-07-30-20-29-30-347.png!
Bug: Allow custom opts for hbck in hbase bin
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Jul/21 13:51

https://issues.apache.org/jira/browse/HBASE-15145 made it so that when you execute {{hbase hbck}}, the regionserver or JAAS opts are added automatically to the command line. This is problematic in some cases depending on what regionserver opts have been set. For instance, one might configure a jmx port for the regionserver but then hbck will fail due to a port conflict if run on the same host as a regionserver. Another example would be that a regionserver might define an {{-Xms}} value which is significantly more than hbck requires.

 

We should make it possible for users to define their own HBASE_HBCK_OPTS which take precedence over the server opts added by default.
Bug: HBase "delete" cannot delete data versions smaller than the specified timestamp
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 21/Jul/21 01:14
Affects Version/s: 2.1.0

When I use 'delete' to delete a piece of data, I set a value larger than the timestamp of the data, and then the operation does not take effect. But 'deleteall' is feasible. Is this difference designed on purpose?
Bug: Cell TTL set to -1 should mean never expire, the same as CF TTL
Status: Open | Priority: Minor | Resolution: 
Created: 01/Jul/21 02:32

We met a user who took it for granted that setting the cell TTL to -1 is the same as the CF TTL, which means that it will never expire. However, if the cell TTL is set to -1, the cell will expire immediately. In fact, if the cell TTL is not set, it means that it will never expire, but he just set it...... We should unify the meaning of cell TTL and CF TTL on the value of -1.
Bug: L1 miss metric is incorrect when using CombinedBlockCache
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Jul/21 08:56

In CombinedBlockCache getBlock method


{code:java}
public Cacheable getBlock(BlockCacheKey cacheKey, boolean caching,
      boolean repeat, boolean updateCacheMetrics) {
    // TODO: is there a hole here, or just awkwardness since in the lruCache getBlock
    // we end up calling l2Cache.getBlock.
    // We are not in a position to exactly look at LRU cache or BC as BlockType may not be getting
    // passed always.
    return l1Cache.containsBlock(cacheKey)?
        l1Cache.getBlock(cacheKey, caching, repeat, updateCacheMetrics):
        l2Cache.getBlock(cacheKey, caching, repeat, updateCacheMetrics);
  }
{code}
If L1 does not contains cacheKey, it will never check it in L1, so that L1 miss metrics is always 0.
Bug: Minor cleanups to reduce checkstyle warnings on backup code
Status: Open | Priority: Trivial | Resolution: 
Created: 17/Aug/21 02:56
Affects Version/s: 3.0.0-alpha-2

As suggested in this PR --> [https://github.com/apache/hbase/pull/3359#pullrequestreview-716511415]

Created this issue to clean up Backup classes to reduce checkstyle warnings.
Bug: Fix typo for file 'hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java'
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 31/Aug/21 10:38

Fix typo for file 'hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java'
{code:bash}
hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java:1823: mulitple ==> multiple

hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java:3562: compatiblity ==> compatibility

hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java:3864: supportted ==> supported

{code}
Bug: Rectify the expired TODO comment in CombinedBC
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 21/Jul/21 02:39

In the method getBlock in CombinedBC, there is a TODO comment as follows:

{code:java}
@Override
  public Cacheable getBlock(BlockCacheKey cacheKey, boolean caching,
      boolean repeat, boolean updateCacheMetrics) {
    // TODO: is there a hole here, or just awkwardness since in the lruCache getBlock
    // we end up calling l2Cache.getBlock.
    // We are not in a position to exactly look at LRU cache or BC as BlockType may not be getting
    // passed always.
    boolean existInL1 = l1Cache.containsBlock(cacheKey);
    if (!existInL1 && updateCacheMetrics && !repeat) {
      // If the block does not exist in L1, the containsBlock should be counted as one miss.
      l1Cache.getStats().miss(caching, cacheKey.isPrimary(), cacheKey.getBlockType());
    }
    return existInL1 ?
        l1Cache.getBlock(cacheKey, caching, repeat, updateCacheMetrics):
        l2Cache.getBlock(cacheKey, caching, repeat, updateCacheMetrics);
  }
{code}
The TODO comment is expired. While in CombinedBC, L2 is not the victim handler of L1.
Bug: DELETE endpoint in REST API does not support deleting binary row keys/columns
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 23/Oct/23 05:53
Affects Version/s: 2.4.17

h2. Notes

This is the first time I have raised an issue in the ASF Jira. Please let me know if there's anything I need to adjust on the issue to fit in with your development flow.

I have marked the priority as "blocker" because this issue blocks me as a user of the HBase REST API from deploying an effective solution for our setup. Please feel free to change this if the Priority field has another meaning to you.

I have also chosen 2.4.17 as the affected version because this is the version I am running, however looking at the source code on GitHub in the default branch, I think many other versions would be affected.
h2. Description of Issue

The DELETE operation in the [HBase REST API|https://hbase.apache.org/1.2/apidocs/org/apache/hadoop/hbase/rest/package-summary.html#operation_delete] requires specifying row keys and column families/offsets in the URI (i.e. as UTF-8 text). This makes it impossible to specify a delete operation via the REST API for a binary row key or column family/offset, as single bytes with a decimal value greater than 127 are not valid in UTF-8.

Percent-encoding these "high" values does not work around the issue, as the HBase REST API uses Java's {{URLDecoder.Decode(percentEncodedString, "UTF-8")}} function, which replaces any percent-encoded byte in the range {{%80}} to {{%FF}} with the [replacement character|https://en.wikipedia.org/wiki/Specials_(Unicode_block)#Replacement_character]. Even if this were not the case, the row-key is ultimately [converted to a byte array|https://github.com/apache/hbase/blob/rel/2.4.17/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RowSpec.java#L60-L100] using UTF-8 encoding, wherein code points >127 are encoded across multiple bytes, corrupting the user-supplied row key.
h2. Proposed Solution

I do not believe it is possible to allow encoding of arbitrary bytes in the URL for the DELETE endpoint without breaking compatibility for any users who may have been unknowingly UTF-8 encoding their binary row keys. Even if it were possible, the syntax would likely be terse.

Instead, I propose a new version of the DELETE endpoint that would accept row keys and column families/offsets in the request _body_ (using Base64 encoding for the JSON and XML formats, and bare binary for protobuf). This new endpoint would follow the same conventions as the PUT operations, except that cell values would not need to be specified (unless the user is performing a check-and-delete operation).

As an additional benefit, using the request body could potentially allow for deleting multiple rows in a single request, which would drastically improve the efficiency of my use case.
Bug: Remove the xml-maven-plugin in root pom on branch-3
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 19/Dec/23 03:34

We only generate ref guide on master branch, so we have already deleted related resources on other branches.

But on branch-3, we still have xml=maven-plugin for including hbase-default.xml in ref guide so it will fail the build.

Let's remove this plugin.
Bug: Race between RegionRemoteProcedureBase and rollback operation could lead to ROLLEDBACK state be persisent to procedure store
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 07/Dec/23 06:22

And then cause load procedures failure.

This is because we do not execute RegionRemoteProcedureBase.persistAndWake method in PEWorker, so even if we hold the procedureExecutionLock in rollback, they could be executed concurrently.

So it is possible that in rollback, we set the state to ROLLEDBACK and delete the procedure, and then in persistAndWake, we persist the ROLLEDBACK state to procedure store.
Bug: Document spark.hadoopRDD.ignoreEmptySplits issue for Spark Connector
Status: Open | Priority: Critical | Resolution: 
Created: 23/Nov/23 20:04

For Spark 3.2.0+ the connector needs 
spark.hadoopRDD.ignoreEmptySplits=false to work correctly.

This crucial piece of information is not documented either in the README, or in the main Hbase connector section.
Bug: There could be holes in stack ids when loading procedures
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 17/Nov/23 11:46

Found this when implementing HBASE-28199, as after HBASE-28199 we will suspend procedures a lot, so a missed scenario has been covered and it will fail some UTs with corrupted procedures when loading.

I think this issue should be fixed separately as it affects all active branches.

Let me try to implement a UT first.
Bug: AsyncFSWAL may hang
Status: Open | Priority: Critical | Resolution: 
Created: 16/Nov/23 04:01

Saw this in the UT environment before when checking why TestClusterScopeQuotaThrottle.

And when implementing HBASE-28199, it is very easy to reproduce the problem while executing some other UTs, like TestAdmin, and the problem disappear after chaning to use FSHLog.

Need to dig more as now AsyncFSWAL is the default implementation.