av9ash/GitBugs huggingface dataset:
Bug: UNASSIGN proc indefinitely stuck on dead rs
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 14/Apr/24 18:50

One scenario we noticed in production -

we had DisableTableProc and SCP almost triggered at similar time

2024-03-16 17:59:23,014 INFO [PEWorker-11] procedure.DisableTableProcedure - 
Set <TABLE_NAME> to state=DISABLING

2024-03-16 17:59:15,243 INFO [PEWorker-26] procedure.ServerCrashProcedure - 
Start pid=21592440, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure 
<regionserver>, splitWal=true, meta=false

DisabeTableProc creates unassign procs, and at this time ASSIGNs of SCP is not completed

{{2024-03-16 17:59:23,003 DEBUG [PEWorker-40] procedure2.ProcedureExecutor - LOCK_EVENT_WAIT pid=21594220, ppid=21592440, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=<TABLE_NAME>, region=<regionhash>, ASSIGN}}

UNASSIGN created by DisableTableProc is stuck on the dead regionserver and we had to manually bypass unassign of DisableTableProc and then do ASSIGN.

If we can break the loop for UNASSIGN procedure to not retry if there is scp for that server, we do not need manual intervention?, at least the DisableTableProc can go to a rollback state?
Bug: replication quota leak when peer changes
Status: Open | Priority: Critical | Resolution: 
Created: 27/May/24 13:25

When peer changes, replication closes the reader and shipper created earlier. However, after the specified timeout, the shipper still does not automatically close (It was interrupted, but it didn't close properly). The existing code simply returns without releasing quota. Not cleaning buffer usage.
In one practice of my company, in this case, the quota was full because it was not released in time, so wal reader could not continue read new data and replication had a backlog.

 

The log is as follows:

2024-05-20 20:00:00,796 WARN [RpcServer.default.FPRWQ.Fifo.read.handler=70,queue=1,port=16020] regionserver.ReplicationSourceShipper: Shipper clearWALEntryBatch method timed out whilst waiting reader/shipper thread to stop. Not cleaning buffer usage. Shipper alive: peer1; Reader alive: false
2024-05-20 20:00:01,351 WARN peer=peer1, can't read more edits from WAL as buffer usage 268435456B exceeds limit 268435456B
Bug: ModifyTableProcedure should not reset TRSP on region node when closing unused region replicas
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 09/May/24 14:24

Found this when digging HBASE-28522.

First, this is not safe as MTP does not like DTP where we hold the exclusive lock all the time.
Second, even if we hold the exclusive lock all the time, as showed in HBASE-28522, we may still hang there forever because SCP will not interrupt the TRSP.
Bug: Optimize the verifyCopyResult logic in ExportSnapshot
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 12/Jun/24 01:47

In [HBASE-28625|https://issues.apache.org/jira/browse/HBASE-28625], we added checksum logic comparison for ExportSnapshot files, but the checksum verification scenarios were too simple. This issue aims to address the existing problem. For detailed discussion, please refer to PR [5950|https://github.com/apache/hbase/pull/5950].
Bug: Losing exception from scan RPC can lead to partial results
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 15/May/24 14:05

This was discovered in Apache Impala using HBase 2.2 based branch hbase client and server. It is not clear yet whether other branches are also affected.

The issue happens if the server side of the scan throws an exception and closes the scanner, but at the same time, the client gets an rpc connection closed error and doesn't process the exception sent by the server. Client then thinks it got a network error, which leads to retrying the RPC instead of opening a new scanner. But then when the client retry reaches the server, the server returns an empty ScanResponse instead of an error, leading to closing the scanner on client side without returning any error.

A few pointers to critical parts:
region server:
1st call throws exception leading to closing (but not deleting) scanner:
[https://github.com/apache/hbase/blob/0c8607a35008b7dca15e9daaec41ec362d159d67/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L3539]
2nd call (retry of 1st) returns empty results:
[https://github.com/apache/hbase/blob/0c8607a35008b7dca15e9daaec41ec362d159d67/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L3403]

client:
some exceptions are handled as non-retriable at RPC level and are only handled through opening a new scanner:
[https://github.com/apache/hbase/blob/0c8607a35008b7dca15e9daaec41ec362d159d67/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java#L214]
[https://github.com/apache/hbase/blob/0c8607a35008b7dca15e9daaec41ec362d159d67/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java#L367]

This mechanism in the client only works if it gets the exception from the server. If there are connection issues during the RPC then the client won't really know the state of the server.
Bug: The StochasticLoadBalancer should support discrete evaluations for replica distribution
Status: Open | Priority: Major | Resolution: 
Created: 10/Apr/24 18:54

I have a larger write up available [here|https://docs.google.com/document/d/1jA8Ghs86v7b-53j5DcsdbPnOXxbHjewkIBFi1E4S1pY/edit?usp=sharing].

Secondary replica balancing squashes all other cost considerations.

Basically there are a few cost functions with relatively huge default multipliers. For example `PrimaryRegionCountSkewCostFunction` has a default multiplier of 100,000. Meanwhile things like StoreFileCostFunction have a multiplier of 5. Having any multiplier of 100k, while others are single digit, basically makes the latter category totally irrelevant from balancer considerations.

I understand that it's critical to distribute a region's replicas across multiple hosts/racks, but I don't think we should do this at the expense of all other balancer considerations.

For example, maybe we could have two types of balancer considerations: costs (as we do now), and conditionals (for the more discrete considerations, like ">1 replica of the same region should not exist on a single host"). This would allow us to prioritize replica distribution _and_ maintain consideration for things like storefile balance.
Bug: REST ScannerModel doesn't support includeStartRow/includeStopRow
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/May/24 14:38

includeStartRow/includeStopRow should be transparently supported.
The current behaviour is limited and confusing, as the user would rightly expect this to work via the REST interface.

The only problem is that adding them may break backwards compatibility.
Need to test if the XML unmarshaller can handle nonexistent fields.
Bug: Enable setting blockcache on-heap sizes in bytes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/May/24 09:37

Specifying blockcache and memcache sizes as a percentage of heap is not always ideal. Sometimes it's easier to specify exact values rather than backing into a percentage. Let's introduce new configuration settings (perhaps named similarly to {{hbase.bucketcache.size}}) that accept byte values. Even nicer would be if these settings accepted human-friendly byte values like {{512m}} or {{10g}}.
Bug: Allow replication to different target table
Status: Open | Priority: Major | Resolution: 
Created: 04/Apr/24 15:19

At the moment, HBase replication assumes the source and target table have the same name. Our company is looking into a use case that would require replicating to an arbitrary target table. 

 

*Potential Solution:*

It seems that the WALKey stores the table name of it's corresponding edit should be applied to.

We can add a setting to ReplicationPeerConfig that allows users to specify a source to target table mapping. This mapping can then be used to overwrite the WALKey's table before shipping the edit to the target cluster.
Bug: ScanResumer.resume would perform unnecessary scan when close AsyncTableResultScanner
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Apr/24 03:20
Affects Version/s: 2.6.0

When we invoke {{AsyncTableResultScanner.close}},  {{AsyncTableResultScanner.resultQueue}} is cleared and {{AsyncTableResultScanner.closed}} is set to true, and we do not need any more scan results. But if there is a {{ScanResumser}}, {{ScanResumer.resume}} would be invoked to perform another unnecessary scan on {{RegionServer}} and call {{AsyncTableResultScanner.onNext}} again when {{ScanResponse}} is received.  {{AsyncTableResultScanner.onNext}}  would do nothing else but just discard scan results because {{AsyncTableResultScanner.closed}} is true.  We could save this unnecessary scan on {{RegionServer}} and close scanner directly .
Bug: Implement HTTP session support in REST server and client for default auth
Status: Open | Priority: Major | Resolution: 
Created: 05/Apr/24 11:32

The REST server (and java client) currently does not implement sessions.

While is not  necessary for the REST API to work, implementing sessions would be a big improvement in throughput and resource usage.

* It would make load balancing with sticky sessions possible
* It would save the overhead of performing authentication for each request

 The gains are particularly big when using SPENGO:

* The full SPENGO handshake can be skipped for subsequent requests
* When Knox performs SPENGO authentication for the proxied client, it access the identity store each time. When the session is set, this step is only perfomed on the initial request.

The same change has resulted in spectacular performance improvements for Phoenix Query Server when implemented in Avatica.
Bug: Optimise BucketCache usage upon regions splits/merges.
Status: Reopened | Priority: Major | Resolution: 
Created: 15/May/24 16:48
Affects Version/s: 2.6.0

This proposal aims to give more flexibility for users to decide whether or not blocks from a parent region should be evict, and also optimise cache usage by resolving file reference blocks to the referred block in the cache.

Some extra context:

1) Originally, the default behaviour on splits was to rely on the "hbase.rs.evictblocksonclose" value to decide if the cached blocks from the parent split should be evicted or not. Then the resulting split daughters get open with refs to the parent file. If hbase.rs.prefetchblocksonopen is set, these openings will trigger a prefetch of the blocks from the parent split, now with cache keys from the ref path. That means, if "hbase.rs.evictblocksonclose" is false and “hbase.rs.prefetchblocksonopen” is true, we will be duplicating blocks in the cache. In scenarios where cache usage is at capacity and added latency for reading from the file system is high (for example reading from a cloud storage), this can have a severe impact, as the prefetch for the refs would trigger evictions. Also, the refs tend to be short lived, as compaction is triggered on the split daughters soon after it’s open.

2) HBASE-27474 has changed the original behaviour described above, to now always evict blocks from the split parent upon split is completed, and skipping prefetch for refs (since refs are short lived). The side effect is that the daughters blocks would only be cached once compaction is completed, but compaction itself will run slower since it needs to read the blocks from the file system. On regions as large as 20GB, the performance degradation reported by users has been severe.

This proposes a new “hbase.rs.evictblocksonsplit” configuration property that makes the eviction over split configurable. Depending on the use case, the impact of mass evictions due to cache capacity may be higher, in which case users might prefer to keep evicting split parent blocks. Additionally, it modifies the way we handle refs when caching. HBASE-27474 behaviour was to skip caching refs to avoid duplicate data in the cache as long as compaction was enabled, relying on the fact that refs from splits are usually short lived. Here, we propose modifying the search for blocks cache keys, so that we always resolve the referenced file first and look for the related referenced file block in the cache. That way we avoid duplicates in the cache and also expedite scan performance on the split daughters, as it’s now resolving the referenced file and reading from the cache.
Bug: There is a possibility that data cannot be obtained during reverse fuzzy query.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jun/24 06:35
Affects Version/s: 2.4.14

this is my example:

1.  

create 'abcd','f'

put 'abcd','111311','f:name','aaaaa'

put 'abcd','111444','f:name','aaaaa'

put 'abcd','111511','f:name','aaaaa'

put 'abcd','111611','f:name','aaaaa'

put 'abcd','111446','f:name','aaaaa'

put 'abcd','111777 ','f:name','aaaaa'

put 'abcd',' 111777','f:name','aaaaa'

 

2.  When I don't use the reversed query, I can get the data.

scan 'abcd', \{FILTER => org.apache.hadoop.hbase.filter.FuzzyRowFilter.new(Arrays.asList(Pair.new(Bytes.toBytesBinary('111433'),Bytes.toBytesBinary("\xFF\xFF\xFF\xFF\x02\x02"))))}

!image-2024-06-03-14-26-57-645.png!

 

3. When I use the reversed query, I can not get the data.

scan 'abcd', \{REVERSED=>TRUE, FILTER => org.apache.hadoop.hbase.filter.FuzzyRowFilter.new(Arrays.asList(Pair.new(Bytes.toBytesBinary('111433'),Bytes.toBytesBinary("\xFF\xFF\xFF\xFF\x02\x02"))))}

!image-2024-06-03-14-27-17-516.png!

 

4. 

The test shows that the following two issues may be related to this issue: HBASE-26232
How to reproduce the issue resolved by this issue?
Currently, I solve the fuzzy query by rolling back the code of this issue. Is there a better solution?
Bug: More sensible client meta operation timeout default
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/May/24 13:54
Affects Version/s: 2.4.17

Documented behavior in the HBase reference for client meta operation timeout {{hbase.client.meta.operation.timeout}} default is that it will be set to the configured client operation timeout, but implementation is that it defaults to the default client operation timeout of 20 minutes. I propose to set the default meta operation to what is currently specified in the hbase reference: 

From "Timeout settings" in the hbase reference:
{panel}
A higher-level timeout is hbase.client.operation.timeout which is valid for each client call. When an RPC call fails for instance for a timeout due to hbase.rpc.timeout it will be retried until hbase.client.operation.timeout is reached. Client operation timeout for system tables can be fine tuned by setting hbase.client.meta.operation.timeout configuration value. When this is not set its value will use hbase.client.operation.timeout
{panel}
There seem to be two very different dependencies on meta operation timeout:
 # End to end operation timeout for system table operations (2.x and 3)
 # Timeout to acquire {{userRegionLock}} to initiate meta scan in {{locateRegionInMeta}} (2.x blocking client only, see HBASE-28730 for more detail/related work)

For case 1 I believe it makes sense from a user perspective that meta operation timeout, which is meant to apply to a specific subset of operations, will respect the 'general' operation timeout that is configured if the more specific meta operation timeout is not explicitly set. 

For case 2 blocking client, the default meta timeout value defeats the purpose of the `userRegionLock` timeout if one has a typical setup where {{hbase.client.operation.timeout}} << 20 minutes and {{hbase.client.meta.operation.timeout}} is not explicitly set, which can lead to operations taking much longer than the configured operation timeout to actually timeout if there is e.g meta slowness and/or contention around userRegionLock on 2.x, see HBASE-28730 for more detail/related work.
Bug: list_namespace not working after an incorrect user input
Status: Open | Priority: Major | Resolution: 
Created: 13/Jun/24 02:45
Affects Version/s: 2.6.0

When using hbase-3.0.0 or 2.6.0, there's a shell bug related to failure handling.

If user inputs an incorrect *list_namespace* command, hshell throws an exception. However, it has a side effect on the following *list_namespace* command: the result becomes empty (incorrect).
h1. Reproduce

Execute the following 2 commands can reproduce this bug
 * The first command is an incorrect list_namespace command, which causes and exception.
 * The second command is a correct list_namespace command, its return value is incorrect (empty).

{code:java}
list_namespace, 'ns.*'
list_namespace{code}
Here's the execution result

The return result of the second command is incorrect.
{code:java}
hbase:002:0> list_namespace, 'ns.*'
Traceback (most recent call last):
SyntaxError ((hbase):2: syntax error, unexpected end-of-file)
list_namespace, 'ns.*'
                      ^
hbase:003:0> list_namespace
hbase:004:0>  {code}
The expected output of list_namespace is
{code:java}
hbase:001:0> list_namespace
NAMESPACE                                                                                                                             
default                                                                                                                               
hbase                                                                                                                                 
2 row(s)
Took 0.6820 seconds {code}
h1. Root Cause

This could be a bug in shell related to list_namespace. Restarting the shell restores normal functionality of the list_namespace command.
Bug: PrefixFilter should use SEEK_NEXT_USING_HINT 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/May/24 17:52

Looking at PrefixFilter, I have noticed that it doesn't use the SEEK_NEXT_USING_HINT mechanism.

AFAICT, we could safely set the the prefix as a next row hint, which could be a huge performance win.

Of course, ideally the user would set the scan startRow to the prefix, which avoids the problem, but the user may forget to do that, or may use the filter in a filterList that doesn't allow for setting the start/stop rows close tho the prefix.
Bug: Refactor direct interactions of Reference file creations to SFT interface
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/May/24 06:10

HBASE-27826 introduces tracking of Link/Reference files with StoreFileTracker interface and with that, we can have the FileBasedStoreFileTracker implementation to track Link/Reference files only using .filelist file itself and not create actual link/ref files (virtual links).

To support the same, we need to refactor all the direct interactions of Ref/Link file creations to SFT layer and this changes starts off with moving the current Reference file operations/logic to StoreFileTrackerBase.
Post that we should be able to override/evolve the representation to handle Virtual Links.
Bug: Add separate fields for column family and qualifier in REST message formats
Status: Open | Priority: Major | Resolution: 
Created: 02/May/24 04:16

The current format uses the archaic column field, which requires extra processing and copying to encode/decode the CF and CQ at both the server and client side.

We need to:
- Add a version field to the requests, to be enabled by clients that support the new format
- Add the new fields to the JSON, XML and protobuf formats, and logic to use them.

This should be doable in a backwards-compatible manner, with the server falling back to the old format if it receives an unversioned request.
Bug: Authentication failure when running hbase-spark in local mode
Status: In Progress | Priority: Major | Resolution: 
Created: 18/Apr/24 04:58
Affects Version/s: connector-1.0.0

h2. Problem

When running Spark in local mode, hbase-spark fails to authenticate to a Kerberos secured HBase cluster. The error message is:
{quote}No matching SASL authentication provider and supporting token found from providers for user: xxx@XXX.XXX (auth:PROXY)
{quote}
That is because {{applyCreds}} changes the authentication method of the current user to {{{}PROXY{}}}, when it should still be {{KERBEROS}} for local mode to run correctly.
h2. Suggested solution

To fix this, I propose removing {{{}applyCreds{}}}. The function is no longer needed and should be removed:

1. Because we should not change the authentication mode of the current user in local mode
2. And because the purpose of the function is no longer valid. It is not doing anything meaningful since the broadcasting of the user credentials was removed in this commit:
[https://github.com/apache/hbase-connectors/commit/75e41365207408f5b47d5925469a49fd60078b5e]

A pull request is on the way.
h2. Testing

The fix was manually tested against Kerberos secured HBase 2.4.17 + Hadoop 3.3.5 cluster using Spark 3.5.1, both in local mode and with Yarn master, running the following Python code.
{code:java}
df = (spark.read.format("org.apache.hadoop.hbase.spark")
                .option("hbase.columns.mapping", "key STRING :key, state STRING info:state")
                .option("hbase.table", "hbase:meta").load())
df.first()
{code}
Bug: Enable setting memcache on-heap sizes in bytes
Status: Open | Priority: Major | Resolution: 
Created: 17/May/24 09:38

Specifying blockcache and memcache sizes as a percentage of heap is not always ideal. Sometimes it's easier to specify exact values rather than backing into a percentage. Let's introduce new configuration settings (perhaps named similarly to {{hbase.bucketcache.size}}) that accept byte values. Even nicer would be if these settings accepted human-friendly byte values like {{512m}} or {{10g}}.
Bug: asyncwal should attempt to recover lease if close fails
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jun/24 19:20

AsyncFSWAL does not verify that WAL gets closed properly.

For example, HDDS-10609 found that Ozone close may fail because it runs out of space. In such a case, the executor thread responsible for closing WAL file simply exits, and the WAL remains open.

It is okay for HDFS WAL to stay open, but Ozone does not support renaming open files, and therefore WAL archiving failed.

See HDDS-10609 for more details.
Bug: Docs around configuring backups can lead to unexpectedly disabling other features
Status: In Progress | Priority: Major | Resolution: 
Created: 28/May/24 17:42
Affects Version/s: 2.6.0

In our documentation for enabling backups, we suggest that the user set the following:
{code:java}
<property>
  <name>hbase.master.logcleaner.plugins</name>
  <value>org.apache.hadoop.hbase.backup.master.BackupLogCleaner,...</value>
</property>
<property>
  <name>hbase.master.hfilecleaner.plugins</name>
  <value>org.apache.hadoop.hbase.backup.BackupHFileCleaner,...</value>
</property> {code}
A naive user will set these and not know what to do about the ",..." part. In doing so, they will unexpectedly be disabling all of the default cleaners we have. For example here are the defaults:
{code:java}
<property>
  <name>hbase.master.logcleaner.plugins</name>
  <value>org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner,org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner,org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner</value>
</property>
<property>
  <name>hbase.master.hfilecleaner.plugins</name>
  <value>org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner,org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner</value>
</property> {code}
So basically disabling support for hbase.master.logcleaner.ttl and hbase.master.hfilecleaner.ttl.

There exists a method BackupManager.decorateMasterConfiguration and BackupManager.decorateRegionServerConfiguration. They are currently javadoc'd as being for tests only, but I think we should call these in HMaster and HRegionServer. Then we can only require the user to set "hbase.backup.enable" and very much simplify our docs here.