av9ash/GitBugs huggingface dataset:
Bug: Throw exception if region has same start key when writePartitions use HFileOutputFormat2
Status: Open | Priority: Minor | Resolution: 
Created: 21/Jan/22 08:29

Some exception info is not friendly in MapReduce task log such as:
{code:java}
2022-01-21 06:23:39,156 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.IllegalArgumentException: Can't read partitions file
	at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:116)
	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:719)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:169)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1865)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.io.IOException: Wrong number of partitions in keyset
	at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:90)
	... 10 more
 {code}
when I write hfile use HFileOutputFormat2 in a MapReduce task.

I can not understand what happend to my job. And got some wrong suggestions such as: trucate table or add cancelDeleteOnExit code in my code. Those are useless and stupid. 

 

*The real reason is my table has same start key ,and I didn't find this problem in hbck web ui overlaps list immediately.* 

{*}So{*}， I thanks it's a good idea to check same start key in writePartitions(), we can print each same start key  and throw a same start key exception if my table regions has same start key.

It is a more friendly exception info than 'Can't read partitions file'.

Here is my code  [https://github.com/Mengqi777/hbase/commit/d8dc093695d02cfb16ba303389de9f7cef778ee1]

May I create a pull request ?
Bug: Throw exception if region has same start key when writePartitions use HFileOutputFormat2
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 21/Jan/22 08:29

Some exception info is not friendly in MapReduce task log such as:
{code:java}
2022-01-21 06:23:39,156 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.IllegalArgumentException: Can't read partitions file
	at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:116)
	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:719)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:169)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1865)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.io.IOException: Wrong number of partitions in keyset
	at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:90)
	... 10 more
 {code}
when I write hfile use HFileOutputFormat2 in a MapReduce task.

I can not understand what happend to my job. And got some wrong suggestions  via google/baidu such as: trucate table or add cancelDeleteOnExit code in my code. Those are useless and stupid. 

 

*The real reason is my table has same start key ,and I didn't find this problem in hbck web ui overlaps list immediately.* 

{*}So{*}， I thanks it's a good idea to check same start key in writePartitions(), we can print each same start key  and throw a same start key exception if my table regions has same start key.

It is more friendly exception info than 'Can't read partitions file'.

Here is my code  [https://github.com/Mengqi777/hbase/commit/d8dc093695d02cfb16ba303389de9f7cef778ee1]

can I create a pull request ?
Bug: Throw exception if region has same start key when writePartitions use HFileOutputFormat2
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 21/Jan/22 08:29

Some exception info is not friendly in MapReduce task log such as:
{code:java}
2022-01-21 06:23:39,156 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.IllegalArgumentException: Can't read partitions file
	at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:116)
	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:719)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:169)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1865)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.io.IOException: Wrong number of partitions in keyset
	at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:90)
	... 10 more
 {code}
when I write hfile use HFileOutputFormat2 in a MapReduce task.

I can not understand what happend to my job. And got some wrong suggestions such as: trucate table or add cancelDeleteOnExit code in my code. Those are useless and stupid. 

 

*The real reason is my table has same start key ,and I didn't find this problem in hbck web ui overlaps list immediately.* 

{*}So{*}， I thanks it's a good idea to check same start key in writePartitions(), we can print each same start key  and throw a same start key exception if my table regions has same start key.

It is more friendly exception info than 'Can't read partitions file'.

Here is my code  [https://github.com/Mengqi777/hbase/commit/d8dc093695d02cfb16ba303389de9f7cef778ee1|https://github.com/Mengqi777/hbase/commit/d8dc093695d02cfb16ba303389de9f7cef778ee1],]

can I create a pull request ?
Bug: Account for HBASE-24500 in regionInfoMismatch tool
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Jan/22 19:14

Had a coworker try to use the RegionInfoMismatch tool I added in HBASE-26656. Curiously, the tool failed on the sanity check I added.
{noformat}
Aborting: sanity-check failed on updated RegionInfo. Expected encoded region name 736ee6186975de6967cd9e9e242423f0 but got 323748c77dde5b05982df0285b013232.
Incorrectly created RegionInfo was: {ENCODED => 323748c77dde5b05982df0285b013232, NAME => 'test4,,1642405560420_0002.323748c77dde5b05982df0285b013232.', STARTKEY => '', ENDKEY => ''}

{noformat}
I couldn't understand why the tool wasn't working until I hooked up a debugger and realized that the problem wasn't in my code :). The version of HBase on the system did not have the fix from HBASE-24500 included which meant that I was hitting the same "strange behavior", as Duo put it, in the RegionInfoBuilder "copy constructor".

While the versions of HBase which do not have this fix are EOL in terms of Apache releases, we can easily work around this in operator-tools (which may be used by any hbase 2.x release still in the wild).
Bug: The threadGroup.destroy in ProcedureExecutor.join fails in some tests
Status: Open | Priority: Minor | Resolution: 
Created: 14/Jan/22 15:31

Always see it dumps the threads in the group to stdout.

This is an example
{noformat}
java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10]
    Thread[HFileArchiver-1,5,PEWorkerGroup]
{noformat}

I guess the problem is that we create a thread pool when executing a procedure so the thread pool will inherit the thread group.

Not a big problem but still a bit nasty...
Bug: Unintuitive AccesControlClient.getUserPermissions() semantics
Status: Open | Priority: Minor | Resolution: 
Created: 07/Jan/22 07:39

The behaviour of the AccesControlClient.getUserPermissions() calls is unintuitive.

It takes a tablename regex, and return the union of all permissinons on all tables that it matches.
While the returned UserPermission objects do have the information on the object they apply to, this still requires post-processing the results.

To get the permissions for a single table, one has to either do something like

{code:java}
Admin.getTablePermission(conn, "^" + tableName.getNameWithNamespaceInclAsString() + "$")
{code}

or post-process the results of the call.

We should add some methods that return the permission for a single table / family / qualifier without making the caller jump though hoops, or at least call out the non-intuitive behavior in the Javadoc, and advise on how to use the API to get the results the caller likely wants.
Bug: [Documentation] Correct description of WAL directory 
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 07/Feb/22 10:11
Affects Version/s: 2.4.9

h3. [Write Ahead Log (WAL)|https://hbase.apache.org/book.html#purpose.wal]
{quote}The WAL resides in HDFS in the _/hbase/WALs/_ directory, with subdirectories per region.
{quote}
The subdirectory should belong to RegionServer.
Bug: KafkaProxy enablePeer parameter mismatch
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 24/Jan/22 10:04

In KafkaProxy class, enable peer by setting "e"
{code:java}
options.addOption("e", "enablepeer", false,
    "enable peer on startup (defaults to false)"); {code}
but it actually uses "a":
{code:java}
if (commandLine.hasOption('a')){
  createPeer=true;
}

if (commandLine.hasOption("a")){
  enablePeer=true;
}{code}
Bug: When StoreFileTracker is FILE, unable to recognize StoreFileListFile after upgrade from 2.5 to 2.6
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 06/Dec/24 21:49
Affects Version/s: 2.5.10

h1. Symptoms

After upgrading from HBase 2.5.10 to HBase 2.6.1, I observed an inconsistent read across versions.

Before the upgrade, I created a table with one row. Invoking a count command returns 1. After upgrading to 2.6.1, execute the same count command, it returns 0.
{code:java}
count 'test'

Old Version Result:
1 row(s)
Took 0.5582 seconds
=> 1

New Version Result: 
0 row(s)
Took 1.4439 seconds
=> 0 {code}
h1. Reproduce

It can be reproduced deterministically with the following steps

Step1: Start up the HBase cluster in 2.5.10 and execute the following commands
{code:java}
create 'test', 'cf', CONFIGURATION => {'hbase.store.file-tracker.impl' => 'FILE'}
put 'test', 'row', 'cf:q', 'value'{code}
Step2: Execute the count command in the old version
{code:java}
count 'test'

1 row(s)
Took 0.5582 seconds
=> 1 {code}
Step3: Stop the entire cluster, upgrade to 2.6.1 and execute the same count command
{code:java}
count 'test'

0 row(s)
Took 0.0842 seconds
=> 0{code}
h1. Analysis

The names of {{StoreFileListFile}} differ between the two versions:
 * In 2.5.10: {{f1}} or {{f2}}
 * In 2.6.1: {{{}f1.{timestamp{}}}} or {{{}f2.{timestamp{}}}}

The addition of a timestamp to the file name was introduced in HBASE-26791.

It can be resolved by handling missing timestamps by setting them to 0.

However, in the case of downgrading versions, the above approach does not work. A workaround is available:
 * Change the storefile tracker implementation from {{FILE}} > {{MIGRATION > DEFAULT}} before downgrading.
Bug: Static resources are not loaded on REST web UI pages in dev mode
Status: Patch Available | Priority: Major | Resolution: 
Created: 20/Nov/24 11:38
Affects Version/s: 2.5.11

The HBase REST server component has a web UI (exposes an info servlet on the
hbase.rest.info.port, default: 8085).
 
While working on HBASE-28832 noticed that with the git cloned project, started REST server, on these  REST web UI pages the static resources (images, CSS, JS files) are not loaded (server responds with HTTP 404 Not Found). 
 
!image-2024-11-20-12-38-16-447.png|width=453,height=323!

 

In the downloaded HBase 2.5.9 distribution (production release) this was still working:

!screenshot-1.png|width=543,height=228!
Bug: Enhancing WALPlayer for Continuous Backup Capabilities
Status: In Progress | Priority: Major | Resolution: 
Created: 21/Nov/24 12:26
Affects Version/s: 2.6.0

*Add Support for Bulk-Loaded Files in WALPlayer*
Currently, WALPlayer lacks support for bulk-loaded files. We need to add functionality to handle bulk-loaded files both during the batching of WALs to generate HFiles and while replaying the WALs
Bug: Developing a Custom ReplicationEndpoint to Support External Storage Integration
Status: In Progress | Priority: Major | Resolution: 
Created: 21/Nov/24 12:12
Affects Version/s: 2.6.0

*Develop a Custom Replication Endpoint*
Implement a custom replication endpoint to support the backup of WALs to external storage systems, such as HDFS-compliant storages (including HDFS, S3, ADLS, and GCS via respective Hadoop connectors).

*Support for Bulk-loaded Files*
Add functionality to back up bulk-loaded files in addition to regular WALs.

*Ensure Process Durability*
Ensure the backup process is durable, with no WALs being missed, even in the event of issues in the cluster.
Bug: Replace some deprecated calls
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Dec/24 07:45
Affects Version/s: 4.0.0-alpha-1

While working on https://issues.apache.org/jira/browse/HBASE-28832 I noticed that there are several calls to deprecated methods.
 * {{java.net.URLEncoder.encode()}} - Deprecated. The resulting string may vary depending on the platform's default encoding. Instead, use the encode(String,String) method to specify the encoding.
 * java.net.URLEncoder.encode(String) -> java.net.URLEncoder.encode(String, Charset)
 * org.apache.hadoop.util.StringUtils.humanReadableInt(long) -> org.apache.hadoop.util.StringUtils.TraditionalBinaryPrefix.long2String(long, "", 1)
 * org.apache.hadoop.fs.FileSystem.getLength(Path) -> getFileStatus(Path).getLen()
 * org.apache.hadoop.hbase.ServerName.getStartcode() -> org.apache.hadoop.hbase.ServerName.getStartCode()

These will not be replaced as we'd need to add commons-text dependency which was decided no to do (see PR):
 * {{org.apache.commons.lang3.StringEscapeUtils.escapeHtml4()}} - Deprecated As of 3.6, use Apache Commons Text [StringEscapeUtils|https://commons.apache.org/proper/commons-text/javadocs/api-release/org/apache/commons/text/StringEscapeUtils.html] instead
 * {{org.apache.commons.lang3.StringEscapeUtils.escapeXml()}} - Deprecated As of 3.6, use Apache Commons Text [StringEscapeUtils|https://commons.apache.org/proper/commons-text/javadocs/api-release/org/apache/commons/text/StringEscapeUtils.html] instead
Bug: Soft drop for destructive table actions
Status: Open | Priority: Major | Resolution: 
Created: 16/Oct/24 18:34

When we administratively drop a table column or entire table, or truncate a table, the process begins rapidly. Procedures are scheduled for immediate execution that then modify or remove descriptors and state in META and on disk, and take unrecoverable actions at the HDFS layer. Although HFiles are copied to the archive in a destructive action, recovery scenarios are not automatic and involve some operator labor to reconstruct the table and re-import the archived data. If the HFileCleaner is not properly configured to facilitate such recovery then some data is not recoverable soon after procedure execution commences and all affected data is not recoverable within minutes. A customer faced with such an accident will be unhappy because the recovery scenarios available to them from this will involve either a restore from backup or from an earlier snapshot, and any changes committed more recently than the time of the last backup or last snapshot will be lost. 

An effective solution is very simple: We can easily prevent the deletion of the HFiles of a deleted table or table column family by taking a snapshot of the table immediately prior to taking any destructive actions. We set a TTL on the snapshot so housekeeping of truly unwanted HFiles remains no touch. Because we take a table snapshot all table structure and metadata is also captured and saved so fast recovery is possible, as either a restore from snapshot, or a clone from snapshot to a new table. For as long as the snapshot is retained it is straightforward to recover the table data by either restoring the table from the snapshot or cloning the snapshot to a new table, at the operator’s discretion. 

No manual actions are required to see the table or column family (or families) truly dropped. Once the snapshot TTL expires all the HFiles related to the dropped table become eligible for deletion. When the HFileCleaner chore executes after that time the HDFS level file deletes will commence with associated reduction in storage requirements. 

Design document is attached. 

I have a *working implementation* of this proposal based on a fork of branch-2.5.
Bug: RSMobFileCleanerChore may close the StoreFileReader object which is being used by Compaction thread
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Nov/24 12:33
Affects Version/s: 2.5.10

For MOB table, RSMobFileCleanerChore is responsible for cleaning MOB files that are no longer referebced by the region located in the current RegionServer.
RSMobFileCleanerChore get the  the information of MOB files by reading the storefile, as fallow.
```java
// RSMobFileCleanerChore.chore()
sf.initReader();
byte[] mobRefData = sf.getMetadataValue(HStoreFile.MOB_FILE_REFS);
byte[] bulkloadMarkerData = sf.getMetadataValue(HStoreFile.BULKLOAD_TASK_KEY);
// close store file to avoid memory leaks
sf.closeStoreFile(true);
```

There is an issue in here, if the StoreFileReader was not created by RSMobFileCleanerChore, but RSMobFileCleanerChore closed it, which will cause the thread that created the object to be unusable, resuting ERROR finally.

 

Reproduction:

This is an occasional problem, but the probability of its occurrence can be increased by making the following modifications.
1. Setting hbase.master.mob.cleaner.period from 24h to 10s, and restart hbase.
2. Puting some mob data into a MOB table.
3. At the same time, executing compaction command for the MOB table, and it is possible that this problem may occur.

 

The error logs as follow.

ERROR: java.io.IOException: Cannot invoke "org.apache.hadoop.hbase.regionserver.StoreFileReader.getMaxTimestamp()" because the return value of "org.apache.hadoop.hbase.regionserver.HStoreFile.getReader()" is null
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:512)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
        at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
        at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Caused by: java.lang.NullPointerException: Cannot invoke "org.apache.hadoop.hbase.regionserver.StoreFileReader.getMaxTimestamp()" because the return value of "org.apache.hadoop.hbase.regionserver.HStoreFile.getReader()" is null
        at org.apache.hadoop.hbase.regionserver.DefaultStoreFileManager.lambda$getUnneededFiles$3(DefaultStoreFileManager.java:235)
Bug: Introduce "pitr" Command for Point-in-Time Recovery/Restore
Status: In Progress | Priority: Major | Resolution: 
Created: 21/Nov/24 12:39
Affects Version/s: 2.6.0

The PITR command restores table data to a specific point in time using a combination of full backups, incremental backups, and WALs. If no end time is specified, it defaults to the current time.

{*}Command{*}:
{code:java}
$ hbase pitr [-t <table_name[,table_name]>] 
[-s <backup_set_name>] 
[-q <name>] 
[-c] 
[-m <target_tables>] 
[-o]
[--to-datetime <end_time>] {code}
{*}Options{*}:
 * -to-datetime <end_time>: _(Optional)_ Specifies the end time for restoration. Defaults to the current time if not provided.

 

{*}Description{*}:
 * For each specified table:
 * Locate the most recent full backup taken before the -to-datetime.
 * Apply the full backup, followed by all incremental backups up to the specified point.
 * Replay the WALs to bring the data to the exact state as of -to-datetime.
Bug: POC: Ensure Continuous Backup Durability with No Missed WALs, Even During Cluster Issues
Status: Resolved | Priority: Major | Resolution: Implemented
Created: 04/Dec/24 05:19

This task focuses on ensuring the durability of the continuous backup process for WALs, even during cluster issues, as part of implementing a custom replication endpoint for backing up WALs and bulk-loaded files to external storage systems.
Bug: Abort RS if unable to sync internal markers.
Status: Open | Priority: Major | Resolution: 
Created: 22/Oct/24 21:30
Affects Version/s: 2.5.8

RS kept on running even if it was unable to write replication marker for 5 minutes. But this issue is not specific to just replication marker. It applies to compaction marker as well as region event marker (like open, close).
Sample exception trace:
{noformat}
2024-10-09 10:12:21,659 ERROR [regionserver/regionserver-33:60020.Chore.3] regionserver.ReplicationMarkerChore - Exception whil
e sync'ing replication tracker edit
org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=15030132, WAL system stuck?
        at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:171)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:876)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:802)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.doSync(FSHLog.java:836)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.lambda$sync$3(AbstractFSWAL.java:602)
        at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.sync(AbstractFSWAL.java:602)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.sync(AbstractFSWAL.java:592)
        at org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullMarkerAppendTransaction(WALUtil.java:169)
        at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:146)
        at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeReplicationMarkerAndSync(WALUtil.java:230)
        at org.apache.hadoop.hbase.replication.regionserver.ReplicationMarkerChore.chore(ReplicationMarkerChore.java:99)
        at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:161)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:107)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
{noformat}
In this case there was namenode crash and all the hdfs clients on regionservers didn't failover to new active namenode due to our inefficient failover configuration parameters.