av9ash/GitBugs huggingface dataset:
Bug: Hbase master is not up due to NotServingRegionException: hbase:meta,,1 is not online
Status: Open | Priority: Major | Resolution: 
Created: 24/Nov/22 14:17
Affects Version/s: 2.4.13

Hbase master is in initializing state and never become ready.

it is not up due to following error

"NotServingRegionException: hbase:meta,,1 is not online on hbase-regionserver-0.hbase-regionserver.default.svc.cluster.local, ..."

when we observe this:

when region server split into multiple region and there is restart of hbase regionserver and master.
{code:java}
INFO  [main] util.HBaseFsck: Validating mapping using HDFS state
Number of live region servers: 1
Number of dead region servers: 2
Master: hbase-master-0.hbase-master.default.svc.cluster.local,....
Number of backup masters: 0
Average load: 0.0
Number of requests: 0 
Number of regions: 0
Number of regions in transition: 0
INFO  [main] client.RpcRetryingCallerImpl: Call exception, tries=6, retries=16, started=4333 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
    at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2830)
    at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1075)
    at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:384)
    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:371)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:351)
, details=, see https://s.apache.org/timeout {code}
workaround:

to make it up, we are deleting /hbase/meta-regionserver using "hbase zkcli delete   /hbase/meta-regionserver"

 

is there any way to prevent this to occur by setting some properties in hbase-site.xml
Bug: hbase client metrics per user specified identity on hconnections.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Nov/22 23:07

At present, hbase client metrics is per each individual hconnection and with a pre-configured scope in conf or cluster ID plus connection ID which has generated hash code as suffix. If a client has more than one connections and the scope configuration is common among all connections, the metrics might override each other.

I am proposing connections to share a common metrics object if the connections have a common configured scope.

If a connection identity is not provided, client metrics will continue working per hconnection and has a connection ID as it scope. I.e. no behavior change.
Bug: Finish the 2.5.2 release
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Dec/22 14:33

# Move release binaries from dist-dev to dist-release
# Publish maven artifacts on repository.a.o
# Tag rel/2.5.2 and push it
# Mark 2.5.2 as release on jira
# Send anouncement email
Bug: Lock contention when doing multiple parallel preads due to StoreFileReader reuse
Status: Open | Priority: Major | Resolution: 
Created: 29/Nov/22 14:58

In HStoreFile, we reuse the StoreFileReader created during HStoreFile initialization when creating a StoreFileScanner for preads:

[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStoreFile.java#L545]

When using S3 as hbase storage, we noticed this caused lock contention when multiple clients were doing preads in parallel:
{noformat}
...
"RpcServer.default.FPBQ.Fifo.handler=38,queue=8,port=16020" #125 daemon prio=5 os_prio=0 tid=0x00007fc11d83c000 nid=0x73f2 waiting for monitor entry [0x00007fc1154e6000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:73)
        - waiting to lock <0x00007fc4c2769660> (a org.apache.hadoop.fs.s3a.S3AInputStream)
...
"RpcServer.default.FPBQ.Fifo.handler=37,queue=7,port=16020" #124 daemon prio=5 os_prio=0 tid=0x00007fc11d83a000 nid=0x73f1 waiting for monitor entry [0x00007fc1155e7000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:73)
        - waiting to lock <0x00007fc4c2769660> (a org.apache.hadoop.fs.s3a.S3AInputStream)
...
"RpcServer.default.FPBQ.Fifo.handler=36,queue=6,port=16020" #123 daemon prio=5 os_prio=0 tid=0x00007fc11d838000 nid=0x73f0 runnable [0x00007fc1156e8000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
...
at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:216)
        - locked <0x00007fc4c2769660> (a org.apache.hadoop.fs.s3a.S3AInputStream)
... {noformat}
 We should create a new instance of StoreFileReader for each StoreFileScanner when doing preads, instead,
Bug: Improve HFileLinkCleaner to validate back reference links ahead the next traverse 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Nov/22 17:43
Affects Version/s: 2.5.2

We found a a race in the CleanerChore related to back reference links. When the HFileLinkCleaner runs for a file it can make 2 decisions depending on the file types.
 - HFiles, The cleaner for HFile deletion only checks if the .links-<> directory is present with files. 
 - Back reference links, the cleaner checks if the forward link is still available in the data directory.

The logic and order how the cleaner checks these 2 files matters. When the back reference is checked first it can remove both the reference and the HFile from the archive, however, when it first runs for the HFile then only the back-reference is removed. In this case, the HFile is only deleted in the next iteration of the CleanerChore, and it could be very slow if the list of files are huge in case of using object store.

The goal of this task is to improve traverse of the archived HFile, reusing the list of found back reference files, and immediately apply the checks for the Back reference links.
Bug: Upgrade hbase-thirdparty to 4.1.3 and upgrade Jackson for CVE-2022-42003/42004
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Oct/22 00:35

Jackson 2.13.4 fixes CVE-2022-42003 and databind 2.14.0-rc1 fixes CVE-2022-42004. 

Move jackson.version to 2.13.4.
Move jackson.databind.version to 2.14.0-rc1.
Bug: Recovered replication may be blocked if enabled hbase.separate.oldlogdir.by.regionserver
Status: Open | Priority: Major | Resolution: 
Created: 09/Nov/22 07:11
Affects Version/s: 2.4.15

In other PR, I got a failed UT
{code:java}
[ERROR] Failures: 
[ERROR] org.apache.hadoop.hbase.replication.TestReplicationKillMasterRSWithSeparateOldWALs.killOneMasterRS
[ERROR]   Run 1: TestReplicationKillMasterRSWithSeparateOldWALs>TestReplicationKillMasterRS.killOneMasterRS:47->TestReplicationKillRS.loadTableAndKillRS:84 Waited too much time for queueFailover replication. Waited 61065ms.
[ERROR]   Run 2: TestReplicationKillMasterRSWithSeparateOldWALs>TestReplicationKillMasterRS.killOneMasterRS:47->TestReplicationKillRS.loadTableAndKillRS:84 Waited too much time for queueFailover replication. Waited 58864ms.
[ERROR]   Run 3: TestReplicationKillMasterRSWithSeparateOldWALs>TestReplicationKillMasterRS.killOneMasterRS:47->TestReplicationKillRS.loadTableAndKillRS:84 Waited too much time for queueFailover replication. Waited 57103ms. {code}
This should be caused by a bug.

If enabled {_}hbase.separate.oldlogdir.by.regionserver{_}, old wals will be moved into different dir by regionserver name like root/oldWALs/server1/wal1 . For recovered replication,  can't convert wal path(like root/oldWALs/wal1) into such paths, and throws FileNotFoundException.
Bug: Tags cannot be written to the hfile during bulkload
Status: Open | Priority: Major | Resolution: 
Created: 01/Nov/22 08:08

When I store hdfs file data into hbase through bulkload, I want to give different data different ttl, so I use tags in mapper to create per cell ttl for each cell, but I found that this ttl does not take effect.

I have the bulkload driver

 
{code:java}
job.setInputFormatClass(TextInputFormat.class);
TextInputFormat.setInputPaths(job, inputFile);
job.setOutputFormatClass(HFileOutputFormat2.class);
HFileOutputFormat2.setOutputPath(job, outputPath);
HFileOutputFormat2.configureIncrementalLoad(job, table, conn.getRegionLocator(TableName.valueOf(tableName)));
job.waitForCompletion(true);
LoadIncrementalHFiles loader = new LoadIncrementalHFiles(conf);
loader.doBulkLoad(outputPath, admin, table, conn.getRegionLocator(TableName.valueOf(tableName))); {code}
 

and the mapper

 
{code:java}
public class BulkloadTTLTestMapper extends Mapper<LongWritable, Text, ImmutableBytesWritable, Put> {
  public void map(LongWritable key, Text value, Mapper.Context context) throws IOException, InterruptedException {
    String values = value.toString();
    String[] lines = values.split(",");
    byte[] rowkey = Bytes.toBytes(lines[0]);
    ImmutableBytesWritable putRowKey = new ImmutableBytesWritable(rowkey);
    Put put = new Put(rowkey);
    Cell cell1 = new KeyValue(Bytes.toBytes(lines[0]), Bytes.toBytes("d"), Bytes.toBytes("field1"), HConstants.LATEST_TIMESTAMP, Bytes.toBytes(lines[1]),new Tag[]{new Tag(TagType.TTL_TAG_TYPE, Bytes.toBytes(100000L))});
    put.add(cell1);
    Cell cell2 = new KeyValue(Bytes.toBytes(lines[0]), Bytes.toBytes("d"), Bytes.toBytes("field2"), HConstants.LATEST_TIMESTAMP, generateValue(lines[2]),new Tag[]{new Tag(TagType.TTL_TAG_TYPE, Bytes.toBytes(200000L))});
    put.add(cell2);
    context.write(putRowKey, put);
  }
} {code}
 

I checked the execution steps through debug, and I found that the tags were not written into hfile. By reading the source code, I found that in org.apache.hadoop.hbase.protobuf.ProtobufUtil.toMutation()

 
{code:java}
if (cell.getTagsLength() > 0) {
   valueBuilder.setTags(ByteStringer.wrap(cell.getTagsArray(),
       cell.getTagsOffset(), cell.getTagsLength()));
}{code}
 

This code was deleted at [issue15198|https://issues.apache.org/jira/browse/HBASE-15198], which made it impossible for me to write tags into hfile.

If i just add 
{code:java}
conf.set("hbase.client.rpc.codec", "org.apache.hadoop.hbase.codec.CellCodecWithTags"){code}
It is Useless, because bulkload does not use these Codecs.

Only when I add back those two lines of code that were deleted at issue15198, I could write tags into hfile and make per cell ttl take effect.

So I want to know is there any other way to help me carry tags in bulkload besides adding these two lines of code back.

Thanks.
Bug: Add an admin method to get replication enabled state
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Oct/22 08:55

When implementing HBASE-27109, it is a bit surprise to me that there is no way for me to get the enabled state of a replication peer directly through Admin interface, I need to call listReplicationPeers to find out the ReplicationPeerDescription and then get the enabled state.

I think we should add a method in Admin interface for this.
Bug: Update all our python scripts to use python3
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Oct/22 11:49

./dev-support/create-release/prepend_changes.py
./dev-support/create-release/prepend_releasenotes.py
./dev-support/submit-patch.py
./dev-support/checkstyle_report.py
./dev-support/checkcompatibility.py
./dev-support/git-jira-release-audit/git_jira_release_audit.py
./dev-support/flaky-tests/findHangingTests.py
./dev-support/flaky-tests/report-flakies.py
./dev-support/gen_redirect_html.py
Bug: Spotbugs 4.7.2 report a lot of logging errors when generating report
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Oct/22 07:20

Like this

{noformat}
Unexpected problem occured during version sanity check
11:27:32  Reported exception:
11:27:32  java.lang.AbstractMethodError: org.apache.logging.slf4j.SLF4JServiceProvider.getRequestedApiVersion()Ljava/lang/String;
11:27:32  	at org.slf4j.LoggerFactory.versionSanityCheck(LoggerFactory.java:297)
11:27:32  	at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:141)
11:27:32  	at org.slf4j.LoggerFactory.getProvider(LoggerFactory.java:421)
11:27:32  	at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:407)
11:27:32  	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:356)
11:27:32  	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:382)
11:27:32  	at edu.umd.cs.findbugs.PluginLoader.<clinit>(PluginLoader.java:113)
11:27:32  	at edu.umd.cs.findbugs.DetectorFactoryCollection.getCoreResource(DetectorFactoryCollection.java:353)
11:27:32  	at edu.umd.cs.findbugs.SystemProperties.loadPropertiesFromConfigFile(SystemProperties.java:84)
11:27:32  	at edu.umd.cs.findbugs.SystemProperties.<clinit>(SystemProperties.java:67)
11:27:32  	at edu.umd.cs.findbugs.DetectorFactoryCollection.<clinit>(DetectorFactoryCollection.java:57)
11:27:32  	at edu.umd.cs.findbugs.workflow.Filter.main(Filter.java:693)
{noformat}

Checked the dependencies, it uses slf4j-api-2.0.0 and log4j2 2.18.0 together, which may be caused by LOG4J2-3139.

In spotbugs 4.7.3 they bumped log4j2 to 2.19.0, which should have fixed the problem.

https://github.com/spotbugs/spotbugs/issues/2176

Let's also do an upgrade.
Bug: metrics method removeHistogramMetrics trigger serious memory leak
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Oct/22 02:42
Affects Version/s: 2.4.15

the method removeHistogramMetrics will cause the metricsMap leak!
because the put op in the map is the baseName but the remove is not , the origin code add the histogramSuffixe follow the baseName when remove.

!https://user-images.githubusercontent.com/13758677/196915944-71197086-2d53-4a85-8b5e-0a61441e8f64.png!

!https://user-images.githubusercontent.com/13758677/196916259-196e3e8a-cde0-4989-8b64-de13b860522d.png!
Bug: Use jdk 11.0.16 instead of 11.0.16.1 for branch-2.4
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Oct/22 13:37

In 2.4.x we still support hadoop 3.2.0, which is still on jetty 9.3.x, which can not parse jva version like 11.0.16.1

https://github.com/eclipse/jetty.project/issues/2090

This causes bunch of UT failures in nightly job for branch-2.4 and finally cause almost all UTs to be in flaky list.

Usually for a mature minor release line, we will not bump its hadoop version, so let's downgrade the jdk 11 version to 11.0.16 to fix the nightly and flaky jobs.
Bug: DefaultMobStoreCompactor should delete MobStoreFile cleanly when compaction is failed
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Oct/22 03:46
Affects Version/s: 3.0.0-alpha-3

When compaction is failed, {{DefaultMobStoreCompactor.performCompaction}}  deletes the {{MobStoreFile}} which is writing, but for {{OptimizedMode}}, {{DefaultMobStoreCompactor.performCompaction}}  may create multiple MobStoreFiles, if compaction is failed, all theses MobStoreFiles are useless and should be deleted altogether. We should better not depend on {{RSMobFileCleanerChore}} to clean them because it is mainly used for archive already compacted MobStoreFiles.
Bug: metrics method removeHistogramMetrics trigger serious memory leak
Status: Resolved | Priority: Major | Resolution: Abandoned
Created: 21/Oct/22 02:23
Affects Version/s: 2.4.14

!https://user-images.githubusercontent.com/13758677/196915944-71197086-2d53-4a85-8b5e-0a61441e8f64.png!

the method removeHistogramMetrics will cause the metricsMap leak!
because the put op in the map is the baseName but the remove is not , the origin code add the histogramSuffixe follow the baseName when remove.

 

!https://user-images.githubusercontent.com/13758677/196916259-196e3e8a-cde0-4989-8b64-de13b860522d.png!
Bug: Make "/prometheus" endpoint accessible from HBase UI
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Oct/22 08:41
Affects Version/s: 2.6.0

Prometheus metrics feature were added in HBASE-20904, but it's not in the UI yet, so the only option to use it is the URL.

I would like to make a change in the web UI, so it can be accessed via a dropdown menu there and I would like to include the description option, so metrics can be viewed with or without description.
Bug: Deprecate getStats and setStatistics in Result
Status: Open | Priority: Minor | Resolution: 
Created: 02/Dec/22 21:55

setStatistics is already IA.Private, but it has never been called. It replaced a deprecated addResults method, which was removed in HBASE-14703. 

The getter getStats() now always returns a null value, since nothing in the codebase ever calls setStatistics.

We should deprecate these methods to keep our API clear and concise.
Bug: Client meta cache clear by exception metrics are missing some cases
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Nov/22 16:23
Affects Version/s: 2.4.15

MetricsConnection has metrics for meta cache clears by server and region, and also metrics by exception type which triggered the clear. The metric by exception type is currently missing at least one instance (in AsyncRequestFutureImpl) where the cache is cleared due to an exception.

We should do an audit of cache clear calls and ensure all the appropriate ones are tracked by exception too.
Bug: Expose table and region storefiles accessed days and size to the metrics
Status: Open | Priority: Minor | Resolution: 
Created: 12/Nov/22 17:24
Affects Version/s: 3.0.0-alpha-4

At present, we have no metrics to show the hot and cold status of region/table storefiles, but from the perspective of storage cost, users are very concerned about the hot and cold status of their table data.
So, it's necessary to expose region/table storefiles accessed days and size to the metrics.
Bug: Remove protobuf 2 dependencies
Status: Open | Priority: Minor | Resolution: 
Created: 19/Oct/22 22:01
Affects Version/s: 2.4.16

Protobuf 2 is subject to CVE-2015-5237, CVE-2021-22569, and CVE-2022-3171 and scanners checking compliance with zero tolerance CVE policies will hit on HBase because a dependency on com.google.protobuf:protobuf-java:2.5.0 is still exported by 2.x versions. 

It is impossible to remove that dependency by upgrade to Protobuf 3 everywhere given our compatibility guidelines but nonetheless it is a problem for downstream consumers of HBase with such security policies. 

HBase 2 uses Protobuf 3 internally for RPC, in hbase-protocol-shaded.

We still depend on some protobuf-java v2.5 classes however because we use the Protobuf 2 compiler to compile the IDL in the backwards compatible hbase-protocol module. Perhaps we can extract and produce a scaffold from protobuf-java v2.5, just enough so the generated classes from the IDL will compile, and ship that scaffold included, so the dependency on protobuf-java 2.5 proper can be dropped. Protobuf is MIT licensed so partial appropriation of the source would be allowable.
Bug: Use ReadWriteLock for region scanner readpoint map 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Nov/22 04:44
Affects Version/s: 3.0.0-alpha-3

Currently we manage the concurrency between the RegionScanner and getSmallestReadPoint by synchronizing on the scannerReadPoints object. In our production, we find that many read threads are blocked by this when we have a heavy read load. 

we need to get smallest read point when 
a. flush a memstore 
b. compact memstore/storefile 
c. do delta operation like increment/append
Usually the frequency of these operations is much less than read requests. 

It's a little expensive to use an exclusive lock here because for region scanners, what it need to do is just calcaulating readpoint and putting the readpoint in the scanner readpoint map, which is thread-safe. Multiple read threads can do this in parallel without synchronization.

Based on the above consideration, maybe we can replace the synchronized lock with readwrite lock. It will help improve the read performance if the bottleneck is on the synchronization here.

!jstack.png!
Bug: Add BulkLoad bandwidth throttling
Status: In Progress | Priority: Minor | Resolution: 
Created: 08/Dec/22 09:21

#  Background:
During bulk load HFile copy, the server performance is affected to prevent the bandwidth of the server from being full. We can design a throttler to limit the speed of copy and thus limit the bandwidth used by copy
 #  Function:
 2.1 Dynamically Updating Traffic Limiting Configurations
 2.2 Node copy HFile Traffic Limiting
 2.3 Synchronous copy HFile of the Active and Standby Clusters is unlimited
 #  Mode of Use:
hbase.regionserver.bulkload.node.bandwidth is greater than 0 open unit Byte/sec
Update the limit value with update_all_config or update_config 'node'
 #  Effect:

// Begin 5M/sec
<property>
  <name>hbase.regionserver.bulkload.node.bandwidth</name>
  <value>5242880</value>
  <source>hbase-site.xml</source>
</property>

// After update_all_config to 10M/sec
<property>
  <name>hbase.regionserver.bulkload.node.bandwidth</name>
  <value>5242880</value>
  <source>hbase-site.xml</source>
</property>

!image-2023-02-02-13-10-06-619.png!
Bug: Remove unneeded distcp log cleanup after incremental backups
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Dec/22 01:07
Affects Version/s: 3.0.0-alpha-3

During the completion step of incremental backups, the [TableBackupClient|https://github.com/apache/hbase/blob/2c3abae18aa35e2693b64b143316817d4569d0c3/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/TableBackupClient.java#L343-L355] ensures distcp logs are cleaned up. However, [DistCp|https://github.com/apache/hadoop/blob/b87c0ea7ebde3edc312dcc8938809610a914df7f/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java#L465-L476] already ensures that the metafolder, where the distcp logs are stored, is cleaned up via the [CopyCommitter|https://github.com/apache/hadoop/blob/b743d56eb4bf350448bd315540fde4f029175082/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java#L189].

Additionally, when running in an external yarn cluster external to hbase, this causes issues as it assumes a common filesystem.

The TableBackupClient cleanup method should be able to be safely removed.

e: After digging in to this even more, the `_distcp_log` path used in the `TableBackupClient` is not actually the correct path used by distcp.
Bug: Fix several problems after HBASE-27304
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Nov/22 15:11

In the HBASE-27304, we support using IP to expose master/rs service to client.  But there are several problems that need to be fixed, see #[4732|https://github.com/apache/hbase/pull/4732] for details. This issue is used to track and fix these problems.