av9ash/GitBugs huggingface dataset:
Bug: BucketCache statistics should not count evictions by hfile
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jul/22 00:09

The eviction metrics are helpful for determining how much access-related churn there is in your block cache. The LruBlockCache properly ignores evictions by invalidation of HFile, but the BucketCache tracks them. 

We should make the BucketCache work similarly so that one can get an accurate view of the evictions occurring through too much random access or too large working set.
Bug: Backport HBASE-22120 (Replace HTrace with OpenTelemetry) to branch-2.4
Status: Open | Priority: Major | Resolution: 
Created: 28/Jul/22 02:00
Affects Version/s: 2.4.0

list of commits from upstream/master need to be backported

{code:java}
HBASE-25373 https://github.com/apache/hbase/commit/302d9ea8b888762a5a20a5ba5c2be7bc239afaef 
HBASE-25401 https://github.com/apache/hbase/commit/242028671535dfed67ec13d9ed0d6067f3ccfd04 
HBASE-25424 https://github.com/apache/hbase/commit/57960fa8fa7228d65b1a4adc8e9b5b1a8158824d 
HBASE-23898 https://github.com/apache/hbase/commit/805b2ae2ad0f6325515d46043ff01e4e2c7a9f59 
HBASE-25454 https://github.com/apache/hbase/commit/dcb78bd4bda4a4ae13d863df8aec266031e5bc93
HBASE-25481 https://github.com/apache/hbase/commit/ae2c62ffaad5ba4c976b0a79c10a365edf2844fd
HBASE-25455 https://github.com/apache/hbase/commit/03e12bfa4ad62ecc6eee6a2c68d431bea2d5c473
HBASE-25484 https://github.com/apache/hbase/commit/2be2c63f0d3917a243b74af9754cbfc805b858d1
HBASE-25535 https://github.com/apache/hbase/commit/bb8c4967f8ce2c89ebaf1ddc5d8a1bf55f1e20d3
HBASE-25591 https://github.com/apache/hbase/commit/f6ff519dd0c7fe0e3ae3c175eefee27a26a065a4
HBASE-25617 https://github.com/apache/hbase/commit/8d68f8cd1c8613be1b499eaa99f46806b2743294
HBASE-25616 https://github.com/apache/hbase/commit/8399293e21127df3ffdcb757242e4cb5964c7e99
HBASE-25723 https://github.com/apache/hbase/commit/7f90c2201f6a17d2e2d031505c35ae7c2b1ed7ea
HBASE-25732 https://github.com/apache/hbase/commit/8df9bebdd367d52a32b08c18a7cf4f9c2d712071
HBASE-25733 https://github.com/apache/hbase/commit/b71488998970a3353086a34736ed1edab527f673
HBASE-23762 https://github.com/apache/hbase/commit/be4503d9f82f044fbfce21c8a42d0b1684607238
HBASE-25778 https://github.com/apache/hbase/commit/f36e1539648bbaee84c626fd54d1605baebf3c5a
{code}
Bug: Purge FutureReturnValueIgnored warnings from error prone
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jul/22 05:26

We used to introduce a FutureUtils to purge the FutureReturnValueIgnored warnings in the async client implementation, to make sure we catch the exception from the callback.

For netty, we could apply the same trick. And there is another problem for netty that channel.write and some other methods will also return a ChannelFuture but usually we just do nothing on it, this will generated a FutureReturnValueIgnored warning too. We could add a logging listener to log the errors if any.
Bug: Fix checking for encoded block size when deciding if block should be closed
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jul/22 12:35
Affects Version/s: 2.4.13

On HFileWriterImpl.checkBlockBoundary, we useed to consider the unencoded and uncompressed data size when deciding to close a block and start a new one. That could lead to varying "on-disk" block sizes, depending on the encoding efficiency for the cells in each block.

HBASE-17757 introduced the hbase.writer.unified.encoded.blocksize.ratio property, as ration of the original configured block size, to be compared against the encoded size. This was an attempt to ensure homogeneous block sizes. However, the check introduced by HBASE-17757 also considers the unencoded size, which in the cases where encoding efficiency is higher than what's configured in hbase.writer.unified.encoded.blocksize.ratio, it would still lead to varying block sizes.

This patch changes that logic, to only consider encoded size if hbase.writer.unified.encoded.blocksize.ratio property is set, otherwise, it will consider the unencoded size. This gives a finer control over the on-disk block sizes and the overall number of blocks when encoding is in use.
Bug: Avoid data inconsistent between primary and secondary replicas for the new region replication framework
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 19/Jul/22 09:54
Affects Version/s: 3.0.0-alpha-3

For the new region replication framework introduced by HBASE-26233, when we write the {{Mutation}} on primary region replica , if {{HRegion.doMiniBatchMutate}}  throws a exception, for example ,{{WAL.sync}} throws a {{TimeoutIOException}} because sync timeout, the writing on primary region replica is failed and we could not query the {{Mutation}} on the primary region replica because of {{Mutation}} is not written to the {{MemStore}}, but the replication to the second replica may be successful because we always execute {{MultiVersionConcurrencyControl.WriteEntry.runCompletionAction}} in {{MultiVersionConcurrencyControl.complete}} , and we could get the {{Mutation}} on the secondary region replica, this inconsistency should be  avoid even if it is short. 
But for {{WALUtil.writeMarker}}, we should always replicate to the second replica even though {{WAL.sync}} is failed, because we just want to sync region state to secondary replica.
Bug: Clean up netty worker/thread pool configuration
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jul/22 18:45
Affects Version/s: 2.5.0

Refer to https://issues.apache.org/jira/browse/HBASE-27112?focusedCommentId=17565952&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17565952
{quote}
"server instanceof HRegionServer" will be true for both HMaster and HRegionServer classes, hence it seems we are only using "hbase.netty.worker.count" essentially during both master and regionservers Netty server initialization. "hbase.netty.eventloop.rpcserver.thread.count" is used for bunch of other implementations of Server interface (used in UTs only)
{quote}
Bug: When Hbase uses GET to query data, some rowkeys cannot be queried，ERROR: java.io.TOException: java.lang.NoclassDefFoundError: org/apache/hadoop/hdfs/DFSInputStreamS2
Status: Closed | Priority: Major | Resolution: Not A Bug
Created: 08/Jul/22 09:00

!image-2022-07-08-16-59-52-808.png!
Bug: Add test coverage for SimpleRpcServer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jul/22 18:06

Add test coverage for SimpleRpcServer.

Improve the way we test both SimpleRpcServer and NettyRpcServer: Use LoadTestKVGenerator to generate random values with varying sizes between 1000 bytes and 1M bytes, and also to verify them when reading the values back.

Add secure test coverage for both SimpleRpcServer and NettyRpcServer.
Bug: Rework tracing configuration
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jul/22 12:29
Affects Version/s: 2.5.0

I'd like to make some adjustments to how we manage tracing configuration in `conf/hbase-env.sh`. I'd like to take advantage of the fact that OpenTelemetry can read its configuration from environment variables and make use of this where possible, only falling back to passing properties into the process launch configuration when it's necessary. I think this will DRY up tracing configuration and make it easier to manage in a container environment.
Bug: Add some docs about exporting and importing snapshots using Aliyun object storage service(OSS)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jul/22 05:53
Affects Version/s: 3.0.0-alpha-3

[Aliyun Object Storage Service|https://www.aliyun.com/product/oss](Aliyun OSS) is widely used, particularly popular among China’s cloud users, and it provides cloud object storage for a variety of use cases. [Hadoop file system|http://hadoop.apache.org/docs/current/hadoop-aliyun/tools/hadoop-aliyun/index.html] supports OSS since version 2.9.1. Now, we can export and import snapshots using Aliyun object storage service(OSS).
Bug: Support regionserver to connect to HMaster proxy port
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 08/Jul/22 05:29

Regionservers get active master address from Zookeeper/Master registry and tries to make RPC calls to master.

For security concerns, regionservers might require making connection to a different proxy port of master rather than it's original port retrieved from Zookeeper.

Configs:
 # hbase.master.expose.proxy.port: Master can use this config (int) to expose new proxy port on active and backup master znodes.
 # hbase.client.consume.master.proxy.port: Clients/Regionservers can use this config (boolean) to determine whether to connect to active master on new proxy port that master has exposed or continue using original port of master for connection.
Bug: Support using IP to expose master/rs servers for some special scenarios
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Aug/22 16:02
Affects Version/s: 3.0.0-alpha-4

In our environment, when accessing the HBase cluster from another cluster (such as the Flink cluster), you cannot use the hostname to access the HBase cluster, but you can use IP. Because it is possible that the hostname of the HBase cluster automatic expansion node is not maintained in the Flink cluster. So we should support using IP to expose master/rs servers for this special scenarios.
Bug: RitDuration histogram metric is broken
Status: Open | Priority: Minor | Resolution: 
Created: 20/Sep/22 19:41
Affects Version/s: 2.5.0

Looks like the method which updates it in MetricsAssignmentManager, updateRitDuration, got broken somewhere along the way. It's no longer used. We should wire this back up.

https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetricsAssignmentManager.java#L82
Bug: Add timeout to JavaRegexEngine
Status: Open | Priority: Minor | Resolution: 
Created: 23/Aug/22 04:10

Java regular expression engine is based on backtracking. Thus, a malicious regular expression may result in a ReDoS.
When ReDoS occurs in the HBase, the region server's handler is occupied as a result it cannot process the user's request.
It would be nice to have a timeout for system protection.
The engine does not have timeout.
The charAt method is called during pattern matching, and the charAt method checks if it is timeout.
Bug: Separate meta read requests from master and client 
Status: Open | Priority: Minor | Resolution: 
Created: 02/Sep/22 14:31
Affects Version/s: 3.0.0-alpha-3

If we have a large number of store files in a single region or the response from hdfs is slow, the region transition can be slow, the client may put a lot of pressure on the meta server when retrying. This may block the master system read requests. Maybe we can set a special priority for the master request to isolate read requests from master and client.
Bug: MasterRpcService#setRegionStateInMeta does not support replica region encodedNames or region names
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Jul/22 17:25
Affects Version/s: 2.4.13

MasterRpcServices#setRegionStateInMeta does not support replica region names, it assumes the primary region only. This makes HBCK2's setRegionState for replica region fails.
Bug:  MetricsSource lastShippedTimeStamps ConcurrentModificationException cause RegionServer crash
Status: In Progress | Priority: Minor | Resolution: 
Created: 22/Sep/22 15:31

022-09-20 14:14:40,332 ERROR [regionserver/hostname1:16020] regionserver.HRegionServer: ***** ABORTING region server hostname1,16020,1663147531495: Unhandled: null *****
 8587 java.util.ConcurrentModificationException
 8588     at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442)
 8589     at java.util.HashMap$ValueIterator.next(HashMap.java:1471)
 8590     at org.apache.hadoop.hbase.replication.regionserver.MetricsSource.getTimestampOfLastShippedOp(MetricsSource.java:321)
 8591     at org.apache.hadoop.hbase.replication.regionserver.ReplicationLoad.buildReplicationLoad(ReplicationLoad.java:80)
 8592     at org.apache.hadoop.hbase.replication.regionserver.Replication.buildReplicationLoad(Replication.java:264)
 8593     at org.apache.hadoop.hbase.replication.regionserver.Replication.refreshAndGetReplicationLoad(Replication.java:253)
 8594     at org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1436)
 8595     at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1243)
 8596     at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1065)
Bug: BufferedMutator#mutate supports mutation type verification
Status: Open | Priority: Minor | Resolution: 
Created: 13/Sep/22 15:44
Affects Version/s: 3.0.0-alpha-4

At present, BufferedMutator#mutate method only tell the user in the comment that it only supports Put / Delete.  But it is not verified in the code implementation, so we need to deal with it.
Bug: Artifacts with resolved profiles
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Aug/22 23:46

Brainstorming/Discussion. The maven-flatten-plugin makes it so published poms are 'flattened'. The poms contain the runtime-necessary dependencies only, 'build' and 'test' dependencies and plugins are dropped, versions are resolved out of properties, and so on. The published poms are the barebones minimum needed to run.

With a switch, the plugin can also make it so the produced poms have all profiles 'resolved' – making it so the produced poms have all resolved hadoop2 or hadoop3 dependencies baked-in – based off which profile we used building.

(I've been interested in this flattening technique since I ran into a downstreamer using hbase from a gradle build. Gradle does not respect profiles. You can't specify that the gradle build pull in hbase with hadoop3 dependencies using 'profiles'. I notice too our [~gjacoby] , [~apurtell] et al. up on the dev list talking about making a hadoop3 set of artifacts...who might be interested in this direction).

The attached patch adds the flatten plugin so folks can take a look-see. It uncovers some locations where our versioning on dependencies is not explicit. The workaround practiced here was adding hadoop2/hadoop3 profiles into sub-modules that were missing them or moving problematic dependencies that were outside of profiles under profiles in sub-modules that had them already. For the latter, if the dependency specified excludes, the excludes were moved up to the parent pom profile (parent pom profiles have dependencyManagement sections... sub-modules have explicit dependency mentions... checks with dependency:tree seem to show excludes continue to be effective).

This is the switch that flattens profiles:           <embedBuildProfileDependencies>true</embedBuildProfileDependencies>

This is the sort of complaint we had when the flatten plugin was having trouble figure dependency versions – particularly hadoop versions

{{[ERROR] Failed to execute goal org.codehaus.mojo:flatten-maven-plugin:1.3.0:flatten (flatten) on project hbase-hadoop2-compat: 3 problems were encountered while building the effective model for org.apache.hbase:hbase-hadoop2-compat:2.5.1-SNAPSHOT}}

{{[ERROR] [WARNING] 'build.plugins.plugin.version' for org.codehaus.mojo:flatten-maven-plugin is missing. @}}

{{[ERROR] [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-mapreduce-client-core:jar is missing. @}}

{{[ERROR] [ERROR] 'dependencies.dependency.version' for javax.activation:javax.activation-api:jar is missing. @}}
Bug: Time based metrics will be reset after any get request
Status: Open | Priority: Minor | Resolution: 
Created: 23/Aug/22 08:15
Affects Version/s: 2.4.13

Jmx metrics can be query by http request.

But metrics will be reset after any get request.

The root cause may be the implement of Histogram's method "snapshot"
{code:java}
@Override
public Snapshot snapshot() {
  return histogram.snapshotAndReset();
} {code}
It will call snapshot and reset at the same time.

I think it should not be reset cause we may need history metrics.
Bug: Spark-hbase support for 'startWith' predicate.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Sep/22 23:10
Affects Version/s: 3.0.0-alpha-3

Currently, spark-hbase connector doesn't support stringStartWith predicate and completely ignores it. This is a disadvantage compared to the Apache Phoenix connector and the old SHC (Hortonworks spark-hbase connector). It would be nice if we also have this functionality in the actual connector.
Bug: numOpenConnections metric is one less than the actual 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Sep/22 02:02
Affects Version/s: 2.0.0

In HBASE-17263，serverChannel add to allChannels，so NumOpenConnections exclude serverChannel from the count.
{code:java}
public NettyRpcServer(final Server server, final String name,
    final List<BlockingServiceAndInterface> services,
    final InetSocketAddress bindAddress, Configuration conf,
    RpcScheduler scheduler) throws IOException {
  super(server, name, services, bindAddress, conf, scheduler);
  ...
  try {
    serverChannel = bootstrap.bind(this.bindAddress).sync().channel();
    LOG.info("NettyRpcServer bind to address=" + serverChannel.localAddress()
        + ", hbase.netty.rpc.server.worker.count=" + workerCount
        + ", useEpoll=" + useEpoll);
    allChannels.add(serverChannel);
  } catch (InterruptedException e) {
    throw new InterruptedIOException(e.getMessage());
  }
  ...
}

@Override
public int getNumOpenConnections() {
  // allChannels also contains the server channel, so exclude that from the count.
  return allChannels.size() - 1;
}{code}
In HBASE-17263，serverChannel not add to allChannels，so NumOpenConnections  is one less than the actual.
{code:java}
public NettyRpcServer(Server server, String name, List<BlockingServiceAndInterface> services,
    InetSocketAddress bindAddress, Configuration conf, RpcScheduler scheduler)
    throws IOException {
  ...
  try {
    serverChannel = bootstrap.bind(this.bindAddress).sync().channel();
    LOG.info("NettyRpcServer bind to address=" + serverChannel.localAddress());
  } catch (InterruptedException e) {
    throw new InterruptedIOException(e.getMessage());
  }
  ...
} {code}
In HBASE-21610, I think it's actually influenced by this bug.
Bug: Improve sasl connection failure log message to include server
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Aug/22 20:18
Affects Version/s: 1.7.2

We were seeing timeout issues in our deployment in the saslConnect path in the blockingRPC client.  Knowing which server we failed to connect to would help isolate and understand the issue.

 

 
{code:java}
ipc.BlockingRpcConnection - Exception encountered while connecting to the server : 
java.io.IOException: Connection reset by peer {code}
 

 

Change the log message to something like

 
{code:java}
ipc.BlockingRpcConnection - Exception encountered while connecting to the server myhost.realm.net: 
java.io.IOException: Connection reset by peer {code}
 

See relevant code [here|https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java#L398]
Bug: Remove dumping of EOFException while reading WAL with ProtobufLogReader
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 28/Sep/22 18:59

The log processing tooling that helps extract and analyze Exceptions from regionserver logs can read EOFException while reseting seeking to original position as part of ProtobufLogReader implementation.

Common logs:
{code:java}
2022-09-28 17:02:00,288 DEBUG [20%2C1664323516467,1] wal.ProtobufLogReader - Encountered a malformed edit, seeking back to last good position in file, from 187159 to 187158
java.io.EOFException: Partial PB while reading WAL, probably an unexpected EOF, ignoring. current offset=187159
	at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:390)
	at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:104)
	at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:92)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:258)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:172)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:101)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.tryAdvanceStreamAndCreateWALBatch(ReplicationSourceWALReader.java:251)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:148) {code}
{code:java}
2022-09-28 11:02:10,792 DEBUG [20%2C1664323193648,1] wal.ProtobufLogReader - Encountered a malformed edit, seeking back to last good position in file, from 112026775 to 112026303
java.io.EOFException: EOF  while reading 296 WAL KVs; started reading at 112026367 and read up to 112026775
	at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:418)
	at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:104)
	at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:92)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:258)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:172)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:101)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:222)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:157)
Caused by: java.io.EOFException: Only read 6
	at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:406)
	... 7 more {code}
After looking at these logs, it seems that having EOFException even at DEBUG level is not helping much because reseting seek to a different position is going to be expected often. We should remove dumping EOFException in these cases.
Bug: Quoted string argument with spaces passed from command line are propagated wrongly to the underlying java class
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 31/Aug/22 17:26
Affects Version/s: 2.5.0

Running the following command : 
/hbase/bin/hbase org.apache.hadoop.hbase.mapreduce.Import -Dmapreduce.map.java.opts="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$HADOOP_HOME/logs/@taskid@_map.hprof" SYSTEM.CATALOG /tmp
fails with error. It is due to the fact that [read -r -a CMD_ARGS <<< "$@" |https://github.com/apache/hbase/blob/master/bin/hbase#L878] in the hbase script would split the command line args (i.e. "$@"). Instead a straightforward array copy can be used. 

 !Screenshot 2022-08-31 at 10.52.11 PM.png!