av9ash/GitBugs huggingface dataset:
Bug: NettyHBaseSaslRpcServerHandler.channelRead0 forget to record "AUTH_FAILED_FOR" auditlog for an exception.
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 09/Dec/22 13:29

In other methods such as SimpleServerRpcConnection.saslReadAndProcess, they always record "AUTH_FAILED_FOR" for an exception,  and "AUTH_SUCCESSFUL_FOR" after task is completed like this: 
{code:java}
  private void saslReadAndProcess(ByteBuff saslToken) throws IOException, InterruptedException {
    ......
      } catch (IOException e) {
        ......
        // attempting user could be null
        RpcServer.AUDITLOG.warn("{}{}: {}", RpcServer.AUTH_FAILED_FOR, clientIP,
          saslServer.getAttemptingUser());
        throw e;
      }
      ......
      if (saslServer.isComplete()) {
        ......
        RpcServer.AUDITLOG.info(RpcServer.AUTH_SUCCESSFUL_FOR + ugi);
        ......
      }
    }
  }
{code}
but NettyHBaseSaslRpcServerHandler.channelRead0 only record "AUTH_SUCCESSFUL_FOR" in finishSaslNegotiation, and just throw Exception without record "AUTH_FAILED_FOR": 
{code:java}
protected void channelRead0(ChannelHandlerContext ctx, ByteBuf msg) throws Exception {
      ......
      if (saslServer.isComplete()) {
        conn.finishSaslNegotiation();
        ......
      }
  }

void finishSaslNegotiation() throws IOException {
    ......
    RpcServer.AUDITLOG.info(RpcServer.AUTH_SUCCESSFUL_FOR + ugi);
  }
{code}
So I think an exceptionCaught should be called here: 

{code:java}
  public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
    LOG.error("Error when doing SASL handshade, provider={}", conn.provider, cause);
    Throwable sendToClient = HBaseSaslRpcServer.unwrap(cause);
    doResponse(ctx, SaslStatus.ERROR, null, sendToClient.getClass().getName(),
      sendToClient.getLocalizedMessage());
    rpcServer.metrics.authenticationFailure();
    String clientIP = this.toString();
    // attempting user could be null
    RpcServer.AUDITLOG.warn("{}{}: {}", RpcServer.AUTH_FAILED_FOR, clientIP,
      conn.saslServer != null ? conn.saslServer.getAttemptingUser() : "Unknown");
    NettyFutureUtils.safeClose(ctx);
  }
{code}
Bug: Ensure that the baseStagingDir created by SecureBulkLoadManager has the correct permissions
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 23/Dec/22 09:17

In the start function of SecureBulkLoadManager, there is the following code segment
{code:java}
  public void start() throws IOException {
    ......
    fs = FileSystem.get(conf);
    baseStagingDir = new Path(CommonFSUtils.getRootDir(conf), HConstants.BULKLOAD_STAGING_DIR_NAME);
    ......
    if (!fs.exists(baseStagingDir)) {
      fs.mkdirs(baseStagingDir, PERM_HIDDEN);
    }
  }
{code}
It can be seen that the run function will use the mkdirs to create the path when baseStagingDir does not exist, and the parameters passed in include the path variable baseStagingDir and a permission 700. But we haven't confirmed whether the permission is correctly assigned to the file.

The above question is raised because there are two mkdir functions of hadoop, namely 
{code:java}
mkdirs(Path f, FsPermission permission)
{code}
 and 
{code:java}
mkdirs(FileSystem fs, Path dir, FsPermission permission)
{code}
, and the first one is used here. The permissions of this function will be affected by the underlying umask. Although 700 here will hardly be affected by umask, but I think from a rigorous point of view, we should have one more permission check and permission grant here.
Bug: Use reflect to read HDFS ReadStatistics
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 08/Dec/22 06:46

 
{code:java}
java.lang.NoSuchMethodError: org.apache.hadoop.hdfs.client.HdfsDataInputStream.getReadStatistics()Lorg/apache/hadoop/hdfs/DFSInputStream$ReadStatistics;
	at org.apache.hudi.org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.updateInputStreamStatistics(FSDataInputStreamWrapper.java:249)
	at org.apache.hudi.org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.close(FSDataInputStreamWrapper.java:296)
	at org.apache.hudi.org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.closeStreams(HFileBlock.java:1825)
	at org.apache.hudi.org.apache.hadoop.hbase.io.hfile.HFilePreadReader.close(HFilePreadReader.java:107)
	at org.apache.hudi.org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.close(HFileReaderImpl.java:1421)
	at org.apache.hudi.io.storage.HoodieHFileReader.close(HoodieHFileReader.java:218)
	at org.apache.hudi.table.action.commit.HoodieMergeHelper.runMerge(HoodieMergeHelper.java:152)
	at org.apache.hudi.table.HoodieSparkCopyOnWriteTable.handleUpdateInternal(HoodieSparkCopyOnWriteTable.java:230)
	at org.apache.hudi.table.HoodieSparkCopyOnWriteTable.handleUpdate(HoodieSparkCopyOnWriteTable.java:221)
	at org.apache.hudi.table.action.compact.HoodieCompactor.compact(HoodieCompactor.java:227)
	at org.apache.hudi.table.action.compact.HoodieCompactor.lambda$compact$57154431$1(HoodieCompactor.java:138) {code}
Bug: Support replace <FILE-PATH> in GC_OPTS for ZGC
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Nov/22 09:33
Affects Version/s: 2.4.15

HBase support replace -Xloggc:<FILE-PATH> for G1 or CMS。ZGC changed the format，like this：
{code:java}
-Xlog:safepoint,classhisto*=trace,age*,gc*=info:file=<FILE-PATH>:time,tid,tags:filecount=1,filesize=500M{code}
Can  we only replace <FILE-PATH> instead of -Xloggc:<FILE-PATH> to support ZGC？

[https://github.com/apache/hbase/blob/master/bin/hbase-daemon.sh#L174]
{code:java}
if [ -n "$SERVER_GC_OPTS" ]; then  
  export SERVER_GC_OPTS=${SERVER_GC_OPTS/"-Xloggc:<FILE-PATH>"/"-Xloggc:${HBASE_LOGGC}"}
fi
if [ -n "$CLIENT_GC_OPTS" ]; then  
  export CLIENT_GC_OPTS=${CLIENT_GC_OPTS/"-Xloggc:<FILE-PATH>"/"-Xloggc:${HBASE_LOGGC}"}
fi {code}
Bug: Optionally disable sorting directories by size in CleanerChore
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Nov/22 15:37

The CleanerChore computes the size of the directories to start the cleaning with the largest directory first. This computation can take a long time over object stores when the archive directory is huge.

Make it configurable to skip the directory sorting in the CleanerChore.
Bug: result of DirectMemoryUtils#getDirectMemorySize may be wrong
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Oct/22 05:23
Affects Version/s: 3.0.0-alpha-3

If the parameter is set repeatedly, the latter will take effect. For example, if we set 

-Xms30g -Xmx30g -XX:MaxDirectMemorySize=40g -XX:MaxDirectMemorySize=50g

the MaxDirectMemorySize will be set as 50g.
Bug: Optionally limit the amount of plans executed in the Normalizer
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Nov/22 18:23

My company (Hubspot) is starting to use {{{}SimpleRegionNormalizer{}}}. We turn the normalizer switch on for 30 minutes each day, when our database traffic is at a low point. We're using the {{hbase.normalizer.throughput.max_bytes_per_sec}} setting to create a rate limit. I've found that while the {{SimpleRegionNormalizer}} only produces new plans for 30 minutes each day, the plans often take many hours to execute. This leds to region splits, merges, and moves occurring in our HBase clusters during hours we'd prefer them not to.

I propose a new setting {{{}hbase.normalizer.plans_size_limit.mb{}}}.

This will allow HBase administrators to limit the number of plans processed by a run of {{{}RegionNormalizerWorker{}}}, by forcing it to stop processing new plans once the cumulative region size limits are exceeded. This will give you a way to limit approximately how long it takes to execute the plans. Because the current limit to execute plans is primarily determined by a per-byte rate limit, I propose that the new settings also work on a similar basis. This will make it feasible to reason about how your rate limit and your size limits interact.
Bug: Search order for locations in  HFileLink
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Oct/22 21:02

Found that search order for locations is following the order of these locations added to HFileLink object. 

 

setLocations(originPath, tempPath, mobPath, archivePath);

archivePath is the last one to be searched. For most cases, hfile exists in archivePath, so we can move archivePath to the first parameter to avoid unnecessary NN query.
Bug: Run flaky test job more often
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Oct/22 18:26

We currently run the flaky tests once per 4 hours and keep a backlog of 50 runs. That means it takes a perfect test a bit over 8 days to get back out of being penalized.

we regularly are leaving a bunch of executors idle (see attached PNG) and I think we should run the tests more often.

we use an executor per branch for a flaky test run. the duration depends on the specific tests that are flagged, but it has a timeout of 2 hours.

as an example the master branch tests currently take < 30 minutes (see other attached PNG).
Bug: Failure while connecting to zk if HBase is running in standalone mode in a container
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Oct/22 15:55

With [this change|https://issues.apache.org/jira/browse/HBASE-23993] zookeeper will only listen for client connections on 127.0.0.1 in standalone mode. If HBase is running in standalone mode inside a docker container and docker port forwarding is being used to connect to HBase inside container then, any client trying to connect to zk will have destination IP as container's private IP rather than loopback IP. This will cause zk to reject all client connections being initiated from outside the container. 

Thus, we need a way to make zk listen for client connections on IP other than loopback. zk already provides a property named `clientPortAddress` to specify the address to listen for client connections. And, we can specify zk properties using `hbase.zookeeper.property.` prefix. Thus, the proposal is to allow user to specify the IP address that zk will listen on using `hbase.zookeeper.property.clientPortAddress` and if its not specified then default to loopback as per current behaviour.

This way, the tests will still remain unimpacted and HBase can also be used in standalone mode in a container.
Bug:  Update and clean up bcprov-jdk15on dependency
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Oct/22 10:41
Affects Version/s: 3.0.0-alpha-4

{noformat}
$ env JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home mvn -T0.5C clean install -DskipTests                                                                
[INFO] Scanning for projects...                                                                                                                                                                             
[WARNING]                                                                                                                                                                                                   
[WARNING] Some problems were encountered while building the effective model for org.apache.hbase:hbase-server:jar:3.0.0-alpha-4-SNAPSHOT                                                                    
[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.bouncycastle:bcprov-jdk15on:jar -> duplicate declaration of version (?) @ line 331, column 17                  
[WARNING]                                                                                                                                                                                                   
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.                                                                                                 
[WARNING]                                                                                                                                                                                                   
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.                                                                                                  
[WARNING]
{noformat}
Bug: Fixing check for "description" request param in JMXJsonServlet.java
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Oct/22 11:23
Affects Version/s: 2.6.0

When trying to access the JMX metrics' description via the "description=true" URL parameter, any value is accepted.

The current version checks only if "description" is in the URL parameter, but doesn't check the parameter value. 

I would like to fix this via checking if the parameter value is 'true' and showing the description only when this condition is met.
Bug: Long running ExportSnapshot fails with Can't find hfile Exception.
Status: Open | Priority: Minor | Resolution: 
Created: 02/Oct/22 11:37

ExportSnapshot Jobs running for more than destination cluster hbase.master.hfilecleaner.ttl value, are filing with {_}Can't find hfile: <hile> in the real or archive folders{_}. Copied HFiles in archive folder is getting deleted at the Destination cluster by SnapshotHFileCleaner cleaner.

 
 # Export snapshot moves archived hfiles files to destination archved folders.
 # In progress ExportSnapshot manifest will be there in /hbase/.hbase-snapshot/.tmp till it is completed.
 # in SnapshotHFileCleaner flow, where it is ignoring /hbase/.hbase-snapshot/.tmp directory to find the snapshot reference files,

{code:java}
 

private void refreshCache() throws IOException {
  // just list the snapshot directory directly, do not check the modification time for the root
  // snapshot directory, as some file system implementations do not modify the parent directory's
  // modTime when there are new sub items, for example, S3.
  FileStatus[] snapshotDirs = FSUtils.listStatus(fs, snapshotDir,
    p -> !p.getName().equals(SnapshotDescriptionUtils.SNAPSHOT_TMP_DIR_NAME)); {code}
 # As in progress snapshot reference is missed by SnapshotHFileCleaner. TimeToLiveHFileCleaner marks the HFiles older(coped before hbase.master.hfilecleaner.ttl) than hbase.master.hfilecleaner.ttl to delete from in progress ExportSnapshots dir.
 # This is causing ExportSnapshot to fail at the verification stage.

 

Workaround:

increase hbase.master.hfilecleaner.ttl value to more than the Snapshot ExportSnapshot job run time in the destination cluster.

 

I think this issue needs to be fixed in SnapshotHFileCleaner flow so that long-running ExportSnapshot jobs can succeed.
Bug: Fix comment syntax errors
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 13/Dec/22 02:33

I found some sytax errors in the comments while reading the source code.

such as the fallowing examples:

at hbase-asyncfs.src.main.java.org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.java(line 95, 121)

at hbase-asyncfs.src.main.java.org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.java(line 87, 107)

at hbase-balancer.src.main.java.org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer.java(line 53, 59)

......

In fact, I did a simple scan and there were quite a few errors, although only a few affected the reading. I think maybe we can fix it, even if it's not in a hurry.
Bug: Add file `.git-blame-ignore-revs` for `git blame`
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 01/Dec/22 09:14

We introduce spotless to format the code, but it is inconvenient for `git blame` to view the changes. 
We can ignore some commits through `--ignore-rev`.

git v2.23 support `--ignore-rev`
```bash
git config blame.ignoreRevsFile .git-blame-ignore-revs
```
Bug: Add a note for RegionMerge tool. 
Status: Open | Priority: Trivial | Resolution: 
Created: 18/Nov/22 21:13

Ref: [https://github.com/apache/hbase-operator-tools/blob/master/hbase-tools/README.md]

NOTE: 

Do not perform region merge operations on phoenix slated tables. It may affect the region boundaries and produce incorrect query results.
Bug: Flaky Test testClone in TestTaskMonitor
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 10/Nov/22 21:53

{{`testClone`}} is detected as flaky by [[NonDex|https://github.com/TestingResearchIllinois/NonDex]|https://github.com/TestingResearchIllinois/NonDex] since the origin task monitor and the cloned one are represented as JSON objects, which are in a non-deterministic order.
Bug: Remove TestRemoteTable.testLimitedScan
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 16/Oct/22 15:57
Affects Version/s: 2.4.15

The TestRemoteTable.testLimitedScan was originally added in HBASE-24267 for 3.0.0-alpha-1 but with the backport of HBASE-27401 the test was added to branch-2.4, branch-2.5, and branch-2 where the limit support is not available.

This causes test failure on these branches.
Bug: resource leak for store flush during high load
Status: Open | Priority: Blocker | Resolution: 
Created: 17/Jan/23 14:19
Affects Version/s: 2.4.15

2023-01-17 14:01:00,294 WARN  [Close-WAL-Writer-3] asyncfs.FanOutOneBlockAsyncDFSOutputHelper: complete file /hbase/WALs/prod-storage-4,16020,1673962204907/prod-storage-4%2C16020%2C1673962204907.1673964006883 not finished, retry = 0
2023-01-17 14:08:02,681 ERROR [regionserver/prod-storage-4:16020-shortCompactions-0] util.ResourceLeakDetector: iter(StoreFlusher.java:68)
    org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:75)
    org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:992)
    org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2358)
    org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2837)
    org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2579)
    org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2551)
    org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2421)
    org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:588)
    org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:558)
    org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$1000(MemStoreFlusher.java:64)
    org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:340)
Bug: Rename TestConnectionImplementation in hbase-it to fix javadoc failure
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 24/Jan/23 22:41
Affects Version/s: 2.4.16

discussed during release of 2.5.3RC0 [https://lists.apache.org/thread/b34lgz3yy8vkv4fbbxj4mvtjyjrp4m6p]

I found a javadoc problem introduced by HBASE-27498 with the same test class shared the same package prefix such that the javadoc failed with a unclear java.lang.NullPointerException. 

- hbase-it/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementation.java 
- hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementation.java

after renaming hbase-it/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementation.java to hbase-it/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementationCacheMasterState.java should fix this issue.
Bug: CatalogJanitor can cause data loss due to errors during cleanMergeRegion
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 19/Jan/23 04:25

In CatalogJanitor.cleanMergeRegion, there is the following check:
{code:java}
HRegionFileSystem regionFs = null;
try {
  regionFs = HRegionFileSystem.openRegionFromFileSystem(this.services.getConfiguration(), fs,
    tabledir, mergedRegion, true);
} catch (IOException e) {
  LOG.warn("Merged region does not exist: " + mergedRegion.getEncodedName());
}

if (regionFs == null || !regionFs.hasReferences(htd)) {
 .. do the cleanup ..
} {code}
 

I think the assumption here is that an IOException would only be thrown if a region doesn't exist? We had a very poorly timed NameNode failover, during CatalogJanitor run, after a merge. The NameNode failover caused the openRegionFromFileSystem call to fail, which logged:
{code:java}
WARN org.apache.hadoop.hbase.master.janitor.CatalogJanitor: Merged region does not exist: 32c71224852c5a4b94a3ba271b4fcb15 {code}
This region did in fact exist and had not fully compacted, so there were still some lingering reference files.

The cleanup process moves the parent regions to the archive directory, but the default TTL for those files in the archive directory is only 5 minutes. After that they are cleaned up and the data is now unrecoverable.

This resulted in FileNotFoundExceptions trying to read or open this region. Our only course of action was to move the lingering reference files aside, so the data is unrecoverable.
Bug: Region replica replication sometimes orphans WAL queue entries during recovery
Status: Open | Priority: Critical | Resolution: 
Created: 13/Mar/23 12:36
Affects Version/s: 2.5.0

Running with timeline-consistent read replicas and {{hbase.region.replica.replication.enabled=true}}, we're seeing some region servers have WAL queue entires that never clear. This appears to correlate with SCP and recovery of replication queues. The result is WALs that build up, consuming dangerous amounts of space on HDFS. Remediation requires disabling and removing the {{region_replica_replication}} peer, which forces an impacted region server to abort with the message "Failed to operate on replication queue". We then delete the zk entry, which unlocks the WAL and the cleaner chore can sweep them.
Bug: PB's parseDelimitedFrom can successfully return when there are not enough bytes
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 26/Feb/23 06:58

Found this when writing some UTs for parsing partial header and trailer, WALHeader.parseDelimitedFrom can return successfully when there are only two bytes in the stream(only the length, actually).

So I know why in the past we have a followingKvCount == 0 check in ProtobufLogReader, as we just want to prevent the partial PB message.

This is a very critial problem, for me I think we should provide our own implementation of parseDelimitedFrom for some critical usages, for example, when reading WAL entries. If there are not enough data, we just throw exception out instead of returning a partial PB message.
Bug: Should not return false when WALKey has no following KVs while reading WAL file
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 16/Feb/23 09:47

In the current implementation

{code}
      if (!walKey.hasFollowingKvCount() || 0 == walKey.getFollowingKvCount()) {
        LOG.trace("WALKey has no KVs that follow it; trying the next one. current offset={}",
          this.inputStream.getPos());
        seekOnFs(originalPosition);
        return false;
      }
{code}

Here we just return false, seek back to the original position. I think the intention here is that it means the data is not available yet and we should try to read them next time.

But this class is not only used for replication, it is also used by splitting, return false will make the reader.next return null, and WALSplitter will think the WAL file has been fully read and complete the splitting task. If there are still other WAL entries in the file, we will miss reading them and cause data loss.

And in fact, the following kv count is a field in a pb message, so it is impossible that now it is 0 but later it will become a value greater than 0, as we use writeDelimited to write the message, there is a size in front of the message, if we read it successfully, we can make sure the message is complete/ So seeking back in replication is also an useless operation.

So here we propose we still need to return true here, so the upper layer are free to skip or not, but they still need to read other entries.