av9ash/GitBugs huggingface dataset:
Bug: list_quota_snapshots fails with ‘ERROR NameError: uninitialized constant Shell::Commands::ListQuotaSnapshots::TABLE’
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Mar/22 12:36
Affects Version/s: 2.4.11

The list_quota_snapshots command fails with 
{noformat}
ERROR NameError: uninitialized constant Shell::Commands::ListQuotaSnapshots::TABLE{noformat}
regardless of the parameters it's called with.

Apparently, it used to work in HBase 2.2.

I don't know enough about Ruby to tell why it used to work, and what broke this exactly,  but using qualified constants fixes the problem.
Bug: HFileArchiver can remove referenced StoreFiles from the archive
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jan/22 00:46
Affects Version/s: 0.95.0

There is a comment in {{HFileArchiver#resolveAndArchiveFile}}:
{code:java}
// if the file already exists in the archive, move that one to a timestamped backup. This is a
// really, really unlikely situtation, where we get the same name for the existing file, but
// is included just for that 1 in trillion chance.
{code}

In reality, we did encounter this frequently enough to cause problems. More details will be included and linked in a separate issue.

But regardless of how we get into this situation, we can consider a different approach to solving it. If we assume store files are immutable, and a store file with the same name and location already exists in the archive, then it can be safer to assume the file was already archived successfully, and react accordingly.
Bug: Load rate calculator for cost functions should be more precise
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Mar/22 02:33
Affects Version/s: 2.4.11

The rate calculator should avoid negative values, e.g. if a region is moved from RS1 to RS2, the request count loads in the balancer cache maybe [100,200,0,100,200], then the region load cost calculated by CostFromRegionLoadAsRateFunction#getRegionLoadCost will be (100-200+100+100)/4=25, while the real cost is (100+0+100+200)/4=100. 
{code:java}
protected double getRegionLoadCost(Collection<BalancerRegionLoad> regionLoadList) {
  Iterator<BalancerRegionLoad> iter = regionLoadList.iterator();
  if (!iter.hasNext()) {
    return 0;
  }
  double previous = getCostFromRl(iter.next());
  if (!iter.hasNext()) {
    return 0;
  }
  double cost = 0;
  do {
    double current = getCostFromRl(iter.next());
    cost += current - previous;
    previous = current;
  } while (iter.hasNext());
  return Math.max(0, cost / (regionLoadList.size() - 1));
} {code}
We should change the cost accumulate codes to,
{code:java}
cost += current >= previous ? current - previous : current; {code}
Bug: Crash Meta-RS and active HMaster successively make cluster out of service
Status: Open | Priority: Major | Resolution: 
Created: 28/Mar/22 14:32
Affects Version/s: 2.4.8

We got a cluster of 2 master nodes (hm1, hm2) and 3 slaves nodes (rs1, rs2, rs3)

 
 # hm1 becomes active
 # rs3 becomes the meta RS
 # rs3 crashes
 # hm1 crashes. At this time, the meta-region-server in ZooKeeper is still rs3
 # hm2 becomes active, but stucks in "master.HMaster: hbase:meta,,1.1588230740 is NOT online; state=\{1588230740 state=OPEN, ts=1648007404659, server=rs3,16020,1648007272771}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined."
Bug: nightly yetus tests don't reflect failure when required environment variables are missing.
Status: Open | Priority: Major | Resolution: 
Created: 26/Mar/22 07:17

the nightly yetus wrapper shows an error when we're missing some needed environment variables, but the stage shows success still.

e.g
{code:java}
[2022-03-24T22:35:39.450Z] [ERROR] Required environment variable 'DEBUG' is not set.
[2022-03-24T22:35:39.451Z] [ERROR] Required environment variable 'USE_YETUS_PRERELEASE' is not set.
[2022-03-24T22:35:39.451Z] [ERROR] Please set the required environment variables before invoking. If this error is  on Jenkins, then please file a JIRA about the error. {code}
Bug: Update branch-specific nightly test handling to allow for earlier release line feature branches
Status: Open | Priority: Major | Resolution: 
Created: 26/Mar/22 07:09

due to limitations in ASF infra our branch protection rules prohibit force pushes or deletions on branches that start with "master" or "branch-".

Our current nightly tests assume that feature branches specific to branch-2 or branch-1 will start with the release branch name rather than the feature Jira key. However, doing so would mean we need to file INFRA jiras to clean up afterwards.

 

We can't change the limitations of branch protection rules, so we should update the nightly tests to allow for feature branches named like {{{}HBASE-XXXXX-branch-2{}}}.
Bug: nightly integration test claims success while showing error parsing shell output
Status: Open | Priority: Major | Resolution: 
Created: 26/Mar/22 06:59

current master branch claims that the integration test is succeeding, but looking at the detailed output shows a failure to parse the shell output:

 
{code:java}
Starting up Hadoop
	waiting for Hadoop to finish starting up.
	waiting for Hadoop to finish starting up.
	waiting for Hadoop to finish starting up.
	waiting for Hadoop to finish starting up.
Verifying configs
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
/home/jenkins/jenkins-home/workspace/HBase_Nightly_master/output-integration/hadoop-3/hbase-conf/core-site.xml: valid
/home/jenkins/jenkins-home/workspace/HBase_Nightly_master/output-integration/hadoop-3/hbase-conf/hbase-site.xml: valid
OK
Listing HDFS contents
Starting up HBase
running master, logging to /home/jenkins/jenkins-home/workspace/HBase_Nightly_master/hbase-install/bin/../logs/hbase-jenkins-master-jenkins-hbase12.out
	retry waiting for hbase to come up.
Setting up table 'test:example' with 1,000 regions
writing out example TSV to example.tsv
uploading example.tsv to HDFS
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
Importing TSV via shaded client artifact for HBase - MapReduce integration.
Verifying row count from import.
/home/jenkins/jenkins-home/workspace/HBase_Nightly_master/component/dev-support/hbase_nightly_pseudo-distributed-test.sh: line 418: [: hbase:002:0> : integer expression expected
Hadoop client jars not given; getting them from 'hadoop classpath' for the example.
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
Building shaded client example.
Running shaded client example. It'll fetch the set of regions, round-trip them to a file in HDFS, then write them one-per-row into the test table.
Checking on results of example program.
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
Verifying row count from example.
/home/jenkins/jenkins-home/workspace/HBase_Nightly_master/component/dev-support/hbase_nightly_pseudo-distributed-test.sh: line 531: [: hbase:002:0> : integer expression expected
ERROR: Only found hbase:002:0>  rows.
Shutting down HBase {code}
Bug: TestRegisterPeerWorkerWhenRestarting fails with IllegalStateException
Status: Open | Priority: Major | Resolution: 
Created: 22/Mar/22 04:54

{noformat}
2022-03-21T12:25:09,067 ERROR [RS_REFRESH_PEER-regionserver/b0aef29cdb13:0-0.replicationSource,1] regionserver.ReplicationSource(434): Unexpected exception in RS_REFRESH_PEER-regionserver/b0aef29cdb13:0-0.replicationSource,1 currentPath=null
java.lang.IllegalStateException: Source should be active.
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.initialize(ReplicationSource.java:553) ~[classes/:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
2022-03-21T12:25:09,068 ERROR [RS_REFRESH_PEER-regionserver/b0aef29cdb13:0-0.replicationSource,1] regionserver.HRegionServer(2284): ***** ABORTING region server b0aef29cdb13,33413,1647865503466: Unexpected exception in RS_REFRESH_PEER-regionserver/b0aef29cdb13:0-0.replicationSource,1 *****
java.lang.IllegalStateException: Source should be active.
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.initialize(ReplicationSource.java:553) ~[classes/:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
2022-03-21T12:25:09,069 ERROR [RS_REFRESH_PEER-regionserver/b0aef29cdb13:0-0.replicationSource,1] regionserver.HRegionServer(2291): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint]
2022-03-21T12:25:09,072 ERROR [RS_REFRESH_PEER-regionserver/b0aef29cdb13:0-0.replicationSource,1] regionserver.ReplicationSource(434): Unexpected exception in RS_REFRESH_PEER-regionserver/b0aef29cdb13:0-0.replicationSource,1 currentPath=null
java.lang.IllegalStateException: Source should be active.
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.initialize(ReplicationSource.java:553) ~[classes/:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
2022-03-21T12:25:09,072 ERROR [RS_REFRESH_PEER-regionserver/b0aef29cdb13:0-0.replicationSource,1] regionserver.HRegionServer(2284): ***** ABORTING region server b0aef29cdb13,41089,1647865503422: Unexpected exception in RS_REFRESH_PEER-regionserver/b0aef29cdb13:0-0.replicationSource,1 *****
java.lang.IllegalStateException: Source should be active.
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.initialize(ReplicationSource.java:553) ~[classes/:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
2022-03-21T12:25:09,072 ERROR [RS_REFRESH_PEER-regionserver/b0aef29cdb13:0-0.replicationSource,1] regionserver.HRegionServer(2291): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint]
{noformat}
Bug: HBase client does not honor UserGroupInformation.doAs
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 24/Feb/22 13:31

Despite passing necessary UserInformation to the RegionServer, which does authorize the request, the async and block clients do not work correctly with the following access pattern:
{code:java}
Connection connection = ConnectionFactory.createConnection();
Table table = connection.getTable(name);

UserGroupInformation proxy = UserGroupInformation.createProxyUser(
  "testUser",
  UserGroupInformation.getCurrentUser()
); 

Result result = proxy.doAs(() -> table.get(get));{code}
In this case, you would expect the get to be executed as "testUser", but instead it is executed as whichever user created the initial connection. This can be verified by checking the security logger on the RegionServer side.

The reason for this is we stash the current User onto the actual ConnectionImplementation, and we pass that through all calls in the stack when executing an RPC. I think the appropriate way would be to replace usage of this stashed User with a call to UserGroupInformation.getCurrentUser() in RpcConnection, where sasl is negotiated and headers generated.
Bug: Memstore flush fencing issue for SFT
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Mar/22 16:39
Affects Version/s: 2.6.0

The scenarios is the following:
 # rs1 is flushing file to S3 for region1
 # rs1 loses ZK lock
 # region1 gets assigned to rs2
 # rs2 opens region1
 # rs1 completes flush and updates sft file for region1
 # rs2 has a different “version” of the sft file for region1

The flush should fail at the end, but the SFT file gets overwritten before that, resulting in potential data loss.

 

Potential solutions include:
 * Adding timestamp to the tracker file names. This and creating a new tracker file when an rs open the region would allow us to list available tracker files before an update and compare the found timestamps to the one stored in memory to verify the store still owns the latest tracker file
 * Using the existing timestamp in the tracker file content. This would also require us to create a new tracker file when a new rs opens the region, but instead of listing the available tracker files, we could try to load and de-serialize the last tracker file and compare the timestamp found in it to the one stored in memory.
Bug: Fix flaky TestSnapshotFromMaster.testSnapshotHFileArchiving
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Mar/22 07:19

After investigating, finally I found that the problem is we just use tryLock in SnapshotFileCache.getUnreferencedFiles. If we failed to grab lock, we will skip cleaning of the archive files.

We should add some retries in the test to make it more stable.
Bug: AsyncFSWAL.unackedAppends should clear after  transfered to  AsyncFSWAL.toWriteAppends
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 11/Jan/22 11:47
Affects Version/s: 2.4.9

When {{ASyncFSWAL}} syncing to HDFS failed,  {{AsyncFSWAL.unackedAppends}} are transfered to {{AsyncFSWAL.toWriteAppends}} to avoid data loss, but {{AsyncFSWAL.unackedAppends}} itself is not cleared. I think there is no need to continue retain them in {{AsyncFSWAL.unackedAppends}} because we would open a new HDFS pipeline to resend the {{AsyncFSWAL.unackedAppends}}. 
BTW :  It would also simplify the logic for fixing HBASE-25905, current fix for HBASE-25905 is somewhat hard to understand. I think the problem to cause HBASE-25905 is  that {{AsyncFSWAL.unackedAppends}}  could not exactly reflect the *unacked* for current HDFS pipeline. If we clear {{AsyncFSWAL.unackedAppends}} after transferring them to {{AsyncFSWAL.toWriteAppends}}, HBASE-25905 could also avoid.
Bug: EOL HBase 2.3.x
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jan/22 05:45

VOTE thread is here:

[https://lists.apache.org/thread/v3gfoyf255rpvql4h3g68ong5p4ps4rs]
 * remove from [https://hbase.apache.org/downloads]
 * delete from dist svn repo [https://dist.apache.org/repos/dist/release/hbase/] (which will remove the release from downloads.apache.org)
 * send an announcement to user@hbase and dev@hbase
 * delete Jenkins jobs
Bug: --hadoop-profile is not honored in branch-2 PR builds
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 16/Mar/22 13:32
Affects Version/s: 2.4.12

I noticed in the PR builds for HBASE-26834 that the {{HADOOP_PROFILE}} is apparently not honored. This is problematic for the JDK11 build, which only works on Hadoop3, which must be enabled via this variable.
Bug: Rsgroup: move machine to group error report table is disabledrsgroup
Status: Open | Priority: Major | Resolution: 
Created: 17/Mar/22 06:58
Affects Version/s: 2.1.0

1. Start rsgroup in cluster

2. Add a group

3. Move a machine into a group and report the following error

!微信图片_20220317150240(1).png!
Bug: TestAdminShell2 is failing because of we referenced a deleted method
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Mar/22 05:32

{noformat}
src/test/ruby/hbase/admin2_test.rb:70:in `block in test_Get_detailed_status'
/home/zhangduo/hbase/code/hbase-shell/src/test/ruby/test_helper.rb:161:in `capture_stdout'
     67:     end
     68:
     69:     define_test 'Get detailed status' do
^[[48;5;16;38;5;226;1m  => 70:       output = capture_stdout { admin.status('detailed', '') }^[[0m
     71:       puts "Status output:\n#{output}"
     72:       # Some text which isn't in the simple output
     73:       assert output.include? 'regionsInTransition'
src/test/ruby/hbase/admin2_test.rb:70:in `block in test_Get_detailed_status'
uri:classloader:/hbase/admin.rb:926:in `status'
Error: ^[[48;5;16;38;5;226;1mtest_Get_detailed_status(Hbase::StatusTest)^[[0m:
  NoMethodError: undefined method `getMaster' for #<Java::OrgApacheHadoopHbase::ClusterMetricsBuilder::ClusterMetricsImpl:0x4eee1637>
  Did you mean?  get_master_name
{noformat}

It is introduced by HBASE-26730.
Bug: Rest server should not use a large Header Cache.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Feb/22 18:20
Affects Version/s: 2.4.9

In the RESTServer we set the HeaderCache size to DEFAULT_HTTP_MAX_HEADER_SIZE (65536). That's not compatible with jetty-9.4.x because the cache size is limited by Character.MAX_VALUE - 1  (65534) there. According to the Jetty source code comments, it's possible to have a buffer overflow in the cache for higher values and that might lead to wrong/incomplete values returned by cache and following incorrect header handling.  
There are a couple of ways to fix it:
1. change the value of DEFAULT_HTTP_MAX_HEADER_SIZE to 65534
2. make header cache size configurable and set its size separately from the header size. 

I believe that the second would give us more flexibility.
Bug: Introduce path configuration for system coprocessors
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jan/22 00:08
Affects Version/s: 2.5.0

Currently when region server starts, system coprocessors are normally loaded as part of the local classpath and the jar is stored in the local storage; in short words, Operator would have to copy the jar to each host (or a shared file system) and construct it as part of HBASE_CLASSPATH. 

although operator may have been doing the presetup of copying jar and making it available to the HBASE_CLASSPATH without any issue, it could be helpful if we provide an alternative method and centralize this configuration in hbase-site similar to the support of table-level coprocessor, e.g. configuring {{hbase.coprocessor.region.classes}} with the local/remote path along with the classname with a specific path token in the form of {{className|priority|path}}.


Similarly in HBASE-23710, it provided the priority configuration via hbase-site that this new improvement aligns with the same purpose, and further help to simplify the deployment of System Coprocessors to e.g. hdfs or supported cloud storage.
Bug: Backport "HBASE-26714 Introduce path configuration for system coprocessors" to branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Feb/22 21:35
Affects Version/s: 2.5.0

backport HBASE-26714 to branch-2, mainly it's due to the unclear cherry pick of the change of HBaseCommonTestUtility to HBaseCommonTestingUtility
Bug: Threads shared EMPTY_RESULT may lead to unexpected client job down.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jan/22 03:40

Currently, we use a pre-created EMPTY_RESULT in the ProtoBuf.util to reduce the object creation. But these objects could be shared by multi client threads. The Result#cellScannerIndex related methods could throw confusing exception and make the client job down. Could refine the logic of these methods.

The precreated objects in ProtoBufUtil.java:

{code:java}
private final static Cell[] EMPTY_CELL_ARRAY = new Cell[]{};
private final static Result EMPTY_RESULT = Result.create(EMPTY_CELL_ARRAY);
final static Result EMPTY_RESULT_EXISTS_TRUE = Result.create(null, true);
final static Result EMPTY_RESULT_EXISTS_FALSE = Result.create(null, false);
private final static Result EMPTY_RESULT_STALE = Result.create(EMPTY_CELL_ARRAY, null, true);
{code}

Result#advance 
{code:java}
public boolean advance() {
    if (cells == null) return false;
    cellScannerIndex++;
    if (cellScannerIndex < this.cells.length) {
      return true;
    } else if (cellScannerIndex == this.cells.length) {
      return false;
    }
    // The index of EMPTY_RESULT could be incremented by multi threads and throw this exception.
    throw new NoSuchElementException("Cannot advance beyond the last cell");
  }
{code}

Result#current

{code:java}
public Cell current() {
    if (cells == null
            || cellScannerIndex == INITIAL_CELLSCANNER_INDEX
            || cellScannerIndex >= cells.length)
      return null;
   // Although it is almost impossible,
   // We can arrive here when the client threads share the common reused EMPTY_RESULT.
    return this.cells[cellScannerIndex];
  }
{code}

In this case, the client can easily got confusing exceptions even if they use different connections, tables in different threads.
We should change the if condition cells == null to isEmpty() to avoid the client crashed from these exception.
Bug: Revert the changes on hbase-daemon.sh after switching to log4j2 properties file
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Mar/22 07:57

In HBASE-19577, we changed to use two ENVVARs to specify log level and appender, so we also changed hbase-daemon.sh to use two ENVVARs. In HBASE-26723 we switched back to use one ENVVAR, so we should also revert the changes in hbase-daemon.sh.
Bug: YCSB section of docs refers to unexisting github repo
Status: Open | Priority: Major | Resolution: 
Created: 07/Mar/22 16:45

in [the YCSB section|https://hbase.apache.org/book.html#ycsb] of the Apache HBase book the [Ted Dunning's YCSB repo|https://github.com/tdunning/YCSB] is mentioned.

This repo doesn't exist anymore, maybe to be replaced by [Brian Frank Cooper|https://github.com/brianfrankcooper/YCSB/] ones?
Bug: Ban reload4j and logback dependencies
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Mar/22 11:40

Multiple logging framework on classpath can lead to strange behaviors, so we need to make sure that there is only one framework on our classpath.

We used to ban log4j when switching to log4j2, now since hadoop is going to switch to reload4j for current release branches and zookeeper has switched to logback, we should exclude them too.
Bug: TestCanaryTool.testWriteTableTimeout is flaky
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Mar/22 04:43

{noformat}
Wanted but not invoked:
appender.append(<custom argument matcher>);
-> at org.apache.hadoop.hbase.tool.TestCanaryTool.testWriteTableTimeout(TestCanaryTool.java:310)

However, there were exactly 2 interactions with this mock:
appender.getName();
-> at org.apache.logging.log4j.core.config.AbstractConfiguration.addLoggerAppender(AbstractConfiguration.java:859)

appender.getName();
-> at org.apache.logging.log4j.core.config.AppenderControl.<init>(AppenderControl.java:56)
{noformat}

May be related to the recent log4j2 change.
Bug: [hboss] Limit synchronization from hot path of HBoss APIs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Mar/22 23:16
Affects Version/s: hbase-filesystem-1.0.0-alpha2

Problem:-

While using HBase with S3, timeouts were observed while restoring a snapshot of a table having many regions and HFiles, and the problem regressed as we added more restores or snapshots in parallel.

 

Root cause:-

HBoss has some of the APIs interacting with the cache as synchronized, but as these are in the hot path of the interaction with the filesystem, threads are blocked until one is done, causing all threads working on the filesystem to run serially. In the case of restore_snapshot, which includes getting a list of completed snapshots, validating each store file, creating an HFile link in the target table, archiving files, etc. Therefore the problem is highlighted more with restore snapshot.

 

Solution:-

Remove the synchronization as those seem unnecessary, change cache data structure to a thread-safe which provides the required level concurrency.
Bug: TestRegionReplicaReplicationError should inject error in replicateToReplica method
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Mar/22 03:20
Affects Version/s: 3.0.0-alpha-2

{{TestRegionReplicaReplicationError}} added by HBASE-26233 now injects error in {{RSRpcServices.replay}}, but  {{RSRpcServices.replay}} in fact is not invoked, seems we should inject error in {{RSRpcServices.replicateToReplica}}.