av9ash/GitBugs huggingface dataset:
Bug: [Flakey Tests] A couple of fixes and cleanups
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Mar/20 18:56

Here's a few cleanups and flakey fixes accumulated in last few days:

{code}
    hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupMajorCompactionTTL.java
     Remove spurious assert. Just before this it waits an arbitrary 10
     seconds. Compactions could have completed inside this time. The spirit
     of the test remains.

    hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java
     Get log cleaner to go down promptly; its sticking around. See if this
     helps with TestMasterShutdown

    hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
     We get a rare NPE trying to sync. Make local copy of SyncFuture and see
     if that helps.

    hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionAdminApi.java
     Compaction may have completed when not expected; allow for it.

    hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestBlockEvictionFromClient.java
     Add wait before testing. Compaction may not have completed. Let
     compaction complete before progressing and then test for empty cache.

    hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java
     Less resources.

    hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java
     Less resources.

    hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
     Wait till online before we try and do compaction (else request is
     ignored)

    hbase-server/src/test/java/org/apache/hadoop/hbase/tool/TestCanaryTool.java
     Disable test that tests for timeout that fails randomly w/ mockito complaint on some mac os
     x's.
{code}
Bug: Address thread safety issue with Map used in BufferCallBeforeInitHandler.
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 09/Mar/20 23:48

id2Call is a HashMap and has a call back method that accesses it that can be run via an executor as well as another method accessing it that can be run from a different thread.

id2Call should likely be a ConcurrentHashMap to be shared like this.
Bug: TestFromClientSide3 and subclasses often fail on testScanAfterDeletingSpecifiedRowV2.
Status: Open | Priority: Major | Resolution: 
Created: 13/Feb/20 11:10

This test method fails a fair amount on me with something like:

TestFromClientSide3WoUnsafe>TestFromClientSide3.testScanAfterDeletingSpecifiedRowV2:236 expected:<3> but was:<2>

I had a hunch that it might be due to interference from other test methods running first so I tried changing the table name for just this method to be unique - still fails.

However, when I just run testScanAfterDeletingSpecifiedRowV2 on it's own without the methods, it does not seem to fail so far.
Bug: TestReplicationEndpoint appears to fail a lot in my attempts for a clean test run locally.
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 11/Feb/20 21:56

This test is failing for me like 30-40% of the time. Fail seems to usually be as below. I've tried increasing the wait timeout but that does not seem to help at all.

{code}
[ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 105.145 s <<< FAILURE! - in org.apache.hadoop.hbase.replication.TestReplicationEndpoint[ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 105.145 s <<< FAILURE! - in org.apache.hadoop.hbase.replication.TestReplicationEndpoint[ERROR] org.apache.hadoop.hbase.replication.TestReplicationEndpoint.testInterClusterReplication  Time elapsed: 38.725 s  <<< FAILURE!java.lang.AssertionError: Waiting timed out after [30,000] msec Failed to replicate all edits, expected = 2500 replicated = 2476 at org.junit.Assert.fail(Assert.java:89) at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:203) at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:137) at org.apache.hadoop.hbase.replication.TestReplicationEndpoint.testInterClusterReplication(TestReplicationEndpoint.java:235){code}
Bug: [Flakey Tests] Allow that server may not yet be cleared from DeadServers in TestHBCKSCP
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Mar/20 06:01

Fails 4/11 times in branch-2 flakey-test runs. Fails at line #157 where we check to see if server still in dead server list. It may not have cleared yet. Thats ok. Let the test pass. We've already asserted server is cleared from meta, the point of HBCKSCP.
Bug: Switch back to ZKConnectionRegistry by default at least in test
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 25/Feb/20 07:23

For now, MasterRegistry can not deal with master restart, as it can not load the new master address automatically.

I see there is a invalidateConnection method in HBaseTestingUtilities but it needs a very big refactoring to make all the UTs work like this.

So here I suggest we switch back to ZKConnectionRegistry by default, and open a new feature branch to finish the TODOs and the refactoring on UTs.

As now it is already a big problem for me as I want to merge a feature branch back to master but the state of UTs are a mess.
Bug: Snapshot owner cannot delete snapshot when ACL is enabled and Kerberos is not enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Feb/20 11:10
Affects Version/s: 2.2.3

When ACL is enabled and Kerberos is not enabled, the snapshot owner cannot delete the snapshot. This is because the owner of the snapshot cannot be taken during permission verification. By investigation, found that only after HBase has enabled security authentication, the owner will be set when doing snapshot. 

SnapshotManager#takeSnapshotInternal
{code:title=SnapshotManager.java|borderStyle=solid}
RpcServer.getRequestUser().ifPresent(user -> {
  if (User.isHBaseSecurityEnabled(master.getConfiguration())) {
    builder.setOwner(user.getShortName());
  }
});
{code}
Bug: Fail fast when bulkLoadHFiles method catch some IOException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Mar/20 03:56

In production environment, we usually do bulkload huge amount hfile . It reasonable  fail fast when any  IOException occur

 

hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
{code:java}
public Map<byte[], List<Path>> bulkLoadHFiles(Collection<Pair<byte[], String>> familyPaths,
    boolean assignSeqId, BulkLoadListener bulkLoadListener,
      boolean copyFile, List<String> clusterIds, boolean replicate) throws IOException {
  ......
  try {
    this.writeRequestsCount.increment();

    // There possibly was a split that happened between when the split keys
    // were gathered and before the HRegion's write lock was taken.  We need
    // to validate the HFile region before attempting to bulk load all of them
    List<IOException> ioes = new ArrayList<>();
    List<Pair<byte[], String>> failures = new ArrayList<>();
    for (Pair<byte[], String> p : familyPaths) {
      byte[] familyName = p.getFirst();
      String path = p.getSecond();

      HStore store = getStore(familyName);
      if (store == null) {
        IOException ioe = new org.apache.hadoop.hbase.DoNotRetryIOException(
            "No such column family " + Bytes.toStringBinary(familyName));
        ioes.add(ioe);
      } else {
        try {
          store.assertBulkLoadHFileOk(new Path(path));
        } catch (WrongRegionException wre) {
          // recoverable (file doesn't fit in region)
          failures.add(p);
        } catch (IOException ioe) {
          // unrecoverable (hdfs problem)
          ioes.add(ioe);
        }
      }
    }

    // validation failed because of some sort of IO problem.
    if (ioes.size() != 0) {
      IOException e = MultipleIOException.createIOException(ioes);
      LOG.error("There were one or more IO errors when checking if the bulk load is ok.", e);
      throw e;
    }
{code}
Bug: refactor  loadBalancer implements for rsgroup balance by table to  achieve overallbalanced
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Mar/20 03:24
Affects Version/s: 2.2.0

 now can not achieve overallbalanced when use rsgroup balancer and by table is on,

because balance every table actually use the clusterload only contain one table's load.

we should use clusterload contain all this rsgroup table's load to balance overall

 

 hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
{code:java}
  public boolean balance(boolean force) throws IOException {
    ......
    boolean isByTable = getConfiguration().getBoolean("hbase.master.loadbalance.bytable", false);
    Map<TableName, Map<ServerName, List<RegionInfo>>> assignments =
      this.assignmentManager.getRegionStates()
        .getAssignmentsForBalancer(tableStateManager, this.serverManager.getOnlineServersList(),
          isByTable);
    for (Map<ServerName, List<RegionInfo>> serverMap : assignments.values()) {
      serverMap.keySet().removeAll(this.serverManager.getDrainingServersList());
    }

//Give the balancer the current cluster state.
    this.balancer.setClusterMetrics(getClusterMetricsWithoutCoprocessor());
    this.balancer.setClusterLoad(assignments);

    List<RegionPlan> plans = new ArrayList<>();
    for (Entry<TableName, Map<ServerName, List<RegionInfo>>> e : assignments.entrySet()) {
      List<RegionPlan> partialPlans = this.balancer.balanceCluster(e.getKey(), e.getValue());
      if (partialPlans != null) {
        plans.addAll(partialPlans);
      }
    }
{code}
now do refactor:
 # add method 'balanceTable' in interface LoadBalancer
 # SimpleLoadBalancer and StochasticLoadBalancer do the real 'balanceTable' , and 'balanceTable' is not support in BaseLoadBalancer and RSGroupBasedLoadBalancer
 # RSGroupBasedLoadBalancer invoke balanceCluster , and pass GroupClusterLoad to internal balacer by group
 # internal balancer balance cluster invoke 'balanceTable'
Bug: Update license for jetty-client
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Mar/20 00:03
Affects Version/s: 3.0.0-alpha-1

After HBASE-22103, compiling on Haddop 3.3.0 has the following error:
{{mvn clean install -Dhadoop.profile=3.0 -Dhadoop.version=3.3.0-SNAPSHOT -DskipTests -Dmaven.javadoc.skip=true}}
{noformat}
This product includes Jetty :: Asynchronous HTTP Client licensed under the Apache Software License - Version 2.0.

ERROR: Please check ^^^^^^^^^^^^ this License for acceptability here:

https://www.apache.org/legal/resolved

If it is okay, then update the list named 'non_aggregate_fine' in the LICENSE.vm file.
If it isn't okay, then revert the change that added the dependency.

More info on the dependency:

<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-client</artifactId>
<version>9.4.20.v20190813</version>
{noformat}

This is caused by YARN-8778 which added dependency on org.eclipse.jetty.websocket:websocket-client, and the Jetty 9.4 update in HADOOP-16152.

{noformat}
[INFO] +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:3.3.0-SNAPSHOT:compile
[INFO] |  +- org.apache.hadoop:hadoop-yarn-client:jar:3.3.0-SNAPSHOT:compile
[INFO] |  |  +- org.eclipse.jetty.websocket:websocket-client:jar:9.4.20.v20190813:compile
[INFO] |  |  |  +- org.eclipse.jetty:jetty-client:jar:9.4.20.v20190813:compile
[INFO] |  |  |  \- org.eclipse.jetty.websocket:websocket-common:jar:9.4.20.v20190813:compile
[INFO] |  |  |     \- org.eclipse.jetty.websocket:websocket-api:jar:9.4.20.v20190813:compile
{noformat}

Propose: update hbase-resource-bundle/src/main/resources/supplemental-models.xml to update the license text for jetty-client.
Bug: [regression]  Increase RESTServer buffer size back to 64k
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Mar/20 14:41
Affects Version/s: 2.2.0

HBASE-14492 is not longer present in our current releases after HBASE-12894. Unfortunately our RESTServer is not extending HttpServer which means that {{DEFAULT_MAX_HEADER_SIZE}} is not being set and HTTP requests with a very large header can still cause connection issues for clients. A quick fix is just to add the settings to the {{HttpConfiguration}} configuration object. A long term solution should be to re-factor services that create an HTTP server and normalize all configuration settings across all of them.
Bug: Add modules to mapreduce job classpaths
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Mar/20 15:10
Affects Version/s: 2.1.10

Modularization of hbase-server introduced new top-level modules that are missing from MapReduce jobs' classpath.

CompactionTool job in MapReduce mode required ReplicationUtils class which is moved to the hbase-replication module.
Bug: Literate builder API for version management in schema
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jan/20 19:21

Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance regarding their usage. Almost all combinations of these four settings make sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the behavior with TTL easier to conceive when creating the schema. This could take the form of a literate builder API for ColumnDescriptor or an extension to an existing one. 

Let me give you a motivating example: We may want to retain all versions for a given TTL, and then only a specific number of versions after that interval elapses. This can be achieved with VERSIONS=INT_MAX, TTL=_retention_interval_, KEEP_DELETED_CELLS=TTL, MIN_VERSIONS=_num_versions_ . This is not intuitive though because VERSIONS has been used to specify the number of versions to retain (_num_versions_ in this example) since HBase version 0.1, so this is going to be a source of confusion - I've seen it in practice. 

A literate builder API, by way of the way we design its method names, could let a user describe more or less in speaking language how they want version retention to work, and internally the builder API could set the low level schema attributes.
Bug: hbck2 assigns command should accept one or more files containing a list of region names
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Mar/20 19:43
Affects Version/s: hbase-operator-tools-1.0.0

Currently, *assigns* command accepts a list of region names as parameter. When dealing with a very large number of regions, though, passing all regions encoded name as the command parameters becomes too much overhead for the CLI. We should accept a path to one or more file that contains these encoded regions, one per line. That way, this command tails nicely into an operator's incantation using grep/sed over log files.
Bug: get_slowlog_responses filters with AND/OR operator support
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Mar/20 17:47
Affects Version/s: 2.3.0

As of now, get_slowlog_responses support filters for various fields but tries to match either of multiple filters i.e. default operator is "OR". We can support user provided "AND" operator so that only entries that match all the filters can be returned as result.
Bug: [Flakey Tests] TestSecureExport.testVisibilityLabels fails with SaslException
Status: Open | Priority: Major | Resolution: 
Created: 14/Mar/20 20:59

It fails on one of the three hosts I'm using to run tests. It fails every time. Here is the complaint:

{code}
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Bad file descriptor (Write failed))]; Host Details : local host is: "stack.example.org/X.X.X.X"; destination host is: "localhost":61294;
        at org.apache.hadoop.hbase.coprocessor.TestSecureExport.lambda$testVisibilityLabels$7(TestSecureExport.java:422)
        at org.apache.hadoop.hbase.coprocessor.TestSecureExport.testVisibilityLabels(TestSecureExport.java:426)
{code}

(Line numbers are a little off because I've been messing around in code... trying to figure how to get it to pass).

I'm going to just disable the failing test in sub-issue until someone has a chance to look at it. This is an old one but its stopping me make progress on test stability on one of my machines and figuring this sort of failure is of low priority for myself at the moment at least.
Bug: When the data directory is set to EC,the amount of storefiles reaches a relatively high level,heap Occupancy Percent is above heap occupancy alarm watermark,resulting in frequent gc and eventual failure
Status: Reopened | Priority: Major | Resolution: 
Created: 26/Mar/20 03:34
Affects Version/s: 2.0.1

When the data directory is set to EC, during the continuous load process, regionserver is found down ,to view the log:

regionserver.HeapMemoryManager: heapOccupancyPercent 0.91630864 is now below the heap occupancy alarm watermark (0.95)

regionserver.HeapMemoryManager: heapOccupancyPercent 0.969546 is above heap occupancy alarm watermark (0.95)

[JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1254ms
 GC pool 'G1 Young Generation' had collection(s): count=2 time=115ms
 GC pool 'G1 Old Generation' had collection(s): count=1 time=1487ms

In particular, GC pauses occurred 283 times in half an hour.

The memory analysis found that most of the memory footprint was generated by HeepByteBuffer,and DFSStripedInputStream produces the HeepByteBuffer

See HDFS-14308，but after the fix, hbase did not return to normal.
Bug: Extend HBASE-16209 strategy (Provide an ExponentialBackOffPolicy sleep between failed region open requests) to region close and split requests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Mar/20 17:47
Affects Version/s: 1.6.0

In HBASE-16209 we provide an ExponentialBackOffPolicy sleep between failed region open requests. This should be extended to also apply to region close and split requests. Will reduce the likelihood of FAILED_CLOSE transitions in production by being more tolerant of temporary regionserver loading issues, e.g. CallQueueTooBigException.
Bug: HBase shell list_peers command is displayed incorrectly
Status: Open | Priority: Major | Resolution: 
Created: 18/Mar/20 06:48

In 2.X version, I found hbase shell list_peers command displays java object id not the human readable peer messages
{code:java}
hbase(main):002:0> list_peers
 PEER_ID CLUSTER_KEY ENDPOINT_CLASSNAME STATE REPLICATE_ALL NAMESPACES TABLE_CFS BANDWIDTH SERIAL
0 row(s)
Took 0.3336 seconds 
=> #<Java::JavaUtil::ArrayList:0x308a9264>
{code}
Bug: "Packaging and Integration" check fails
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Mar/20 23:29
Affects Version/s: 2.3.0

Looks like this is running with Hadoop 2.8.5, despite HBASE-23986

{noformat}
[2020-03-24T09:14:00.930Z] ++ ls -1 /home/jenkins/jenkins-slave/workspace/HBase_Nightly_master/hadoop-2.8.5-bin.tar.gz
[2020-03-24T09:14:00.930Z] ++ head -n 1
[2020-03-24T09:14:00.930Z] + artifact=/home/jenkins/jenkins-slave/workspace/HBase_Nightly_master/hadoop-2.8.5-bin.tar.gz
[2020-03-24T09:14:00.930Z] + tar --strip-components=1 -xzf /home/jenkins/jenkins-slave/workspace/HBase_Nightly_master/hadoop-2.8.5-bin.tar.gz -C hadoop-2
[2020-03-24T09:14:06.186Z] + /home/jenkins/jenkins-slave/workspace/HBase_Nightly_master/component/dev-support/hbase_nightly_pseudo-distributed-test.sh --single-process --working-dir output-integration/hadoop-2 --hbase-client-install hbase-client hbase-install hadoop-2/bin/hadoop hadoop-2/share/hadoop/yarn/test/hadoop-yarn-server-tests-2.8.5-tests.jar hadoop-2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.5-tests.jar hadoop-2/bin/mapred
[2020-03-24T17:50:28.859Z] Sending interrupt signal to process
[2020-03-24T17:50:30.005Z] Sending interrupt signal to process
[2020-03-24T17:50:42.558Z] + echo '(x) {color:red}-1 client integration test{color}
[2020-03-24T17:50:42.558Z] --Failed when running client tests on top of Hadoop 2. [see log for details|https://builds.apache.org/job/HBase%20Nightly/job/master/1677//artifact/output-integration/hadoop-2.log]. (note that this means we didn'\''t run on Hadoop 3)'
[2020-03-24T17:50:42.558Z] + exit 2
[2020-03-24T17:50:42.866Z] script returned exit code 2
{noformat}
Bug: [hbase-thirdparty] Make release 3.2.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jan/20 04:23
Affects Version/s: 2.3.0

Update the dependencies in hbase-thirdparty. In particular, pb goes from 3.7 to 3.11 with a few perf improvements.

Would like to update the hbase-thirdparty to include in the coming hbase-2.3.0.
Bug: Balancer keeps balancing indefinitely 
Status: Open | Priority: Major | Resolution: 
Created: 10/Mar/20 17:16
Affects Version/s: 2.0.2

Before raising this issue - am not sure if this got fixed directly or indirectly in other latest versions of hbase.
The steps are 
1) Create a cluster and create some tables.  (assume we have RS 1,2,3, 4 and 5)
2) After the table creation and some ops done, the cluster was restarted. Due to this some regions are in RIT. the RIT in  progress was to be assigned to RS 3.
3) After the cluster comes back RS 3 and 4 are stopped.  (RS 3 will have newer timestamp)
4) Now the master that comes up sees there are some RIT in place and tries to load the entries to process the procedures again. As part of this the RegionStateStore is populated with the old RS 3 hostname. (older timestamp). This adds to the ServerStateNode creating a RS 3 with old timestamp as one server.
5) Now after the master restarts and all regions assigned, the balancer infinitely tries to balance the region to the RS 3 (old timestamp server) thinking it is part of the cluster. 
6)the other problem is the MoveProcedure has the target as RS 3 (with old timestamp) but the AM realizes that it is a down server and move it to the one of the active server. But this is not recorded anywhere.
I will continue to check the latest code if this case is valid.
Bug: Replicate slow/large RPC calls to HDFS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Mar/20 17:02
Affects Version/s: 1.7.0

We should provide capability to replicate complete slow and large RPC logs to HDFS or create new system table in addition to Ring Buffer. This way we don't lose any of slow logs and operator can retrieve all the slow/large logs. Replicating logs to HDFS / creating new system table should be configurable.
Bug: Optionally reject multi() requests with very high no of rows
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Mar/20 10:58

As part of multi() request, RegionServer might receive requests with higher rows than row size threshold defined by hbase.rpc.rows.warning.threshold, and we provide warning log. To improve CPU consumption, we should introduce a config that would let RegionServer reject such requests other than providing warning log. This is an attempt to not let any client overburden CPU computation available to RegionServers.