av9ash/GitBugs huggingface dataset:
Bug: [Flakey Tests] TestAsyncRegionAdminApi, TestRegionMergeTransactionOnCluster fixes and debug
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Mar/20 21:46

Add some fixes, some hopeful changes that will hopeful result in fixes, and then a bit of debug:

{code}
 hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteTableProcedure.java
  Edit of log about archiving that shows in middle of a table create;
  try to make it less disorientating.

 hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionAdminApi.java
  Loosen assert. Compaction may have produced a single file only. Allow
  for this.

 hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncTableGetMultiThreaded.java
  Make this test less furious given it is inline w/ a bunch of unit
  tests.

 hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java
  Add debug

 hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaObserverChoreRegionReports.java
  Add wait on quota table to show up before moving forward; otherwise,
  attempt at quota setting fails.

 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
  Debug

 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransactionOnCluster.java
  Remove asserts that expected regions to still have a presence in fs
  after merge when a catalogjanitor may have cleaned up parent dirs.

 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicas.java
  Catch exception on way out and log it rather than let it fail test.

 hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestSnapshotScannerHDFSAclController.java
  Wait on acl table before proceeding.
{code}
Bug: Rubocop configuration needs updated
Status: Open | Priority: Major | Resolution: 
Created: 25/Mar/20 17:00
Affects Version/s: 2.4.0

From https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2565/consoleFull

{noformat}
15:30:40  Running rubocop against identified ruby scripts.
15:30:40  The following cops were added to RuboCop, but are not configured. Please set Enabled to either `true` or `false` in your `.rubocop.yml` file:
15:30:40   - Style/HashEachMethods
15:30:40   - Style/HashTransformKeys
15:30:40   - Style/HashTransformValues
{noformat}

Is out rubocop versions somehow not pinned? HBASE-23943 was just fixed.
Bug: [flakey test] TestAccessController3
Status: Open | Priority: Major | Resolution: 
Created: 25/Mar/20 18:24
Affects Version/s: 2.4.0

Saw this failure on JDK8/Hadoop2, https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2565/

Not much to go on.

{noformat}
java.lang.AssertionError: region server should have aborted due to FaultyAccessController
	at org.apache.hadoop.hbase.security.access.TestAccessController3.tearDownAfterClass(TestAccessController3.java:205)
{noformat}
Bug: [Flakey Test] TestReplicationStatus
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 23/Feb/20 05:59

Fails about 20% of the time even when i run locally. Spent time trying to untangle but this is an awkward one. It is old. Subclasses TestReplicationBase which itself needs cleanup. Tried adding barriers to wait on events but holding up main thread messes up the test. Needs deeper dive.
Bug: Spotbugs: Return value of putIfAbsent is ignored
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 20/Mar/20 20:24

Looks like a regression from HBASE-23561.

[https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1301/2/artifact/yetus-general-check/output/branch-spotbugs-hbase-server-warnings.html]
{quote}
 Return value of putIfAbsent is ignored, but node is reused in org.apache.hadoop.hbase.master.assignment.RegionStates.createRegionStateNode(RegionInfo)
 Bug type RV_RETURN_VALUE_OF_PUTIFABSENT_IGNORED (click for details)
 In class org.apache.hadoop.hbase.master.assignment.RegionStates
 In method org.apache.hadoop.hbase.master.assignment.RegionStates.createRegionStateNode(RegionInfo)
 Called method java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(Object, Object)
 Type org.apache.hadoop.hbase.master.assignment.RegionStateNode
 Value loaded from node
 At RegionStates.java:[line 133]
{quote}
Bug: Turn down flakey rerun rate on all but hot branches
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Mar/20 22:49
Affects Version/s: 1.3.6

Keep flakies running on branch-2.3 and branch-2.2 every four hours.

Have it so flakies run twice a day on master and branch-2/2.4 instead of every 4 hours.

Have it so we run once a day on 2.1, 1.4, and 1.3.

Reporting is done once a day still (untouched).
Bug: [flakey test] TestZooKeeper
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 14/Mar/20 01:09
Affects Version/s: 3.0.0-alpha-1

I observed a test failure in {{TestZooKeeper#testMasterSessionExpired}} on my local rig. On a casual read of the logs from {{testMasterSessionExpired}}, it appears we have a faulty assumption related to master MTTR; the master abort is logged ~1250ms after ZK session close, which seems entirely too fast to me. Once the master aborts, the damage is done and the test cannot recover.

The first re-run passes. Surefire does not keep logs of successful tests, so I don't know the timing between events in the successful run.
Bug: https in xmlns URIs breaks maven-release-plugin:update-versions
Status: Open | Priority: Major | Resolution: 
Created: 18/Mar/20 17:58
Affects Version/s: 2.3.0

Trying to use {{mvn release:update-versions}} to update our version on {{branch-2}}, getting

{noformat}
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-release-plugin:3.0.0-M1:update-versions (default-cli) on project hbase: Execution default-cli of goal org.apache.maven.plugins:maven-release-plugi
n:3.0.0-M1:update-versions failed: The namespace xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" could not be added as a namespace to "project": The namespace prefix "xsi" collides with an additional na
mespace declared by the element -> [Help 1]
{noformat}

Looks like it's because these xmlns values have actual meaning to the plugin, and having https uris breaks things. Removing the https bits allows the plugin to proceed.
Bug: Use loopback for zk standalone server in minizkcluster
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Mar/20 03:37

Use loopback for minizkcluster.  HBASE-23919 shows how sometimes unit tests fail in setup because the minizkcluster won't come up. It looks like client is trying to stat the server but though the server is trying to poll a socket bound to 0.0.0.0, the server never notices the client pings. Using loopback, I don't see these odd failures anymore.
Bug: Coprocessor existence check for READ level privilege
Status: Reopened | Priority: Major | Resolution: 
Created: 18/Mar/20 23:14

Currently getTableDescriptor requires a user to have Admin or Create permissions. A client might need to get table descriptors to act accordingly eg. based on an attribute set or a CP loaded. It should not be necessary for the client to have create or admin privileges just to read the descriptor, execute and/or read permission should be sufficient?
Bug: Provide a harness for running test-patch locally
Status: Open | Priority: Major | Resolution: 
Created: 17/Mar/20 19:46

This is more of brainstorm or wish than prescriptive. It would be really nice to be able to run locally Yetus {{test-patch}} the way we do for PreCommit and Nightly jobs. There are two use-cases I have in mind. It would be useful for folks who want to hack on the build infrastructure without "doing it live". It would also be handy just for running precommit or nightly checks locally, without waiting for shared resources or polluting the public space with experiments.

I imagine a new dev-support tool that uses docker or vagrant to spin up the environment in a way that mimics the Jenkins build pipeline as close as possible. Maybe even uses the {{Jenkinsfiles}} directly with something like [jenkinsfile-runner|https://github.com/jenkinsci/jenkinsfile-runner].
Bug: move_namespaces_rsgroup is not moving namespace into desired rsgroup
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 27/Feb/20 10:47
Affects Version/s: 2.2.3

When creating a namespace and specifying a rs group in hbase.rsgroup.name, the namespace gets associated with the mentioned rs group. However later, the namespace does not move to another rs group when using "move_namespace_rsgroup".

 

Steps to reproduce the issue:
 # create a rs group 'r1' and add a region server in it.
 # *create_namespace 'namespace',\{METHOD => 'set', 'hbase.rsgroup.name' => 'r1'}*
 # *describe_namespace 'namespace'*
 # *move_namespaces_rsgroup 'default',['namespace']*
 # *describe_namespace 'namespace'*

Before moving the namespace into another rs group, it will show rsgroup r1 but even after the step 4, it still shows same description of the namespace. This bug is not allowing me to remove a rs group because it keeps on telling "r1 cannot be deleted as namespace 'namespace' is still associated with the rs group 'r1' ".
Bug: Have test runs use less resources
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 10/Mar/20 15:36

Our tests can create thousands of threads all up in the one JVM. Using less means less memory, less contention, and hopefully, likelier passes.

I've been studying the likes of TestNamespaceReplicationWithBulkLoadedData to see what it does as it runs (this test puts up 4 clusters with replication between). It peaks at 2k threads. After some configuration and using less HDFS, can get it down to ~800 threads and about 1/2 the memory-used.

(HDFS is a profligate offender. DataXceivers (Server and Client), jetty threads, Volume threads (async disk 'worker' then another for cleanup...), image savers, ipc clients -- new thread per incoming connection w/o bound (or reuse), block responder threads, anonymous threads, and so on. Many are not configurable or boundable or are hard-coded; e.g. each volume gets 4 workers. Biggest impact was to be had by downing the count of data nodes. TODO: a follow-on that turns down DN counts in all tests)

I've been using Java Flight Recorder during this study. Here is how you get a flight recorder for the a single test run:
{code:java}
MAVEN_OPTS=" -XX:StartFlightRecording=disk=true,dumponexit=true,filename=recording.jfr,settings=profile,path-to-gc-roots=true,maxsize=1024m"  mvn  test -Dtest=TestNamespaceReplicationWithBulkLoadedData -Dsurefire.firstPartForkCount=0 -Dsurefire.secondPartForkCount=0 {code}
i.e. start recording on mvn launch, bound the size of the recording, and have the test run in the mvn context (DON'T fork). Useful is connecting to the running test at the same time from JDK Mission Control. We do the latter because the thread reporting screen is overwhelmed by the count of running threads and if you connect live, you can at least get a 'live threads' graph w/ count as the test progresses. Useful.

When the test finishes, it dumps a .jfr file which can be opened in JDK MC. I've been compiling w/ JDK8 and then running w/ JDK11 so I can use JDK MC Version 7, the non-commercial latest. Works pretty well.

Let me put up a patch for tests that cuts down thread counts where we can.
Bug: Backport CacheCompactedBlocksOnWrite: HBASE-23066, HBASE-23588, HBASE-23350 to branch-1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/20 16:29
Affects Version/s: 1.6.0

Backport HBASE-23066 and all relevant subtasks to branch-1:

HBASE-23066: Create a config that forces to cache blocks on compaction 

HBASE-23588: Cache index blocks and bloom blocks on write if CacheCompactedBlocksOnWrite is enabled

HBASE-23350: Make compaction files cacheonWrite configurable based on threshold
Bug: TestHFileWriterV3 should have tests for all data block encodings
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jan/20 03:53
Affects Version/s: 1.6.0

TestHFileWriterV3 tests include writing and reading HFiles for default data blocks encoding. However, since this test has efficient way of testing by looping through encoded/un-encoded keys and values and keeping offsets updated etc, it would be really good to have such test for each encoded data blocks. For that, we should explore using respective seekers e.g. RowIndexV1Seeker for ROW_INDEX_V1 encoded data blocks etc.
Bug: Priority configuration for system coprocessors
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jan/20 19:15

Currenty HBase allows operators to set system region coprocessors via hbase-site.xml to be loaded on each table in a cluster (or alternately, all tables but system tables). HBase assumes that the first loaded system coprocessor gets the first, or SYSTEM priority, with each subsequent system coproc getting incremented by 1. As a reminder, in HBase _lower_ priorities go first. 

It can be useful for an operator to be able to define a coprocessor on each table that needs a different priority. For example, an operator might want a coproc to load on each table _last_, so that it can enforce some system invariant and know that no other coproc will interfere with it. 

I propose adding optional priority config to the hbase-site.xml configuration, separated from each coproc class in the comma-separated list by a special character (perhaps a pipe, such as table coprocs use) that's not used in class names. 

The region coprocessor host will parse the priority if present and use it when instantiating the coproc.
Bug: Resolve the TODO of FSTableDescriptor's construct method
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Feb/20 01:43

Resolve the TODO in FSTableDescriptor's construct method.

{code:java}
-   * @param metaObserver Used by HMaster. It need to modify the META_REPLICAS_NUM for meta table descriptor.
-   *                     see HMaster#finishActiveMasterInitialization
-   *                     TODO: This is a workaround. Should remove this ugly code...
{code}
Bug: The method setClusterLoad of SimpleLoadBalancer is incorrect when balance by table 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Mar/20 07:21
Affects Version/s: 2.2.2

now if in parameter clusterLoad is by table, for example
{code:java}
table1=>
     server1=>[table1,region1]
     server2=>[]
table2=>
    server1=>[table2,region1]
    server2=>[]
{code}
then, the member variable serverLoadList is:
{code:java}
[{server1, load 1}{server2, load 0}{server1, load 1} {server2, load 0}]
{code}
the cluster will be considered balanced  in method overallNeedsBalance
Bug: [operator tools] Add forbiddennapis plugin to pom.xml so that we can permanently ban references to hbase I.A. private classes from hbck2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Mar/20 11:50
Affects Version/s: hbase-operator-tools-1.0.0

Following suggestion from [~elserj] on PR for HBASE-23791, this adds _forbiddenapis_ plugin to hbck2 pom.xml file, defining a ban to *MetaTableAccessor.* This also gets rid of one remaining reference to *MetaTableAccessor* on *HBCKMetaTableAccessor* that we missed to detect in HBASE-23791.
Bug: ROW_INDEX_V1 encoder should consider the secondary index size with the encoded data size tracking
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Feb/20 04:48

Copied from parent issue comments

"In case of Row Index this tracking is not accounting the row offsets index being written. Because on the go these are kept in memory only and written at the end of block write. This end block decision was based on size check only which did NOT include this offsets index."
Bug: [Flakey Tests] Down the flakies re-run ferocity; it makes for too many fails.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Mar/20 16:51

The parent issue upped the parallelism of the flakey reruns – see the second panel on this page [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2/lastSuccessfulBuild/artifact/dashboard.html] This upped the flakie list length. It also may have overcommitted the test-running machine. Let me down the rate to something more mild.
Bug: [operator tools] Remove reference to I.A. Private interface MetaTableAccessor
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Feb/20 15:54

While trying to use newly command _extraRegionsInMeta_ added by HBASE-23371, [~daisuke.kobayashi] noticed it was not working properly on some deployments not including another patch merged in HBASE-22758, that changed *MetaTableAccessor* interface:
{noformat}
$ hbase hbck -j hbase-operator-tools-1.0.0.1.0.0.0-11/hbase-hbck2/hbase-hbck2.jar extraRegionsInMeta -f default:cluster_test
Regions that had no dir on the FileSystem and got removed from Meta: 0
ERROR:
	There were following errors on at least one table thread:
java.lang.NoSuchMethodError: org.apache.hadoop.hbase.MetaTableAccessor.deleteRegionInfos(Lorg/apache/hadoop/hbase/client/Connection;Ljava/util/List;) 
{noformat}
Since *MetaTableAccessor* is IA Private, and HBCK2 is aimed to evolve independently of hbase project, ideally we should not rely in IA Private interfaces.

There's already an existing *HBCKMetaTableAccessor* on hbck2, with some *MetaTableAccessor* original methods used in hbck2 re-implemented. This PR removes all dependencies to *MetaTableAccessor* currently existing in hbck2, re-implementing some of the required methods on *HBCKMetaTableAccessor*.

Thanks for finding and reporting it, [~daisuke.kobayashi]!
Bug: Backport HBASE-23553 to branch-2.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Mar/20 08:35

HBASE-23553 is fixed in branch-2.2 but there are some conflicts on branch-2.1 backport. HBASE-23553 is required HBASE-6028(Turning compaction on/off feature), and HBASE-6028 is used only in test-case "TestTableSnapshotScanner#testMergeRegion" for this issue. Thought only to modify HBASE-23553 test-case without HBASE-6028 backport otherwise we may end up with several other conflicts on HBASE-6028 backport.
Bug: Revisit Callable converting all exceptions (including Runtimes) to IOE
Status: Open | Priority: Major | Resolution: 
Created: 28/Feb/20 21:49

This came up in the review of HBASE-23904 where in one location, we swapped out the old Callable for a different idiom. The refactor happened to let out RuntimeExceptions rather than wrap them as we've done up to this. Discussion in the issue wondered if this not the proper behavior. This issue is about deciding if it is the case and if so, then the retrofit of the code base to not wrap REs in IOEs.
Bug: Replace usages of HColumnDescriptor(byte [] familyName) with builder pattern
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Feb/20 11:54
Affects Version/s: 3.0.0-alpha-1

Replace HColumnDescriptor(byte [] familyName) with ColumnFamilyDescriptorBuilder.ModifyableColumnFamilyDescriptor as we need to remove HColumnDescriptor(byte [] familyName) from 3.0.
Bug: Inconsistent library versions notice.
Status: Open | Priority: Major | Resolution: 
Created: 01/Mar/20 12:30

Hi. I have implemented a tool to detect library version inconsistencies. Your project have 1 inconsistent library.

Take junit:junit for example, this library is declared as version 4.13 in hbase-rest, 4.12 in hbase-archetypes/hbase-shaded-client-project and etc... Such version inconsistencies may cause unnecessary maintenance effort in the long run. For example, if two modules become inter-dependent, library version conflict may happen. It has already become a common issue and hinders development progress. Thus a version harmonization is necessary.

Provided we applied a version harmonization, I calculated the cost it may have to harmonize to all upper versions including an up-to-date one. The cost refers to POM config changes and API invocation changes. Take junit:junit for example, if we harmonize all the library versions into 4.13-beta-1. The concern is, how much should the project code adapt to the newer library version. We list an effort table to quantify the harmonization cost.

The effort table is listed below. It shows the overall harmonization effort by modules. The columns represents the number of library APIs and API calls(NA,NAC), deleted APIs and API calls(NDA,NDAC) as well as modified API and API calls(NMA,NMAC). Modified APIs refers to those APIs whose call graph is not the same as previous version. Take the first row for example, if upgrading the library into version 4.13-beta-1. Given that 1375 APIs is used in module hbase-server, 0 of them is deleted in a recommended version(which will throw a NoMethodFoundError unless re-compiling the project), 1 of them is regarded as modified which could break the former API contract.
||Index||Module||NA(NAC)||NDA(NDAC)||NMA(NMAC)||
|1|hbase-server|1375(4528)|0(0)|1(1)|
|2|hbase-common|143(547)|0(0)|0(0)|
|3|hbase-mapreduce|108(389)|0(0)|0(0)|
|4|..|..|..|..|


 
Also we provided another table to show the potential files that may be affected due to library API change, which could help to spot the concerned API usage and rerun the test cases. The table is listed below.




||Module||File||Type||API||
|hbase-it|hbase-it/src/test/java/org/apache/hadoop/hbase/mttr/IntegrationTestMTTR.java|modify|org.junit.Assume.assumeFalse(boolean)|
|hbase-archetypes/hbase-shaded-client-project|hbase-archetypes/hbase-shaded-client-project/src/test/java/org/apache/hbase/archetypes/exemplars/shaded_client/TestHelloHBase.java|modify|org.junit.Assert.assertEquals(java.lang.String,java.lang.Object,java.lang.Object)|
|hbase-server|hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHBaseWalOnEC.java|modify|org.junit.Assume.assumeNoException(java.lang.String,java.lang.Throwable)|
|4|..|..|..|



If you are interested, you can have a more complete and detailed report in the attached PDF file.
Bug: [Flakey Test] TestMasterShutdown#testMasterShutdownBeforeStartingAnyRegionServer
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 27/Feb/20 17:06

{noformat}
java.lang.AssertionError: Found unexpected number of snapshots expected:<2> but was:<1>
	at org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testGetCompletedSnapshotsWithCleanup(TestSnapshotFromMaster.java:313)
{noformat}