av9ash/GitBugs huggingface dataset:
Bug: [Flakey Test] TestAsyncTableRSCrashPublish times out VPN enabled
Status: Closed | Priority: Major | Resolution: Fixed
Created: 15/Feb/20 17:09

This is an odd one. If I put up VPN on my local machine, this TestAsyncTableRSCrashPublish test times out. If I take down the VPN, the test passes. This is actually an awkward issue to trace since the udp multicast stuff doesn't log much about its operation. Digging, the VPN adds a utun8 network interface to my box. When this eighth interface comes on line, the TestAsyncTableRSCrashPublish test will select this new interface, which is ipv4, for the master to publish events on, but the clients will choose the seventh utun interface, which is ipv6, for listening. If no vpn, both pick an en?? interface and all just works. I've messed around with the test and tried having the master add the networkinterface its chosen to the configuration, and then had the clients use same network interface, but somehow clients insist on listening on ipv6. No matter how I mess -- even insisting on ipv4 on client and server (see coming attached code), the clients don't get the master broadcast.

Attached code is my hacking trying to make sense of this. Let me make a trivial subtask to add the logging at least in case someone runs into this in prod, they'll have a chance for figuring out mismatched networking (currently there is no way to figure w/o changing code).
Bug: Revert binary incompatible change and remove reflection
Status: Closed | Priority: Major | Resolution: Fixed
Created: 14/Jan/20 16:52
Affects Version/s: 2.1.9

Based on the feedback from Sean on the branch-1 [backport|https://github.com/apache/hbase/pull/1034] of HBASE-23622 the introduced binary incompatible change in  {{ByteRangeUtils}} should be reverted. Also the reflection parts in {{CommonFSUtils}} should be removed.
Bug: Dynamically turn on RingBuffer for slow/large RPC logs
Status: Open | Priority: Major | Resolution: 
Created: 05/Mar/20 17:15
Affects Version/s: 1.7.0

Make hbase.regionserver.slowlog.buffer.enabled dynamically configurable and accordingly turn on/off RingBuffer in RegionServers to start/stop storing complete slow RPC logs for operator to retrieve.
Bug: Move HBaseClusterManager.CommandProvider and it's different implementations as seperate class/interface
Status: Open | Priority: Major | Resolution: 
Created: 03/Jan/20 12:03

HBaseClusterManager.CommandProvider and it's implementations are private to HBaseClusterManager and hbase-it package. 

If any one wants to do custom implementations of ClusterManager Interface, they cant's use this code as it is not visible outside the hbase-it package. 

This Jira is also dependents on [HBASE-23639|https://issues.apache.org/jira/browse/HBASE-23639]
Bug: ZNodeClearer#clear() throws IllegalArgumentException 
Status: Patch Available | Priority: Major | Resolution: 
Created: 09/Jan/20 08:24
Affects Version/s: 2.1.5

Precondition:

hbase.balancer.tablesOnMaster = true

hbase.balancer.tablesOnMaster.systemTablesOnly = true

In case of master-rs collocation HBASE_ZNODE_FILE is overwritten by regionserver process in HRegionServer#handleReportForDutyResponse() here is how it looks on master server:

[hadoop@master-node data]$ cat hbase-hadoop-master.znode 

/hbase/rs/master-node,16000,1578371505720

When ZNodeClearer#clear(), the rsZNode is also splicing with the contents of the file, causing an exception：

java.lang.IllegalArgumentException: Invalid path string "/hbase/rs//hbase/rs/master-node,16000,1578371503003" caused by empty node name specified @10

        at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:99)

        at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:851)

        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:166)

        at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeFailSilent(ZKUtil.java:1259)

        at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeFailSilent(ZKUtil.java:1251)

        at org.apache.hadoop.hbase.ZNodeClearer.clear(ZNodeClearer.java:188)

        at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:144)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)

        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)

        at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3117)
Bug: [Flakey Test] TestExportSnapshotNoCluster.testSnapshotWithRefsExportFileSystemState
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Feb/20 16:51
Affects Version/s: 1.7.0

{{TestExportSnapshotNoCluster.testSnapshotWithRefsExportFileSystemState}} fails with

{noformat}
java.lang.IllegalArgumentException: Wrong FS: file:/home/jenkins/jenkins-slave/workspace/HBase_Nightly_branch-2@2/component/hbase-mapreduce/target/test-data/878a5107-35a3-90ea-50ef-d2a3c32a50dc/.hbase-snapshot/tableWithRefsV1, expected: hdfs://localhost:44609
	at org.apache.hadoop.hbase.snapshot.TestExportSnapshotNoCluster.testSnapshotWithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:110)
	at org.apache.hadoop.hbase.snapshot.TestExportSnapshotNoCluster.testSnapshotWithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:90)
{noformat}
Bug: lack of HMaster parameters in the SnapshotLogCleaner leads to NLP
Status: Open | Priority: Major | Resolution: 
Created: 06/Jan/20 06:20
Affects Version/s: 1.2.11

NullPointException occurs when the SnapshotCleaner calls the init function to initialize the HMaster parameter. Similar to the construction method of HFileCleaner, because the construction method of LogCleaner did not have HMaster parameter, leading to errors. Therefore, the construction method of LogCleaner solves the problem by adding HMaster parameters.

The Log:
{code:java}
java.lang.NullPointerExceptionjava.lang.NullPointerException at org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner.init(SnapshotLogCleaner.java:74) at org.apache.hadoop.hbase.master.cleaner.TestSnapshotAndLogCleaner.testSnapshotLogCleaning(TestSnapshotAndLogCleaner.java:68){code}
Bug: [HBCK2] Expose Multi-Region merge feature via hbck2
Status: Open | Priority: Major | Resolution: 
Created: 11/Feb/20 00:49

HBase Master can do multi-region merge from: 2.0.6,2.1.6,2.2.1,2.3.0,3.0.0

HBASE-22827 aims at exposing the feature via shell from: 2.3.0 & 3.0.0

Let's use this Jira to track, exposing the access to the multi-region merge feature from hbck2 for the versions where shell feature isn't available but the master supports the multi merge feature & hbck service supports mergeRegions api: i.e. 2.1.9+, 2.2.4+ and of course 2.3.0+,3.0.0+ where both shell/hbck2 would be able to do the task for us.
Bug: HBase fails to run on Hadoop 3.3.0/3.2.2/3.1.4 due to jetty version mismatch
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Feb/20 03:52

HBase master branch is currently on Jetty 9.3, and latest Hadoop 3 (unreleased branches trunk, branch-3.2 and branch-3.1) bumped Jetty to 9.4 to address a vulnerability CVE-2017-9735.

(1) Jetty 9.3 and 9.4 are quite different (there are incompatible API changes) and HBase won't start on the latest Hadoop 3.
(2) In any case, HBase should update its Jetty dependency to address the vulnerability.
Fortunately for HBase, updating to Jetty 9.4 requires no code change other than the maven version string.

More tests are needed to verify if HBase can run on older Hadoop versions if its Jetty is updated.
Bug: Add trace support for simple apis in async client
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Feb/20 14:49

The asynchronous hbase client framework (HBASE-16833) was added in HBase 2.0 and doesn't have HTrace traces. Let's add the OpenTracing traces as part of the OT migration project.

===== Update =====
Use OpenTelemetry to implement tracing in async client. And first we will implement tracing for simple APIs, such as get, put and so on. Will file a separated issue to implement trace support for scan related API as it much more complicated.
Bug: [SimpleRpcServer] Subsequent requests will have no response in case of request IO mess up
Status: Open | Priority: Major | Resolution: 
Created: 01/Mar/20 19:35

Code in SimpleServerRpcConnection#readAndProcess work like this
{code:java}
public int readAndProcess() throws IOException, InterruptedException {
  …
  if (data == null) {
    …
    initByteBuffToReadInto(dataLength);
    …
  }
  count = channelDataRead(channel, data);
  if (count >= 0 && data.remaining() == 0) { // count==0 if dataLength == 0
    process();
  }
  return count;
}
{code}
In case of request IO mess up, _data.remaining()_ may be greater than 0, so _process()_ method will not be executed.

There are some cleanup operations in _process()_
{code:java}
private void process() throws IOException, InterruptedException {
  data.rewind();
  try {
    ..
  } finally {
    dataLengthBuffer.clear(); // Clean for the next call
    data = null; // For the GC
    this.callCleanup = null;
  }
}
{code}
If _process()_ not executed, variable _data_ will always not null, and _data.remaining()_ will always be greater than 0, so _process()_ will never be executed again, and subsequent requests will have no response, this has been occured in our product env.
Bug: HBase Bad Substitution ERROR on hadoop-functions.sh
Status: Open | Priority: Major | Resolution: 
Created: 31/Mar/20 23:52
Affects Version/s: 2.2.3

Any Time i make a call to hbase (e.g. "hbase version" or "hbase-daemon.sh start master", i receive this error message:

*/usr/hdeco/hadoop/bin/../libexec/hadoop-functions.sh: line 2366: HADOOP_ORG.APACHE.HADOOP.HBASE.UTIL.GETJAVAPROPERTY_USER: bad substitution*


*/usr/hdeco/hadoop/bin/../libexec/hadoop-functions.sh: line 2461: HADOOP_ORG.APACHE.HADOOP.HBASE.UTIL.GETJAVAPROPERTY_OPTS: bad substitution*

 

"hbase version" does provide version information after this error message, but i am unable to start the hbase master, so i am unable to use hbase further.

 

I have never posted in any forum before, so let me know if more information is needed.
Bug: [OpenTracing] Declare HTrace is unusable in the user doc
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jan/20 22:40

The trace doesn't work at all in HBase 2.0 and above after HBASE-18601 (the trace doesn't get picked up at the server side). We should make a note in the user doc stating it is
 # unusable
 # deprecated in HBase 2.x because HTrace is in Attic.
 # removed from HBase 3.0 and above.
Bug: Change nightly poll from cron @daily to pollSCM @daily; i.e. run nightly if a change ONLY
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Mar/20 21:34

Change build on branch-1.3, 1.4, 2.1, and feature branches  HBASE-23162-branch-1 and   HBASE-22114-branch-1 to be pollSCM @daily -- i.e. poll once a day and if change run nightly -- rather than build every night regardless.

See https://lists.apache.org/thread.html/r5dca2cacc123f2e5719c622add6853ac62b56b2a77885fe0b2eb53c3%40%3Cdev.hbase.apache.org%3E for dev list discussion on downing our nightly load.
Bug: [flakey test] TestVerifyBucketCacheFile
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Mar/20 19:28
Affects Version/s: 3.0.0-alpha-1

I see sporadic failures in this test class. Sometimes a failure on {{assertTrue(file.delete())}}, an inconsistent annoyance. However, this one looks more sinister.

{noformat}
2020-03-12 12:11:35,059 ERROR [Time-limited test] bucket.BucketCache(312): Can't restore from file[/Users/ndimiduk/repos/apache/hbase/hbase-server/target/test-data/5e5c5f5f-d5c2-94b2-8ce9-cf561f4f19f7/bucket.persistence] because of 
java.io.IOException: Mismatch of checksum! The persistent checksum is ���&amp#20;B&amp#16;k�&amp#3;��2�Ӏk, but the calculate checksum is �o���r��w��c��4
	at org.apache.hadoop.hbase.io.hfile.bucket.PersistentIOEngine.verifyFileIntegrity(PersistentIOEngine.java:55)
	at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.parsePB(BucketCache.java:1158)
	at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.retrieveFromFile(BucketCache.java:1106)
	at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.<init>(BucketCache.java:310)
	at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.<init>(BucketCache.java:258)
	at org.apache.hadoop.hbase.io.hfile.bucket.TestVerifyBucketCacheFile.testRetrieveFromFile(TestVerifyBucketCacheFile.java:116)
{noformat}
Bug: New L1 cache : AdaptiveLRU
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/20 08:17

Hi!

I first time here, correct me please if something wrong.

All latest information is here:

[https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]

I want propose how to improve performance when data in HFiles much more than BlockChache (usual story in BigData). The idea - caching only part of DATA blocks. It is good becouse LruBlockCache starts to work and save huge amount of GC.

Sometimes we have more data than can fit into BlockCache and it is cause a high rate of evictions. In this case we can skip cache a block N and insted cache the N+1th block. Anyway we would evict N block quite soon and that why that skipping good for performance.

---

Some information below isn't  actual

---

 

 

Example:

Imagine we have little cache, just can fit only 1 block and we are trying to read 3 blocks with offsets:
 124
 198
 223

Current way - we put the block 124, then put 198, evict 124, put 223, evict 198. A lot of work (5 actions).

With the feature - last few digits evenly distributed from 0 to 99. When we divide by modulus we got:
 124 -> 24
 198 -> 98
 223 -> 23

It helps to sort them. Some part, for example below 50 (if we set *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip others. It means we will not try to handle the block 198 and save CPU for other job. In the result - we put block 124, then put 223, evict 124 (3 actions).

See the picture in attachment with test below. Requests per second is higher, GC is lower.

 
 The key point of the code:
 Added the parameter: *hbase.lru.cache.data.block.percent* which by default = 100
  
 But if we set it 1-99, then will work the next logic:
  
  
{code:java}
public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean inMemory) {   
  if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())      
    if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) 
      return;    
... 
// the same code as usual
}
{code}
 

Other parameters help to control when this logic will be enabled. It means it will work only while heavy reading going on.

hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run eviction process that start to avoid of putting data to BlockCache
 hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to evicted each time that start to avoid of putting data to BlockCache

By default: if 10 times (100 secunds) evicted more than 10 MB (each time) then we start to skip 50% of data blocks.
 When heavy evitions process end then new logic off and will put into BlockCache all blocks again.
  

Descriptions of the test:

4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.

4 RegionServers

4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)

Total BlockCache Size = 48 Gb (8 % of data in HFiles)

Random read in 20 threads

 

I am going to make Pull Request, hope it is right way to make some contribution in this cool product.
Bug: [OpenTracing] Add shaded JaegerTracing tracer to hbase-thirdparty
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 07/Feb/20 20:57

JaegerTracing pulls in lots of dependencies. Some, like libthrift (0.13.0) conflicts the one ships in HBase (0.12.0).

Additionally, not everyone may want to use Jaeger.

I propose to shade JaegerTracing and its dependencies into an uber jar, place it as a hbase-thirdparty artifact. As an added benefit, this makes the management of tracers in the HBase's dependency tree much easier. Finally, we can follow the same suit and provide Zipkin tracer support in the future.
Bug: Backport [HBASE-21874 Bucket cache on Persistent memory] to branch-2.2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jan/20 18:31
Affects Version/s: 2.2.2

We had patch for backporting HBASE-21874 into branch-2.2 attached to that original jira, but it never went through and per discussions there, decided to close it and open later a separate Jira for the backport.
Bug: PreCommit-HBASE-Build ignores the `ATTACHMENT_ID` provided by PreCommit-Admin
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 24/Feb/20 22:27

It appears that we've dropped the parameter {{ATTACHMENT_ID}} from our {{PreCommit-HBASE-Build}} job. (It may be the case that Yetus's {{test-patch}} doesn't support specifying what attachment id to test.) The result, I believe, is that when a Jira issue has patches attached for more than one branch, we only get precommit testing on the patch for master.
Bug: Coverage for Assign and Unassign of Regions on RegionServer on failure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Mar/20 19:32

Looking at 'HBASE-23984 [Flakey Tests] TestMasterAbortAndRSGotKilled fails in teardown', and at UnassignRegionHandler, AssignRegionHandler, CloseRegionHandler, and the work that is done inline w/ request vs that which to the side in executors, we need more coverage and specification of what happens around the edges. This coverage would be more to see if holes in our handling currently in a unit test case context before we see it out on clusters.

HBASE-23984  addresses holes where UnassignRegionHandler and AssignRegionHandler could skip out w/o clearing Regions from the RegionServer#regionsInTransitionInRS Map of Regions In Transition if failed open or close because the RegionServer is aborting.

Other holes seem lurking. On exception, we were leaving entries in the RegionServer# submittedRegionProcedure map added by HBASE-2204; not the end of the world but they should be cleared on error? HBASE-23984 adds clearning from submittedRegionProcedure but then procedures even if failed get added to the cache of procedures... so if we try to run the procedure again against this server it won't be scheduled.

interesting stuff.

This issue is about adding tests that fail assign/unassign/close on the RegionServer side making sure RS state is left in a good condition on fail.
Bug: Improve performance of move_servers_rsgroup and move_tables_rsgroup by using async region move API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Mar/20 15:40

Currently move_servers_rsgroup and move_tables_rsgroup commands and APIs are taking lot of time.
In my test environment, to move a server with 100 regions it takes around 137 seconds.
Similarly it takes around same time to move a table with 100 regions to other group.

The time taken in rsgroup meta update is  negligible. Almost all the time is taken in region moment. This is happening because region is moved serially using  getAssignmentManager().move(region) API

API getAssignmentManager().moveAsync(regionplan)  can be used to move the regions in parallel to improve the performance of region group move servers and tables commands and APIs
Bug: [Flakey Tests] TestServerSideScanMetricsFromClientSide and NullPointerException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Mar/20 15:56

This fails for me locally sporadically. 
{code}
 -------------------------------------------------------------------------------
 Test set: org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide
 -------------------------------------------------------------------------------
 Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.893 s <<< FAILURE! - in org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide
 org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync  Time elapsed: 0.325 s  <<< ERROR!
 java.lang.NullPointerException
{code}

Log pinpoints the problem a little better
{code}
 Exception in thread "testTable.asyncPrefetcher" java.lang.NullPointerException
   at org.apache.hadoop.hbase.client.ClientAsyncPrefetchScanner$PrefetchRunnable.run(ClientAsyncPrefetchScanner.java:183)
   at java.lang.Thread.run(Thread.java:748)
 java.lang.NullPointerException
   at org.apache.hadoop.hbase.client.ClientAsyncPrefetchScanner$PrefetchRunnable.run(ClientAsyncPrefetchScanner.java:174)
   at java.lang.Thread.run(Thread.java:748)
 java.lang.NullPointerException
   at org.apache.hadoop.hbase.client.ClientAsyncPrefetchScanner$PrefetchRunnable.run(ClientAsyncPrefetchScanner.java:174)
   at java.lang.Thread.run(Thread.java:748)
 2020-03-26 22:18:15,160 ERROR [Listener at localhost/53268] hbase.TestServerSideScanMetricsFromClientSide(202): FAIL
 java.lang.NullPointerException
   at org.apache.hadoop.hbase.client.ClientAsyncPrefetchScanner$PrefetchRunnable.run(ClientAsyncPrefetchScanner.java:174)
   at java.lang.Thread.run(Thread.java:748)
{code}

It didn't make sense to me since complaint was about a 'lock' that is created as a final data member. But then this issue happened for my compañero, [~ndimiduk]... and he saw that the call is from a Thread inner class in this class's grandparent that is set running on construction; i.e. the lock *could* be null at the time of reference.
Bug: Backport HBASE-22480 to branch-2.1
Status: Patch Available | Priority: Major | Resolution: 
Created: 08/Mar/20 05:18
Affects Version/s: 2.0.6

My HBase 2.0/2.1 cluster experienced severe performance drop after running reversed scans for weeks plus 1~2 major compaction. When debug into regionserver logs, it appears the bucket cache is full for every ~100 ms and its eviction can't work properly because ref count has become negative: 
{code:java}
(every ~100 ms)
Free started because "Full!"
This block AAA is still referred by -1 readers. Can not be freed now
This block BBB is still referred by -2 readers. Can not be freed now
This block CCC is still referred by -3 readers. Can not be freed now
...
This block ... is still referred by (negative number) readers. Can not be freed now
{code}
There have been patch HBASE-22480 by [~binlijin] addressing this ref count problem in 2.2+. I'd like to backport it to 2.1 (understand that 2.0 is EOL)
Bug: [Flakey Tests] Disable TestClusterScopeQuotaThrottle#testUserNamespaceClusterScopeQuota
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Mar/20 20:40

testUserNamespaceClusterScopeQuota in TestClusterScopeQuotaThrottle is just spewing:

{code}
2020-03-23 13:33:23,996 INFO  [Listener at localhost/50281] hbase.ChoreService(157): Chore ScheduledChore name=QuotaRefresherChore, period=1800000, unit=MILLISECONDS is enabled.
2020-03-23 13:33:25,000 INFO  [Listener at localhost/50281] hbase.ChoreService(157): Chore ScheduledChore name=QuotaRefresherChore, period=1800000, unit=MILLISECONDS is enabled.
2020-03-23 13:33:26,002 INFO  [Listener at localhost/50281] hbase.ChoreService(157): Chore ScheduledChore name=QuotaRefresherChore, period=1800000, unit=MILLISECONDS is enabled.
2020-03-23 13:33:27,003 INFO  [Listener at localhost/50281] hbase.ChoreService(157): Chore ScheduledChore name=QuotaRefresherChore, period=1800000, unit=MILLISECONDS is enabled.
{code}

If I run it standalone it passes. Fails in nightly currently.

Rather than spend time on this, disabling for now. It came in a good while ago as part of HBASE-21820