av9ash/GitBugs huggingface dataset:
Bug: ReplicationSource should clear buffer usage on ReplicationSourceManager upon termination
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Aug/20 19:20
Affects Version/s: 2.2.6

Following investigations on the issue described by [~elserj] on HBASE-24779, we found out that once a peer is removed, thus killing peers related *ReplicationSource* instance, it may leave *ReplicationSourceManager.totalBufferUsed* inconsistent. This can happen if *ReplicationSourceWALReader* had put some entries on its queue to be processed by *ReplicationSourceShipper,* but the peer removal killed the shipper before it could process the pending entries. When *ReplicationSourceWALReader* thread add entries to the queue, it increments *ReplicationSourceManager.totalBufferUsed* with the sum of the entries sizes. When those entries are read by *ReplicationSourceShipper,* *ReplicationSourceManager.totalBufferUsed* is then decreased. We should also decrease *ReplicationSourceManager.totalBufferUsed* when *ReplicationSource* is terminated, otherwise those unprocessed entries size would be consuming *ReplicationSourceManager.totalBufferUsed __*indefinitely, unless the RS gets restarted. This may be a problem for deployments with multiple peers, or if new peers are added.**
Bug: checkAndPut (or checkAndMutate) might return false when the row is mutated successfully
Status: Open | Priority: Major | Resolution: 
Created: 28/Sep/20 15:55
Affects Version/s: 1.2.11

In the client, when the MutateRequest times out, we retry the operation in org.apache.hadoop.hbase.client.RpcRetryingCallerImpl#callWithRetries. If the server received the request but the client failed to get a response, the server returns processed=false  because the value is already there. So the value false is returned, even though the checkAndPut was successful in the first attempt. It should return true if processed=false because it's already been processed AND it is a retry operation.

 

Example RpcRetryingCallerImpl inside checkAndPut:

                       request

client  -------{-}{{-}}o{{-}}{-}--------> server

  

client  --------------------> server (processing)

 

client  --------------------> server (processed)

(timed out)

 

                      retry request

client  -------{-}{{-}}o{{-}}{-}--------> server

  

client  --------------------> server (processing)

 

client   ------------------> server (already processed)

 

                     response (processed = false)

client  <-----{-}{{-}}o{{-}}{-}------- server

 

checkAndPut returns false, even though it's successful.

 

In 2.1.0, I could only reproduce it three times by accident, but it always happened in 1.2.11. In 2.1.0, I could only reproduce it systematically by cleaning the response with the debugger before it got to the hconnection thread.

Repro steps
 * Create a breakpoint in the exception in org.apache.hadoop.hbase.client.RpcRetryingCallerImpl#callWithRetries to make sure we get the exception and retry.
 * Create an infinite loop to create different rows with checkAndPut.
 * Start running with the disabled breakpoints.
 * Enable the breakpoint.
 * Pause all threads and verify that we are waiting for a response in the IPC thread. Wait for 1-2 minutes. This will cause a timeout.

 * Continue and verify that an exception is triggered.
 * Add a breakpoint to verify the response.
 * Continue and check the response and the returned value.
Bug: HBase on Cloud Blob FS : Provide config to skip HFile archival while table deletion 
Status: Open | Priority: Major | Resolution: 
Created: 04/Jul/20 07:42

When we delete a table as part of delete of table from FS, we do below things
1. Rename to table directory to come under /hbase/.tmp. This is an atomic rename op
2. Go through each of HFiles under every region:cf and archive that one by one. (Rename the file from .tmp path to go to /hbase/archive)
3. Delete the table dir under .tmp dir

In case of HDFS this is not a big deal as every rename op is just a meta op (Though the HFiles archival is a costly only as there will be so many calls to NN based the table's regions# and total storesfiles#)  But on Cloud blob based FS impl, this is a concerning op. Every rename will be a copy blob op. And we are doing it twice per each of the HFiles in this table !

The proposal here is to provide a config option (default to false) to skip this archival step.
We can provide another config to even avoid the .tmp rename? The atomicity of the Table delete can be achieved by HM side procedure and proc WAL. In table delete the 1st step is to delete the table form META anyways
Bug: Auto scale RSGroup
Status: Open | Priority: Major | Resolution: 
Created: 24/Jul/20 07:45
Affects Version/s: 3.0.0-alpha-1

In current use, if RSs go offline or online, we must manually move RSs in or out RSGroups.

Now we can configure how many servers rsgroups need base on HBASE-24431 , and then add an AutoScaleChore to periodically check and move servers.
Bug: Purge protobuf java 2.5.0 dependency
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jul/20 13:35

On master branch, we have removed the hbase-protocol module so in general, we do not need to depend on protobuf 2.5.0 directl. Especially for hadoop 3.3.0, hadoop will not depend on 2.5.0 any more, we should make sure hbase do not introduce protobuf 2.5.0 too.
Bug: Clean up peer metrics when disabling peer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jul/20 07:35
Affects Version/s: 2.2.5

  Supposed that we have an peer with id 1,  when execute shell cmd disable_peer  '1'  and enable_peer '1', then i can see the SizeOfLogQueue metric of all regionservers   +1 ,  after 10 times disable_peer ops  , it will increase to 11, and  it will never decrease to 1 in fulture .

  I can see the function ReplicationSourceManager.refreshSources(peerId) is called , it will terminate the previous replication source and create a new one.  and found the note  //Do not clear metrics in the bellow code block:
{code:java}
ReplicationSourceInterface toRemove = this.sources.put(peerId, src);
if (toRemove != null) {
  LOG.info("Terminate replication source for " + toRemove.getPeerId());
  // Do not clear metrics
  toRemove.terminate(terminateMessage, null, false);
}
{code}
 this cause the wrong number of sizeOfLogQueue, i think it's a sub issue of (HBASE-23231)
Bug: make sure the next hbase 1.y release has Hadoop 2.10 as a minimum version
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Sep/20 19:37

Our reference guide list of prerequisites still has Hadoop 2.8 and 2.9 listed for HBase 1 releases.

* [hadoop 2.8 is EOM|https://lists.apache.org/thread.html/r348f7bc93a522f05b7cce78a911854d128a6b1b8bd8124bad4d06ce6%40%3Cuser.hadoop.apache.org%3E]
* [hadoop 2.9 is EOM|https://lists.apache.org/thread.html/r16b14cce9504f7a9d228612c6b808e72d8dd20863c78be51a7e04ed5%40%3Cuser.hadoop.apache.org%3E]

The current list in the reference guide for HBase 1.6 is just the 1.5 list copied. we should update it to remove 2.8 and 2.9 and make sure we're no longer doing build/test based on those versions for branch-1.
Bug: Clear cached service kerberos ticket in case of SASL failures thrown from server side
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jul/20 02:13

We setup a SASL connection using different mechanisms like Digest, Kerberos from master to RS for various activities like region assignment etc. In case of SASL connect failures, we try to dispose of the SaslRpcClient and try to relogin from the keytab on the client side. However the relogin from keytab method doesn't clear off the service ticket cached in memory unless TGT is about to expire within a timeframe.

This actually causes an issue where there is a keytab refresh that happens because of expiry  on the RS server and throws a SASL connect error when Master reaches out to the RS server with the cached service ticket that no longer works with the new refreshed keytab. We might need to clear off the service ticket cached as there could be a credential refresh on the RS server side when handling connect failures
Bug: Improve performance of SKIP vs SEEK logic
Status: Closed | Priority: Major | Resolution: Fixed
Created: 15/Jul/20 00:27
Affects Version/s: 1.7.0

In our testing of HBase 1.3 against the current tip of branch-1 we saw a 30% slowdown in scanning scenarios.

We tracked it back to HBASE-17958 and HBASE-19863.
Both add comparisons to one of the tightest HBase has.

[~bharathv]
Bug: Ignore TestAsyncTableRSCrashPublish
Status: Closed | Priority: Major | Resolution: Fixed
Created: 12/Aug/20 06:47

[ERROR] Failures: 
[ERROR] TestAsyncTableRSCrashPublish.test:94 Waiting timed out after [60,000] msec

 

I meet this failure many times when runAllTests. And other developers meet this too when vote RC. Let's ignore this first and enable this after parent issue resolved.
Bug: Direct insert HFiles and Persist in-memory HFile tracking
Status: Closed | Priority: Major | Resolution: Implemented
Created: 21/Jul/20 05:22
Affects Version/s: 3.0.0-alpha-1

We propose a new feature (a new store engine) to remove the {{.tmp}} directory used in the commit stage for common HFile operations such as flush and compaction to improve the write throughput and latency on object stores. Specifically for S3 filesystems, this will also mitigate read-after-write inconsistencies caused by immediate HFiles validation after moving the HFile(s) to data directory.

Please see attached for this proposal and the initial result captured with 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN, and workload C RUN result.

The goal of this JIRA is to discuss with the community if the proposed improvement on the object stores use case makes senses and if we miss anything should be included.

Improvement Highlights
 1. Lower write latency, especially the p99+
 2. Higher write throughput on flush and compaction 
 3. Lower MTTR on region (re)open or assignment 
 4. Remove consistent check dependencies (e.g. DynamoDB) supported by file system implementation
Bug: Small Updates to Functionality of Shell IRB Workspace
Status: Closed | Priority: Major | Resolution: Fixed
Created: 31/Jul/20 13:38
Affects Version/s: 3.0.0-alpha-1

There are a few amendments I want to make to the first patch for shell IRB workspaces:
 # Hide the new warning: "irb: warn: can't alias help from irb_help."
 # Split Shell::Shell#eval_io into eval_io and exception_handler. This will be a better separation of concerns for both usage and testing.
 ## Why is this change so important? At the moment, eval_io may raise SystemExit, which would cause the ruby test executor to quit without running all tests. The method eval_io also used to refer to a global variable $fullTraceback, which is a poor separation of concerns.
 # Allow finding script2run in the load path. While undocumented, the 2.x shell did this, so we may need to do this for compatibility.
Bug: Generalize MasterRegistry, etc  to support both Master and Regionserver
Status: Open | Priority: Major | Resolution: 
Created: 15/Sep/20 08:02

Creating a jira to capture a discussion in [HBASE-11288|https://issues.apache.org/jira/browse/HBASE-11288?focusedCommentId=17194403&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17194403]. 

The discussion came out of avoiding the master from being part of the read/write path. 

As per [~andrew.purtell@gmail.com]

"Generalizing the work to "bootstrap servers" that could be operating in either of master or regionserver roles seems fine but please let's make that a new JIRA or at least its own subtask."

This is currently a place holder if HBASE-11288 implementation of split meta goes in as it would not make sense to add it otherwise. If it does I will get to implementing this then.
Bug: Edit of log messages around async WAL Replication; checkstyle fixes; and a bugfix
Status: Closed | Priority: Major | Resolution: Fixed
Created: 18/Sep/20 19:43

Edit of  logging around region replicas: shortening and adding context.

Checkstyle fixes in edited files while I was in there.

Bug fix in AssignRegionHandler – was using M_RS_CLOSE_META to open
 a Region instead of a M_RS_OPEN_META.

 

Main reason for this issue is making the substantial adjacent  issue HBASE-25055 smaller in size/easier to review.
Bug: Add ReplicationSource for meta WALs; add enable/disable when hbase:meta assigned to RS
Status: Closed | Priority: Major | Resolution: Fixed
Created: 17/Sep/20 05:32

Add ReplicationSource that feeds on hbase:meta WAL files. Add enabling this source when hbase:meta is opened and hbase:meta region replicas are configured ON. Disable the source when the hbase:meta Region moves away.
Bug: Allow configuring WALEntry filters on ReplicationSource
Status: Closed | Priority: Major | Resolution: Fixed
Created: 04/Aug/20 20:54
Affects Version/s: 2.4.0

The parent issue is about enabling memstore replication of meta Regions. As-is, the ReplicationSource is hardcoded to filter out hbase:meta WALEntries; they are not forwarded for Replication.

This issue is all internals making it so can create an instance of ReplicationSource with a different base set of WALEntry filters. We also add a means of filtering WALs by path name.
Bug: A tool to rewrite corrupted HFiles
Status: Resolved | Priority: Major | Resolution: Feedback Received
Created: 20/Aug/20 20:22

Typically I have been dealing with corrupted HFiles (due to loss of hdfs blocks) by just removing them. However, It always seemed wasteful to throw away the entire HFile (which can be hundreds of gigabytes), just because one hdfs block is missing (128MB).

I think there's a possibility for a tool that can rewrite an HFile by skipping corrupted blocks. 

There can be multiple types of issues with hdfs blocks but any of them can be treated as if the block doesn't exist:
1. All the replicas can be lost
2. The block can be corrupted due to some bug in hdfs (I've recently run into HDFS-15186 by experimenting with EC).

At the simplest the tool can be a local mapreduce job (mapper only) with a custom HFile reader input that can seek to next DATABLK to skip corrupted hdfs blocks.
Bug: Table regions details on master GUI display slowly.
Status: Resolved | Priority: Major | Resolution: Abandoned
Created: 16/Sep/20 01:49
Affects Version/s: 2.3.2

When a table has a large number of regions (e.g.,a single table contains more than 100,000 regions), it takes about 20 to 30 minutes to display the table regions on the master GUI, which is unacceptable to users. After testing, we find that web page rendering takes up the most time,and this can be solved by pagination query.
Bug: Bulk load performance is degraded in HBase 2 
Status: Open | Priority: Major | Resolution: 
Created: 22/Jul/20 05:02
Affects Version/s: 2.2.3

in our Test,It is observed that Bulk load performance is degraded in HBase 2 .

 Test Input: 

1: Table with 500 region(300 column family)

2:  data =2 TB

Data Sample

18600000001201502051000000068110,18600000001,20150205,5,404,735412,2938,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111111111111111111111111111111111111111111111111111111111111111111111111111111111

3: Cluster: 7 node(2 master+5 Region Server)

 4: No of Container Launched are same in both case

HBase 2 took 10% more time then HBase 1.3 where test input is same for both cluster

 
|Feature|HBase 2.2.3
 Time(Sec)|HBase 1.3.1
 Time(Sec)|Diff%|Snappy lib:
  |
|BulkLoad|21837|19686.16|-10.93|Snappy lib:
 HBase 2.2.3: 1.4
 HBase 1.3.1: 1.4|
Bug: HBase RSGroup: Table region details under RS Groups are in the onder fromat like HBase 1.x
Status: Open | Priority: Major | Resolution: 
Created: 20/Aug/20 17:32
Affects Version/s: 2.2.3

HBase RSGroup: Table region details under RS Groups are in the older format like HBase 1.x.

 

in HBase2, Master UI will contain table details

 
 
 
||Namespace||Name||State||Regions||Description||
||OPEN||OPENING||CLOSED||CLOSING||OFFLINE||FAILED||SPLIT||Other||

but under Regroup, table region details are different which are same as in HBase1.x

 
||Namespace||Table||Stats||Online Regions||Offline Regions||Failed Regions||Split Regions||Other Regions||Description||
Bug: Optimize BufferNode Lock
Status: Patch Available | Priority: Major | Resolution: 
Created: 27/Aug/20 11:44
Affects Version/s: 2.2.3

[https://github.com/apache/hbase/blob/047e0618d290a09a4a269b00548fe17691e31787/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.java#L373]

during startup, a large number of OpenRegionProcedures are generated, which are added to the BufferNode. However, the BufferNode has some "synchronized" methods, These methods may affect MTTR
Bug: reduce invalid subprocedure task
Status: Patch Available | Priority: Major | Resolution: 
Created: 27/Aug/20 08:20
Affects Version/s: 2.2.3

[https://github.com/apache/hbase/blob/047e0618d290a09a4a269b00548fe17691e31787/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/snapshot/RegionServerSnapshotManager.java#L165]

[https://github.com/apache/hbase/blob/047e0618d290a09a4a269b00548fe17691e31787/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/flush/RegionServerFlushTableProcedureManager.java#L146]

 

if involvedRegions is null or empty, rs should skip subprocedure.
Bug: the RSGroupBasedLoadBalancer#retainAssignment throws NPE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Sep/20 01:37
Affects Version/s: 2.2.3

when BaseLoadBalancer#
https://github.com/apache/hbase/blob/8bfa2cb2eedcf050b26a28961e1b77dbf3cd8c95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java#L1433

If the result of the BaseLoadBalancer#retainAssignment is null, the RSGroupBasedLoadBalancer#retainAssignment will throw NPE.

https://github.com/apache/hbase/blob/8bfa2cb2eedcf050b26a28961e1b77dbf3cd8c95/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java#L206
Bug: WAL replay should ignore 0-length files
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/20 19:58
Affects Version/s: 2.3.1

I overdrove a small testing cluster, filling HDFS. After cleaning up data to bring HBase back up, I noticed all masters -refused to start- abort. Logs complain of seeking past EOF. Indeed the last wal file name logged is a 0-length file. WAL replay should gracefully skip and clean up such an empty file.

{noformat}
2020-09-16 19:51:30,297 ERROR org.apache.hadoop.hbase.master.HMaster: Failed to become active master
java.io.EOFException: Cannot seek after EOF
        at org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1448)
        at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:66)
        at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initInternal(ProtobufLogReader.java:211)
        at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initReader(ProtobufLogReader.java:173)
        at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:64)
        at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:168)
        at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:323)
        at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:305)
        at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:293)
        at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:429)
        at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:4859)
        at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:4765)
        at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:1014)
        at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:956)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7496)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegionFromTableDir(HRegion.java:7454)
        at org.apache.hadoop.hbase.master.region.MasterRegion.open(MasterRegion.java:269)
        at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:309)
        at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104)
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:949)
        at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2240)
        at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:622)
        at java.base/java.lang.Thread.run(Thread.java:834)
{noformat}
Bug: Reject to add a peer which replicate to itself earlier
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jul/20 03:15

Now there are one check in ReplicationSource#initialize method
{code:java}
// In rare case, zookeeper setting may be messed up. That leads to the incorrect
// peerClusterId value, which is the same as the source clusterId
if (clusterId.equals(peerClusterId) && !replicationEndpoint.canReplicateToSameCluster()) {
  this.terminate("ClusterId " + clusterId + " is replicating to itself: peerClusterId "
      + peerClusterId + " which is not allowed by ReplicationEndpoint:"
      + replicationEndpoint.getClass().getName(), null, false);
  this.manager.removeSource(this);
  return;
}
{code}
This check should move to AddPeerProcedure's precheck.
Bug: Generic NamedQueue framework for recent in-memory history
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jul/20 18:31

As per the discussion on parent jira, we should come up with named queue (online ring buffer) to serve recent history for multiple use-cases like slowlog, balancer decision, other region activities e.g flush, compaction, split, merge etc.

Since we already have slow/large rpc logs in ring buffer (HBASE-22978), as part of this Jira, the proposal is to refactor slowlog provider to get generic payload for ring buffer and based on event type (slow_log is the only one for now), we can have separate internal in-memory queues.

After this refactor, it should be relatively simpler to use the same framework and create more cases like parent Jira (balancer decision in ring buffer).