av9ash/GitBugs huggingface dataset:
Bug: NPE in MigrateReplicationQueueFromZkToTableProcedure recovery
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 20/Jul/23 04:56
Affects Version/s: 3.0.0-alpha-4

MigrateReplicationQueueFromZkToTableProcedure will restore the disabled state of replication log cleaner barrier when replayed in master recovery,
{code:java}
@Override
protected void afterReplay(MasterProcedureEnv env) {
  if (getCurrentState() == getInitialState()) {
    // do not need to disable log cleaner or acquire lock if we are in the initial state, later
    // when executing the procedure we will try to disable and acquire.
    return;
  }
  if (!env.getReplicationPeerManager().getReplicationLogCleanerBarrier().disable()) {
    throw new IllegalStateException("can not disable log cleaner, this should not happen");
  }
} {code}


however the replicationPeerManager has not been initialized when replay procedure, which will cause a nullPointerException and master to abort.

Maybe better to add a check after the initialization of replicationPeerManager to determine whether replication log cleaner barrier needs to be disabled ?
Bug: Bytes.toInt(byte[] bytes)-why set default offset as zero
Status: Open | Priority: Blocker | Resolution: 
Created: 29/Mar/21 05:55

As we know, when using Bytes.tobytes(int), the method convert the int value to byte-array with big endian.

So, when we try to extract the right int-value from byte-array, we use method 'Bytes.toInt(byte[] bytes)',  and int the method, it sets the offset as zero.

 

it shows like this 

/**
 * Converts a byte array to an int value
 * @param bytes byte array
 * @return the int value
 */
public static int toInt(byte[] bytes) {
 return toInt(bytes, 0, SIZEOF_INT);
}

My question is : 

why don't we set the offset as (bytes.length - SIZEOF_INT)?

 

Recently, we have long-value stored in hbase, as we try to extract, we decode the byte-array as int-value, it leads some incorrect-value like 0.

 

The usage of decoding-int-value from long-value-byte-array is wrong, I'm just curious about the start point of setting offset as 0
Bug: HBASE-24260 breaks JDK7 compilation in branch-1
Status: Resolved | Priority: Blocker | Resolution: Duplicate
Created: 07/Jan/21 03:27

Found in PR-2799. 

{code}
public static String getZKQuorum(Configuration conf) {
    String port =
      Integer.toString(conf.getInt(HConstants.ZOOKEEPER_CLIENT_PORT, 2181));
    String[] serverHosts = conf.getStrings(HConstants.ZOOKEEPER_QUORUM, "localhost");
    for (int i = 0; i < serverHosts.length; i++) {
      serverHosts[i] = serverHosts[i] + ":" + port;
    }
    return String.join(",", serverHosts);
  }
{code}

Fails to compile 
{quote}
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project hbase-it: Compilation failure
[ERROR] /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2799/src/hbase-it/src/main/java/org/apache/hadoop/hbase/chaos/ChaosUtils.java:[46,18] cannot find symbol
[ERROR] symbol:   method join(java.lang.String,java.lang.String[])
[ERROR] location: class java.lang.String
{quote}

because String.join() is only available since JDK8.

CC: [~lokiore] [~ndimiduk]
Bug: Fix NPE in ReplicationSourceManager as well as avoid permanently unreplicated data due to EOFException from WAL
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 22/Feb/21 21:48

There seems to be a major issue with how we handle the EOF exception from WALEntryStream. 

Problem:

When we see EOFException, we try to handle it and remove it from the log queue, but we never try to ship the existing batch of entries. *This is a permanent data loss in replication.*

 

Secondly, we do not stop the reader on encountering the EOFException and thus if EOFException was on the last WAL, we still try to process the WALEntry stream and ship the empty batch with lastWALPath set to null. This is the reason of NPE as below which *crash* the region server. 
{code:java}
2021-02-16 15:33:21,293 ERROR [,60020,1613262147968] regionserver.ReplicationSource - Unexpected exception in ReplicationSourceWorkerThread, currentPath=nulljava.lang.NullPointerExceptionat org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:193)at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceShipperThread.updateLogPosition(ReplicationSource.java:831)at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceShipperThread.shipEdits(ReplicationSource.java:746)at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceShipperThread.run(ReplicationSource.java:650)2021-02-16 15:33:21,294 INFO [,60020,1613262147968] regionserver.HRegionServer - STOPPED: Unexpected exception in ReplicationSourceWorkerThread
{code}
Bug: Handle the NoNode exception in remove log replication and avoid RS crash
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 17/Feb/21 18:30
Affects Version/s: 1.7.0

Should not crash the region server it there is a NoNode exception while removing the log
 We should look into the excpetion and if it is NoNode we shouldn't crash. There might be a possiblity the node was deleted as part of peer tear down.
{code:java}
@Override
public void removeLog(String queueId, String filename) {
try { 
  String znode = ZKUtil.joinZNode(this.myQueuesZnode, queueId); 
  znode = ZKUtil.joinZNode(znode, filename); ZKUtil.deleteNode(this.zookeeper, znode); }
catch (KeeperException e) { 
  this.abortable.abort("Failed to remove wal from queue (queueId=" + queueId + ", filename=" + filename + ")", e); }
}
{code}
This was the exception observed on region servers:
{code:java}
2021-02-16 20:11:58,567 FATAL [95922885,xyz_peer] regionserver.HRegionServer - ABORTING region server regionserver-111,60020,1613495922885: Failed to remove wal from queue (queueId=xyz_peer, filename=regionserver-111%2C60020%2C1613495922885.1613505863058)
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/replication/rs/regionserver-111,60020,1613495922885/xyz_peer/regionserver-111%2C60020%2C1613495922885.16135058630
58
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:114)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:890)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:238)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1341)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1330)
        at org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.removeLog(ReplicationQueuesZKImpl.java:142)
        at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.cleanOldLogs(ReplicationSourceMana
ger.java:232)
        at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.cleanOldLogs(ReplicationSourceMana
ger.java:222)
        at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(Replica
tionSourceManager.java:198)
        at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceShipperThread.updateLogP
osition(ReplicationSource.java:831)
        at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceShipperThread.shipEdits(
ReplicationSource.java:746)
        at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceShipperThread.run(Replic
ationSource.java:650)
{code}
Bug: meta replica state is not respected during active master switch
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 05/Mar/21 19:31
Affects Version/s: 2.0.6

We saw this warning in master log.

WARN org.apache.hadoop.hbase.master.assignment.AssignmentManager: No RegionStateNode for hbase:meta,,1_0003 but reported as up on server1.example.com,16020,1614958467735; closing...

 

The root cause is that meta replica regions are in zookeeper, and these state are not iterated by the new active master so it loses track.
Bug: Scan#setSmall blindly sets ReadType as PREAD
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 08/Mar/21 04:15

setSmall(boolean small) takes a boolean param and it might get called with false also. But with out considering that, we set read type as PREAD.  
Phoenix clones Scan object and do 
newScan.setSmall(scan.isSmall());
So this makes ALL types of scans from phoenix as PREAD type now. (Even full table scan)
Bug: Nightly build uses JDK11 for branch-2.2
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 08/Jan/21 17:41
Affects Version/s: 2.2.7

Nightly build for branch-2.2 is using JDK11 accidentally. Yetus tries to set JAVA_HOME to JDK8 but the path does not exist and it falls back to JDK11.
{noformat}
[2021-01-07T19:11:14.729Z] WARNING: JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 does not exist. Dockermode: attempting to switch to another.
[2021-01-07T19:11:14.729Z] WARNING: Setting /usr/lib/jvm/java-11-openjdk-amd64 as the JAVA_HOME. {noformat}
With JDK11 there are a lot of compilation errors.
Bug: Fix error prone problems on branch-2.2
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 06/Jan/21 12:34
Affects Version/s: 2.2.7

Nightly builds on branch-2.2 do not succeed for more than a month. General checks show a few error prone warnings. These were already corrected on branch-2.3+ (HBASE-23753). A recent error prone upgrade to 2.4.0 on branch-2.2 requires these code fixes.

{noformat}
[ERROR] /home/jenkins/jenkins-home/workspace/HBase_HBase_Nightly_branch-2.2/component/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java:[1347,18] error: [ProtoFieldNullComparison] Protobuf fields cannot be null.
    (see https://errorprone.info/bugpattern/ProtoFieldNullComparison)
  Did you mean 'if ((!proto.getCellList().isEmpty() && !values.isEmpty()) ||'?
[ERROR] /home/jenkins/jenkins-home/workspace/HBase_HBase_Nightly_branch-2.2/component/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionInfoBuilder.java:[122,9] error: [ComparableType] Type of Comparable (RegionInfo) is not the same as implementing class (MutableRegionInfo).
    (see https://errorprone.info/bugpattern/ComparableType)
[INFO] 2 errors 
{noformat}
Bug: Improve the shell's 'status replication' command output
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Mar/21 02:04
Affects Version/s: 2.4.2

The output of the shell's 'status "replication"' command could use some cleaning up.

The basic format is:

version VERSION
N live servers
SERVER_1:
       SOURCE: PeerID=PEERID, AgeOfLastShippedOp=160347, SizeOfLogQueue=49, TimeStampsOfLastShippedOp=Thu Mar 25 01:55:21 UTC 2021, Replication Lag=161013
       SINK  : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Thu Mar 25 01:49:53 UTC 2021
...

Issues:
- Per line format is KEY=VALUE, ... . Most KEYs are CamelCase but some have spaces. Should all be camel case for consistency.
- Age is printed as long and timestamp is string (local TZ) format. I think the string formatting of the timestamp makes it difficult to read. Lets just print timestamps as longs. Or provide a non-default option for string formatted timestamps.
- There is no sense of effective rate or bandwidth. Each source and sink line emitted should provide some sense of current workload: rpcs/sec, bytes/sec, cells/sec... something.
Bug: Size of log queue metric is incorrect in branch-1/branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Mar/21 20:36
Affects Version/s: 1.7.0

In HBASE-25539 I did some refactoring for adding a new metric "oldestWalAge" and tried to consolidate update to all the metrics related to ReplicationSource class (size of log queue and oldest wal age) at one place.  In that refactoring introduced one bug where I am decrementing twice from size of log queue metric whenever we remove a wal from Replication source queue.
Bug: release script generated vote template has incorrect staging area
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Feb/21 03:59

I noticed while making the hbase-thirdparty 3.5.0 RC0 that the generated vote email showed the location of the RC using just the version and RC number:

https://dist.apache.org/repos/dist/dev/hbase/3.5.0RC0/

The correct location needs to include the project name when the release is for something other than the main project, e.g.:

https://dist.apache.org/repos/dist/dev/hbase/hbase-thirdparty-3.5.0RC0/
Bug: Offload the compaction job to independent Compaction Server
Status: Open | Priority: Major | Resolution: 
Created: 30/Mar/21 07:04

The basic idea is add a role "CompactionServer" to take the Compaction job. HMaster is responsible for scheduling the compaction job to different CompactionServer.

[design doc|https://docs.google.com/document/d/1exmhQpQArAgnryLaV78K3260rKm64BHBNzZE4VdTz0c/edit?usp=sharing]

Suggestions are welcomed. Thanks.
Bug: Optimize TableName.valueOf from O(n) to O(1).  We can get benefits when the number of tables in the cluster is greater than dozens
Status: Patch Available | Priority: Major | Resolution: 
Created: 15/Jan/21 03:17
Affects Version/s: 1.2.12

Now, TableName.valueOf will try to find TableName Object in cache linearly(code show as below). So it is too slow when we has  thousands of tables on cluster.
{code:java}
// code placeholder
for (TableName tn : tableCache) {
  if (Bytes.equals(tn.getQualifier(), qns) && Bytes.equals(tn.getNamespace(), bns)) {
    return tn;
  }
}{code}
I try to store the object in the hash table, so it can look up more quickly. code like this
{code:java}
// code placeholder
TableName oldTable = tableCache.get(nameAsStr);{code}
 

In our cluster which has tens thousands of tables. (Most of that is KYLIN table). 
 We found that in the following two cases, the TableName.valueOf method will severely restrict our performance.
  
 Common premise: tens of thousands table in cluster
 cause: TableName.valueOf with low performance. (because we need to traverse all caches linearly)
  
 Case1. Replication
 premise1: one of table write with high qps, small value, Non-batch request. cause too much wal entry

premise2: deserialize WAL Entry includes calling the TableName.valueOf method.

Cause: Replicat Stuck. A lot of WAL files pile up.

 

Case2. Active Master Start up

NamespaceStateManager init should init all RegionInfo, and regioninfo init will call TableName.valueOf.  It will cost some time if TableName.valueOf is slow.
Bug: The delayed FlushRegionEntry should be removed when we need a non-delayed one
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Mar/21 07:56

The regionserver periodically check all the regions, if one not flushed for long time, then it will create a delayed FlushRegionEntry, the delay range is 0~300s.

During the delay time, if many data are written to the region suddenly, we can not do the flush immediately due to the existing one in regionsInQueue, then the RegionTooBusyException will occur.

It is better to improve the logic here, that the delayed entry should be replaced by the non-delayed one.
Bug: Provide a switch that allows avoiding reopening all regions when modifying a table to prevent RIT storms.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Feb/21 04:06
Affects Version/s: 3.0.0-alpha-1

Under normal circumstances, modifying a table will cause all regions belonging to the table to enter RIT. Imagine the following two scenarios：
 # Someone entered the wrong configuration （e.g. negative 'hbase.busy.wait.multiplier.max' value） when altering the table, causing thousands of online regions to fail to open, leading to online accidents.
 # Modify the configuration of a table, but this modification is not urgent, the regions are not expected to enter RIT immediately.

-'alter_lazy' is a new command to modify a table without reopening any online regions except those regions were assigned by other threads or split etc.-

 

Provide an optional lazy_mode for the alter command to modify the TableDescriptor without the region entering the RIT. The modification will take effect when the region is reopened.
Bug: Close region may stuck when region is compacting and skipped most cells read
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Mar/21 08:28
Affects Version/s: 1.7.1

We found in our cluster about stop region stuck. The region is compacting, and its store files has many TTL expired cells. Close region state marker(HRegion#writestate.writesEnabled) is not checked in compaction, because most cells were skipped. 

!RS-region-state.png|width=698,height=310!

 

!Master-UI-RIT.png|width=693,height=157!

 

HBASE-23968 has encountered similar problem, but the solution in it is outer the method

InternalScanner#next(List<Cell> result, ScannerContext scannerContext), which will not return if there are many skipped cells, for current compaction scanner context. As a result, we need to return in time in the next method, and then check the stop marker.
Bug: Should not use a byte array as a key of HashMap or an element of HashSet
Status: Open | Priority: Major | Resolution: 
Created: 13/Feb/21 13:16

I sometimes face the code where we use a byte array as a key of HashMap (ConcurrentHashMap) or an element of HashSet, which could cause very confusing bugs.

The following code is an example where we use a byte array as a key of HashMap:
https://github.com/apache/hbase/blob/326835e8372cc83092e0ec127650438ff153476a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceIdAccounting.java#L94

In Java, we can't use a byte array as a key of HashMap or an element of HashSet directly. One solution for it is to use TreeMap (ConcurrentSkipListMap) or TreeSet (ConcurrentSkipListSet) with Bytes.BYTES_COMPARATOR instead of HashMap and HashSet as follows:

{code}
Map<byte[], String> map1 = new TreeMap<>(Bytes.BYTES_COMPARATOR);
Map<byte[], String> map2 = new ConcurrentSkipListMap<>(Bytes.BYTES_COMPARATOR);

Set<byte[]> set1 = new TreeSet<>(Bytes.BYTES_COMPARATOR);
Set<byte[]> set2 = new ConcurrentSkipListSet<>(Bytes.BYTES_COMPARATOR);
{code}

We should fix the existing ones in this Jira.
Bug: Document suggested settings for PressureAwareCompactionThroughputController
Status: Open | Priority: Major | Resolution: 
Created: 04/Mar/21 17:59
Affects Version/s: 2.3.0

After upgrading a cluster to 2.3, we noticed that compactions were taking much longer. We tracked this down to the {{PressureAwareCompactionThroughputController}}, which became default as part of our upgrade. Our default configuration settings set the lower bound: 50MB/s, upper bound: 100MB/s. These settings make sense if you have a 1gbit NIC (~120 MB/s). For a 10gbit NIC, they’re quite inadequate.

Let's document this component explicitly and mention it as a call-out in the section on upgrading. Bonus points if we can provide suggested configuration values for the common networking configurations.
Bug: When unconfigured, PressureAwareCompactionThroughputController should sniff NIC speed and use appropriate settings
Status: Open | Priority: Major | Resolution: 
Created: 04/Mar/21 18:01

After upgrading a cluster to 2.3, we noticed that compactions were taking much longer. We tracked this down to the PressureAwareCompactionThroughputController, which became default as part of our upgrade. Our default configuration settings set the lower bound: 50MB/s, upper bound: 100MB/s. These settings make sense if you have a 1gbit NIC (~120 MB/s). For a 10gbit NIC, they’re quite inadequate.

Suggestion: instead of using fixed values as the default for these configurations, we should sniff the NIC speed and configure for an optimal value according to available bandwidth.
Bug: [JDK17] reflective access Field.class.getDeclaredField("modifiers") not supported
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jan/21 15:06
Affects Version/s: 2.3.3

The reflective access
{code:java}
Field.class.getDeclaredField("modifiers")
{code}
in HFileSystem.java:334 leads to a warning (and probably error?):

 
{code:java}
java.lang.NoSuchFieldException: modifiers
	at java.base/java.lang.Class.getDeclaredField(Class.java:2417)
	at org.apache.hadoop.hbase.fs.HFileSystem.addLocationsOrderInterceptor(HFileSystem.java:334)
	at org.apache.hadoop.hbase.fs.HFileSystem.addLocationsOrderInterceptor(HFileSystem.java:291)
	at org.apache.hadoop.hbase.fs.HFileSystem.<init>(HFileSystem.java:96)
	at org.apache.hadoop.hbase.fs.HFileSystem.get(HFileSystem.java:465)
	at org.apache.hadoop.hbase.HBaseTestingUtility.getTestFileSystem(HBaseTestingUtility.java:3330)
	at org.apache.hadoop.hbase.HBaseTestingUtility.getNewDataTestDirOnTestFS(HBaseTestingUtility.java:565)
	at org.apache.hadoop.hbase.HBaseTestingUtility.setupDataTestDirOnTestFS(HBaseTestingUtility.java:554)
	at org.apache.hadoop.hbase.HBaseTestingUtility.getDataTestDirOnTestFS(HBaseTestingUtility.java:527)
	at org.apache.hadoop.hbase.HBaseTestingUtility.getDefaultRootDirPath(HBaseTestingUtility.java:1415)
	at org.apache.hadoop.hbase.HBaseTestingUtility.createRootDir(HBaseTestingUtility.java:1446)
	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1157)
	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1144)
	at foo.Main.main(Main.java:11)
{code}
when running the following code:

 
{code:java}
public static void main(String[] args) throws Exception {
    HBaseTestingUtility utility = new HBaseTestingUtility(HBaseConfiguration.create());
    utility.startMiniCluster(StartMiniClusterOption.builder().numRegionServers(3).build());
}{code}
From my knowledge this results from the more restrictive reflection protection of java.base classes in the newer java versions.

 

Related to HBASE-22972
Bug: Sync WAL stuck when prepare flush cache will prevent flush cache and cause OOM
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 31/Mar/21 11:52
Affects Version/s: 1.4.13

We call HRegion#doSyncOfUnflushedWALChanges when preparing to flush cache. But this WAL sync may stuck, and abort the flush of cache. 

!prepare-flush-cache-stuck.png|width=519,height=246!

If we cannot aware of this problem in time, RS will OOM kill.

I think we should force abort RS when sync stuck in preparing, like in committing snapshots.
Bug: RoundRobinTableInputFormat
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Feb/21 21:24

[~huaxiang] figured we needed a round robin on the result returned out TableInputFormat to better distribute task on a production cluster at our place-of-employ.

This seems to be an old problem. See this helpful, old Flipkart blog: https://tech.flipkart.com/is-data-locality-always-out-of-the-box-in-hadoop-not-really-2ae9c95163cb