av9ash/GitBugs huggingface dataset:
Bug: skip empty log cleaner delegate class names (WAS => cleaner.CleanerChore: Can NOT create CleanerDelegate= ClassNotFoundException)
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 31/Jul/20 22:10

2020-07-31 00:19:49,839 WARN [master/ps0753:16000:becomeActiveMaster] cleaner.CleanerChore: Can NOT create CleanerDelegate=

java.lang.ClassNotFoundException:

at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
 at java.lang.Class.forName0(Native Method)
 at java.lang.Class.forName(Class.java:264)
 at org.apache.hadoop.hbase.master.cleaner.CleanerChore.newFileCleaner(CleanerChore.java:173)
 at org.apache.hadoop.hbase.master.cleaner.CleanerChore.initCleanerChain(CleanerChore.java:155)
 at org.apache.hadoop.hbase.master.cleaner.CleanerChore.<init>(CleanerChore.java:105)
 at org.apache.hadoop.hbase.master.cleaner.HFileCleaner.<init>(HFileCleaner.java:139)
 at org.apache.hadoop.hbase.master.cleaner.HFileCleaner.<init>(HFileCleaner.java:120)
 at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1424)
 at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1025)
 at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2189)
 at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:609)
 at java.lang.Thread.run(Thread.java:748)

 

This is the config:

 

<property>
 <name>hbase.master.hfilecleaner.plugins</name>
 <value>
 </value>
 <description>

<property>
Bug: The current checkcompatibility.py script can not compare master and rel/2.0.0
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 03/Jul/21 10:33

There are basically two problems.

One is the findbugs-maven-plugin, on rel/2.0.0, its version is 3.0.0, and it will lead the below error
{noformat}
[ERROR] Failed to execute goal org.codehaus.mojo:findbugs-maven-plugin:3.0.0:findbugs (default) on project hbase: Unable to parse configuration of mojo org.codehaus.mojo:findbugs-maven-plugin:3.0.0:findbugs for parameter pluginArtifacts: Cannot assign configuration entry 'pluginArtifacts' with value '${plugin.artifacts}' of type java.util.Collections.UnmodifiableRandomAccessList to property of type java.util.ArrayList -> [Help 1]
{noformat}

The other is because of opentelemetry. It packs some jar in the fat agent jar, and the name of the directory for holding the extracted jar content still ends .jar, which makes the current script think it is a jar and cause problems.
{noformat}
java.io.FileNotFoundException: /home/zhangduo/hbase/code/target/compat-check/dst/hbase-assembly/target/dependency/opentelemetry-javaagent-1.0.1-all-jar/META-INF/licenses/okhttp-3.14.9.jar (是一个目录)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at java.io.FileInputStream.<init>(FileInputStream.java:93)
        at sun.tools.jar.Main.run(Main.java:307)
        at sun.tools.jar.Main.main(Main.java:1288)
ERROR: can't extract '/home/zhangduo/hbase/code/target/compat-check/dst/hbase-assembly/target/dependency/opentelemetry-javaagent-1.0.1-all-jar/META-INF/licenses/okhttp-3.14.9.jar'
INFO:root:Results: {}
Traceback (most recent call last):
  File "./checkcompatibility.py", line 539, in <module>
    main()
  File "./checkcompatibility.py", line 535, in main
    args.compare_warnings))
  File "./checkcompatibility.py", line 233, in compare_results
    if compare_tool_results_count(tool_results, check, issue_type, known_count)]
  File "./checkcompatibility.py", line 252, in compare_tool_results_count
    return tool_results[check][issue_type] > known_count
KeyError: 'binary'
{noformat}
Bug: [create-release] The indent of checkcompatibility.py is broken after HBASE-25473
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 02/Jul/21 14:43

{noformat}
zhangduo@zhangduo-ubuntu:~/hbase/code/dev-support$ ./checkcompatibility.py 
  File "./checkcompatibility.py", line 251
    return tool_results[check][issue_type] > known_count
                                                       ^
IndentationError: unindent does not match any outer indentation level
{noformat}
Bug: JVM core dump in the replication path
Status: Resolved | Priority: Critical | Resolution: Duplicate
Created: 15/Jul/21 18:00
Affects Version/s: 2.3.5

When replication is turned on, we found the following code dump in the region server. 

I checked the code dump for replication. I think I got some ideas. For replication, when RS receives walEdits from remote cluster, it needs to send them out to final RS. In this case, NettyRpcConnection is deployed, calls are queued while it refers to ByteBuffer in the context of replicationHandler (returned to the pool once it returns). Code dump will happen since the byteBuffer has been reused. Needs ref count in this asynchronous processing.

 

Feel free to take it, otherwise, I will try to work on a patch later.

 

 
{code:java}
Stack: [0x00007fb1bf039000,0x00007fb1bf13a000],  sp=0x00007fb1bf138560,  free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 28175 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.write(Ljava/io/OutputStream;Z)I (21 bytes) @ 0x00007fdbbbb2663c [0x00007fdbbbb263c0+0x27c]
J 14912 C2 org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.writeRequest(Lorg/apache/hbase/thirdparty/io/netty/channel/ChannelHandlerContext;Lorg/apache/hadoop/hbase/ipc/Call;Lorg/apache/hbase/thirdparty/io/netty/channel/ChannelPromise;)V (370 bytes) @ 0x00007fdbbb94b590 [0x00007fdbbb949c00+0x1990]
J 14911 C2 org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.write(Lorg/apache/hbase/thirdparty/io/netty/channel/ChannelHandlerContext;Ljava/lang/Object;Lorg/apache/hbase/thirdparty/io/netty/channel/ChannelPromise;)V (30 bytes) @ 0x00007fdbb972d1d4 [0x00007fdbb972d1a0+0x34]
J 30476 C2 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.write(Ljava/lang/Object;ZLorg/apache/hbase/thirdparty/io/netty/channel/ChannelPromise;)V (149 bytes) @ 0x00007fdbbd4e7084 [0x00007fdbbd4e6900+0x784]
J 14914 C2 org.apache.hadoop.hbase.ipc.NettyRpcConnection$6$1.run()V (22 bytes) @ 0x00007fdbbb9344ec [0x00007fdbbb934280+0x26c]
J 23528 C2 org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(J)Z (106 bytes) @ 0x00007fdbbcbb0efc [0x00007fdbbcbb0c40+0x2bc]
J 15987% C2 org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run()V (461 bytes) @ 0x00007fdbbbaf1580 [0x00007fdbbbaf1360+0x220]
j  org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run()V+44
j  org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run()V+11
j  org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run()V+4
{code}
Bug: Sum of multiplier of cost functions is not populated properly when we have a shortcut for trigger
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 29/Sep/21 21:41

We have a couple of scenarios that we force balancing:
 * idle servers
 * co-hosted regions

The code path quit before populating the sum of multiplier of cost functions. This causes wrong value reported in logging. As below, the weighted average is not divide by total weight. This causes inconsistent log among iterations.
{quote}2021-09-24 21:46:57,881 INFO org.apache.hadoop.hbase.master.balancer.S*tocha*sticLoadBalancer: Running balancer because at least one server hosts replicas of the same region.

2021-09-24 21:46:57,881 INFO org.apache.hadoop.hbase.master.balancer.S*tocha*sticLoadBalancer: Start S*tocha*sticLoadBalancer.balancer, initial weighted average imbalance=6389.260497305375, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06659036267913739); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05296760285663541); MoveCostFunction : (multiplier=7.0, imbalance=0.0, balanced); ServerLocalityCostFunction : (multiplier=25.0, imbalance=0.46286750487559114); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.2569525347374165); TableSkewCostFunction : (multiplier=500.0, imbalance=0.3760689783169534); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0553889913899139); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.05854089790897909); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.06969346106898068); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.07834116112410174); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.12533769793201735); StoreFileCostFunction : (multiplier=5.0, imbalance=0.06921401085082914);  computedMaxSteps=5577401600
{quote}
Bug: AbstractFSWALProvider#getArchivedLogPath doesn't look for wal file in all oldWALs directory.
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 21/Jul/21 14:48
Affects Version/s: 2.4.4

Below is the code for AbstractFSWALProvider#getArchivedLogPath

{code:java}
 public static Path getArchivedLogPath(Path path, Configuration conf) throws IOException {
    Path rootDir = CommonFSUtils.getWALRootDir(conf);
    Path oldLogDir = new Path(rootDir, HConstants.HREGION_OLDLOGDIR_NAME);
    if (conf.getBoolean(SEPARATE_OLDLOGDIR, DEFAULT_SEPARATE_OLDLOGDIR)) {
      ServerName serverName = getServerNameFromWALDirectoryName(path);
      if (serverName == null) {
        LOG.error("Couldn't locate log: " + path);
        return path;
      }
      oldLogDir = new Path(oldLogDir, serverName.getServerName());
    }
    Path archivedLogLocation = new Path(oldLogDir, path.getName());
    final FileSystem fs = CommonFSUtils.getWALFileSystem(conf);

    if (fs.exists(archivedLogLocation)) {
      LOG.info("Log " + path + " was moved to " + archivedLogLocation);
      return archivedLogLocation;
    } else {
      LOG.error("Couldn't locate log: " + path);
      return path;
    }
  }
{code}

This method is called from the following places.
[AbstractFSWALProvider#openReader|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractFSWALProvider.java#L524]

[ReplicationSource#getFileSize|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java#L399]

[WALInputFormat.WALRecordReader#nextKeyValue|https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/WALInputFormat.java#L220]

All of the above calls are trying to find the log in archive path after they couldn't locate the wal in walsDir and they are not used for moving a log file to archive directory.
But we will look for archive path within serverName directory only if conf key is true.
Cc [~zhangduo]
Bug: SEEK_NEXT_USING_HINT is ignored on reversed Scans
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 27/Aug/21 17:22
Affects Version/s: 2.4.5

It turns out that the {{SEEK_NEXT_USING_HINT}} feature is broken for reversed scans. That's because the implementation hard-codes an assumption that the scan is forward.

Here's the mistake: [https://github.com/keyserbrian1/hbase/blob/735bcf85e9b59a71981babe2f5da51978d61d8d3/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java#L740|https://github.com/keyserbrian1/hbase/blob/735bcf85e9b59a71981babe2f5da51978d61d8d3/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java#L740]

The bug sneaks past the existing tests because the mistake isn't detectable by looking at the result of a naive reversed scan.

Attached is a PR that includes a fix and tests designed to detect if filter hints are actually used.
Bug: Splitted WAL files are deleted rather than archiving
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 25/Aug/21 13:51
Affects Version/s: 2.2.7

When procedure based WAL split is enabled, we are deleting splitted WAL files immediately rather than moving to archive path.

 https://github.com/apache/hbase/blob/67e9c6422d6d5b39586ffe529f55b76d8b8939ff/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/SplitWALRemoteProcedure.java#L99
Bug: New replication gets stuck or data loss when multiwal groups more than 10
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 26/Jul/21 02:57
Affects Version/s: 1.7.1

{code:java}
void preLogRoll(Path newLog) throws IOException {
  recordLog(newLog);
  String logName = newLog.getName();
  String logPrefix = DefaultWALProvider.getWALPrefixFromWALName(logName);
  synchronized (latestPaths) {
    Iterator<Path> iterator = latestPaths.iterator();
    while (iterator.hasNext()) {
      Path path = iterator.next();
      if (path.getName().contains(logPrefix)) {
        iterator.remove();
        break;
      }
    }
    this.latestPaths.add(newLog);
  }
}
{code}
ReplicationSourceManager use _latestPaths_ to track each walgroup's last WALlog and all of them will be enqueue for replication when new replication  peer added。

If we set hbase.wal.regiongrouping.numgroups > 10, says 12, the name of WALlog group will be _regionserver.null0.timestamp_ to _regionserver.null11.timestamp_。*_String.contains_* is used in _preoLogRoll_ to replace old logs in same group, leads when _regionserver.null1.ts_ comes, _regionserver.null11.ts_ may be replaced, and *_latestPaths_ growing with wrong logs*.

Replication then partly stuckd as _regionsserver.null1.ts_ not exists on hdfs, and data may not be replicated to slave as _regionserver.null11.ts_ not in replication queue at startup.

Because of [ZOOKEEPER-706|https://issues.apache.org/jira/browse/ZOOKEEPER-706], if there is too many logs in zk _/hbase/replication/rs/regionserver/peer_, remove_peer may not delete this znode, and other regionserver can't not pick up this queue for replication failover.
Bug: Resolve dependency conflicts of hbase-endpoint third-party libraries
Status: Open | Priority: Major | Resolution: 
Created: 19/Jul/21 11:29
Affects Version/s: 2.0.0

Hi, Our project uses “hbase-endpoint”, but the protobuf version is 2.5 version, which conflicts with 3.0 in our project. I want to merge into a "hbase-shaded-endpoint" submission to resolve "hbase-endpoint" dependency issue of version 2.0
Bug: Allow split when store file count larger than the configured blocking file count
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Aug/21 07:12
Affects Version/s: 1.7.1

Currently, region will not split when the number of store files is up to the configed blocking count, by `hbase.hstore.blockingStoreFiles`.
The relevant codes are as follows, 

the CompactSplit#requestSplit() (called by the MemstoreFlusher and CompactionRunner) checks the compaction priority of the region, if the compact priority < PRIORITY_USER, the region will not split.
{code:java}
public synchronized boolean requestSplit(final Region r) {
  // don't split regions that are blocking
  HRegion hr = (HRegion)r;
  try {
    if (shouldSplitRegion() && hr.getCompactPriority() >= PRIORITY_USER) {
      byte[] midKey = hr.checkSplit().orElse(null);
      if (midKey != null) {
        requestSplit(r, midKey);
        return true;
      }
    }
.... {code}
But the region's compact priority is the minimum of all the stores, when the number of storefiles in a store is larger than the configed `hbase.hstore.blockingStoreFiles`, the compact priority will be a negative number, while PRIORITY_USER = 1.
{code:java}
public int getStoreCompactionPriority() {
  int priority = blockingFileCount - storefiles.size();
  return (priority == HStore.PRIORITY_USER) ? priority + 1 : priority;
} {code}
As a result, when a region size is up to the split limit, but its speed of reducing the number of files through compaction is slower than the speed of generating new files(e.g. compacting L0 files to stripes, bulk load, flush memstore), the region will never split. 

The problem is obvious in StripeStoreEngine, though memstore flushing is pending when store file count up to the blocking count, each L0 compaction may generate the stripe count new files to each stripe. And in this scenario, since the store always compact priority to split, the stripe count is larger and larger, the new files generated by compact is more and more, no split in the end. While split can divide the compaction pressure(1 parent compaction + 2 children compaction can be reduced to 2 children compaction).

We can add a configuration to enable split when blocking, not only keep the origin behavior but also support flexible control.
Bug: Introduce a MigrationStoreFileTracker to support migrating from different store file tracker implementations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Aug/21 15:58

As mentioned in this section in the design doc:

https://docs.google.com/document/d/16Nr1Fn3VaXuz1g1FTiME-bnGR3qVK5B-raXshOkDLcY/edit#heading=h.vv41957yf4i
Bug: Persist the StoreFileTracker configurations to TableDescriptor when creating table
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Sep/21 10:36

As discussed in this section in the design doc:

https://docs.google.com/document/d/16Nr1Fn3VaXuz1g1FTiME-bnGR3qVK5B-raXshOkDLcY/edit#heading=h.78r2mdeyquug

If we use different SFT implementation at master side and region server side, it is likely to cause data loss, which is a very serious misconfiguration problem.

A possible solution is to make sure that master and region server always load the configurations about StoreEngine from the same place. To archive this, a possible way is to always set the StoreEngine configurations to the TableDescriptor, even if user does not explicitly set it when creating a table.

And also, when upgrading, we should check whether the existing tables have StoreEngine configurations, if not, we need to set them.
Bug: Should find a suitable way to let users specify the store file tracker implementation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Sep/21 13:55

Now all the implementations are marked as IA.Private, so there is no safe way for users to specify them.

But for simplify, maybe we should not expose the full class name of the implementation classes to users. Just follow the way in WALFactory, to introduce an alias name for the implemention, for example

DEFAULT, FILE, REGION, etc.

Thoughts?
Bug: Add more checks to prevent misconfiguration on store file tracker
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Sep/21 15:31

For example, when creating new tables, if the store file tracker is configured to MigrationStoreFileTracker, we should fail the creation and tell users you should not use this store file tracker for a new table.

And when switching the implementation for store file tracker, we should let end users first switch to MigrationStoreFileTracker, and then to the final target store file tracker. If end users configure wrong store file tracker, we should fail the alter table operation.
Bug: Add support for specifying store file tracker when restoring or cloning snapshot
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Sep/21 16:04

As discussed in HBASE-26280.

https://issues.apache.org/jira/browse/HBASE-26280?focusedCommentId=17414894&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17414894
Bug: The region replication framework should not be built upon the general replication framework
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Aug/21 06:59

At least, at the source path, where we track the edits, we should not make region replication rely on general replication framework.

The difficulty here for switching to a table based storage is that, the WAL system and replication system highly depend on each other. There will be cyclic dependency if we want to store replication peer and queue data in a hbase table.
And after HBASE-18070, even meta wal provider will be integrated together with replication system, which makes things more difficult.

But in general, for region replication, it is not a big deal to lose some edits, a flush can fix everything, which means we do not so heavy tracking system in the general replication system.

We should find a more light-weighted way to do region replication.
Bug: Use P2P communicate between region servers to sync the list for bootstrap node
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Aug/21 15:09

For returning bootstrap nodes to client, we do not need to always track exactly the current live region server. But now, we just set a watcher on zookeeper to watch all the live region servers, on every region server, for large cluster, it will be a big pressure for zookeeper.

So here I think, we could let region servers to communicate with each other to sync the available bootstrap node, like a P2P network.

Will put a simple design doc soon.
Bug: Provide coprocessor hooks for updateConfiguration and clearRegionBlockCache
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Sep/21 22:02

Sorry for the vague title, not sure if it'd be better to create individual jiras. I discovered that the following two AdminService endpoints are not covered by AccessController:

 
 * updateConfiguration
 * clearRegionBlockCache

There may be others, I haven't done a full audit. We should add coprocessor hooks for these and wrap with AccessController, especially since they can both have an affect on the cluster.
Bug: Deprecate MasterRegistry
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Aug/21 09:37

Maybe in some environment we still want to use master as registry endpoint, but this should be controlled at cluster side, not client side.

 

The implementation of MasterRegistry is almost the same with RpcConnectionRegistry except that it uses getMasters instead of getBootstrapNodes to refresh the ‘bootstrap’ nodes connected to. So we could add configs in server side to control what nodes we want to return to client in getBootstrapNodes, i.e, master or region server, then the RpcConnectionRegistry can fully replace the old MasterRegistry. As part of this change, we deprecate the MasterRegistry.

==== Update ====
We send out an email to mailing list, and there is no objections on not returning masters as registry endpoints, so here we decide to just deprecated MasterRegistry.
Bug: [hbase-connectors] Pushdown filters in Spark do not work correctly with long types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Aug/21 14:40
Affects Version/s: 1.0.0

Reading from an HBase table and filtering on a LONG column does not seem to work correctly.

{{Dataset<Row> df = spark.read()
   .format("org.apache.hadoop.hbase.spark")
   .option("hbase.columns.mapping", "id STRING :key, v LONG cf:v")
   ...
   .load();
 df.filter("v > 100").show();}}

Expected behaviour is to show rows where cf:v > 100, but instead an empty dataset is shown.

Moreover, replacing {{"v > 100"}} with {{"v >= 100"}} results in a dataset where some rows have values of v less than 100. 

The problem appears to be that long values are decoded incorrectly as integers in {{NaiveEncoder.filter}}:

{{case LongEnc | TimestampEnc =>
   val in = Bytes.toInt(input, offset1)
   val value = Bytes.toInt(filterBytes, offset2 + 1)
   compare(in.compareTo(value), ops)}}

It looks like that error hasn’t been caught because {{DynamicLogicExpressionSuite}} lack test cases with long values.

The erroneous code is also present in the master branch. We have extended the test suite and implemented a quick fix and will PR on GitHub.