av9ash/GitBugs huggingface dataset:
Bug: java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI
Status: Open | Priority: Major | Resolution: 
Created: 15/Jun/21 04:10
Affects Version/s: 2.3.5

Hi All,

We have Hadoop 3.2.2 and HBase 2.3.5 Versions installed. (java version "1.8.0_291")

 

While bringing up HBase master, I'm seeing following error messages in HBase master log.

 

Other HDFS. clients like Spark,MapReduce,Solr etc are able to write HDFS but HBase is unable to write its meta files in HDFS with following exceptions.

 

> Summary of Error logs from hbase master 

2021-06-15 03:57:45,968 INFO [Thread-7] hdfs.DataStreamer: Exception in createBlockOutputStream
 java.io.IOException: Invalid token in javax.security.sasl.qop: ^DD
 2021-06-15 03:57:45,939 WARN [Thread-7] hdfs.DataStreamer: Abandoning BP-1583998547-10.10.10.3-1622148262434:blk_1073743393_2570
 2021-06-15 03:57:45,946 WARN [Thread-7] hdfs.DataStreamer: Excluding datanode DatanodeInfoWithStorage[10.10.10.3:50010,DS-281c3377-2bc1-47ea-8302-43108ee69430,DISK]
 2021-06-15 03:57:45,994 WARN [Thread-7] hdfs.DataStreamer: DataStreamer Exception
 org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /hbase/data/data/hbase/meta/.tmp/.tableinfo.0000000001 could only be written to 0 of the 1 minReplication nodes. There are 3 datanode(s) running and 3 node(s) are excluded in this operation.
 2021-06-15 03:57:46,023 INFO [Thread-9] hdfs.DataStreamer: Exception in createBlockOutputStream
 java.io.IOException: Invalid token in javax.security.sasl.qop: ^DDI
 2021-06-15 03:57:46,035 INFO [Thread-9] hdfs.DataStreamer: Exception in createBlockOutputStream
 java.io.IOException: Invalid token in javax.security.sasl.qop: ^DD
 2021-06-15 03:57:46,508 ERROR [main] regionserver.HRegionServer: Failed construction RegionServer
 java.io.IOException: Failed update hbase:meta table descriptor
 2021-06-15 03:57:46,509 ERROR [main] master.HMasterCommandLine: Master exiting
 java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster.
 Caused by: java.io.IOException: Failed update hbase:meta table descriptor

 

Not sure what is the root cause behind this. Any comments/suggestions on this is much appreciated.
Bug: [JDK17] Add a JDK17 profile
Status: Resolved | Priority: Major | Resolution: Implemented
Created: 30/Jun/21 05:59

While HBase builds fine with JDK17, tests fail because a number of Java SDK modules are no longer exposed to unnamed modules by default. We need to open them up.

Without which, the tests fail for errors like:
{noformat}
[ERROR] Tests run: 6, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 0.469 s <<< FAILURE! - in org.apache.hadoop.hbase.rest.model.TestNamespacesModel
[ERROR] org.apache.hadoop.hbase.rest.model.TestNamespacesModel.testBuildModel  Time elapsed: 0.273 s  <<< ERROR!
java.lang.ExceptionInInitializerError
        at org.apache.hadoop.hbase.rest.model.TestNamespacesModel.<init>(TestNamespacesModel.java:43)
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make protected final java.lang.Class java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int) throws java.lang.ClassFormatError accessible: module java.base does not "opens java.lang" to unnamed module @56ef9176
        at org.apache.hadoop.hbase.rest.model.TestNamespacesModel.<init>(TestNamespacesModel.java:43)
{noformat}
Bug: Fix Jersey for hbase-server processes
Status: Reopened | Priority: Major | Resolution: 
Created: 03/May/21 23:12
Affects Version/s: 2.5.0

I spent some time trying to use Jersey from within the Master and it's not working. To summarize, we have unshaded resources from both jersey-server-1.19 and jersey-server-2.32 on the hbase-server classpath. Jersey's initialization uses ServiceLoader to look up concrete implementation classes of {{javax.ws.rs}} classes at runtime. Because we do not shade {{javax.ws.rs}} in hbase-thirdparty-jersey, an attempt to use shaded jersey-2.x still results in loading unshaded jersey-1.x jars, leading to an error like this

{noformat}
java.lang.AbstractMethodError: javax.ws.rs.core.UriBuilder.uri(Ljava/lang/String;)Ljavax/ws/rs/core/UriBuilder;
	at javax.ws.rs.core.UriBuilder.fromUri(UriBuilder.java:96)
	at org.apache.hbase.thirdparty.org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:275)
	at org.apache.hbase.thirdparty.org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
	at org.apache.hbase.thirdparty.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
	at org.apache.hbase.thirdparty.org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
	at org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:112)
{noformat}

We cannot override what version of these classes are loaded at runtime via Java property because Jersey's load order implementation checks system properties as a last resort, not first thing as is claimed by javadoc.

So I can think of two solutions.
# One is to shade {{javax.ws.rs}} in hbase-thirdparty-jersey. This would shade both the interfaces and the resource files that are referenced at runtime, allowing for an entirely isolated jersey container instantiate.
# Another idea is to add a custom {{ClassLoader}} that is inserted before jersey is initialized. This would filter out resources that are "banned", allowing our desired implementation through.

Between these, I think (1) is better, but I don't know what else might break. I've made an effort of both, but with neither approach can I get a jersey environment to response from my resource class... either because the solution is incomplete, or because I don't have the jersey environment configured properly.

See also some discussion that happened over on Slack, https://apache-hbase.slack.com/archives/C13K8NVAM/p1618857521051700
Bug: msg=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized
Status: Open | Priority: Major | Resolution: 
Created: 25/Apr/21 18:51

Hi Team,

I am getting error "apache.hadoop.hbase.PleaseHoldException Master is initializing".

When In have checked the status of region server with status 'detailed' command then getting same region server as a live node and dead also.

I have tried to removing /hbase-unsecure and to restart hbase master and region server but no luck.

Even, I have restarted  Hbase services in sequence( ZK, region, Master) multiple times but still the issue didn't resolve. 

 

[main] client.RpcRetryingCallerImpl: Call exception, tries=7, retries=36, started=8209 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
 at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2977)
Bug: RSGroupBasedLoadBalancer online status never updates after being set to true for the first time
Status: Open | Priority: Major | Resolution: 
Created: 26/Apr/21 16:24

Once the RSGroupBasedLoadBalancer is “online” (it has found the hbase:meta and hbase:rsgroup tables), it will never update the status again. That means if hbase:meta or hbase:rsgroup ever go offline, the balancer doesn’t update its status to “offline,” so some of the code paths will go through the “online” code path even though the catalog tables aren’t available to be read from or written to (in particular, anything that calls RSGroupInfoManagerImpl#flushConfig).

Also, in the RSGroupInfoManagerImpl#flushConfig code path, the call to write to hbase:rsgroup comes before the update to the rsGroupMap and tableMap which are stored in memory (see order of [these lines of code|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java#L664-L670]), so if hbase:rsgroup goes offline after the RSGroupBasedLoadBalancer is already marked as “online,” exceptions thrown while trying to write to an offline hbase:rsgroup table prevent the in-memory rsGroupMap and tableMap from being updated. In terms of the order just mentioned, in-memory state should be updated first.
Bug: Improve efficiency of SnapshotHFileCleaner
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/May/21 09:06
Affects Version/s: 2.0.0

We have met same problems of thousands threads in HBASE-22867, but after this issue, the cleaner becomes more inefficient.

From the jstack we can see that most dir-scan threads are blocked at SnapshotHFileCleaner#getDeletableFiles,
{code:java}
"dir-scan-pool-19" #694 daemon prio=5 os_prio=0 tid=0x0000000002ab1800 nid=0x26a7e waiting for monitor entry [0x00007fb0a9913000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner.getDeletableFiles(SnapshotHFileCleaner.java:74)
        - waiting to lock <0x00007fb148737048> (a org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner)
        at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteFiles(CleanerChore.java:498)
        at org.apache.hadoop.hbase.master.cleaner.CleanerChore.lambda$traverseAndDelete$1(CleanerChore.java:246)
        at org.apache.hadoop.hbase.master.cleaner.CleanerChore$$Lambda$41/1187372779.act(Unknown Source)
        at org.apache.hadoop.hbase.master.cleaner.CleanerChore.deleteAction(CleanerChore.java:358)
        at org.apache.hadoop.hbase.master.cleaner.CleanerChore.traverseAndDelete(CleanerChore.java:246)
        at org.apache.hadoop.hbase.master.cleaner.CleanerChore.lambda$null$2(CleanerChore.java:255)
        at org.apache.hadoop.hbase.master.cleaner.CleanerChore$$Lambda$38/2003131501.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745){code}
and all the HFileCleaner threads are waiting at the delete tasks queue,
{code:java}
"gha-data-hbase0002:16000.activeMasterManager-HFileCleaner.large.2-1621210982419" #358 daemon prio=5 os_prio=0 tid=0x00007fb967fc0000 nid=0x266f2 waiting on condition [0x00007fb0c57d6000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007fb1486db9f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:106)
        at org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:264)
        at org.apache.hadoop.hbase.master.cleaner.HFileCleaner$1.run(HFileCleaner.java:233)
{code}
So it's need to increase the speed of scanning files. But since the getDeletableFiles is a synchronized method, increasing the number of scan-dir threads can not solve this problem. 

After looking through the codes in SnapshotHFileCleaner and SnapshotFileCache, I think the lock granularity in them should be optimized.
Bug: javadoc generation fails on openjdk-11.0.11+9
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jun/21 14:49
Affects Version/s: 2.4.3

I'm trying to build javadoc for HBase 2.4.3 on jdk11. The command I'm running is as follows:
{code:java}
JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.11.0.9-2.el8_4.x86_64/  mvn -Phadoop-3.0 -Phadoop.profile=3.0 -Dhadoop-three.version=3.2.2 -Dhadoop.guava.version=27.0-jre -Dslf4j.version=1.7.25 -Djetty.version=9.3.29.v20201019 -Dzookeeper.version=3.5.7 -DskipTests -Dcheckstyle.skip=true site{code}
I've tried this with maven 3.5.4 and 3.6.2. Based on JAVA_HOME above, jdk11.0.11+9.

{{The error is as follows:}}
{code:java}
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-site-plugin:3.7.1:site (default-site) on project hbase: Error generating maven-javadoc-plugin:3.2.0:aggregate-no-fork report:
 [ERROR] Exit code: 1 - javadoc: warning - The old Doclet and Taglet APIs in the packages
 [ERROR] com.sun.javadoc, com.sun.tools.doclets and their implementations
 [ERROR] are planned to be removed in a future JDK release. These
 [ERROR] components have been superseded by the new APIs in jdk.javadoc.doclet.
 [ERROR] Users are strongly recommended to migrate to the new APIs.
 [ERROR] javadoc: error - invalid flag: -author
 [ERROR]
 [ERROR] Command line was: /usr/lib/jvm/java-11-openjdk-11.0.11.0.9-2.el8_4.x86_64/bin/javadoc -J-Xmx2G @options @packages
 [ERROR]
 [ERROR] Refer to the generated Javadoc files in '/hbase/rpm/build/BUILD/hbase-2.4.3/target/site/apidocs' dir.
 [ERROR] -> [Help 1]
 [ERROR]
 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
 [ERROR] Re-run Maven using the -X switch to enable full debug logging.
 [ERROR]
 [ERROR] For more information about the errors and possible solutions, please read the following articles:
 [ERROR] [Help 1]{code}
I believe this is due to the yetus doclet org.apache.yetus.audience.tools.IncludePublicAnnotationsStandardDoclet. Commenting this doclet out from the userapi and testuserapi reportSets in pom.xml fixes the build.

 
 I noticed hbase 2.4.3 depends on audience-annotations 0.5.0, which is very old. I tried updating to 0.13.0, but that did not help.
Bug: Add flexibility to backup replication in case replication filter throws an exception
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/May/21 06:00
Affects Version/s: 1.6.0

There may be situations when the wal entry filter might result in some temporary issues but expected to recover at some point in time. In this case, we should have an option to backup replication and retry until the wal entry filter recovers instead of just aborting the region server.
Bug: Close PENDING_OPEN regions make regions FAILED_CLOSE
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 16/Apr/21 13:11
Affects Version/s: 1.7.1

There is a case on our production cluster,
 # RS1 crash;
 # bulk assign some RS1 regions to RS2, including region A;
 # RS2 crash;
 # bulk assign RS2 regions to RS3, including region A;
 # the first assign of A to RS2 failed and reassign A;
 # close A on RS3, but A is PENDING_OPEN on RS3;
 # A FAILED_CLOSE;
Bug: Setting NAMESPACES when adding a replication peer still requires scope definition at CF level
Status: In Progress | Priority: Major | Resolution: 
Created: 01/Jun/21 10:15
Affects Version/s: 2.3.5

A common ask from our customers is for an easy way to enable replication for all tables within a namespace, but we had noticed that setting *NAMESPACES* option when adding a peer has no effect on enabling replication for the defined namespaces tables, it still replicates only entries where the *CF REPLICATION_SCOPE* is set to '1'.

The problem is due to the chaining of *ScopeWALEntryFilter* and *NamespaceTableCfWALEntryFilter* together. This is currently done on *BaseReplicationEndpoint* class, but these two filters are not compatible, as *ScopeWALEntryFilter.filterCells* logic filters out cells from CFs where REPLICATION_SCOPE property is set to 0, before *NamespaceTableCfWALEntryFilter.filterCells* can have a chance to apply it's own logic, based on namespace definition.

-This PR changes the logic described above, making  *ScopeWALEntryFilter* and *NamespaceTableCfWALEntryFilter* mutual exclusive, allowing for entries from all tables within the defined namespaces to be replicated without needing to have *REPLICATION_SCOPE* set to true.-

Changing the original approach: Adding boolean operators to *ChainWALEntryFilter,* allowing operators to define either *AND/OR* junction logic for the chained filters.
Bug: Move StoreFlushContext out of HStore and make it pluggable
Status: Closed | Priority: Major | Resolution: Fixed
Created: 24/May/21 14:50
Affects Version/s: 3.0.0-alpha-1

Currently, StoreFlushContext is directly implemented and instantiated inside HStore class. This implementation assumes hfiles are always flushed into temp dir first, and its commit implementation moves these files into the actual family dir. In order to allow for the direct flushes (no temp, nor renames), we need to make StoreFlushContext implementations pluggable in HStore.
Bug: Row commit sequencer
Status: Open | Priority: Major | Resolution: 
Created: 05/Jun/21 22:38

Use a row commit sequencer in HRegion to ensure that only the operations that mutate disjoint sets of rows are able to commit within the same clock tick. This maintains the invariant that more than one mutation to a given row will never be committed in the same clock tick.

Callers will first acquire row locks for the row(s) the pending mutation will mutate. Then they will use RowCommitSequencer.getRowSequence to ensure that the set of rows about to be mutated do not overlap with those for any other pending mutations in the current clock tick. If an overlap is identified, getRowSequence will yield and loop until there is no longer an overlap and the caller's pending mutation can succeed.
Bug: Introduce EnvironmentEdge.currentTimeAdvancing
Status: Resolved | Priority: Major | Resolution: Abandoned
Created: 25/May/21 01:23

Introduce new {{EnvironmentEdge#currentTimeAdvancing}} which ensures that when the current time is returned, it is the current time in a different clock tick from the last time the {{EnvironmentEdge}} was used to get the current time.

When processing mutations we substitute the {{Long.MAX_VALUE}} timestamp placeholder with a real placeholder just before committing the mutation. The current code gets the current time for timestamp substitution while under row lock and mvcc. We will simply use {{EnvironmentEdge#currentTimeAdvancing}} instead of {{EnvironmentEdge#currentTime}} at this point in the code to ensure we have seen the clock tick over. When processing a batch of mutations (doMiniBatchMutation etc) we will call {{currentTimeAdvancing}} only once. This means the client cannot bundle cells with wildcard timestamps into a batch where those cells must be committed with different timestamps. Clients must simply not submit mutations that must be committed with guaranteed distinct timestamps in the same batch. Easy to understand, easy to document, and it aligns with our design philosophy of the client knows best.

It is not required to handle batches as proposed. We could guarantee a distinct timestamp for every mutation in a batch. Count the number of mutations, call this M. Acquire all row locks and get the current time. Then, wait for at least M milliseconds. Then, set the first mutation timestamp with this value and increment by 1 for all remaining. Then, do the rest of mutation processing as normal. I don't think this extra waiting to reserve the range of timestamps is necessary. See reasoning in above paragraph. Mentioned here for sake of discussion.

It will be fine to continue to use {{EnvironmentEdge#currentTime}} everywhere else. In this way we will only potentially spin wait where it matters, and won't suffer serious overheads during batch processing.
Bug: Backport HBASE-22120 (Replace HTrace with OpenTelemetry) to branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/May/21 15:14
Affects Version/s: 2.5.0

list of commits from upstream/master need to be backported 

{code}
HBASE-25373 https://github.com/apache/hbase/commit/302d9ea8b888762a5a20a5ba5c2be7bc239afaef 
HBASE-25401 https://github.com/apache/hbase/commit/242028671535dfed67ec13d9ed0d6067f3ccfd04 
HBASE-25424 https://github.com/apache/hbase/commit/57960fa8fa7228d65b1a4adc8e9b5b1a8158824d 
HBASE-23898 https://github.com/apache/hbase/commit/805b2ae2ad0f6325515d46043ff01e4e2c7a9f59 
HBASE-25454 https://github.com/apache/hbase/commit/dcb78bd4bda4a4ae13d863df8aec266031e5bc93
HBASE-25481 https://github.com/apache/hbase/commit/ae2c62ffaad5ba4c976b0a79c10a365edf2844fd
HBASE-25455 https://github.com/apache/hbase/commit/03e12bfa4ad62ecc6eee6a2c68d431bea2d5c473
HBASE-25484 https://github.com/apache/hbase/commit/2be2c63f0d3917a243b74af9754cbfc805b858d1
HBASE-25535 https://github.com/apache/hbase/commit/bb8c4967f8ce2c89ebaf1ddc5d8a1bf55f1e20d3
HBASE-25591 https://github.com/apache/hbase/commit/f6ff519dd0c7fe0e3ae3c175eefee27a26a065a4
HBASE-25617 https://github.com/apache/hbase/commit/8d68f8cd1c8613be1b499eaa99f46806b2743294
HBASE-25616 https://github.com/apache/hbase/commit/8399293e21127df3ffdcb757242e4cb5964c7e99
HBASE-25723 https://github.com/apache/hbase/commit/7f90c2201f6a17d2e2d031505c35ae7c2b1ed7ea
HBASE-25732 https://github.com/apache/hbase/commit/8df9bebdd367d52a32b08c18a7cf4f9c2d712071
HBASE-25733 https://github.com/apache/hbase/commit/b71488998970a3353086a34736ed1edab527f673
HBASE-23762 https://github.com/apache/hbase/commit/be4503d9f82f044fbfce21c8a42d0b1684607238
HBASE-25778 https://github.com/apache/hbase/commit/f36e1539648bbaee84c626fd54d1605baebf3c5a
{code}
Bug: ReadOnlyZKClient APIs - CompletableFuture.get() calls can cause threads to hang forver when ZK client create throws Non IOException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/May/21 12:02

This is applicable for zk client versions which is not having fix for ZOOKEEPER-2184.
Now we are on zookeeper 3.5.7 on active 2.x branches. Still its better to handle this case in our code.
Bug: TestSlowLogAccessor.testHigherSlowLogs repeatable failure
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 28/Apr/21 22:31
Affects Version/s: 2.4.3

{noformat}
 [ERROR] TestSlowLogAccessor.testHigherSlowLogs:211 Waiting timed out after [7,000] msec{noformat}

Repeatable failure.
Bug: Support Read/Write request size cost functions in StochasticLoadBalancer
Status: Open | Priority: Major | Resolution: 
Created: 02/Apr/21 06:59
Affects Version/s: 1.7.1

Currently, StochasticLoadBalancer has several cost functions to reflect pressure of regions, they are Read/Write/CP RequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction. Memstore and storefile cost functions are used to reflect the write io pressure. But when some regions has high read io pressures, balancer can not aware of it and then make some actions.

Like request count cost functions in StochasticLoadBalancer, we could also support request size cost functions. Because in some circumstances, request size  is the most important factor reflecting the pressure of RSes.