av9ash/GitBugs huggingface dataset:
Bug: TestJMXConnectorServer fails with BindException
Status: Open | Priority: Major | Resolution: 
Created: 07/Jun/21 16:17

Test needs to accommodate port clashes as TestJMXListener et al. does.

{code}
2021-06-05 05:13:47,401 ERROR [master/d1cb58dfe2a6:0:becomeActiveMaster] coprocessor.CoprocessorHost(416): The coprocessor org.apache.hadoop.hbase.JMXListener threw java.rmi.server.ExportException: Port already in use: 49932; nested exception is: 
	java.net.BindException: Address already in use (Bind failed)
java.rmi.server.ExportException: Port already in use: 49932; nested exception is: 
	java.net.BindException: Address already in use (Bind failed)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:335)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:243)
	at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:412)
	at java.rmi/sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147)
	at java.rmi/sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:234)
	at java.rmi/sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:220)
	at java.rmi/sun.rmi.registry.RegistryImpl.<init>(RegistryImpl.java:205)
	at java.rmi/java.rmi.registry.LocateRegistry.createRegistry(LocateRegistry.java:203)
	at org.apache.hadoop.hbase.JMXListener.startConnectorServer(JMXListener.java:134)
	at org.apache.hadoop.hbase.JMXListener.start(JMXListener.java:209)
	at org.apache.hadoop.hbase.coprocessor.BaseEnvironment.startup(BaseEnvironment.java:72)
	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.checkAndLoadInstance(CoprocessorHost.java:283)
	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadSystemCoprocessors(CoprocessorHost.java:168)
	at org.apache.hadoop.hbase.master.MasterCoprocessorHost.<init>(MasterCoprocessorHost.java:158)
	at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:897)
	at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2091)
	at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:507)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.BindException: Address already in use (Bind failed)
	at java.base/java.net.PlainSocketImpl.socketBind(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:436)
	at java.base/java.net.ServerSocket.bind(ServerSocket.java:395)
	at java.base/java.net.ServerSocket.<init>(ServerSocket.java:257)
	at java.base/java.net.ServerSocket.<init>(ServerSocket.java:149)
	at java.rmi/sun.rmi.transport.tcp.TCPDirectSocketFactory.createServerSocket(TCPDirectSocketFactory.java:45)
	at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:670)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:324)
	... 17 more
{code}
Bug: Shell alter command (modify operation) sometimes times out
Status: Open | Priority: Major | Resolution: 
Created: 05/Jun/21 04:42
Affects Version/s: 2.1.4

Execution of an alter command like the following on a non-interactive shell sometimes times out:
{code:java}
echo "alter 'aeris_v2', {NAME=>'e', VERSIONS=>'43200', MIN_VERSIONS=>'3', TTL=>'7776000'}, NANOSECOND_TIMESTAMPS => 'true'" | hbase shell -n{code}
A similar issue was reported in [HBASE-21795|https://issues.apache.org/jira/browse/HBASE-21795], but based on its description it should be resolved on the HBase versions we are running (2.1.4, 2.2.4).

The hbasemaster and regionserver logs attached are relevant to one occurrence of the issue when the command started at 2021-05-29 05:25 and timed out 10 minutes later at 2021-05-29 05:35.

Note that the affected table appeared to be modified even though the shell command timed out.

The hbase shell debug output attached is from another similar failure where we managed to toggle the shell d flag to capture extra debug information. The capture indicates the zookeeper client session being closed at the 1-minute mark and the command failing at the 10-minute mark with a timeout.

If there is a way to output even more debugging information (e.g. stack traces) from the shell process other than specifying the "d" flag to help troubleshoot the issue, please let us know.
Bug: TestWALEntryStream#testCleanClosedWALs test is failing.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/May/21 13:32
Affects Version/s: 2.3.6

We are seeing the following test failure. TestWALEntryStream#testCleanClosedWALs
 This test was added in HBASE-25924. I don't think the test failure has anything to do with the patch in HBASE-25924.
 Before HBASE-25924, we were *not* monitoring _uncleanlyClosedWAL_ metric. In all the branches, we were not parsing the wal trailer when we close the wal reader inside ReplicationSourceWALReader thread. The root cause was when we add active WAL to ReplicationSourceWALReader, we cache the file size when the wal was being actively written and once the wal was closed and replicated and removed from WALEntryStream, we did reset the ProtobufLogReader object but we didn't update the length of the wal and that was causing EOF errors since it can't find the WALTrailer with the stale wal file length.

The fix applied nicely to branch-1 since we use FSHlog implementation which closes the WAL file sychronously.

But in branch-2 and master, we use _AsyncFSWAL_ implementation and the closing of wal file is done asynchronously (as the name suggests). This is causing the test to fail. Below is the test.
{code:java}
  @Test
  public void testCleanClosedWALs() throws Exception {
    try (WALEntryStream entryStream = new WALEntryStream(
      logQueue, CONF, 0, log, null, logQueue.getMetrics(), fakeWalGroupId)) {
      assertEquals(0, logQueue.getMetrics().getUncleanlyClosedWALs());
      appendToLogAndSync();
      assertNotNull(entryStream.next());
      log.rollWriter();  =======> This does an asynchronous close of wal.
      appendToLogAndSync();
      assertNotNull(entryStream.next());
      assertEquals(0, logQueue.getMetrics().getUncleanlyClosedWALs());
    }
  }
{code}
In the above code, when we roll writer, we don't close the old wal file immediately so the ReplicationReader thread is not able to get the updated wal file size and that is throwing EOF errors.
 If I add a sleep of few milliseconds (1 ms in my local env) between rollWriter and appendToLogAndSync statement then the test passes but this is *not* a proper fix since we are working around the race between ReplicationSourceWALReaderThread and closing of WAL file.
Bug: TestRESTServerSSL fails because of jdk bug
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/May/21 22:35

[~weijing329] identified issue in TestRESTServerSSL when using jdk8 292+. It came up in comment in the parent issue. I verified it fails for me using jdk8 v292. Here is the failure....

```[INFO] Running org.apache.hadoop.hbase.rest.TestRESTServerSSL
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.4 s <<< FAILURE! - in org.apache.hadoop.hbase.rest.TestRESTServerSSL
[ERROR] org.apache.hadoop.hbase.rest.TestRESTServerSSL  Time elapsed: 1.387 s  <<< ERROR!
java.security.NoSuchAlgorithmException: unrecognized algorithm name: PBEWithSHA1AndDESede
	at org.apache.hadoop.hbase.rest.TestRESTServerSSL.beforeClass(TestRESTServerSSL.java:74)```

For workaround, see https://github.com/bcgit/bc-java/issues/941
Bug: Remove deprecated ZK command 'rmr' in hbase-cleanup.sh
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/May/21 04:12
Affects Version/s: 2.5.0

Deprecated command 'rmr' was removed from ZooKeeper since 3.6.0(see: ZOOKEEPER-3411).

It's better to use 'deleteall' instead.
Bug: Update Compression/TestCompressionTest: LZ4, SNAPPY, LZO
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/May/21 21:05

LZ4 was changed in hadoop-3.3.1 to use a more amenable library, one that did not require gymnastics installing lz4 native libs everywhere; rather, you just add a jar to the classpath. See HADOOP-17292.

Similar was done for SNAPPY. See HADOOP-17125.

What this means is that our TestCompressionTest passes for hadoop before 3.3.1 but at 3.3.1, the assert that SNAPPY and LZ4 compressors should fail -- because no lib installed -- now no longer asserts.

While in here, LZO is GPL and requires extra install to setup [1]. When TestCompressionTest runs, it emits the below for the LZO check. The check is kinda useless.

{code}
2021-05-28T10:05:36,513 WARN  [Time-limited test] util.CompressionTest(75): Can't instantiate codec: lzo
org.apache.hadoop.hbase.DoNotRetryIOException: Compression algorithm 'lzo' previously failed test.
{code}

I think best thing for now is to comment out the asserts that LZ4 and SNAPPY do NOT work when binary cannot be found -- since this holds only if hadoop < 3.3..1; the test is a little weak anyways.



1. https://stackoverflow.com/questions/23441142/class-com-hadoop-compression-lzo-lzocodec-not-found-for-spark-on-cdh-5
Bug: Exclude jakarta.activation-api
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/May/21 16:29
Affects Version/s: 2.3.0

Hadoop 3.3.1 replaced its dependency of javax.activation 1.2.0 with jakarta.activation 1.2.1.

They are essentially the same thing (they even have the same classpath name), but Eclipse took over JavaEE development and therefore changed group/artifact id. (https://stackoverflow.com/questions/46493613/what-is-the-replacement-for-javax-activation-package-in-java-9)

See HADOOP-17049 for more details. Hadoop 3.3.0 updated jackson-databind to 2.10 which shades jakarta.activation, causing classpath conflict.

The solution to this issue will be similar to HBASE-22268
Bug: TestHBaseConfiguration#testDeprecatedConfigurations is broken with Hadoop 3.3
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/May/21 22:48
Affects Version/s: 2.5.0

The test TestHBaseConfiguration#testDeprecatedConfigurations was added recently by HBASE-25861 to address the usage of Hadoop Configuration addDeprecations API.

However, the API's behavior was changed to fix a bug.
Bug: Correct the usage of Configuration#addDeprecation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/May/21 03:31
Affects Version/s: 2.5.0

When I was solving HBASE-25745 ([PR3139|https://github.com/apache/hbase/pull/3139]), I found that our use of Configuration#addDeprecation API was wrong. 

 

At present, we will call Configuration#addDeprecation in the static block for the deprecated configuration. But after testing, it is found that this does not complete backward compatibility. When user upgrades HBase and does not change the deprecated configuration to the new configuration, he will find that the deprecated configuration does not effect, which may not be consistent with expectations. The specific test results can be seen in the PR above, and we can found the calling order of Configuration#addDeprecation is very important.

 

Configuration#addDeprecation is a Hadoop API, looking through the Hadoop source code, we will find that before creating the Configuration object, the addDeprecatedKeys() method will be called first: [https://github.com/apache/hadoop/blob/b93e448f9aa66689f1ce5059f6cdce8add130457/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/HdfsConfiguration.java#L34] .
Bug: Add target Region to connection exceptions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Apr/21 06:01

We spent a bit of time making it so exceptions included the remote host name. Looks like we can add the target Region name too with a bit of manipulation; will help figuring hot-spotting or problem Region on serverside.  For example, here is what I was seeing recently on client-side when a RS was was timing out requests:

{code}
2021-04-06T02:18:23.533Z, RpcRetryingCaller{globalStartTime=1617675482894, pause=100, maxAttempts=4}, org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to ps0989.example.org/1.1.1.1:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=88369369,methodName=Get], waitTime=5006, rpcTimeout=5000
        at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:145)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:383)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:357)
...
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to ps0989.bot.parsec.apple.com/17.58.114.206:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=88369369,methodName=Get], waitTime=5006, rpcTimeout=5000
        at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:209)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:378)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:89)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:409)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:405)
        at org.apache.hadoop.hbase.ipc.Call.setTimeout(Call.java:110)
        at org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:136)
        at org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672)
        at org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747)
        at org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472)
        ... 1 more
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=88369369,methodName=Get], waitTime=5006, rpcTimeout=5000
        at org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:137)
        ... 4 more
{code}

I wanted the region it was hitting. I wanted to know if it was a server problem or a Region issue. If clients only having issue w/ one Region, then I could focus on it.

After the PR the exception (from another context) looks like this:

{code}
org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to address=127.0.0.1:12345, regionInfo=hbase:meta,,1.1588230740 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: error
....
{code}
Bug: Data is not getting replicated to secondary replicas after altering a table to enable region replication (async Wal replication)
Status: Open | Priority: Major | Resolution: 
Created: 28/May/21 13:07

Data is not getting replicated to secondary replicas after altering a table to enable region replication (async Wal replication)
*Test Steps:*
1)Create table with one replica 
create 'table1','cf'
2)Alter table with region replica=3
alter 'table1',\{REGION_REPLICATION => '3'}
3)write data to table1 
*Observation:*
1)Data is not getting replicated to secondary replicas this is happening when alter table after the peer is created for region replication.
2)In table UI page also you can observe write requests showing for secondary replicas=0
Bug: Move MetaTableAccessor out of hbase-balancer module
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Apr/21 13:48

It should not be there.

The only reason we have to put it there is the favor node balancer. The favor node balancer does not work well so maybe we could just purge it.

==== update ====

With HBASE-25926 in place we could just move it directly.
Bug: RegionServer failed to start due to IllegalThreadStateException in AuthenticationTokenSecretManager.start
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/May/21 11:34

RegionServer failed to complete initialization and aborted during AuthenticationTokenSecretManager#leaderElector start.

Observed following WARN log,
{noformat}
2021-05-03 07:59:01,848 | WARN  | RS-EventLoopGroup-1-6 | Thread leaderElector[ZKSecretWatcher-leaderElector:56] is stopped or not alive | org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager.retrievePassword(AuthenticationTokenSecretManager.java:153)
2021-05-03 07:59:01,848 | INFO  | RS-EventLoopGroup-1-6 | Thread leaderElector [ZKSecretWatcher-leaderElector:56] is started | org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager.retrievePassword(AuthenticationTokenSecretManager.java:156)
2021-05-03 07:59:01,854 | INFO  | ZKSecretWatcher-leaderElector | Found existing leader with ID: RS-IP-PORT-StartCode | org.apache.hadoop.hbase.zookeeper.ZKLeaderManager.waitToBecomeLeader(ZKLeaderManager.java:130)
{noformat}
As per the code, AuthenticationTokenSecretManager#leaderElector is started while retrieving password before AuthenticationTokenSecretManager#start, 
 [https://github.com/apache/hbase/blob/8c2332d46532135723cc7a6084a2a125f3d9d8db/hbase-server/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java#L155]

So IllegalThreadStateException occured during AuthenticationTokenSecretManager#start, 
 [https://github.com/apache/hbase/blob/8c2332d46532135723cc7a6084a2a125f3d9d8db/hbase-server/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java#L107]
{noformat}
2021-05-03 07:59:02,066 | ERROR | main | Failed construction RegionServer | org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:775)
java.lang.IllegalThreadStateException
	at java.lang.Thread.start(Thread.java:708)
	at org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager.start(AuthenticationTokenSecretManager.java:107)
	at org.apache.hadoop.hbase.ipc.NettyRpcServer.start(NettyRpcServer.java:131)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.start(RSRpcServices.java:1695)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:756)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:3270)
	at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:63)
	at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
{noformat}
Bug: Client integration test is failing on master and branch-2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/May/21 02:34

{noformat}
Starting up HBase
127.0.0.1: Host key verification failed.
running master, logging to /home/jenkins/jenkins-home/workspace/HBase_HBase_Nightly_master/hbase-install/bin/../logs/hbase-jenkins-master-jenkins-hbase7.out
2021-05-22T04:16:40,142 INFO  [main] master.HMaster: STARTING service HMaster
2021-05-22T04:16:40,149 INFO  [main] util.VersionInfo: HBase 3.0.0-SNAPSHOT
2021-05-22T04:16:40,149 INFO  [main] util.VersionInfo: Source code repository file:///home/jenkins/jenkins-home/workspace/HBase_HBase_Nightly_master/unpacked_src_tarball revision=Unknown
2021-05-22T04:16:40,149 INFO  [main] util.VersionInfo: Compiled by jenkins on Sat May 22 04:07:41 UTC 2021
2021-05-22T04:16:40,149 INFO  [main] util.VersionInfo: From source with checksum b6959885410c34f4458efd580213907ca50e84a0a900ea1d465d04a1a9480e520419d51da933547e3f517850a2913a8e8aefbcd6ba9e12589fced980910fb941
cat: /home/jenkins/jenkins-home/workspace/HBase_HBase_Nightly_master/output-integration/hadoop-3/hbase-conf//regionservers: No such file or directory
cat: /home/jenkins/jenkins-home/workspace/HBase_HBase_Nightly_master/output-integration/hadoop-3/hbase-conf//regionservers: No such file or directory
	retry waiting for hbase to come up.
	retry waiting for hbase to come up.
	retry waiting for hbase to come up.
	retry waiting for hbase to come up.
	retry waiting for hbase to come up.
	retry waiting for hbase to come up.
	retry waiting for hbase to come up.
	retry waiting for hbase to come up.
	retry waiting for hbase to come up.
	retry waiting for hbase to come up.
	retry waiting for hbase to come up.
	retry waiting for hbase to come up.
	retry waiting for hbase to come up.
	retry waiting for hbase to come up.
	retry waiting for hbase to come up.
/home/jenkins/jenkins-home/workspace/HBase_HBase_Nightly_master/component/dev-support/hbase_nightly_pseudo-distributed-test.sh: line 539:  3122 Terminated              sleep "${sleep_time}"
Shutting down HBase
no hbase master found
{noformat}
Bug: Improve logging in "packaging and integration" stage
Status: Open | Priority: Major | Resolution: 
Created: 24/May/21 20:22
Affects Version/s: 3.0.0-alpha-1

We had a nightly build of master [fail inscrutably|https://ci-hadoop.apache.org/blue/organizations/jenkins/HBase%2FHBase%20Nightly/detail/master/298/pipeline/103]. Looks like the build went down some dark hole and didn't make a peep for almost 16h. It would be nice to get some logging from whatever process that was.

{noformat}
[2021-05-20T20:13:45.733Z] Attempting to use run an instance on top of Hadoop 3.
[2021-05-21T11:58:19.827Z] Sending interrupt signal to process
[2021-05-21T11:58:20.683Z] Sending interrupt signal to process
[2021-05-21T11:58:32.771Z] script returned exit code 2
{noformat}
Bug: Ban System.currentTimeMillis with a build enforcer
Status: Open | Priority: Major | Resolution: 
Created: 25/May/21 00:06

We introduced EnvironmentEdgeManager a long time ago as a way to inject alternate clocks (gettimeofday() aka System.currentTimeMillis()) for unit tests. In order for this to be effective, all callers that would otherwise use System.currentTimeMillis() must call EnvironmentEdgeManager.currentTime() instead, except obviously the implementors of EnvironmentEdge.

It's common for contributors to be unaware of this practice and reviewers might not catch it.

It will be much more important to have EnvironmentEdgeManager in use where expected once we have EnvironmentEdge also providing a monotonic clock source. (See parent.)

Introduce a build enforcer that bans System.currentTimeMillis() except where annotated to allow it.