av9ash/GitBugs huggingface dataset:
Bug: Client connection warming API
Status: Open | Priority: Major | Resolution: 
Created: 20/Jul/22 15:15

In a high performance API or low latency stream workers, you often do not want to incur costs on the first few requests. In these cases, you want to warm connections before ever adding to the load balancer or processing group.

Upon first creating a Connection, there are two areas that can slow down the first few requests:
 * Fetching region locations
 * Creating the initial connection to each RegionServer, which sends connection headers, possibly does auth handshakes, etc.

A user can easily work around the first slowness by calling Table.getRegionLocator().getAllRegionLocations().

It's more challenging for a user to warm the actual RegionServer connections. One way we have done this is to use a RegionLocator to fetch all locations for a table, reduce that down to 1 region per server, and then issue a Get to each row. We end up repeating this for every table that a process may connect to, because at the level we do this we can't easily tell which servers have already been warmed. We also have run into various bugs over time, for example where an empty startkey causes a Get to fail.

We can make this easier for the users by providing an API which uses Connection internals to as cheaply as possible warm these connections. I'd propose we add the following:

New Table/AsyncTable method {{{}warmConnections(){}}}. This would do the following:
 * use region locator to fetch all locations (with caching)
 * reduce returned locations to unique ServerNames
 * for each ServerName (with lock):
 ** if already warmed, skip
 ** otherwise, get a connection to that server and send an initial request to trigger socket creation/connection header/etc

With this API, if someone is connecting to multiple tables, they could warm each of them Table in parallel and we'd only create connections to each server once.
Bug: TestRaceBetweenSCPAndTRSP fails in pre commit
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Aug/22 15:48

Seems the PE worker is stuck here. Need dig more.

{noformat}
"PEWorker-5" daemon prio=5 tid=326 in Object.wait()
java.lang.Thread.State: WAITING (on object monitor)
        at java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
        at java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
        at java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
        at java.base@11.0.10/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
        at app//org.apache.hadoop.hbase.master.assignment.TestRaceBetweenSCPAndTRSP$AssignmentManagerForTest.getRegionsOnServer(TestRaceBetweenSCPAndTRSP.java:97)
        at app//org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.getRegionsOnCrashedServer(ServerCrashProcedure.java:288)
        at app//org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.executeFromState(ServerCrashProcedure.java:195)
        at app//org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.executeFromState(ServerCrashProcedure.java:66)
        at app//org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)
        at app//org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:919)
        at app//org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1650)
        at app//org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1396)
        at app//org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)
        at app//org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1962)
        at app//org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread$$Lambda$477/0x0000000800ac1840.call(Unknown Source)
        at app//org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
        at app//org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1989)
{noformat}
Bug: Implement a new table based replication queue storage and make the minimum replication system work
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jul/22 14:33

This is the start point.

The plan is to introduce the new table based replication queue storage implementation, and make simple replication work, no failure recovery, no claim queue, no wal cleaner, etc. The related UTs will be disabled until we finally cover the area in later issues.
Bug: Long running heavily filtered scans hold up too many ByteBuffAllocator buffers
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 20/Jul/22 14:55

We have a workload which is launching long running scans searching for a needle in a haystack. They have a timeout of 60s, so are allowed to run on the server for 30s. Most of the rows are filtered, and the final result is usually only a few kb.

When these scans are running, we notice our ByteBuffAllocator pool usage goes to 100% and we start seeing 100+ MB/s of heap allocations. When the scans finish, the pool goes back to normal and heap allocations go away.

My working theory here is that we are only releasing ByteBuff's once we call {{shipper.shipped(),}} which only happens once a response is returned to the user. This works fine for normal scans which are likely to quickly find enough results to return, but for long running scans in which most of the results are filtered we end up holding on to more and more buffers until the scan finally returns.

We should consider whether it's possible to release buffers for blocks whose cells have been completely skipped by a scan.
Bug: getClusterMetrics NullPointerException when ServerTask status null
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Sep/22 00:00

We had a situation where an hmaster had just started up and something started calling Admin.getClusterMetrics. It seems there were ServerTasks in memory, but at least one had no status. This resulted in a spam of NPE:
{code:java}
[RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=60000] ERROR org.apache.hadoop.hbase.ipc.RpcServer: Unexpected throwable object  
java.lang.NullPointerException    
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterStatusProtos$ServerTask$Builder.setStatus(ClusterStatusProtos.java:14352)    
at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toServerTask(ProtobufUtil.java:3565)    
at org.apache.hadoop.hbase.ClusterMetricsBuilder.lambda$toClusterStatus$4(ClusterMetricsBuilder.java:80)    
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)    
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)    
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)    
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)    
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)    
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)    
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)    
at org.apache.hadoop.hbase.ClusterMetricsBuilder.toClusterStatus(ClusterMetricsBuilder.java:80)    
at org.apache.hadoop.hbase.master.MasterRpcServices.getClusterStatus(MasterRpcServices.java:980)    
at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)    
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:385)    
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)    
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)    
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82) {code}
Bug: Port FileWatcher from ZK to autodetect keystore/truststore changes in TLS connections
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Aug/22 12:05
Affects Version/s: 3.0.0-alpha-3

As a follow-up feature I'd like to port the file watcher functionality from ZooKeeper in order to avoid restarts when keystore/truststore is updated on a running cluster.
Bug: Publish binaries and maven artifacts for both hadoop2 and hadoop3
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Sep/22 15:48

As per the discussion in this thread

https://lists.apache.org/thread/y05gspk4mnxsz6nk7hc5ots8wt50366b

And this is possible after we land HBASE-27340.
Bug: Add config for setting a max actions per normalizer run
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 29/Sep/22 18:14

When you enable the normalizer for the first time on an existing table, the next time it runs it can do a massive amount of merges/splits. This can be painful for callers.

Additionally, if someone were to write a large amount of Puts or Deletes it may cause a merge/split storm as well.

We should add a config which allows operators to limit the amount of actions per run. This way the normalizer can work towards its plan gracefully over the course of a few hours rather than all at once.
Bug: Avoid synchronization in AsyncFSWAL
Status: Patch Available | Priority: Major | Resolution: 
Created: 06/Sep/22 03:08
Affects Version/s: 3.0.0-alpha-3

{{AsyncFSWAL}} uses {{consumeLock}} to synchronize operations between {{AsyncFSWAL.consumeExecutor}} and {{AsyncFSWAL.doReplaceWriter}}   (also {{AsyncFSWAL.doShutdown}}), I think we could further use the event-driven approach again to eliminate the {{consumeLock}}:

* For {{AsyncFSWAL.doReplaceWriter}} ,it could just publish the request for replacing writer event to {{AsyncFSWAL}} and wait for the event to complete.
* For {{AsyncFSWAL.consumeExecutor}} ,it could detect the replacing writer event at its safe point (i.e. writer is broken or no {{FSWALEntry}} to write) and executes replacing writer event by itself, and after it completes , it notifies {{LogRoller}} thread that the event is completed. Replacing writer is a very lightweight operation and only replacing writer is completed, {{AsyncFSWAL.consumeExecutor}} could continue to write WAL, so it is harmless for {{AsyncFSWAL.consumeExecutor}} and could avoid using consumeLock.

The main difference between the PR and the current implementation is {{AsyncFSWAL.consumeExecutor}} replacing the writer by itself.
Bug: EOF thrown by WALEntryStream causes replication blocking
Status: Open | Priority: Major | Resolution: 
Created: 01/Sep/22 11:17
Affects Version/s: 2.4.14

In [WALEntryStream#readNextEntryAndRecordReaderPosition|https://github.com/apache/hbase/blob/308cd729d23329e6d8d4b9c17a645180374b5962/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALEntryStream.java#L257], it is possible that we read uncommitted data.  If we read beyond the committed file length, then reopen inputStream and seek back.

In our use, we found that the position where seek back may be exactly the length of the file being written, which may cause EOF.

The thrown EOF is finally caught [ReplicationSourceWALReader.run|https://github.com/apache/hbase/blob/308cd729d23329e6d8d4b9c17a645180374b5962/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceWALReader.java#L158], but [totalBufferUsed|https://github.com/apache/hbase/blob/308cd729d23329e6d8d4b9c17a645180374b5962/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceWALReader.java#L78] is not cleanup up.

After a long run, all peers will go slow and eventually block completely.
Bug: Minimise block addition failures due to no space in bucket cache writers queue by introducing wait time
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Sep/22 17:39
Affects Version/s: 3.0.0-alpha-3

Currently in bucket cache asynchronous caching mechanism introduced where initially the blocks to be cached will be added to queue and writer threads consume the blocks from the queue and write to bucket cache. In case if block writing to bucket cache is slow then there is a chance that  queue of writer threads become full  and following block additions will be failed. In case of slower storages like s3 might introduce latencies even if we enable bigger sizes of bucket cache using ephemeral storages. So we can allow configurable wait time while adding blocks to queue so that chances of queue free up is possible during the wait time and block addition failures can be minimised. To avoid the performance impact of wait time in regular read paths we can use the wait time mainly during background operations like compactions, flushes or prefetches etc.
Bug: Frequent and not useful "Final timeLimitDelta" log lines
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Sep/22 00:30
Affects Version/s: 2.4.14

Introduced by HBASE-27205 Fix tests that rely on EnvironmentEdgeManager in branch-2.4. 

This line should be removed: https://github.com/apache/hbase/blob/e25b2a7cd0b0daddad5ab09a87cd7ed9f59c81b6/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L3236
Bug: Avoid decompressing blocks when reading from bucket cache prefetch threads 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/22 20:29
Affects Version/s: 3.0.0-alpha-4

When prefetching blocks into bucket cache, we had observed a consistent CPU usage around 70% with no other workloads ongoing. For large bucket caches (i.e. when using file based bucket cache), the prefetch can last for sometime and having such a high CPU usage may impact the database usage by client applications.

Further analysis of the prefetch threads stack trace showed that very often, decompress logic is being executed by these threads:
{noformat}
"hfile-prefetch-1654895061122" #234 daemon prio=5 os_prio=0 tid=0x0000557bb2907000 nid=0x406d runnable [0x00007f294a504000]
   java.lang.Thread.State: RUNNABLE
        at org.apache.hadoop.io.compress.snappy.SnappyDecompressor.decompressBytesDirect(Native Method)
        at org.apache.hadoop.io.compress.snappy.SnappyDecompressor.decompress(SnappyDecompressor.java:235)
        at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:88)
        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:105)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        - locked <0x00000002d24c0ae8> (a java.io.BufferedInputStream)
        at org.apache.hadoop.hbase.io.util.BlockIOUtils.readFullyWithHeapBuffer(BlockIOUtils.java:105)
        at org.apache.hadoop.hbase.io.compress.Compression.decompress(Compression.java:465)
        at org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultDecodingContext.prepareDecoding(HFileBlockDefaultDecodingContext.java:90)
        at org.apache.hadoop.hbase.io.hfile.HFileBlock.unpack(HFileBlock.java:650)
        at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1342) {noformat}

This is because *HFileReaderImpl.readBlock* is always decompressing blocks even when *hbase.block.data.cachecompressed* is set to true. 

This patch proposes an alternative flag to differentiate prefetch from normal reads, so that doesn't decompress DATA blocks when prefetching with  *hbase.block.data.cachecompressed* set to true.
Bug: Do not need to throw IllegalStateException when peer is not active in ReplicationSource.initialize
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/22 14:51

The only place where we can reset the sourceRunning flag to false is in terminate method, where we want to stop the replication source. And we already have a interrupted state check in the initialize method to not throw IllegalStateException when peer is terminating, so I think here we could just do not throw IllegalStateExeption at all.

The reason why the interrupted state check does not work is that, in FutureUtils.get, we will wrap InterruptedException with InterruptedIOException, without restoring the interrupted state, but the upper layer does not treat InterruptedIOException specially so the interrupted state is lost. But anway, I do not think we should rely on the interrupted state to determine whether we should abort the region server...
Bug: Bump spotbugs version
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Sep/22 06:40

Recently there are strange spotbugs failures in pre commit builds, likely to be a false positive.

https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4779/2/artifact/yetus-general-check/output/branch-spotbugs-hbase-server-warnings.html

Let's see if upgrading the spotbugs version can solve the problem.
Bug: Add dead region server to SplitLogManager#deadWorkers set as the first step.
Status: Open | Priority: Major | Resolution: 
Created: 21/Sep/22 20:54
Affects Version/s: 1.7.2

Currently we add a dead region server to +SplitLogManager#deadWorkers+ set in SERVER_CRASH_SPLIT_LOGS state. 
Consider a case where a region server is handling split log task for hbase:meta table and SplitLogManager has exhausted all the retries and won't try any more region server. 
The region server which is handling split log task has died. 
We have a check in SplitLogManager where if a region server is declared dead and if that region server is responsible for split log task then we forcefully resubmit split log task. See the code [here|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java#L721-L726].

But we add a region server to SplitLogManager#deadWorkers set in [SERVER_CRASH_SPLIT_LOGS|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java#L252] state. 
Before that it runs [SERVER_CRASH_GET_REGIONS|https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java#L214] state  and checks if hbase:meta table is up. In this case, hbase:meta table was not online and that prevented SplitLogManager to add this RS to deadWorkers list. This created a deadlock and hbase cluster was completely down for an extended period of time until we failed over active hmaster. See HBASE-27382 for more details.

Improvements:
1.  We should a dead region server to +SplitLogManager#deadWorkers+ list as the first step.
Bug: Document native TLS support in Netty RPC
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jul/22 10:38

Add a new section to the HBase book on how a developer can get this going. Should include
 * relevant TLS properties added in X509Util.java which need to be added to hbase-site.xml
 * how to generate a self-signed CA and certs using {{{}keytool{}}}/{{{}openssl{}}}
 * any known limitations
Bug: Use Hadoop Credentials API to retrieve passwords of TLS key/trust stores
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Aug/22 12:24

Based on a discussion in the TLS Jira and mailing list, it would be beneficial to protect the password of trust and key stores for TLS encryption support in Netty RPC.

Hadoop Credentials API is built for the same purpose, extensible and simple to use:

[https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html]
Bug: LogLevel.Servlet not working in 2.5.0
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 27/Sep/22 12:44
Affects Version/s: 2.5.0

I'm upgrading our servers to hbase 2.5.0 and noticed that the LogLevel.Servlet no longer works. I am guessing this is due to the log4j2 change.

According to the servlet output, it sees log class as {{org.slf4j.impl.Log4jLoggerAdapter}} but the log levels seem totally disconnected from reality — setting a log level has no effect, and checking the loglevel of a logger that i had configured in my log4j.properties shows the wrong value. Logging itself seems to be using my log4j.properties correctly, but the LogLevel.Servlet sees and sets a different view

I tried moving aside the {{log4j2.properties}} and also separately tried moving aside the {{{}log4j.properties{}}}, neither helped
Bug: Make balance byTable config take effect when use CLI 
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 23/Sep/22 09:05

For Master triger balance, whether balance by table is depend on 
{code:java}
this.isByTable = conf.getBoolean(HConstants.HBASE_MASTER_LOADBALANCE_BYTABLE, isByTable);{code}
But for CLI, is different.

 

I think we'd better make the balance behavior same.
Bug: HBase master reports a region has been in transition for 50+ years
Status: Open | Priority: Major | Resolution: 
Created: 26/Jul/22 16:09
Affects Version/s: 2.4.10

Every time we upgrade our HBase clusters we get some spurious alerts firing because for a brief period of time the HBase master reports some impossibly high RIT (region in transition) time.

For example:

!image.png|width=747,height=249!

 

The condition resolves itself on its own within a few minutes. I'll try to find some relevant logs and attach them to this issue.
Bug: Add mutual authentication support to TLS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Aug/22 21:26

With HBASE-26666 we now have native TLS on server and client. By default clients validate server certificate on handshake. This issue adds server authentication of clients. We can also add support for custom rules, such as cert CommonName validation.

I've already got a POC running of this, so assigning to me
Bug: Use encoded size for calculating compression ratio in block size predicator
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Sep/22 10:05
Affects Version/s: 3.0.0-alpha-3

In HBASE-27264 we had introduced the notion of block size predicators to define hfile block boundaries when writing a new hfile, and provided the

PreviousBlockCompressionRatePredicator implementation for calculating block sizes based on a compression ratio. It was using the raw data size written to the block so far to calculate the compression ratio, but in the case where encoding is enabled, this could lead to a very high compression ratio and therefore, larger block sizes. We should use the encoded size to calculate compression ratio, instead.

Here's a example scenario:

1) Sample block size when not using the  PreviousBlockCompressionRatePredicator as implemented by HBASE-27264:
{noformat}
onDiskSizeWithoutHeader=6613, uncompressedSizeWithoutHeader=32928 {noformat}

2) Sample block size when using PreviousBlockCompressionRatePredicator as implemented by HBASE-27264 (uses raw data size to calculate compression rate):
{noformat}
onDiskSizeWithoutHeader=126920, uncompressedSizeWithoutHeader=655393
{noformat}

3) Sample block size when using PreviousBlockCompressionRatePredicator with encoded size for calculating compression rate:
{noformat}
onDiskSizeWithoutHeader=54299, uncompressedSizeWithoutHeader=328051
{noformat}