av9ash/GitBugs huggingface dataset:
Bug: FNFE related to 'master:store' when moving archived hfiles to global archived dir
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Mar/24 15:35

Sometimes we encounter the following log.
{code:java}
2024-02-26T18:11:34,934 WARN  [master:store-Store-Compactor] region.MasterRegionFlusherAndCompactor: Failed to move archived hfiles from file:/root/hbase-4.0.0-alpha-1-SNAPSHOT/tmp/hbase/MasterData/archive/data/master/store/1595e783b53d99cd5eef43b6debb2682/state to global dir file:/root/hbase-4.0.0-alpha-1-SNAPSHOT/tmp/hbase/archive/data/master/store/1595e783b53d99cd5eef43b6debb2682/statejava.io.FileNotFoundException: File file:/root/hbase-4.0.0-alpha-1-SNAPSHOT/tmp/hbase/MasterData/archive/data/master/store/1595e783b53d99cd5eef43b6debb2682/state does not exist        at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:733) ~[hadoop-common-3.3.5.jar:?]        at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:2045) ~[hadoop-common-3.3.5.jar:?]        at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:2089) ~[hadoop-common-3.3.5.jar:?]        at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:970) ~[hadoop-common-3.3.5.jar:?]        at org.apache.hadoop.hbase.master.region.MasterRegionUtils.moveFilesUnderDir(MasterRegionUtils.java:42) ~[hbase-server-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]        at org.apache.hadoop.hbase.master.region.MasterRegionFlusherAndCompactor.moveHFileToGlobalArchiveDir(MasterRegionFlusherAndCompactor.java:145) ~[hbase-server-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]        at org.apache.hadoop.hbase.master.region.MasterRegionFlusherAndCompactor.compact(MasterRegionFlusherAndCompactor.java:157) ~[hbase-server-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_231]        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_231]        at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_231] {code}
Analysis:
Against 'master:store', Archived hfiles would first be placed in \{hbase.rootdir}/MasterData/archive/ after compacting, and then moved to global archived dir.
'master:store' has 3 column families:proc,rs,state, if there is empty in one of column families,such as state, no files in this column families will be archived, and \{hbase.rootdir}/MasterData/archive/data/master/store/1595e783b53d99cd5eef43b6debb2682/state would do not exist.
Finally, we will get FileNotFoundException when moving these archived hfiles to global archived dir.
 
Although，Current code can avoid this effecting by catching Exception, However it is maybe better to check the dir exists before moving.
Bug: Confusing logging during backup restore
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Feb/24 16:42
Affects Version/s: 2.6.0

Encountered this while experimenting with the backup/restore functionality.

My setup was as follows:
 * Took several backups (Full1, inc2, inc3)
 * Changed an entry in the "lily_tenant_acme:LILY_SETTINGS" table
 * Attempt a restore (to test if my changed entry is reverted):

{code:java}
$ hbase restore -conf backup-conf.xml s3a://backuprestore-experiments/hbase backup_1709123740345 -t "lily_tenant_acme:LILY_SETTINGS" -m "lily_tenant_acme:LILY_SETTINGS-restored1" -o

24/02/28 16:15:41 WARN org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil: The addDependencyJars(Configuration, Class<?>...) method has been deprecated since it is easy to use incorrectly. Most users should rely on addDependencyJars(Job) instead. See HBASE-8386 for more details.
24/02/28 16:15:58 WARN org.apache.hadoop.hbase.tool.LoadIncrementalHFiles: Skipping non-directory hdfs://hdfsns/user/lily/hbase-staging/bulk_output-lily_tenant_acme-LILY_SETTINGS-restored1-1709136941410/_SUCCESS
24/02/28 16:15:59 WARN org.apache.hadoop.hbase.backup.impl.RestoreTablesClient: Nothing has changed, so there is no need to restore 'lily_tenant_acme:LILY_SETTINGS'
{code}
Based on the final logging line, I presumed my restore operation had failed.

After some investigation however, I found that this was not the case: my change was reverted as expected.

Some code investigation learned me this log message is shown because I was restoring backup `inc3`, and there were no changes between `full1` and `inc3`.

I suggest rephrasing this log message, and changing it to a INFO level.
Bug: Backport "HBASE-27811  Enable cache control for logs endpoint and set max age as 0" to branch-2
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Feb/24 09:20

Not setting the proper header values may cause browsers to store pages within their respective caches. On public, shared, or any other non-private computers, a malicious person may search through the browser cache to locate sensitive information cached during another user's session.

/logs endpoint contains sensitive information that an attacker can exploit.

Any page with sensitive information needs to have the following headers in response:
Cache-Control: no-cache, no-store, max-age=0
Pragma: no-cache
Expires: -1
Bug: Suppress the noisy logging on HBaseAdmin#postOperationResult
Status: Open | Priority: Minor | Resolution: 
Created: 16/Feb/24 06:15

While testing 2.6.0 found that for every admin operation there is INFO log of operation status which is noisy. Would be better to suppress by making it debug. This information is helpful during the operation failure to debug further but feel it's unnecessary to print in successful case.
{noformat}
hbase:002:0> create 't','f'
2024-02-16 06:02:38,887 INFO  [main] client.HBaseAdmin: Operation: CREATE, Table Name: default:t, procId: 65 completed
Created table t

{noformat}
{noformat}
hbase:006:0> flush 't'
2024-02-16 06:03:16,294 INFO  [main] client.HBaseAdmin: Operation: FLUSH, Table Name: default:t, procId: 68 completed
Took 0.3733 seconds

{noformat}
Bug: RegionServer Canary can should use Scan just like Region Canary with option to enable Raw Scan
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Feb/24 22:11

While working on HBASE-28204 to improve Region Canary, It came to notice that RegionServer canary uses the same code as Region Canary to check if a region is accessible. Plus RegionSever Canary doesn't have Raw Scan enabled. 

 

This JIRA aims to enable Raw Scan option for RegionServer Canary and use Scan only just like Region Canary.
Bug: MoveWithAck#isSuccessfulScan for Region movement should use Region End Key for limiting scan to one region only.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Feb/24 23:26

Based on recent learnings and improvements in HBase Canary in HBASE-28204 and HBASE-28356, I noticed that MoveWithAck.java class also uses similar code to check that Region is online after region move.

 
{code:java}
  private void isSuccessfulScan(RegionInfo region) throws IOException {
    Scan scan = new Scan().withStartRow(region.getStartKey()).setRaw(true).setOneRowLimit()
      .setMaxResultSize(1L).setCaching(1).setFilter(new FirstKeyOnlyFilter()).setCacheBlocks(false); {code}
If the region, that was moved, is empty then MoveWithAck#isSuccessfulScan() will end up scanning next region key space, which is not the intent. If multiple regions in sequence are empty, then this could create too many unnecessary scans.  By setting withStopRow(endKeyOfRegion, false) for the scan object, this scan can be bound to only single region.
Bug: Fix casting bug in NettyRpcServer introduced in HBASE-28317
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Jan/24 19:01

I introduced a bug in HBASE-28317 that makes it useless. I tried to up-cast an array of {{java.security.cert.Certificate}} to an array of {{java.security.cert.X509Certificate}}, which doesn't work. I'll fix this in this ticket. This bug causes a lot of log spam and makes {{RpcCallContext.getClientCertificateChain()}} always return empty, but does not otherwise break the HBase server.

I'm also immediately regretting my use of WARN logs when a certificate isn't present, because it turns out this happens on connections from localhost, so there's a lot of WARNs with stacktraces in the logs.
Bug: RpcCallContext should expose client's TLS certificate
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Jan/24 16:02

At my employer we plan on using a coprocessor to log information about some requests to HBase. For this to be useful to us, we need to know who each request is coming from. We use HBase's TLS support with mutual authentication to authenticate clients. I'd like a way to expose the client certificate used on a request to coprocessors. For setups using Kerberos authentication, RpcCall exposes the Kerberos principal shortname via {{getRequestUser()}}, so this would be the TLS equivalent to that.
Bug: Type conversion is no need in method CompactionChecker.chore()
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Jan/24 14:38

In method CompactionChecker.chore(), we iterate the map named onlineRegions ，which the type of value is HRegion, So we do not need to type conversion from Region to HRegion.
 
{code:java}
private final Map<String, HRegion> onlineRegions = new ConcurrentHashMap<>(); {code}
Bug: Set proper error in response for all usages of HttpServer.isInstrumentationAccessAllowed()
Status: Open | Priority: Minor | Resolution: 
Created: 10/Jan/24 11:00

During review https://github.com/apache/hbase/pull/5215, it was found we simply return 200 even if instrumentation is not allowed. While at some places we set proper error. This JIRA is to fix usages of the method and set proper response code.

Review comment: https://github.com/apache/hbase/pull/5215/files#r1318296417

CC: [~ndimiduk]
Bug: Refactor GarbageCollectorMXBean instantiation in process*.jsp
Status: Open | Priority: Minor | Resolution: 
Created: 10/Jan/24 11:05

During review of https://github.com/apache/hbase/pull/5215/ we saw that beans are instantiated based on assumptions around JVM, it is a good idea to refactor code so that we don't get errors when JVM assumptions change in future.

Review comment: https://github.com/apache/hbase/pull/5215/files#r1318304462

CC: [~ndimiduk]
Bug: Fix Typo in Java Method Comment
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 29/Mar/24 03:24
Affects Version/s: 4.0.0-alpha-1

https://github.com/apache/hbase/blob/9adca10e9c106df5119cc067fd610fb19cd1d5ec/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java#L795

 

The java method comment contains a typo. 

The comment incorrectly states "Region name to assign" instead of "Region name to unassign".
Bug: hlog loss result in data loss when regionserver down network card
Status: Open | Priority: Blocker | Resolution: 
Created: 26/Oct/22 08:51
Affects Version/s: 2.2.7

When tested hbase replication, I found that the data in the memory was lost. Through the source code of hbase, I found that after hbase split the meta table, it will delete the entire wal  directory , but hlog is still in the same directory, and which save the data in the memstore. Eventually cause data loss.

The specific process is as follows:
 #  put a few data into hbase table.
 #  turn down the network card of all regionserver nodes.
 #  turn up the network card of all regionserver nodes，regionserver will be killed.
 #  restart the hbase cluster, scan table and find no data in the table.

In hmaster log will print:

      master.MasterWalManager: Log dir for server xxx does not exist

 

The splitLogDistributed function in SplitLogManager.java caused this issue. hmaster will first call splitLogDistributed function to publish the split task of the meta table. After the meta split task is completed, the directory will be deleted. Then hmaster want to publish another normal split task to restore the data again, but no wal directory was found.
{code:java}
    waitForSplittingCompletion(batch, status);
      ...
    for (Path logDir : logDirs) {
      status.setStatus("Cleaning up log directory...");
      final FileSystem fs = logDir.getFileSystem(conf);
      try {
        if (fs.exists(logDir) && !fs.delete(logDir, false)) {
          LOG.warn("Unable to delete log src dir. Ignoring. " + logDir);
        }
      } catch (IOException ioe) {
        FileStatus[] files = fs.listStatus(logDir);
        if (files != null && files.length > 0) {
          LOG.warn("Returning success without actually splitting and "
              + "deleting all the log files in path " + logDir + ": "
              + Arrays.toString(files), ioe);
        } else {
          LOG.warn("Unable to delete log src dir. Ignoring. " + logDir, ioe);
        }
      }
    }{code}
   My English is poor, so if there is anything unclear, can leave me a message.
Bug: Optimize the compaction triggering logic if  table has set TTL
Status: Open | Priority: Critical | Resolution: 
Created: 15/Dec/22 09:42
Affects Version/s: 2.4.15

Although table has a TTL set in the cluster，some regions do not perform periodic major compactions，and the expired TTL data is still stored in these large files. Not trigger the reason is that the periodic check of the thread according to number of files in the region  If > hbase.hstore.compaction.min then major compaction is not triggered, Instead, minor compaction is triggered first. But minor compaction which excludes large files (by the hbase.hstore.compaction.max.size), when the larger files reach a certain number, the expired data under the larger files cannot be deleted normally.
Bug: In memory compaction 'COMPACT' may cause data corruption when adding cells large than maxAlloc(default 256k) size
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 04/Nov/22 07:59

When init 'CellChunkImmutableSegment' for 'COMPACT' action, we not force copy to current MSLab. 

When cell size bigger than maxAlloc, cell will stay in previous chunk which will recycle after segment replace, and we may read wrong data when these chunk reused by others.

!image-2022-11-04-15-46-21-645.png!

 

Timeline:
 # add a cell 'A' bigger than 256K
 # cell 'A' will copy to a chunk 'A' when first compact
 # cell 'A' will retain in chunk 'A' when second compact
 # chunk 'A' recycled after segment swap and close
Bug: RegionServer JVM crash when scan
Status: Open | Priority: Critical | Resolution: 
Created: 28/Oct/22 10:23
Affects Version/s: 2.3.5

jdk8 + hadoop3 + bucketcache + g1 + hbase2.3.5

In our cluster, we found region servers may be crashed in several cases

hs_err_pid117788.log

#
 # A fatal error has been detected by the Java Runtime Environment:

#

#  SIGSEGV (0xb) at pc=0x00007fdc733a9a4e, pid=117788, tid=0x00007fdc0a34b700

#
 # JRE version: Java(TM) SE Runtime Environment (8.0_171-b11) (build 1.8.0_171-b11)

 # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.171-b11 mixed mode linux-amd64 compressed oops)

 # Problematic frame:

 # j  org.apache.hadoop.hbase.ipc.NettyServerRpcConnection.isConnectionOpen()Z+4

#
 # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

#
 # If you would like to submit a bug report, please visit:

 #   [http://bugreport.java.com/bugreport/crash.jsp]

#

 

---------------  T H R E A D  ---------------

 

Current thread (0x00007fd48004f000):  JavaThread "RpcServer.default.FPBQ.Fifo.handler=17,queue=2,port=16020" daemon [_thread_in_Java, id=118639, stack(0x00007fdc0a24b000,0x00007fdc0a34c000)]

 

regionserver日志

2022-10-28 03:33:03,611 WARN  [RpcServer.default.FPBQ.Fifo.handler=25,queue=0,port=16020] ipc.RpcServer: (responseTooSlow):

{"call":"Scan(org.apache.hadoop.h base.shaded.protobuf.generated.ClientProtos$ScanRequest)","starttimems":"1666899153611","responsesize":"19","method":"Scan","param":"scanner_id: 119667009731 85298703 number_of_rows: 1000 close_scanner: false next_call_seq: 0 client_handles_partials: true client_handles_heartbeats: t <TRUNCATED>","processingtimems ":30000,"client":"10.64.143.22:48550","queuetimems":0,"class":"HRegionServer","scandetails":"table: action_user_track_v2 region: action_user_track_v2,\\xE8
x00\\x00\\x00-\\xAA\\xF9\\xFB\\x00\\x00\\x01

{\\xCF\\xA6\\xDE\\x09,1645585735023.efb249fd8f4c26d4911d50f14fc30ce8."}

 

2022-10-28 03:33:03,611 WARN  [RpcServer.default.FPBQ.Fifo.handler=17,queue=2,port=16020] hfile.BlockCacheUtil: Caching an already cached block: 58a6823d3958

49099bb226e18acb79b2_SeqId_6__1032126997. This is harmless and can happen in rare cases (see HBASE-8547)

2022-10-28 03:33:03,611 WARN  [RpcServer.default.FPBQ.Fifo.handler=17,queue=2,port=16020] hfile.BlockCacheUtil: Caching an already cached block: eb1c30882099

45f6b5006db45d5fe286_SeqId_4__1032126997. This is harmless and can happen in rare cases (see HBASE-8547)

2022-10-28 03:33:03,612 WARN  [RpcServer.default.FPBQ.Fifo.handler=17,queue=2,port=16020] hfile.BlockCacheUtil: Caching an already cached block: eb1c30882099

45f6b5006db45d5fe286_SeqId_4__1032149468. This is harmless and can happen in rare cases (see HBASE-8547)
Bug: RegionServer JVM crash when scan
Status: Resolved | Priority: Critical | Resolution: Duplicate
Created: 31/Oct/22 08:25
Affects Version/s: 2.3.5

hs_err_pid790137.log

#

# A fatal error has been detected by the Java Runtime Environment:

#

#  SIGSEGV (0xb) at pc=0x00007f0389b452a6, pid=790137, tid=139652643038976

#

# JRE version: Java(TM) SE Runtime Environment (8.0_20-b26) (build 1.8.0_20-b26)

# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.20-b23 mixed mode linux-amd64 compressed oops)

# Problematic frame:

# V  [libjvm.so+0x5832a6]  G1ParScanThreadState::copy_to_survivor_space(oopDesc*)+0x226

#

# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

#

# If you would like to submit a bug report, please visit:

#   http://bugreport.sun.com/bugreport/crash.jsp

#

 

---------------  T H R E A D  ---------------

 

Current thread (0x00007f038405d000):  GCTaskThread [stack: 0x00007f036a1e2000,0x00007f036a2e3000] [id=790310]

 

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00000000000000e0

 

Registers:

RAX=0x0000000000000000, RBX=0x00007f038a4ee240, RCX=0x0000000000000003, RDX=0x0000000000000001

RSP=0x00007f036a2e1310, RBP=0x00007f036a2e1370, RSI=0x00000004f00ebbe8, RDI=0x00000000080007a8

R8 =0x000000009e01d77d, R9 =0x00007f036a2e1590, R10=0x00000004e7c08208, R11=0x00007f0384994900

R12=0x00000004f00ebbe8, R13=0x00007f038497ca80, R14=0x00007f036a2e1590, R15=0x00007f036a2e1590

RIP=0x00007f0389b452a6, EFLAGS=0x0000000000010246, CSGSFS=0x0000000000000033, ERR=0x0000000000000004

  TRAPNO=0x000000000000000e
Bug: Add block bytes scanned metrics
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Dec/22 14:18

Initially in HBASE-14978, but more recently improved in HBASE-27558 and HBASE-27570, we now track the amount of block bytes scanned by read requests. Random read use-cases (which might read 1 small row from a 64kb block) and heavily filtered scans (which might scan many blocks in order to return a needle in a haystack row) can unknowingly cause RegionServers to do more work than a user expects. Now that we have block IO accounting, we should expose this to the user in the form of jmx metrics and ScanMetrics.
Bug: AsyncRequestFutureImpl unnecessarily clears meta cache for full server
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Dec/22 14:02

In AsyncRequestFutureImpl.receiveGlobalFailure, the meta cache is cleared twice. First the entire cache is cleared for the serverName, then each individual region's cache is further cleared. See [here.|https://github.com/apache/hbase/blob/branch-2/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java#L766-L780]

This behavior was introduced in HBASE-19900, which was a Jira more related to return value corruption rather than meta cache. I believe the change was an accident in the refactoring. Take a look at the commit [here.|https://github.com/apache/hbase/commit/e2636c81f7700a34368d73079375dbce6e180547#diff-7c58ffd83c150488599591ed5a3a068599646ebdbbbfdcd2233386e5472cca35L709]

In the commit, the cleanServerCache call should have been wrapped in tableName == null. Then the updateCachedLocations call should have been wrapped in tableName != null. So we should only go down one path or another based on the existence of a tableName.

Full server cache clears are extremely expensive in a high concurrency client. This is especially true in AsyncProcess, where a single request might hit many servers. We should reinstate the tableName checks so that we only do one or the other.
Bug: AsyncProcess should not clear meta cache for RejectedExecutionException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Nov/22 21:18

Batch requests using AsyncProcess get submitted to a thread pool. If the pool is too small or backing regionservers are overloaded, the pool may reject requests. Currently this causes a cache clear, which only exacerbates issues in most cases.

If there's a real reason to clear cache, the individual failing tasks that actually get accepted into the pool should be enough to appropriately clear the cache. So let's special case REE in AsyncRequestFutureImpl to not clear cache.
Bug: Locating regions for all actions of batch requests can exceed operation timeout
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Nov/22 19:34

This affects both AsyncTable and branch-2's Table. It's a follow-on from HBASE-27487.

In both cases, for a batch request, the first thing that must be done is resolve region locations for all actions in a batch. This is done in sequence for all actions, and the timeout is not checked after each action. If meta is slow or a sufficiently large batch is sent, the time to sequentially resolve all actions may exceed the operation timeout.

Currently, the code will eventually check the operation timeout after all locations have been resolved and before submitting the MultiRequests. We should check the timeout before each call to locate a region, and abort early if it's exceeded.
Bug: Improve BucketAllocatorException log to always include HFile name
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Oct/22 13:52

In HBASE-26444 the logging around BucketAllocatorException was improved. One thing I'm noticing is that I'm often seeing logs like this:
{quote}2022-10-03T13:13:18,284 [main-BucketCacheWriter-0] WARN org.apache.hadoop.hbase.io.hfile.bucket.BucketCache: Most recent failed allocation after 60000 ms; Message: Allocation too big size=771956; adjust BucketCache sizes hbase.bucketcache.bucket.sizes to accomodate if size seems reasonable and you want it cached.
{quote}
According to the logic in the code, this means the failure was for an HFileBlock but the block's getColumnFamily and/or getTableName were null. I'm not sure why that is the case, we don't have any special customizations here. But as a result we don't print any indication of the hfile name. We should add an option to fallback here on the default printing which includes the re.getKey() (HFile name) in all cases.