av9ash/GitBugs huggingface dataset:
Bug: Finish 3.0.0-beta-1 release
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Dec/23 08:11

# Release the artifacts on repository.apache.org 
# Move the binaries from dist-dev to dist-release
# Add xml to download page(via HBASE-28265)
# Push tag 3.0.0-beta-1RC0 as tag rel/3.0.0-beta-1
# Release 3.0.0-beta-1 on JIRA https://issues.apache.org/jira/projects/HBASE/versions/12353291
# Add release data on https://reporter.apache.org/addrelease.html?hbase
# Send announcement email
Bug: Enhance ByteBufferUtils.readVLong to read more bytes at a time
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Dec/23 14:22

Currently, ByteBufferUtils.readVLong is used to decode rows in all data block encodings in order to read the memstoreTs field. For a data block encoding like prefix, ByteBufferUtils.readVLong can surprisingly occupy over 50% of the CPU time in BufferedEncodedSeeker.decodeNext (which can be quite a hot method in seek operations).

 

Since memstoreTs will typically require at least 6 bytes to store, we could look to vectorize the read path for readVLong to read 8 bytes at a time instead of a single byte at a time (like in https://issues.apache.org/jira/browse/HBASE-28025) in order to increase performance.

 

Attached is a CPU flamegraph of a region server process which shows that we spend a surprising amount of time in decoding rows from the DBE in ByteBufferUtils.readVLong.
Bug: MOB HFiles are expired earlier than their reference data
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Dec/23 14:28
Affects Version/s: 2.5.0

I have observed that mob HFiles are expired earlier than their reference data. Upon reviewing the relevant code, it was observed that the standard expired timestamp is not accurately set. The issue arises from the fact that if we do not set the calendar to the MILLISECOND level, a three-digit random number is appended to the end of the expired timestamp, leading to inaccuracies.

For instance, if we write mob data to a Time-To-Live (1 day) table at 22:00 on 12/20/2023, and the ExpiredMobFileCleaner begins to work at 20:00 on 12/21/2023, the timestamp of the mob HFile will be parsed as 1703001600000, and the StandardExpiredTS is calculated as 1703001600314 with the three-digit random number as MILLISECOND. Consequently, the StandardExpiredTS >= MobHfileTS, thus indicating that the mob HFile should be expired. However, it is observed that this mob HFile only exists for 22 hours, which is less than the specified one-day-time threshold.
Bug: Add sun.net.dns and sun.net.util to the JDK11+ module exports in the hbase script
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Dec/23 13:16

As noted in HBASE-28247, HBase can run into module permission issues that are not handled in the current JDK11 options in the hbase startup script.

The surefire test config also includes some JDK17 specific options, we should also add those as needed.

We are not yet aware of any additional JVM options required by Java 21.
Bug: Evaluate using hbase-shaded-client-byo-hadoop for Spark connector
Status: Resolved | Priority: Major | Resolution: Done
Created: 22/Nov/23 12:11

Since 3.2 Spark now uses hadoop-client-api and hadoop-client-runtime.
While we don't actually specify what HBase libraries are needed on the Spark client side for the connector, at least the Cloudera docs specify the classes provided by "hbase mapredcp"
which includes the full unshaded Hadoop JAR set.

Investigate whether  *hbase-shaded-client-byo-hadoop* and the *hbase-client-api* and *hbase-client-runtime* is enough for the connector, and if yes, document how to set the Spark classpath.

Alternatively, if *hbase-shaded-client-byo-hadoop*  is not enough, check if *hbase-shaded-mapreduce* plus the above two shaded Hadoop client JAR provides everything needed.
Bug: Document Spark classpath requirements for the Spark connector
Status: Open | Priority: Major | Resolution: 
Created: 22/Nov/23 12:21

The README for the Spark connector details the classpath requirements for the HBase server side, but does not talk about how to set up the Spark classpath for HBase.

The Cloudera docs [https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/accessing-hbase/topics/hbase-configure-spark-connector.html] suggest using "hbase mapredcp" It is, however inconsistent, as "hbase mapredcp" includes the unshaded hadoop libraries, while the example command line omits the hadoop libraries (and seem to depend on the on the existing Hadoop JARs on the Spark classpath).

Figure this out, and update the documentation.

UPDATE:
SPARK-33618 has reverted to using unshaded Hadoop.
We only have to document the `hbase mapredcp` option.
Bug: Bump jruby to 9.3.13.0 and related joni and jcodings to 2.2.1 and 1.0.58 respectively
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Dec/23 09:40

Given branch-2 including branch-2.6 is already on 9.3.9.0, we should bump to at least 9.3.13.0. This will fix the bundled *org.bouncycastle : bcprov-jdk18on : 1.71* having [CVE-2023-33201|https://nvd.nist.gov/vuln/detail/CVE-2023-33201] from our classpath for the least.

As a follow up can try to bump to latest 9.4.x line. Otherwise I can try to work directly on HBASE-28250 as well, although this may not be straight forward and would require some good testing.

Please let me know what others think.
Bug: The shouldPerformMajorCompaction in StripeCompactionPolicy is not implemented and does not periodically trigger the major of the table
Status: Open | Priority: Major | Resolution: 
Created: 27/Dec/23 07:26

shouldPerformMajorCompaction in StripeCompactionPolicy is not implemented, always returns false,
HRegionServer -> CompactionChecker -> chore() -> s.shouldPerformMajorCompaction() is always false, 
and the s.triggerMajorCompaction(); cannot be called, This causes the forceMajor variable to be false all the time.
Bug: After triggering a Major Compaction, clearing the force flag has no effect.
Status: Open | Priority: Major | Resolution: 
Created: 19/Dec/23 08:43
Affects Version/s: 2.2.6

In the HStore.java class, in the requestCompaction method, after manually triggering Major Compaction, this.forceMajor is set to true.

When clearing the force flag, execute the following code:
{code:java}
this.forceMajor = this.forceMajor && !request.isMajor(); {code}
However, when the table is applied to a Stripe Compaction strategy (other compaction policy may have similar problems), request.isMajor() is always false, because in CompactionRequestImpl the setIsMajor method is not always called. We need to check the calling status of the setIsMajor method. Otherwise, after a major is manually triggered, forceMajor will always be true, and subsequent periodic compactions will also be forced to a major compaction.
Bug: PrefetchExecutor should not run for files from CFs that have disabled BLOCKCACHE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Nov/23 17:36
Affects Version/s: 2.4.17

HFilePReadReader relies on the return of CacheConfig.shouldPrefetchOnOpen return to decide if it should run the PrefetchExecutor for the files. 

Currently, CacheConfig.shouldPrefetchOnOpen returns true if "hbase.rs.prefetchblocksonopen" is set to true at the config, OR PREFETCH_BLOCKS_ON_OPEN is set to true at CF level.

There's also the CacheConfig.shouldCacheDataOnRead, which returns true if both hbase.block.data.cacheonread is set to true at the config AND BLOCKCACHE is set to true at CF level.

If BLOCKCACHE is set to false at CF level, HFilePReadReader will still run the PrefetchExecutor to read all the file's blocks from the FileSystem, but then would find out the given block shouldn't be cached. 

I believe we should change CacheConfig.shouldPrefetchOnOpen to return true only if CacheConfig.shouldCacheDataOnRead is also true.
Bug: BucketCache.blocksByHFile may leak on allocationFailure or if we reach io errors tolerated
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Nov/23 16:00
Affects Version/s: 2.4.17

We add blocks to BucketCache.blocksByHFile on doDrain before we actually had successfully added the block to the cache. We may still fail to cache the block if it is too big to fit any of the configured bucket sizes, or if we fail to write it in the ioengine and reach the tolerated io errors threshold. In such cases, the related block would remain in the BucketCache.blocksByHFile indefinitely.
Bug: PrefetchExecutor should stop once cache reaches capacity
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Oct/23 21:50
Affects Version/s: 2.6.0

The prefetch executor runs a full scan on regions in the background once regions are opened, if the "hbase.rs.prefetchblocksonopen" property is set to true. However, if the store file size is much larger than the cache capacity, we should interrupt the prefetch once it has reached the cache capacity, otherwise it would just be triggering evictions of little value, since we don't have any sense of block priority at that point. It's better to stop the read, and let client reads cause the eviction of LFU blocks and cache the most accessed blocks.
Bug: Put the cached time at the beginning of the block; run cache validation in the background when retrieving the persistent cache
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Oct/23 00:01
Affects Version/s: 2.6.0

In HBASE-28004, we added a "cached time" long at the end of each block on the bucket cache. We also record the cached time in the backing map we persist to disk periodically, in order to retrieve the cache upon crashes/restarts. The persisted backing map includes the last modification time of the cache itself.

On restarts, once we read the backing map from the persisted file, we compare the last modification time of the cache recorded there against the last modification time of the cache. If those differ, it means the cache has been updated after the backing map has been persisted, so the backing map might not be accurate. We then iterate though the backing map entires and compare the entries cached time against the related block in the cache, and if those differ, we remove the entry from the map. 

Currently this validation is made at RS initialisation time, but with caches as large as 1.6TB/30M+ blocks, it can last to an hour, meaning the RS is useless over that time. This PR changes this validation to be performed in the background, whilst direct accesses to a block in the cache would also perform the "cached time" comparison.

This PR also moves the "cached time" to the beginning of the block in the cache, instead of the end. We noticed that with the "cached time" at the end we can fail to ensure consistency at some conditions. Consider the following: 
1) A block B1 of size S gets allocated at offset 0 with cached time T1;
2) The backing map is persisted, containing B1 at offset 0 and cached time T1;
3) B1 is evicted. It's offset in the cache is now free, however its contents are still there, including the cached time T1 at its end;
4) A new block B2 of size S/2 gets allocated at offset 0 with cached time T2;
5) RS crashes before the backing map gets saved, so the persisted backing map still has only the reference to B1, but not B2;
6) At restart, we run the validation. Because B2 was half the size of B1, we haven't overridden B1 cached time from the cache, so we will successfully validate B1, although its content is now half overridden by B2.
Bug: Resource leak for memstore and flush
Status: Open | Priority: Major | Resolution: 
Created: 24/Dec/23 16:43
Affects Version/s: 3.0.0-beta-2


{code:java}
2023-12-16T23:44:13,992 ERROR [RpcServer.default.FPBQ.Fifo.handler=22,queue=1,port=16020] util.ResourceLeakDetector: .hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2845)
        org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2718)
        org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2692)
        org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2683)
        org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1850)
        org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1660)
        org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1615)
        org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1598)
        org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.process(UnassignRegionHandler.java:132)
        org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:750)
2023-12-17T15:07:59,111 ERROR [regionserver/core-1-1:16020-longCompactions-0] util.ResourceLeakDetector: .java:45)
        java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        org.apache.hadoop.hbase.util.ReflectionUtils.instantiate(ReflectionUtils.java:61)
        org.apache.hadoop.hbase.util.ReflectionUtils.newInstance(ReflectionUtils.java:98)
        org.apache.hadoop.hbase.regionserver.HStore.getMemstore(HStore.java:377)
        org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:283)
        org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:6904)
        org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1173)
        org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1170)
        java.util.concurrent.FutureTask.run(FutureTask.java:266)
        java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        java.util.concurrent.FutureTask.run(FutureTask.java:266)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:750)
{code}
Bug: Change zookeeper.zone.parent and restart hbase cluster, hmaster will be stuck waiting for hbase:meta online 
Status: Open | Priority: Major | Resolution: 
Created: 21/Dec/23 10:42
Affects Version/s: 2.2.7

In our scenario, we usually change zookeeper.zone.parent=/hbase-unsecure to /hbase-secure when enabling kerberos authentication. after restart the hbase cluster, we will almost certainly be stuck on master initalization. 

we can see hmaster stuck in waiting for the InitMetaProcedure finish

!image-2023-12-21-18-34-33-586.png|width=1221,height=220!

In the regionserver log, we find the regionserver receive duplicate open meta region request, and it does not respond to hmaster the second time. so the hmaster wait the procedure in a long time.

!image-2023-12-21-18-36-36-282.png|width=1527,height=163!

We suspect that both the servercrash procedure and hmaster startup trigger the procedure for the online meta table.

!image-2023-12-21-18-41-55-819.png|width=714,height=331!

 

hmaster log this, so it will send a assign meta request to regionserver

!image-2023-12-21-18-42-47-764.png|width=720,height=30!

 

So any suggestions to avoid this?
Bug: Fix broken ruby scripts and clean up logging
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Dec/23 18:17
Affects Version/s: 3.0.0-alpha-4

Some of the ruby scripts are broken in 3.x as they are referencing non-existent classes, these were deprecated earlier and removed in 3.x:
 * {{org.apache.hadoop.hbase.client.HBaseAdmin}}
 * {{org.apache.hadoop.hbase.HTableDescriptor}}

Following 4 scripts are failing:
{code:java}
NameError: missing class name org.apache.hadoop.hbase.client.HBaseAdmin
  method_missing at org/jruby/javasupport/JavaPackage.java:253
          <main> at region_status.rb:50
{code}
{code:java}
{NameError: missing class name org.apache.hadoop.hbase.HTableDescriptor
  method_missing at org/jruby/javasupport/JavaPackage.java:253
          <main> at replication/copy_tables_desc.rb:30
{code}
{code:java}
NameError: missing class name org.apache.hadoop.hbase.client.HBaseAdmin
  method_missing at org/jruby/javasupport/JavaPackage.java:253
          <main> at draining_servers.rb:28
{code}
{code:java}
NameError: missing class name org.apache.hadoop.hbase.client.HBaseAdmin
  method_missing at org/jruby/javasupport/JavaPackage.java:253
          <main> at shutdown_regionserver.rb:27
{code}
Bug: hbck should report previously reported regions with null region location
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Oct/23 23:04
Affects Version/s: 2.5.6

Operators bypassed some in progress TRSPs leading to a state where some regions were persistently in transition but hidden. 

Because the master builds its list of regions in transition by tracking TRSP, the bypass of TRSP removed the regions from the RIT list. This was expected, but I will propose a change to RIT tracking on another issue. 

The online hbck chore also did not report the inconsistency. This was not expected.

HBASE-28144 was another issue related to this incident, already fixed. 

Ensure that hbck will report as inconsistent regions where previously a location was reported but now the region location is null, if it is not expected to be offline.
Bug:  Bump jackson version to 2.15.2 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Dec/23 13:16

We should bump jackson to 2.15.2 as it is already move to this in hbase-thirdparty in HBASE-28093 

Also 2.14.1 has [sonatype-2022-6438.|https://github.com/FasterXML/jackson-core/issues/861]
Bug: Sync internal protobuf version for hbase to be same as hbase-thirdparty
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Dec/23 14:29

hbase-thirdparty 4.1.5 uses protobuf 3.24.3 while hbase is still on 3.21.12. This Jira is to sync internal protobuf version for hbase to be same as hbase-thirdparty.
Bug: The configs for using meta replica can conflict
Status: Open | Priority: Major | Resolution: 
Created: 13/Nov/23 07:53
Affects Version/s: 2.5.6

The config of "hbase.locator.meta.replicas.mode" can surpass switching off reading from meta replica by "hbase.meta.replicas.use".

UTs by setting hbase.meta.replicas.use=false and hbase.locator.meta.replicas.mode=LoadBalance can recur the problem.
Bug: ProcedureTestingUtility.restart is broken sometimes after HBASE-28199
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Dec/23 13:34

In ProcedureTestingUtility.restart, we will reuse the same ProcedureExecutor, so when restarting, we need to make sure that all procedures are not executed and then clear the scheduler.

But after HBASE-28199, we may add procedures back to scheduler after a CompletableFuture is completed, so even if all the PEWorker are terminated we could still add things to scheduler, which may break some tests.

We need to find a way to deal with this.
Bug: The snapshot operation encountered an NPE and failed.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Dec/23 02:38
Affects Version/s: 3.0.0-beta-1

Executing a merge region before the snapshot operation is completed will result in an NPE error for the snapshot operation and cause it to fail

Triggering logic
 # Execute the snapshot command.
 # Merge regions before the snapshot is completed.
 # After the merge is completed, the two parent regions will be cleaned up.
 # An NPE will be reported when the snapshot operation is performed on that parent region. The log is as follow:

{code:java}
2023-11-20T23:26:07,061 ERROR [PEWorker-13] procedure2.ProcedureExecutor: CODE-BUG: Uncaught runtime exception: pid=238720, ppid=238714, state=RUNNABLE, hasLock=true; SnapshotRegionProcedure c47539f1d6032ba1a037f5279c22baa0
java.lang.NullPointerException: null
        at org.apache.hadoop.hbase.master.procedure.SnapshotRegionProcedure.execute(SnapshotRegionProcedure.java:160) ~[hbase-server-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
        at org.apache.hadoop.hbase.master.procedure.SnapshotRegionProcedure.execute(SnapshotRegionProcedure.java:58) ~[hbase-server-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
        at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:921) ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1649) ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1395) ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75) ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1961) ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
        at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216) ~[hbase-common-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT]
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1988) ~[hbase-procedure-3.0.0-beta-1-SNAPSHOT.jar:3.0.0-beta-1-SNAPSHOT] {code}
Bug: [SFT] Add description for specifying SFT impl during snapshot recovery
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Dec/23 12:04
Affects Version/s: 4.0.0-alpha-1

HBASE-26286 added an option to clone_snapshot command that allows for specifying the SFT implementation during the snapshot recovery. This really useful when recovering snapshots imported from clusters not using the same SFT impl as the one where we are cloning it. Without this, the cloned/restore table will get created with the the SFT impl of its original cluster, requiring extra conversion steps using the MIGRATION tracker.

This also fix formatting for the "Bulk Data Generator Tool", which is currently displayed as a sub-topic of the SFT chapter. It should have it's own chapter.
Bug: HBCK2 filesystem command to fix version file fails when Master in not up
Status: In Progress | Priority: Major | Resolution: 
Created: 10/Dec/23 10:47
Affects Version/s: hbase-operator-tools-1.2.0

Recently while fixing an issue for a customer, I observed that the hbase master keeps restarting because of missing hbase version file. When trying to fix the issue using the hbck2 filesystem -f command, it kept trying to connect to the hbase master as a pre-check for supported hbase versions, but as the master was not up, the command never progressed beyond the connection creation step. It seems unnecessary to check supported versions in filesystem command as it is all offline checks.
Bug: Include shaded netty-all in hbase-shaded-mapreduce
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 28/Nov/23 13:23
Affects Version/s: 2.4.17

Since ZooKeeper TLS connection is supported within HBase, we need netty-all on the classpath. It's already present in hbase-shaded-client and we need it in hbase-shaded-mapreduce for MR clients too. netty-all is already included in pom.xml, but only with test-scope, so shading plugin doesn't add it to the jar. Patch will override it to compile-scope.
Bug: [JDK17] JVM crashes intermittently on aarch64
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Nov/23 02:59

The crash is related to the C1 JIT compiler. It's crashing while trying to compile RegionServerMetricsWrapperRunnable. See [https://github.com/adoptium/adoptium-support/issues/951] for a bit more details (no resolution yet).

I'm going to experiment with breaking up the run() method a bit to see if it can play nicer with the compiler. The recommendation from the adoptium folks was to try disabling the C1 compiler, which I'd rather not do.