av9ash/GitBugs huggingface dataset:
Bug: New version of Region#getRowLock with timeout
Status: Open | Priority: Major | Resolution: 
Created: 19/Aug/24 22:01
Affects Version/s: 2.6.0

Region APIs are LimitedPrivate for Coprocs. One of the APIs provided by HBase for Coproc use is to acquire row level read/write lock(s):
{code:java}
/**
 * Get a row lock for the specified row. All locks are reentrant. Before calling this function
 * make sure that a region operation has already been started (the calling thread has already
 * acquired the region-close-guard lock).
 * <p>
 * The obtained locks should be released after use by {@link RowLock#release()}
 * <p>
 * NOTE: the boolean passed here has changed. It used to be a boolean that stated whether or not
 * to wait on the lock. Now it is whether it an exclusive lock is requested.
 * @param row      The row actions will be performed against
 * @param readLock is the lock reader or writer. True indicates that a non-exclusive lock is
 *                 requested
 * @see #startRegionOperation()
 * @see #startRegionOperation(Operation)
 */
RowLock getRowLock(byte[] row, boolean readLock) throws IOException; {code}
The implementation by default uses config "hbase.rowlock.wait.duration" as row level lock timeout for both read and write locks. The default value is quite high (~30s).

While updating the cluster level row lock timeout might not be worth for all use cases, having new API that takes timeout param would be really helpful for critical latency sensitive Coproc APIs.

 

The new signature should be:
{code:java}
RowLock getRowLock(byte[] row, boolean readLock, int timeout) throws IOException; {code}
Bug: Use CompletableFuture instead of fixed sleep during the move operation at master 
Status: Patch Available | Priority: Major | Resolution: 
Created: 14/Sep/24 06:45

During move operation master submits the RegionTransitionProcedure and waits for the completion of this procedure, current wait mechanism is based on fixed sleep intervals of 1 second.  The procedure could have finished earlier than this 1 second sleep.. optimize this by using CompletableFuture and return after the completion of TRSP procedure.
Bug: Implement chunked persistence of backing map for persistent bucket cache.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Aug/24 07:47
Affects Version/s: 2.6.0

The persistent bucket cache implementation feature relies on the persistence of backing map to a persistent file. the protobuf APIs are used to serialise the backing map and its related structures into the file. An asynchronous thread periodically flushes the contents of backing map to the persistence file.

The protobuf library has a limitation of 2GB on the size of protobuf messages. If the size of backing map increases beyond 2GB, an unexpected exception is reported in the asynchronous thread and stops the persister thread. This causes the persistent file go out of sync with the actual bucket cache. Due to this, the bucket cache shrinks to a smaller size after a cache restart. Checksum errors are also reported.

This Jira tracks the implementation of introducing chunking of the backing map to persistence such that every protobuf is smaller than 2GB in size.

Thanks,
Janardhan
Bug: Introduce Cache for SFT instances created via StoreFileTrackerFactory
Status: Open | Priority: Major | Resolution: 
Created: 23/Sep/24 13:15

As part of HBASE-28564 the creation of HStoreFile is made SFT aware and anytime a store file is created, it need SFT instance.
Now with this all the interaction of HStorefiles need SFT instance.
In case of FileBasedStoreFileTracker, each instance of it loads the backed .filelist file and this can be a costly operation in S3


This Jira targets to introduce a cache layer at StoreFileTrackerFactory for SFT instances 
per each
_TableName + Region + CF + Mode_ (Write/ReadOnly mode of SFT)

More detailed thought process around the same [here|https://github.com/apache/hbase/pull/5939#discussion_r1759312918]
{code:java}
Every time we create a StoreFileTracker object it will have no state, and so it will either need to go to the filesystem and list the directory or read the tracker file depending on which type it is in order to initialize as soon as we try to use it.
It's fine... because the original code causes IO to happen also.. however,
What do you think about the possibility of reuse? This is a more general question than a comment about this particular call site. Should the StoreFileTrackerFactory cache instances and return the cached instances that match the arguments to StoreFileTrackerFactory.create() rather than make a new instance? Can StoreFileTracker instances be made thread safe so they can be cached and shared?
If we have reuse, and all the relevant filesystem ops go through the StoreFileTracker, then we could potentially save a lot of filesystem or object store IO, because a reused StoreFileTracker would have the ground truth already and would not need to go to the filesystem or object store and do IO in order to e.g. return the StoreFileInfo of a given path.
{code}
----
Bug: Optimise memory utilisation retrieval of bucket-cache from persistence.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/24 05:52
Affects Version/s: 2.7.0

During the persistence of bucket-cache backing map to a file, the backing map is divided into multiple smaller chunks and persisted to the file. This chunking avoids the high memory utilisation of during persistence, since only a small subset of backing map entries need to persisted in one chunk.

However, during the retrieval of the backing map during the server startup, we accumulate all these chunks into a list and then process each chunk to recreate the in-memory backing map. Since, all the chunks are fetched from the persistence file and then processed, the memory requirement is higher.

The retrieval of bucket-cache from persistence file can be optimised to enable the processing of one chunk at a time to avoid high memory utilisation.

Thanks,

Janardhan
Bug: Optimise bucket cache persistence by reusing backmap entry object.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Sep/24 13:58
Affects Version/s: 3.0.0-beta-2

During the persistence of backing map entries into the backing map file, we create a new BackingMapEntry.Builder for each entry in the backing map. This can be optimised by using a single BackingMapEntry.Builder object and using it to build each entry during serialisation.

This Jira tracks the optimisation by avoiding multiple builder objects.

Thanks,
Janardhan
Bug: Support for RS group endpoints in HBase REST
Status: Open | Priority: Major | Resolution: 
Created: 17/Sep/24 10:10

*Description:*
Currently, the HBase REST server does not support endpoints for region server grouping. This feature request is to add new endpoints that will allow users to interact with and manage region server groups through the HBase REST API.

*Goals:*
 # Expose endpoints to create, list, update, and delete region server groups along with other APIs as necessary, supported by Java client
 # Ensure endpoints support necessary authentication and authorization.
 # Provide appropriate error handling and logging for the new endpoints.

*Acceptance Criteria:*
 # New endpoints for region server grouping are added to the HBase REST server.
 # Endpoints are thoroughly documented.
 # Unit and integration tests are created and passing. If there is no IT for existing features also, share manual test details.

*Technical Details:*
 * New endpoints should follow RESTful principles.
 * Authentication/authorization mechanisms must be consistent with existing endpoints.
 * Ensure compatibility with existing HBase version and REST server.

*For contributors looking to make a contribution here, please use following for quickstart:*
 * You can refer relevant code at [https://github.com/apache/hbase/tree/master/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest]
 * You can refer documentation of existing endpoints at [https://hbase.apache.org/book.html#_rest] and design new endpoints in-line with current standards
 * Build HBase code locally. Follow [https://hbase.apache.org/book.html#_build_a_tarball]
 * Start a localhost rest server. Follow [https://hbase.apache.org/book.html#_starting_and_stopping_the_rest_server]
 * Also feel free to use a starter REST postman collection at [https://gist.github.com/NihalJain/7db91823bd140e41953a88eb3df56eae]
Bug: BucketCache.notifyFileCachingCompleted may throw IllegalMonitorStateException 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jul/24 15:37
Affects Version/s: 2.6.0

If the prefetch thread completes reading the file blocks faster than the bucket cache writer threads are able to drain it from the writer queues, we might run into a scenario where BucketCache.notifyFileCachingCompleted may throw IllegalMonitorStateException, as we can reach [this block of the code|https://github.com/wchevreuil/hbase/blob/684964f1c1693d2a0792b7b721c92693d75b4cea/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java#L2106]. I believe the impact is not critical, as the prefetch thread is already finishing at that point, but nevertheless, such error in the logs might be misleading.
Bug: Improve BackupLogCleaner naming, debug logging
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Sep/24 19:08
Affects Version/s: 2.6.0

While implementing HBase's incremental backups across a few hundred clusters, we continue to step on some rakes. Now and again, we find old WALs piling up due to a poorly cleaned up BackupInfo, or a bug in the BackupLogCleaner, etc.

The BackupLogCleaner is difficult to debug for a couple of reasons:
 # It has a lack of useful debug logging
 # It has [a misleadingly named method|https://github.com/HubSpot/hbase/blob/2d08fa67dfe9458260bc0be3ebc7bbd769850190/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/master/BackupLogCleaner.java#L83-L83] (this method returns the newest backup ts, not the oldest)

I'm going to introduce a small refactor that will alleviate these pain points.
Bug: Implement asynchronous retrieval of bucket-cache data from persistence.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Aug/24 14:01
Affects Version/s: 2.6.0

    During the retrieval of data from bucket cache persistence file, a transient structure that stores the blocks ordered by filename is constructed from the backing map entries. The  population of this transient structure is done during the server start-up. This process increases the region-server startup time, if the bucketcache has large number of blocks.

This population happens inline with the server restart and blocks the server for several minutes. This makes the server restart inconvenient for the external users. Restarts during upgrade can run into timeout issues due to this delay in the server startup.

 Hence, the recommendation in this Jira is to make the cache-retrieval asynchronous to the server startup. During a server startup, a new thread is spawn that reads the persistence file and creates the required structures from persistence file. The server continues with the restart and does not wait for the bucket-cache initialisation to complete.  

 Note that the bucket cache is not available immediately for usage and will only be ready to use after the data is repopulated from persistence into memory.

Thanks,
Janardhan
Bug: Use thirdparty protobuf for REST interface in HBase 2.x
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jul/24 18:08
Affects Version/s: 2.5.9

This change has already been done in branch 3+ as part of the protobuf 2.5 removal,
We just need to backport it to 2.x.

This removes the requirement of having unshaded protobuf 2.5.0 on the hbase-rest client classpath.
Bug: Increment Inconsistency in Replication
Status: Open | Priority: Major | Resolution: 
Created: 10/Sep/24 07:18

*Issue:*
Consistency is not achieved for Increment operation in replication.

*Setup:*
Lets have two HBase clusters 'cluster1' and 'cluster2' and both are added as peers to each other in them and both have replication enabled. There is a counter cell with initial value '2' in a table. There is a replication delay from 'cluster1' to 'cluster2'.

*Actions:*
1. Perform increment in 'cluster1' with 1.
2. Perform increment in 'cluster2' with 2.

*Expected Behaviour:*
The value in the counter cell should be 5.

*Actual Behaviour:*
The value in the counter cell is 4.

*Analysis:*
1. After increment in 'cluster1', the value became 2 in 'cluster1'.
2. The replication from 'cluster1' to 'cluster2' gets initiated.
3. But there is a replication delay from 'cluster1' to 'cluster2' and within that timeframe, increment in 'cluster2' performed.
4. Now the value is 4 in 'cluster2' and it got replicated to 'cluster1'.
5. Since the replication is cell-based, not operation based and the 'cluster2' increment is the latest, value 4 from 'cluster2' overrides value 3 in 'cluster1'.

*Steps to reproduce:*
Add a coprocessor in 'cluster2' with 'Thread.sleep' in 'preWALAppend' to simulate replication delay.

*Inference:*
During debugging, it was discovered that the replication is cell-based, meaning the entire cell is being replicated rather than the specific operation being performed. This method works for other operations such as put and delete operations because it resolves the inconsistency problem by utilising the timestamp and version of the cell. However, for increment operations, which rely on the cell's previous value, this method is not successful.
Bug: Logging Enhancement on Configuration Setting in RpcConnection
Status: Open | Priority: Major | Resolution: 
Created: 06/Sep/24 03:00
Affects Version/s: 3.0.0-beta-1

The original implementation of the {{getConnectionHeader()}} method in {{RpcConnection}} does not include detailed logging regarding the configuration of the Crypto AES feature. This lack of logging can lead to several issues: # 
*Undetected Crypto AES Configuration:* If Crypto AES is enabled or disabled without proper logging, administrators may not be aware of the active encryption settings. This can lead to a misconfigured environment where sensitive data is not properly secured, or unnecessary resources are consumed by enabling encryption when it's not required.
 # 
*Troubleshooting Challenges:* Without logs that clearly indicate whether Crypto AES is enabled or disabled, diagnosing issues related to encrypted communication becomes significantly harder. In the event of a communication failure or performance issue, the lack of logs may hinder root cause analysis, delaying resolution.
 # 
*Unintended Configuration Behavior:* If the configuration is altered or misconfigured (e.g., Crypto AES is inadvertently disabled), the absence of logs makes it difficult to confirm that the system is operating as intended. This could lead to security vulnerabilities or unexpected behavior that goes unnoticed.

 
*Expected Behavior:* 
Enhanced logging should provide clear insights into the configuration state of Crypto AES during the establishment of the connection header: * 
If Crypto AES is enabled, the system should log the setup of the Cipher transformation, including the key and default value.
 * 
If Crypto AES is disabled, the system should log this state, ensuring that the administrator is aware and that this is the intended configuration.

 
*How-to-Fix:*
To address these issues, the code can be enhanced by inserting specific log statements that capture the state of the Crypto AES configuration. These logs should be added right before the Cipher transformation is set up or bypassed, ensuring that the configuration's impact on the system is well-documented.
Bug: Enhancing Logging for Connection Implementation in ConnectionFactory
Status: Open | Priority: Major | Resolution: 
Created: 06/Sep/24 02:59
Affects Version/s: 3.0.0-beta-1

The system lacks visibility into the specific connection implementation being used during the creation of a connection. This can lead to issues when troubleshooting connection-related problems, particularly when a custom connection implementation is specified. If the configuration is incorrect or the custom class is inaccessible, it is difficult to diagnose the issue without detailed logs. As a result, users may encounter unexpected behavior without a clear understanding of what went wrong, making it challenging to identify and resolve configuration or implementation errors.
 
*Expected Behavior:* 
The system should log detailed information about the connection implementation being used or attempted during the creation of a connection. If a custom connection class is specified, the system should log the attempt and the outcome, including any errors encountered. If the default implementation is used, this should also be logged.
 
*How-to-Fix:*
To address this issue, we propose introducing logging in the {{createConnection}} method of the {{ConnectionFactory}} class. The logs will provide clear feedback about whether the default asynchronous connection implementation ({{{}ConnectionOverAsyncConnection{}}}) is used or if a custom implementation is specified. Additionally, if a custom connection implementation fails due to an inaccessible or misconfigured class, detailed error logs will be generated. This enhancement will significantly improve the ability to diagnose and resolve connection-related issues, ensuring that any configuration or implementation errors are promptly identified and addressed.
Bug: AsyncTableImpl should call coprocessor callbacks in a defined order
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/24 14:44

To call a coprocessor endpoint asynchronously, you start by calling {{AsyncTable#coprocessorService()}}, which gives you a {{CoprocessorServiceBuilder}}, and a few steps later you can talk to your coprocessor over the network. One argument to {{AsyncTable#coprocessorService()}} is a {{CoprocessorCallback}} object, which contains several methods that will be called during the lifecycle of a coprocessor endpoint call. {{AsyncTableImpl}}'s implementation of {{AsyncTable#coprocessorService()}} wraps your {{CoprocessorCallback}} with its own that delegates the work to a thread pool. A snippet of this:
{code}
      @Override
      public void onRegionComplete(RegionInfo region, R resp) {
        pool.execute(context.wrap(() -> callback.onRegionComplete(region, resp)));
      }
...
      @Override
      public void onComplete() {
        pool.execute(context.wrap(callback::onComplete));
      }
{code}
The trouble with this is that your implementations of {{onRegionComplete()}} and {{onComplete()}} will end up getting called in a random order, and/or at the same time. The tasks of calling them are delegated to a thread pool, and the completion of those tasks is not waited on, so the thread pool can choose any ordering it wants to. Troublingly, {{onComplete()}} can be called before the final {{onRegionComplete()}}, which is an violation of [the contract specified in the {{CoprocessorCallback#onComplete()}} javadoc|https://github.com/apache/hbase/blob/41dd87cd908d4d089d0b8cff6c88c01ed60622c5/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncTable.java#L671].

I discovered this while working on HBASE-28770. I found that {{AsyncAggregationClient#rowCount()}} returns incorrect results 5-10% of the time, and this bug is the reason. Other {{AsyncAggregationClient}} methods I presume are similarly affected.
Bug: Client always send flush requests twice when flush procedure is disabled
Status: Open | Priority: Major | Resolution: 
Created: 02/Sep/24 12:32
Affects Version/s: 2.6.0

When hbase disable flush procedure at server side, whether flush procedure is diabled or not at clien side, client always sends flush requests twice.

In current hbase, There are four situations about flush request:
1. Both server and client enables flush Procedure, Client only sends a flush request by flush procedure.
2. Server disabled flush procedure and client enables flush procedurce, Client sends a flush request by flush procedure, after failure ,it will sends a flush request by old flush implementation
3. Server enables flush procedure and client disables flush procedure, Client only send a flush request by flush procedure
4. Both server and client disables flush Procedure, Client alwalys sends a flush request by flush procedure at first, after failure ,it will sends a flush request by old flush implementation 

 

The situation 1/2 is no problem.
For situation 3, I think client should only send a flush request by old flush implementation instead of flush procedure
For situation 4, I think client should only send a flush request by old flush implementation, instead of first using flush procedure, and then switching to old flush implementation after failing
Bug: [hbase-operator-tools] Upgrade log4j to 2.17.2
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 02/Sep/24 09:30
Affects Version/s: hbase-operator-tools-1.2.0

After HBASE-26723, we use log4j2.properties file instead of log4j2.xml, and upgrade log4j to 2.17.2 to resolve the issue of log4j2.properties not supporting old syntax.

When using hbck2, if the log4j2.properties from the HBase classpath is loaded, an exception will occur due to the old syntax. Therefore, also upgrade hbck log4j2 to 2.17.2.
Bug: [hbase-thirdparty] Finish hbase-thirdparty 4.1.8 release
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Aug/24 19:59

# Release the artifacts on repository.apache.org
 # Move the binaries from dist-dev to dist-release
 # Push tag 4.1.8RC1 as tag rel/4.1.8
 # Release thirdparty-4.1.8 on JIRA https://issues.apache.org/jira/projects/HBASE/versions/12354962
 # Add release data on [https://reporter.apache.org/addrelease.html?hbase]
 # Send announcement email
Bug: mTLS client hostname verification doesn't work with OptionalSslHandler
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Aug/24 16:48
Affects Version/s: 3.0.0-beta-1

Netty's OptionalSslHandler cannot carry hostport information to SslEngine, hence HBASE-27673 fixed the TLS-only case only. We need to have a custom handling for the plaintext-enabled TLS mode in order to support client hostname verification in that case too.
Bug: Fix typo in Jenkinsfile_Github for jdk8 hadoop2 check
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jul/24 14:15

https://github.com/apache/hbase/blob/9dee538f65d84a900724d424c71793dff46e9684/dev-support/Jenkinsfile_GitHub#L314

This line

PR JDK8 Hadoop3 Check Report

Should be

PR JDK8 Hadoop2 Check Report
Bug: Exclude samples and release-documentation zip of jaxws-ri from output tarball
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Aug/24 16:35

Found this while (testing HBASE-28070 and) I was checking lib folder for extracted assembly for master. I guess this must be a problem for all branches fixed for HBASE-28760

Following zip files are there in hbase-4.0.0-alpha-1-SNAPSHOT/lib:
* samples-2.3.2.zip
* release-documentation-2.3.2-docbook.zip