av9ash/GitBugs huggingface dataset:
Bug: Clone Scan in ClientScanner to avoid errors with Scan re-used
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Oct/22 13:31

This has come up before in https://issues.apache.org/jira/browse/HBASE-1774 and https://issues.apache.org/jira/browse/HBASE-4891. The major pushback was around ScanMetrics, which relied on sharing a mutable Scan object.

Since https://issues.apache.org/jira/browse/HBASE-17584, ScanMetrics are available on ResultScanner and the method on Scan was deprecated (removed in master).

I think this issue became pretty urgent in https://issues.apache.org/jira/browse/HBASE-17167, when we started passing mvcc into the Scan object. If a user unknowingly reuses the Scan object, this can seem like data loss since the Scan will return none of the expected data.  We recently hit this in our upgrade from hbase client 1.2 to 2.4.6, where use-cases that had worked in 1.2 suddenly started returning no results in 2.4.6. It's very hard to debug.

I suggest that we now add the clone in master branch. For branch-2, I think we could put it behind a config param to preserve backwards compatibility of Scan.getScanMetrics. If the config param is enabled, scan cloning occurs and Scan.getScanMetrics will be inaccurate. Personally I think this is far better scenario, because data result accuracy is more important than metrics. But we can leave it to the user to decide, and provide a release note.
Bug: Should use 'org.apache.hbase.thirdparty.io.netty.tryReflectionSetAccessible'
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Nov/22 15:59

We current use '-Dio.netty.tryReflectionSetAccessible=true' in pom, but after shading, it should be '-Dorg.apache.hbase.thirdparty.io.netty.tryReflectionSetAccessible=true' now.
Bug: [hbase-connectors] Duplicate result when searching HBase by Spark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Nov/22 08:29

I've used hbase-connector for querying HBase table by Spark, and the following issue is what I've encountered recently:

When I run SQL:
{code:java}
select * from hbase_tabe where rowkey in ('r1', 'r2') or rowkey='r1'
{code}
I've got duplicate row "r1" in results.
In HBaseTableScanRdd, buildGets method forms two duplicate GET request for rowkey "r1". I think we might need a Set to make sure duplicate rowkey won't be request.
Bug: Bump spotless plugin to 2.27.2 and reimplement the 'Remove unhelpful javadoc stubs' rule
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Oct/22 15:20

spotless maven 2.27.2 is released and it contains https://github.com/diffplug/spotless/issues/1359, which allows the replacement to be null, so we can add back the 'Remove unhelpful javadoc stubs' rule now.
Bug: Separate slowlog thresholds for scans vs other requests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Dec/22 15:55

Scans by their nature are able to more efficiently pull back larger response sizes than gets. They also may take longer to execute than other request types. We should make it possible to configure a separate threshold for response time and response time for scans. This will allow us to tune down the thresholds for others without adding unnecessary noise for requests which are known to be slower/bigger.
Bug: Include more request information in slowlog for Scans
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Dec/22 18:19

Currently the slowlog only includes a barebones text format of the underlying protobuf Message fields. This is not a great UX for 2 reasons:
 # Most of the proto fields dont mirror the actual API names in our requests (Scan, Get, etc).
 # The chosen data is often not enough to actually infer anything about the request

Any of the API class's toString method would be a much better representation of the request. On the server side, we already have to turn the protobuf Message into an actual API class in order to serve the request in RSRpcServices. Given slow logs should be a very small percent of total requests, I think we should do a similar parsing in SlowLogQueueService. Or better yet, perhaps we can pass the already parsed request into the queue at the start to avoid the extra work. 

When hydrating a SlowLogPayload with this request information, I believe we should use {{Operation's toMap(int maxCols)}} method. Adding this onto the SlowLogPayload as a map (or list of key/values) will make it easier to consume via downstream automation. Alternatively we could use {{{}toJSON(){}}}.

We should also include any attributes from the queries, as those made aid tracing at the client level.

Edit: because of nuance related to handling multis and the adequacy of info available for gets/puts, we're scoping this issue down to focus on improving the information available on Scan slowlogs
Bug: log duplication issues in MasterRpcServices
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Dec/22 10:23

MasterRpcServices record audit log in privileged operations (grant, revoke) and vital apis like "execMasterService".
{code:java}
  public RevokeResponse revoke(RpcController controller, RevokeRequest request)
    throws ServiceException {
    try {
      ......
        server.cpHost.preRevoke(userPermission); // has audit log in AccessChecker
       ...... // removeUserPermission
        User caller = RpcServer.getRequestUser().orElse(null);
        if (AUDITLOG.isTraceEnabled()) {
          // audit log should record all permission changes
          String remoteAddress = RpcServer.getRemoteAddress().map(InetAddress::toString).orElse("");
          AUDITLOG.trace("User {} (remote address: {}) revoked permission {}", caller,
            remoteAddress, userPermission);
        }
        ......
  }
{code}
but I found a path from *server.cpHost.preRevoke(userPermission);* to {*}AccessChecker audit log{*}, which caused {*}log duplication{*}.
*_+grant/revoke -> AccessController.preGrant/Revoke -> preGrantOrRevoke -> AccessChecker.requireGlobalPermission/... -> logResult+_*
{code:java}
public void requireGlobalPermission(User user, String request, Action perm, String namespace)
    throws IOException {
    AuthResult authResult;
    if (authManager.authorizeUserGlobal(user, perm)) {
      authResult = AuthResult.allow(request, "Global check allowed", user, perm, null);
      authResult.getParams().setNamespace(namespace);
      logResult(authResult);
    } else {
      authResult = AuthResult.deny(request, "Global check failed", user, perm, null);
      authResult.getParams().setNamespace(namespace);
      logResult(authResult);
      throw new AccessDeniedException(
        "Insufficient permissions for user '" + (user != null ? user.getShortName() : "null")
          + "' (global, action=" + perm.toString() + ")");
    }
  }

  public static void logResult(AuthResult result) {
    if (AUDITLOG.isTraceEnabled()) {
      User user = result.getUser();
      UserGroupInformation ugi = user != null ? user.getUGI() : null;
      AUDITLOG.trace(
        "Access {} for user {}; reason: {}; remote address: {}; request: {}; context: {};"
          + "auth method: {}",
        (result.isAllowed() ? "allowed" : "denied"),
        (user != null ? user.getShortName() : "UNKNOWN"), result.getReason(),
        RpcServer.getRemoteAddress().map(InetAddress::toString).orElse(""), result.getRequest(),
        result.toContextString(), ugi != null ? ugi.getAuthenticationMethod() : "UNKNOWN");
    }
  }
{code}
Since *AccessChecker* integrates auditlogs for permission check, I'll delete the log in MasterRpcServices.

There must be more problems like this, I' ll check it later and commit the code.

-There are many "write" operations like "deleteTable", which may cause security problems, should also record an audit log.-
Bug: HTable MetricsTableLatencies not remove trigger memory leak 
Status: Open | Priority: Major | Resolution: 
Created: 15/Nov/22 13:04
Affects Version/s: 2.0.0

MetricsTableLatenciesImpl  histogramsByTable only put but not remove.Maybe we should remove it when table disabled.

supplement:

MetricsTableQueryMeterImpl metersByTable has the same problem.

Recurrence steps:
 # create table: create 'student','data'
 # write data: put 'student','2015001','data:name','Zhangsan'
 # disable table: disable 'student'
 # drop table: drop 'student'
 # visit regionserver:port/jmx
 # search Namespace_default_table_student_metric

Table is deleted, the latency data of the deleted table always in the /jmx endpoint.
Bug: Determine too large requests by response block size rather than cell size
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Dec/22 15:45

We send responses that are too slow or too large to the slow log. The too large is currently based on the response cell size, but the response block size is actually more important for performance. We should enhance the threshold check to use the response block size (or maybe check both sizes).

As part of this, we should also add the response block size to the slow log entries.
Bug: Provide RS coproc ability to attach WAL extended attributes to mutations at replication sink
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Dec/22 23:44

HBase provides coproc ability to enhance WALKey attributes (a.k.a. WAL annotations) in order for the replication sink cluster to build required metadata with the mutations. The endpoint is preWALAppend(). This ability was provided by HBASE-22622. The map of extended attributes is optional and hence not directly used by hbase internally. 

For any hbase downstreamers to build CDC (Change Data Capture) like functionality, it might required additional metadata in addition to the ones being used by hbase already (replication scope, list of cluster ids, seq id, table name, region id etc). For instance, Phoenix uses many additional attributes like tenant id, schema name, table type etc.
We already have this extended map of attributes available in WAL protobuf, to provide us the capability to (de)serialize it. While creating new ReplicateWALEntryRequest from the list of WAL entires, we are able to serialize the additional attributes. Similarly, at the replication sink side, the deserialized WALEntry has the extended attributes available.

At the sink cluster, we should be able to attach the deserialized extended attributes to the newly generated mutations so that the peer cluster can utilize the mutation attributes to re-build required metadata.
Bug: Observed lot of threads blocked in ConnectionImplementation.getKeepAliveMasterService
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Nov/22 06:01
Affects Version/s: 2.5.0

Recently We observed that lot of threads are blocked in method "ConnectionImplementation.getKeepAliveMasterService" during some initialization stages of rolling restart workflow. 

During rolling restart, we make RPC calls to Master using RpcRetryingCallerImpl, so as part of initialization we call "ConnectionImplementation.getKeepAliveMasterService" for each thread. Internally this method do RPC call within a synchronized block to check if master is running (mss.isMasterRunning).

Lots of threads are in blocked state due following synchronized block

synchronized (masterLock) {
   if (!isKeepAliveMasterConnectedAndRunning(this.masterServiceState))

{      MasterServiceStubMaker stubMaker = new MasterServiceStubMaker();      this.masterServiceState.stub = stubMaker.makeStub();    }

   resetMasterServiceState(this.masterServiceState);
 }

In Thread Dump Analyzer (2.4), we get warning that "A lot of threads are waiting for this monitor to become available again.
 This might indicate a congestion. You also should analyze other locks blocked by threads waiting for this monitor as there might be much more threads waiting for it.". Please check attached screenshot  !Screenshot 2022-11-16 at 10.06.59 AM.png|width=1639,height=971!

--------------------

"pool-11-thread-158" #313 prio=5 os_prio=0 tid=0x000055b88bcb8800 nid=0x404e waiting for monitor entry [0x00007fa48aa86000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1336)
    - waiting to lock <0x00000005d30ecb68> (a java.lang.Object)
    at org.apache.hadoop.hbase.client.ConnectionImplementation.getMaster(ConnectionImplementation.java:1327)
    at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:57)
    at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:103)
    at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3019)
    at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3011)
    at org.apache.hadoop.hbase.client.HBaseAdmin.move(HBaseAdmin.java:1458)
    at org.apache.hadoop.hbase.util.MoveWithoutAck.call(MoveWithoutAck.java:58)
    at org.apache.hadoop.hbase.util.MoveWithoutAck.call(MoveWithoutAck.java:33)

-------------------

 

*Proposal:*
We can optimize this flow as follows
1. Use double checked lock for "isKeepAliveMasterConnectedAndRunning(this.masterServiceState)" so that theads don't race for monitor, when master is running.
2. "isKeepAliveMasterConnectedAndRunning()" method should reuse the Globally cached state of isMasterRunning instead of doing expensive Call in for each thread. 

Check PR [https://github.com/apache/hbase/pull/4889] for more details.

Note: The "master" branch uses "AsyncConnectionImpl" so apparently we don't have issues there.
Bug: Close store file readers after region warmup 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Dec/22 09:49

We recently upgraded one of our clusters from version 1.4.13 to 2.4.15. After that we noticed that the number of open FD was increased 10 times more than before, from 1.3k to 15k. Each RS has about 1000 regions.

 

I thouht that it is OK if the number of open FD keeps steady. But the number was increased when regions were balanced. I found that there is a bug related to the region warmup process, which leaks opened FDs, and I fixed it. Please review the PR.

 

But I still have a question about why HBase 2.4 opens much more than 1.4. I am trying to find the answer, but not yet. Is there any person who can answer the question?
Bug: Encapsulate and centralise access to ref count through StoreFileInfo
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Dec/22 13:34
Affects Version/s: 3.0.0-alpha-3

Both {{StoreFileReader}} and {{StoreFileInfo}} have  a {{refCount}}, and the {{refCount}} is currently used in three main ways:
* When a new {{StoreFileScanner}} is created or close, it increases  or decreases the {{StoreFileReader.refCount}}.
* When {{CompactedHFilesDischarger}}  checks a {{HStoreFile}} whether it could be deleted, it check the {{StoreFileInfo.refCount}}.
* When {{HStore.getScanners}} gets {{HStoreFile}} from {{StoreFileManager}}, it  increases  or decreases the {{StoreFileInfo.refCount}}.

The problem here is  {{StoreFileReader.refCount}}  is copied from the {{StoreFileInfo.refCount}}  and the inconsistent usage of the {{refCount}} making the code somewhat hard to understand and causing trace the resource race problems such as HBASE-27484 and HBASE-27519 somewhat difficult. I suggest we should unify these two {{refCount}} and just use {{StoreFileInfo.refCount}}
Bug: Allow namespace admins to clone snapshots created by them
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Nov/22 10:13
Affects Version/s: 2.5.1

Creating a snapshot requires table admin permissions. But cloning it requires global admin permissions unless the user owns the snapshot and wants to recreate the original table the snapshot was based on using the same table name. This puts unnecessary load on the few people having global admin permissions. I would like to relax this rule a bit and allow the owner of the snapshot to clone it into any namespace where they have admin permissions regardless of the table name used.
Bug: HBaseTestingUtility minicluster requires log4j2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Nov/22 08:22

Using the HBaseMiniCluster now requires addition of log4j-api and log4j-core to the classpath. This can be demonstrated by updating hbase-downstreamer for 2.5.1.

https://github.com/saintstack/hbase-downstreamer
Bug: Fix python requirements problem
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Dec/22 13:31

The dependabot report two security problems for python requirements

gitpython <= 3.1.20
certifi < 2022.12.07

And there is also a parse error

InstallationError("Invalid requirement: 'cryptography=3.3.2' (from line 22 of /home/dependabot/dependabot-updater/dependabot_tmp_dir/requirements.txt)\nHint: = is not a valid operator. Did you mean == ?")
Bug: Improve our hbase_docker to be able to build and start standalone clusters other than master branch
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Nov/22 13:12

Find this docker file when trying to upgrade all our docker files from ubuntu 18.04 to 22.04.

While upgrading it to ubuntu 22.04, I think we could also improve it to start cluster for other branches and tags, i.e, the current stable release, which will be better for our end users.
Bug: Another case for FNFE on StoreFileScanner after a flush followed by a compaction
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Dec/22 03:19
Affects Version/s: 2.4.15

Besides the fix for {{HStore.getScanners}} for FNFE on StoreFileScanner after a flush followed by a compaction in HBASE-27484, there is a another case would cause FNFE:
# When {{StoreScanner}} is scanning, there is a flush followed by a compaction. When the flushed is completed in {{HStore.completeFlush}} and after it adds the new {{HStoreFile}} in {{StoreEngine}}, it would notify the {{StoreScanner.updateReaders}} by {{HStore.notifyChangedReadersObservers}} with the new flushed {{HStoreFile}}：
{code:java}
    private boolean completeFlush(List<HStoreFile> sfs, long snapshotId) throws IOException {
    // NOTE:we should keep clearSnapshot method inside the write lock because clearSnapshot may
    // close {@link DefaultMemStore#snapshot}, which may be used by
    // {@link DefaultMemStore#getScanners}.
    storeEngine.addStoreFiles(sfs,
      snapshotId > 0 ? () -> this.memstore.clearSnapshot(snapshotId) : () -> {
      });
    // notify to be called here - only in case of flushes
    notifyChangedReadersObservers(sfs);
    if (LOG.isTraceEnabled()) {
      long totalSize = getTotalSize(sfs);
      String traceMessage = "FLUSH time,count,size,store size,store files ["
        + EnvironmentEdgeManager.currentTime() + "," + sfs.size() + "," + totalSize + ","
        + storeSize + "," + storeEngine.getStoreFileManager().getStorefileCount() + "]";
      LOG.trace(traceMessage);
    }
    return needsCompaction();
  }
{code}
# But if before entering {{StoreScanner.updateReaders}} , the compaction task starts and  it compacts the new flushed {{HStoreFile}} and then {{HStore.closeAndArchiveCompactedFiles}} is called by {{CompactedHFilesDischarger}} to delete the {{HStoreFile}}. 
# When {{StoreScanner.updateReaders}} continues to execute, it would read already deleted {{HStoreFile}} and throws FNFE.
Bug: Evict blocks on split/merge; Avoid caching reference/hlinks if compaction is enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Nov/22 21:03
Affects Version/s: 3.0.0-alpha-3

This change aims to improve block cache usage upon splits/merges. On a split/merge event the following main steps happen:

1) parent regions are closed; 2) daughters are created and opened with refs/hlinks; 3) Compaction is triggered soon after the daughters get online;

With "hbase.rs.evictblocksonclose" set to false, we keep all blocks for the closed regions in 1, then will try to load same blocks again on 2 (since we are using the refs/links for the cache key), just to throw it away and cache the compaction resulting file in 3. 

If the block cache is close to its capacity, blocks from the compacted files in 3 will likely miss the cache.

The proposal here is to always evict blocks for parent regions on a split/merge event, and also avoid caching blocks for refs/hlinks if compactions are enabled.
Bug: Hbase cannot support offpeak periods throughout all the day when hbase.offpeak.start.hour=0 and hbase.offpeak.end.hour=23 setted
Status: Open | Priority: Major | Resolution: 
Created: 17/Nov/22 02:37
Affects Version/s: 2.4.15

Hbase cannot support offpeak periods throughout all the day when hbase.offpeak.start.hour=0 and hbase.offpeak.end.hour=23 setted

 

!image-2022-11-17-10-37-10-989.png!
Bug: CallTimeoutException can cause feedback loop with meta clears
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 07/Dec/22 22:58

In HBASE-27487 and HBASE-27490 we added safeguard which should reduce feedback loop caused by slow meta. We have continued to chaos test the hbase client and have found another case that needs to be handled.

With those two jiras, we no longer allow multiget to exceed operation timeout when meta is slow, and OperationTimeoutExceededExceptions do not clear meta cache. This allows for quicker recovery in many cases.

However, consider the case where you have a 1s RPC timeout and 3s operation timeout. Let's say meta is slow and it takes 2.9 seconds to resolve region locations for a batch. When we go to submit the multi actions to the server, we will only have a 100ms remaining time on our operation timeout. This may not be enough, and it results in a CallTimeoutException.

I use slow meta as an example, but it's possible for any slow regionserver to kick off a feedback loop due to a sudden surge in CallTimeoutException resulting in many clients clearing cache and hitting meta. Even with meta replicas, this just exacerbates any slowness and may become unrecoverable without extreme actions.

I also use multigets as the example here, and I think they are most at risk of this, but this is theoretically possible for all request types. A single Get might retry a few times and the last attempt only has a few milliseconds of remaining time. This could also result in a CallTimeoutException and potentially kick off a feedback loop.

I'm still trying to consider options, and am open to opinions here. This issue affects AyncTable and Table. Here are some raw options I've been weighing:
 * Treat CTE as special (non-clearing)
 ** Problem: what if a failed server continues to timeout and we have no idea that regions have moved?
 ** Can we differentiate CTE vs some sort of SocketTimeoutException (where server is not serving)?

 * Rate limit cache clears from CTE
 * Rate limit all cache clears
 * Rethrow CTE as an OperationTimeoutExceededException when remainingTime < rpcTimeout, thus skipping clear in that case
 ** This is the most targeted solution, but may leave other edge cases.
 ** Problem: what about the case where a server is just running slow, but continually causing cache clears?