Bug: Alter table causes AccessDeniedException at the client side
Status: In Progress | Priority: Major | Resolution: 
Created: 21/Nov/24 02:41
Affects Version/s: 2.4.16

When altering a table there is a chance that the client catches some AccessDeniedExceptions:

 

 
{code:java}
#444021290, table=***, attempt=1/5 failed=1ops, last exception: org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions (user=***, scope=default:***, family=***, params=[table=***,family=***],action=WRITE)
    at org.apache.hadoop.hbase.security.access.AccessController.prePut(AccessController.java:1388)
    at org.apache.hadoop.hbase.coprocessor.RegionObserver.prePut(RegionObserver.java:413)
    at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$23.call(RegionCoprocessorHost.java:907)
    at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$23.call(RegionCoprocessorHost.java:904)
    at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:558)
    at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:631)
    at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.prePut(RegionCoprocessorHost.java:904)
    at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.callPreMutateCPHook(HRegion.java:4160)
    at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.access$800(HRegion.java:3610)
    at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation$1.visit(HRegion.java:3685)
    at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.visitBatchOperations(HRegion.java:3218)
    at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.checkAndPrepare(HRegion.java:3674)
    at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4447)
    at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4380)
    at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1009)
    at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:927)
    at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:892)
    at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2855)
    at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45008)
    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387)
    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) {code}
 

 

This is because HBase updates the table owner when a table is altered. By updating the table, ZKPermissionWatcher receives the permission changes and tries to update the permission cache in AuthManager:

 

[https://github.com/apache/hbase/blob/rel/2.4.16/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AuthManager.java#L185-L192]
{code:java}
  private void updateTableCache(TableName table, ListMultimap<String, Permission> tablePerms) {
    PermissionCache<TablePermission> cacheToUpdate =
      tableCache.getOrDefault(table, new PermissionCache<>());
    clearCache(cacheToUpdate);
    updateCache(tablePerms, cacheToUpdate);
    tableCache.put(table, cacheToUpdate);
    mtime.incrementAndGet();
  } {code}
 

At the same time, AccessController is accessing the same reference of the permission cache in AuthManager:

[https://github.com/apache/hbase/blob/rel/2.4.16/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AuthManager.java#L298-L319]
{code:java}
  public boolean accessUserTable(User user, TableName table, Permission.Action action) {
    if (user == null) {
      return false;
    }
    if (table == null) {
      table = PermissionStorage.ACL_TABLE_NAME;
    }
    if (authorizeUserNamespace(user, table.getNamespaceAsString(), action)) {
      return true;
    }
    PermissionCache<TablePermission> tblPermissions =
      tableCache.getOrDefault(table, TBL_NO_PERMISSION);
    if (hasAccessTable(tblPermissions.get(user.getShortName()), action)) {
      return true;
    }
    for (String group : user.getGroupNames()) {
      if (hasAccessTable(tblPermissions.get(AuthUtil.toGroupEntry(group)), action)) {
        return true;
      }
    }
    return false;
  } {code}
 

I think this is a bug introduced in the previous refactoring: https://issues.apache.org/jira/browse/HBASE-21255