Bug: Unexpected crash in the process of deleting a table causes stale data in ZooKeeper
Status: Open | Priority: Major | Resolution: 
Created: 22/Oct/21 06:26
Affects Version/s: 1.7.1

There is a table {{mytable}} in a HBase cluster with two HMasters and two RegionServers.

1. Current active HMaster is HMaster1;
2. Client request to delete the table {{mytable}};
3. HMaster1 deletes znode in ZooKeeper: /hbase/table/mytable

{code:java}
[org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:238), org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeFailSilent(ZKUtil.java:1367), org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeFailSilent(ZKUtil.java:1359), org.apache.hadoop.hbase.zookeeper.ZKTableStateManager.setDeletedTable(ZKTableStateManager.java:244), org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.deleteAssignmentState(DeleteTableProcedure.java:383), org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.executeFromState(DeleteTableProcedure.java:136), org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.executeFromState(DeleteTableProcedure.java:57), org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139), org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1166), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:954), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:907), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:76), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:481), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java)]
{code}

4. HMaster1 crashes before deleting the znode: /hbase/table-lock/mytable

{code:java}
[org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:238), org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1341), org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1330), org.apache.hadoop.hbase.master.TableLockManager$ZKTableLockManager.tableDeleted(TableLockManager.java:442), org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.markTableAsDeleted(MasterProcedureScheduler.java:895), org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.completionCleanup(MasterProcedureScheduler.java:261), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.procedureFinished(ProcedureExecutor.java:1415), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:974), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:907), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:76), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:481), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java)]
{code}

5. HMaster2 becomes active
6. The table {{mytable}} was deleted, but the znode "/hbase/table-lock/mytable" was left on ZooKeeper.