Bug: Crash HM and META-RS when truncating table causes data loss
Status: Open | Priority: Major | Resolution: 
Created: 25/Mar/22 03:18
Affects Version/s: 2.4.8

I have a hbae cluster of 2 master nodes (C1HM1:172.0.0.2 and C1HM2:172.0.0.3) and 3 slave nodes (C1RS1:172.0.0.4, C1RS2:172.0.0.5, C1RS3:172.0.0.6). The following is the bug triggering process ({color:#de350b}red {color}for crash and exception events; {color:#00875a}green {color}for events in hbase nodes):
 * the active master is C1HM1, the meta server is C1RS2
 * client requests to create a table "FAVMyInfo"
 * {color:#de350b}2022-03-24 01:44:54,317 [MyTest] - INFO - Going to crash node 172.25.0.5（table "FAVMyInfo" exists in the ZooKeeper at this time）{color}
 * {color:#00875a}C1HM1 assigns meta regions to C1RS3{color}
 * client receives ACK about creating table "FAVMyInfo", and then makes some changes to "FAVMyInfo"
 * 2022-03-24 01:46:20,664 [MyTest] - INFO - Client has disabled table "FAVMyInfo" and requests to truncate "FAVMyInfo"
 * {color:#00875a}C1HM1 first deleted table "FAVMyInfo", and then create a new "FAVMyInfo" region;{color}
 * {color:#de350b}2022-03-24 01:46:23,302 [MyTest] - INFO - Going to crash node 172.25.0.2 (at this time, table "FAVMyInfo" does not exist in the ZooKeeper). The node C1HM1 was killed before:{color}

{code:java}
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$BlockingStub.mutate(ClientProtos.java), org.apache.hadoop.hbase.client.ClientServiceCallable.doMutate(ClientServiceCallable.java:55), org.apache.hadoop.hbase.client.HTable$3.rpcCall(HTable.java:534), org.apache.hadoop.hbase.client.HTable$3.rpcCall(HTable.java:529), org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127), org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:108), org.apache.hadoop.hbase.client.HTable.put(HTable.java:538), org.apache.hadoop.hbase.MetaTableAccessor.put(MetaTableAccessor.java:1365), org.apache.hadoop.hbase.MetaTableAccessor.putToMetaTable(MetaTableAccessor.java:1355), org.apache.hadoop.hbase.MetaTableAccessor.updateTableState(MetaTableAccessor.java:1657), org.apache.hadoop.hbase.MetaTableAccessor.updateTableState(MetaTableAccessor.java:1169), org.apache.hadoop.hbase.master.TableStateManager.updateMetaState(TableStateManager.java:174), org.apache.hadoop.hbase.master.TableStateManager.setTableState(TableStateManager.java:84), org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.setEnablingState(CreateTableProcedure.java:376), org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure.executeFromState(TruncateTableProcedure.java:140), org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure.executeFromState(TruncateTableProcedure.java:45), org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:191), org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:956), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1665), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1412), org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:78), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1979), org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java) {code}
 * {color:#de350b}2022-03-24 01:46:46,023 [MyTest] - INFO - Going to crash node 172.25.0.6. The node C1RS3 was killed before:{color}

{code:java}
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java), org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45253), org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:392), org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133), org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:354), org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:334), org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java)]{code}
 * {color:#00875a}C1HM2 takes over the cluster, and unexpectedly deleted table FAVMyInfo{color}
 * 2022-03-24 01:48:13,497 [MyTest] - INFO - client receives response for truncating "FAVMyInfo", no Exceptions here. Then client requests to get a row from "FAVMyInfo", but gets {color:#ff0000}"TableNotFoundException: FAVMyInfo"{color}