Bug: Rolling back from 2.5.0-SNAPSHOT to 2.4.13 fails due to `File does not exist: /hbase/MasterData/data/master/store/.initialized/.regioninfo`
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 27/Jul/22 17:52
Affects Version/s: 2.5.0

I was doing some perf testing with builds of 2.5.0. I rolled back to 2.4.13 and the master won't start. Stack trace ends in,

{noformat}
java.io.FileNotFoundException: File does not exist: /hbase/MasterData/data/master/store/.initialized/.regioninfo                                                                                          
        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86)                                                                                                                            at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76)
        at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:156)                                                                                         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2089)                                                                                                  
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:762)                                                                                         
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:458)                                        
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)                                                      at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:604)                                                                                           
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:572)                                                                                           
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:556)                               
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093)                                                                                                                                           
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1043)                                                                                                                                     
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:971) 
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)                                                                                                               
        at java.base/javax.security.auth.Subject.doAs(Subject.java:439)                                                                                                                                   
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)                                                                                                           
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2976)
{noformat}

When I examine the on-disk file system, I see,

{noformat}
nonroot@namenode-0:~$ hdfs dfs -ls /hbase/MasterData/data/master/store/
Found 3 items
drwxr-xr-x   - nonroot supergroup          0 2022-07-19 17:37 /hbase/MasterData/data/master/store/.initialized
drwxr-xr-x   - nonroot supergroup          0 2022-07-19 17:37 /hbase/MasterData/data/master/store/.tabledesc
drwxr-xr-x   - nonroot supergroup          0 2022-07-27 16:25 /hbase/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682
nonroot@namenode-0:~$ hdfs dfs -ls /hbase/MasterData/data/master/store/.initialized/
nonroot@namenode-0:~$
{noformat}