Bug: After upgrading HBase from 2.1.1 to 2.2.0,  table cannot be read and HMaster crashes
Status: Open | Priority: Major | Resolution: 
Created: 09/Nov/20 23:57
Affects Version/s: 2.1.1

When we upgraded HBASE cluster from 2.1.1 to 2.2.0, we failed to read data by using HBase shell command, and later HMaster crashes. 

Commandline error message:

 
{code:java}
HBase ShellUse "help" to get list of supported commands.Use "exit" to quit this interactive shell.For Reference, please visit: http://hbase.apache.org/2.0/book.html#shellVersion 2.2.0, r3ec693241f2edb0cf78e71739874f2323a86bf1e, Tue Nov  3 02:12:43 UTC 2020Took 0.0043 secondshbase(main):001:0> listTABLETestTable1 row(s)Took 0.5663 seconds=> ["TestTable"]hbase(main):002:0> scan 'TestTable'ROW                                            COLUMN+CELL
ERROR: Unknown table TestTable!
For usage try 'help "scan"'
Took 0.1510 secondshbase(main):003:0>
{code}
Log file:

 

 
{code:java}
2020-11-07 04:03:57,346 ERROR [master/bff37223dfc6:16000:becomeActiveMaster] master.HMaster: Failed to become active masterjava.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILEDat org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:366)at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:312)at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1344)at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1122)at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2353)at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:598)at java.lang.Thread.run(Thread.java:748)Caused by: java.io.IOException: Timedout 300000ms waiting for namespace table to be assigned and enabled: tableName=hbase:namespace, state=ENABLEDat org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:108)at org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:63)at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:248)at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1342)... 4 more2020-11-07 04:03:57,347 ERROR [master/bff37223dfc6:16000:becomeActiveMaster] master.HMaster: Master server abort: loaded coprocessors are: []2020-11-07 04:03:57,347 ERROR [master/bff37223dfc6:16000:becomeActiveMaster] master.HMaster: ***** ABORTING master bff37223dfc6,16000,1604721496292: Unhandled exception. Starting shutdown. *****java.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILEDat org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:366)at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:312)at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1344)at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1122)at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2353)at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:598)at java.lang.Thread.run(Thread.java:748)Caused by: java.io.IOException: Timedout 300000ms waiting for namespace table to be assigned and enabled: tableName=hbase:namespace, state=ENABLEDat org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:108)at org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:63)at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:248)at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1342)... 4 more2020-11-07 04:03:57,348 INFO  [master/bff37223dfc6:16000:becomeActiveMaster] regionserver.HRegionServer: ***** STOPPING region server 'bff37223dfc6,16000,1604721496292' *****2020-11-07 

stopping=bff37223dfc6,16000,1604721496292; zookeeper connection closed.2020-11-07 04:03:57,612 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x1014e7bdcf800002020-11-07 04:03:57,620 ERROR [main] master.HMasterCommandLine: Master exitingjava.lang.RuntimeException: HMaster Abortedat org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:244)at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3198)
{code}
 

Steps to reproduce: 
 # ** Start up a cluster of version 2.1.1 with 3 nodes
 # Use hbase pe to write data. 

 
{code:java}
/hbase/bin/hbase pe --nomapred --oneCon=true --valueSize=10 --rows=100 sequentialWrite 1{code}
 
 # Stop the cluster by: 
 ## run “bash stop-hbase.sh” to shutdown the master.
 ## Set the property hbase.procedure.upgrade-to-2-2 to true in hbase-site.xml
 ## Start the master again and wait until the Master quits 
 # Upgrade the node to 2.2.0
 # After upgrading, try to read the data TestTable, the error message shows. Then, after a while, Master crashes.