Bug: HBase upgrade from 2.3 to master fails because master.HMaster fails to become active master
Status: Open | Priority: Major | Resolution: 
Created: 13/Jun/20 20:59

When we tested upgrading HBase from 2.3 to master(c303f9d329d578d31140e507bdbcbe3aa097042b), the upgraded node failed to start.

The error message is shown below:
{code:java}
2020-06-13 20:23:26,409 ERROR [master/085bbffdc52f:16000:becomeActiveMaster] master.HMaster: Failed to become active master2020-06-13 20:23:26,409 ERROR [master/085bbffdc52f:16000:becomeActiveMaster] master.HMaster: Failed to become active masterjava.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILED at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:379) at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:319) at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1362) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1137) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) at java.lang.Thread.run(Thread.java:748)Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 2 actions: RetriesExhaustedException: 2 times, servers with issues:  at org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.makeError(BufferedMutatorOverAsyncBufferedMutator.java:107) at org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.internalFlush(BufferedMutatorOverAsyncBufferedMutator.java:122) at org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.close(BufferedMutatorOverAsyncBufferedMutator.java:166) at org.apache.hadoop.hbase.master.TableNamespaceManager.migrateNamespaceTable(TableNamespaceManager.java:93) at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:123) at org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:61) at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:249) at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1360) ... 4 more2020-06-13 20:23:26,410 ERROR [master/085bbffdc52f:16000:becomeActiveMaster] master.HMaster: Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint]2020-06-13 20:23:26,410 ERROR [master/085bbffdc52f:16000:becomeActiveMaster] master.HMaster: ***** ABORTING master 085bbffdc52f,16000,1592079787433: Unhandled exception. Starting shutdown. *****java.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILED at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:379) at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:319) at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1362) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1137) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) at java.lang.Thread.run(Thread.java:748)Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 2 actions: RetriesExhaustedException: 2 times, servers with issues:  at org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.makeError(BufferedMutatorOverAsyncBufferedMutator.java:107) at org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.internalFlush(BufferedMutatorOverAsyncBufferedMutator.java:122) at org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.close(BufferedMutatorOverAsyncBufferedMutator.java:166) at org.apache.hadoop.hbase.master.TableNamespaceManager.migrateNamespaceTable(TableNamespaceManager.java:93) at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:123) at org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:61) at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:249) at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1360) ... 4 more
{code}
It can be reproduced through the following steps:
 * Start a single-node HBase 2.3 with the following configuration.
{code:java}
  <property>
    <name>hbase.rootdir</name>
    <value>/var/lib/hbase</value>
  </property>  <property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/var/lib/zookeeper</value>
  </property>{code}

 * Use HBase performance evaluation tool to generate data.
{code:java}
/hbase/bin/hbase pe --nomapred --oneCon=true --valueSize=10 --rows=100 sequentialWrite 1{code}

 * Upgrade the node to master with the same configuration. The new version node failed to start. And there is error message in /hbase/logs.

When I used the default configuration, the error message is the same as before.

I also attached the entire log file. [^hbase--master-085bbffdc52f.log]