Bug: Upgrade from 1.7.2 to 2.6.0 failed: HMaster aborted
Status: Open | Priority: Major | Resolution: 
Created: 04/Sep/24 18:45
Affects Version/s: 2.5.0

I am trying to migrate from 1.7.2 cluster to 2.6.0 (both are released versions). However, I observed that the hmaster crashed during the upgrade process.
h1. Reproduce

Step1: Start up 1.7.2 HBase cluster (1 HDFS, 1 HM, 1 RS).

Step2: Stop the 1.7.2 HBase cluster.

Step3: Upgrade to 2.6.0 HBase cluster.

HMaster will crash with the following exception
{code:java}
2024-09-04T16:04:47,004 WARN  [PEWorker-2] procedure.InitMetaProcedure: Failed to init meta, suspend 1000secs
java.io.IOException: Meta table is not partial, please sideline this meta directory or run HBCK to fix this meta table, e.g. rebuild the server hostname in ZNode for the meta region
        at org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.deleteMetaTableDirectoryIfPartial(InitMetaProcedure.java:199) ~[hbase-server-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.writeFsLayout(InitMetaProcedure.java:78) ~[hbase-server-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.executeFromState(InitMetaProcedure.java:102) ~[hbase-server-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.executeFromState(InitMetaProcedure.java:54) ~[hbase-server-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188) ~[hbase-procedure-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:944) ~[hbase-procedure-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1766) ~[hbase-procedure-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1444) ~[hbase-procedure-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:77) ~[hbase-procedure-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:2092) ~[hbase-procedure-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216) ~[hbase-common-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2119) ~[hbase-procedure-2.6.0.jar:2.6.0]
2024-09-04T16:04:47,005 INFO  [PEWorker-2] procedure2.TimeoutExecutorThread: ADDED pid=1, state=WAITING_TIMEOUT:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta; timeout=1000, timestamp=1725465888005
2024-09-04T16:04:48,045 ERROR [PEWorker-1] procedure2.ProcedureExecutor: Root Procedure pid=1, state=FAILED:INIT_META_WRITE_FS_LAYOUT, exception=org.apache.hadoop.hbase.exceptions.TimeoutIOException via ProcedureExecutor:org.apache.hadoop.hbase.exceptions.TimeoutIOException: Operation timed out after 1.0010 sec; InitMetaProcedure table=hbase:meta does not support rollback but the execution failed and try to rollback, code bug?
org.apache.hadoop.hbase.procedure2.RemoteProcedureException: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Operation timed out after 1.0010 sec
        at org.apache.hadoop.hbase.procedure2.Procedure.setFailure(Procedure.java:768) ~[hbase-procedure-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.Procedure.setTimeoutFailure(Procedure.java:797) ~[hbase-procedure-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.executeTimedoutProcedure(TimeoutExecutorThread.java:131) ~[hbase-procedure-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.execDelayedProcedure(TimeoutExecutorThread.java:109) ~[hbase-procedure-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.run(TimeoutExecutorThread.java:68) ~[hbase-procedure-2.6.0.jar:2.6.0]
Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Operation timed out after 1.0010 sec
        at org.apache.hadoop.hbase.procedure2.Procedure.setTimeoutFailure(Procedure.java:798) ~[hbase-procedure-2.6.0.jar:2.6.0]
        ... 3 more
2024-09-04T16:04:48,058 INFO  [PEWorker-1] procedure2.ProcedureExecutor: Rolled back pid=1, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.exceptions.TimeoutIOException via ProcedureExecutor:org.apache.hadoop.hbase.exceptions.TimeoutIOException: Operation timed out after 1.0010 sec; InitMetaProcedure table=hbase:meta exec-time=1.4160 sec
2024-09-04T16:04:48,059 ERROR [master/hmaster:16000:becomeActiveMaster] master.HMaster: Failed to become active master
java.io.IOException: Failed to initialize meta table
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1077) ~[hbase-server-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2459) ~[hbase-server-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.master.HMaster.lambda$null$0(HMaster.java:590) ~[hbase-server-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187) ~[hbase-common-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:177) ~[hbase-common-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.master.HMaster.lambda$run$1(HMaster.java:587) ~[hbase-server-2.6.0.jar:2.6.0]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Caused by: org.apache.hadoop.hbase.procedure2.RemoteProcedureException: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Operation timed out after 1.0010 sec
        at org.apache.hadoop.hbase.procedure2.Procedure.setFailure(Procedure.java:768) ~[hbase-procedure-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.Procedure.setTimeoutFailure(Procedure.java:797) ~[hbase-procedure-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.executeTimedoutProcedure(TimeoutExecutorThread.java:131) ~[hbase-procedure-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.execDelayedProcedure(TimeoutExecutorThread.java:109) ~[hbase-procedure-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.run(TimeoutExecutorThread.java:68) ~[hbase-procedure-2.6.0.jar:2.6.0]
Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Operation timed out after 1.0010 sec
        at org.apache.hadoop.hbase.procedure2.Procedure.setTimeoutFailure(Procedure.java:798) ~[hbase-procedure-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.executeTimedoutProcedure(TimeoutExecutorThread.java:131) ~[hbase-procedure-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.execDelayedProcedure(TimeoutExecutorThread.java:109) ~[hbase-procedure-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.run(TimeoutExecutorThread.java:68) ~[hbase-procedure-2.6.0.jar:2.6.0]
2024-09-04T16:04:48,060 ERROR [master/hmaster:16000:becomeActiveMaster] master.HMaster: Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.quotas.MasterQuotasObserver]
2024-09-04T16:04:48,060 ERROR [master/hmaster:16000:becomeActiveMaster] master.HMaster: ***** ABORTING master hmaster,16000,1725465872458: Unhandled exception. Starting shutdown. *****
java.io.IOException: Failed to initialize meta table
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1077) ~[hbase-server-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2459) ~[hbase-server-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.master.HMaster.lambda$null$0(HMaster.java:590) ~[hbase-server-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187) ~[hbase-common-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:177) ~[hbase-common-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.master.HMaster.lambda$run$1(HMaster.java:587) ~[hbase-server-2.6.0.jar:2.6.0]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Caused by: org.apache.hadoop.hbase.procedure2.RemoteProcedureException: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Operation timed out after 1.0010 sec
        at org.apache.hadoop.hbase.procedure2.Procedure.setFailure(Procedure.java:768) ~[hbase-procedure-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.Procedure.setTimeoutFailure(Procedure.java:797) ~[hbase-procedure-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.executeTimedoutProcedure(TimeoutExecutorThread.java:131) ~[hbase-procedure-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.execDelayedProcedure(TimeoutExecutorThread.java:109) ~[hbase-procedure-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.run(TimeoutExecutorThread.java:68) ~[hbase-procedure-2.6.0.jar:2.6.0]
Caused by: org.apache.hadoop.hbase.exceptions.TimeoutIOException: Operation timed out after 1.0010 sec
        at org.apache.hadoop.hbase.procedure2.Procedure.setTimeoutFailure(Procedure.java:798) ~[hbase-procedure-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.executeTimedoutProcedure(TimeoutExecutorThread.java:131) ~[hbase-procedure-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.execDelayedProcedure(TimeoutExecutorThread.java:109) ~[hbase-procedure-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.run(TimeoutExecutorThread.java:68) ~[hbase-procedure-2.6.0.jar:2.6.0]
2024-09-04T16:04:48,065 INFO  [master/hmaster:16000:becomeActiveMaster] regionserver.HRegionServer: ***** STOPPING region server 'hmaster,16000,1725465872458' *****
2024-09-04T16:04:48,065 INFO  [master/hmaster:16000:becomeActiveMaster] regionserver.HRegionServer: STOPPED: Stopped by master/hmaster:16000:becomeActiveMaster
2024-09-04T16:04:48,067 INFO  [master/hmaster:16000] regionserver.HRegionServer: Stopping infoServer
2024-09-04T16:04:48,169 INFO  [master/hmaster:16000] handler.ContextHandler: Stopped o.a.h.t.o.e.j.w.WebAppContext@78483404{master,/,null,STOPPED}{file:/hbase/hbase-2.6.0/hbase-webapps/master}
2024-09-04T16:04:48,180 INFO  [master/hmaster:16000] server.AbstractConnector: Stopped ServerConnector@3a7ba368{HTTP/1.1, (http/1.1)}{0.0.0.0:16010}
2024-09-04T16:04:48,180 INFO  [master/hmaster:16000] server.session: node0 Stopped scavenging
2024-09-04T16:04:48,182 INFO  [master/hmaster:16000] handler.ContextHandler: Stopped o.a.h.t.o.e.j.s.ServletContextHandler@b525e3a{static,/static,file:///hbase/hbase-2.6.0/hbase-webapps/static/,STOPPED}
2024-09-04T16:04:48,183 INFO  [master/hmaster:16000] handler.ContextHandler: Stopped o.a.h.t.o.e.j.s.ServletContextHandler@294e4c35{logs,/logs,file:///var/log/hbase/,STOPPED}
2024-09-04T16:04:48,188 INFO  [master/hmaster:16000] regionserver.HRegionServer: aborting server hmaster,16000,1725465872458
2024-09-04T16:04:48,190 INFO  [master/hmaster:16000] regionserver.HRegionServer: stopping server hmaster,16000,1725465872458; all regions closed.
2024-09-04T16:04:48,191 WARN  [OldWALsCleaner-1] cleaner.LogCleaner: Interrupted while cleaning old WALs, will try to clean it next round. Exiting.
2024-09-04T16:04:48,191 WARN  [OldWALsCleaner-0] cleaner.LogCleaner: Interrupted while cleaning old WALs, will try to clean it next round. Exiting.
2024-09-04T16:04:48,194 INFO  [master/hmaster:16000] hbase.ChoreService: Chore service for: master/hmaster:16000 had [] on shutdown
2024-09-04T16:04:48,196 INFO  [master/hmaster:16000] procedure2.RemoteProcedureDispatcher: Stopping procedure remote dispatcher
2024-09-04T16:04:48,196 INFO  [master/hmaster:16000] procedure2.ProcedureExecutor: Stopping
2024-09-04T16:04:48,205 INFO  [master/hmaster:16000] region.RegionProcedureStore: Stopping the Region Procedure Store, isAbort=true
2024-09-04T16:04:48,208 WARN  [master/hmaster:16000] master.ActiveMasterManager: Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null
2024-09-04T16:04:48,208 INFO  [master/hmaster:16000] assignment.AssignmentManager: Stopping assignment manager
2024-09-04T16:04:48,209 INFO  [master/hmaster:16000] region.MasterRegion: Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true
2024-09-04T16:04:48,369 INFO  [master/hmaster:16000] regionserver.HRegion: Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682.
2024-09-04T16:04:48,373 ERROR [master/hmaster:16000] regionserver.HRegion: Memstore data size is 2190 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682.
2024-09-04T16:04:48,374 INFO  [master/hmaster:16000] regionserver.HRegion: Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682.
2024-09-04T16:04:48,374 INFO  [master/hmaster:16000] flush.MasterFlushTableProcedureManager: stop: server shutting down.
2024-09-04T16:04:48,374 INFO  [master:store-WAL-Roller] wal.AbstractWALRoller: LogRoller exiting.
2024-09-04T16:04:48,376 INFO  [master/hmaster:16000] ipc.NettyRpcServer: Stopping server on /192.168.227.2:16000
2024-09-04T16:04:48,511 INFO  [master/hmaster:16000] zookeeper.ZooKeeper: Session: 0x10009e5081e0000 closed
2024-09-04T16:04:48,511 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x10009e5081e0000
2024-09-04T16:04:48,511 INFO  [master/hmaster:16000] regionserver.HRegionServer: Exiting; stopping=hmaster,16000,1725465872458; zookeeper connection closed.
2024-09-04T16:04:48,512 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: HMaster Aborted
        at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:255) ~[hbase-server-2.6.0.jar:2.6.0]
        at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:147) ~[hbase-server-2.6.0.jar:2.6.0]
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) ~[hadoop-common-2.10.2.jar:?]
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:140) ~[hbase-server-2.6.0.jar:2.6.0] {code}
 

Upgrade from 1.7.2 to 2.4.18 can still succeed.

I am wondering whether this is a backward incompatibility or if there are any additional steps I missed to upgrade to 2.6.0.