Bug: upgrading standalone-mode hbase from 2.2 to 2.3 fails because of local filesystem not supporting hflush
Status: Open | Priority: Major | Resolution: 
Created: 26/May/20 02:57
Affects Version/s: 2.3.0

When we tested upgrading HBASE from 2.2 to 2.4/2.5, the upgraded node failed to start.

The error message is shown below:
{code:java}
2020-05-26 02:46:33,124 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster] wal.AsyncFSWALProvider: The RegionServer async write ahead log provider relies on the ability to call hflush for proper operation during component failures, but the current FileSystem does not support doing so. Please check the config value of 'hbase.wal.dir' and ensure it points to a FileSystem mount that has suitable capabilities for output streams.2020-05-26 02:46:33,124 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster] wal.AsyncFSWALProvider: The RegionServer async write ahead log provider relies on the ability to call hflush for proper operation during component failures, but the current FileSystem does not support doing so. Please check the config value of 'hbase.wal.dir' and ensure it points to a FileSystem mount that has suitable capabilities for output streams.2020-05-26 02:46:33,130 INFO  [master/9f0ca3fedcda:16000:becomeActiveMaster] wal.AbstractFSWAL: Closed WAL: AsyncFSWAL 9f0ca3fedcda%2C16000%2C1590461189948:(num 1590461193109)2020-05-26 02:46:33,134 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster] master.HMaster: Failed to become active masterjava.io.IOException: cannot get log writer at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:118) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createAsyncWriter(AsyncFSWAL.java:704) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:710) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:128) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:838) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:548) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:489) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:157) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:297) at org.apache.hadoop.hbase.master.store.LocalRegion.createWAL(LocalRegion.java:193) at org.apache.hadoop.hbase.master.store.LocalRegion.bootstrap(LocalRegion.java:213) at org.apache.hadoop.hbase.master.store.LocalRegion.create(LocalRegion.java:310) at org.apache.hadoop.hbase.master.store.LocalStore.create(LocalStore.java:148) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:969) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) at java.lang.Thread.run(Thread.java:748)Caused by: org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException: hflush at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:71) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:169) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:166) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:105) ... 17 more2020-05-26 02:46:33,135 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster] master.HMaster: ***** ABORTING master 9f0ca3fedcda,16000,1590461189948: Unhandled exception. Starting shutdown. *****java.io.IOException: cannot get log writer at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:118) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createAsyncWriter(AsyncFSWAL.java:704) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:710) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:128) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:838) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:548) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:489) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:157) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:297) at org.apache.hadoop.hbase.master.store.LocalRegion.createWAL(LocalRegion.java:193) at org.apache.hadoop.hbase.master.store.LocalRegion.bootstrap(LocalRegion.java:213) at org.apache.hadoop.hbase.master.store.LocalRegion.create(LocalRegion.java:310) at org.apache.hadoop.hbase.master.store.LocalStore.create(LocalStore.java:148) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:969) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) at java.lang.Thread.run(Thread.java:748)Caused by: org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException: hflush at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:71) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:169) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:166) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:105) ... 17 more
{code}
It can be reproduced through the following steps:
 # Start a single-node Hbase branch-2.2.
 # Use hbase pe to write data. 
{code:java}
/hbase/bin/hbase pe --nomapred --oneCon=true --valueSize=10 --rows=100 sequentialWrite 1{code}

      3. Upgrade the node to branch-2.3/master.  After upgraded, as the log suggested, hbase failed to start.