Bug: Exception on standalone (master) shutdown
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Aug/20 21:24
Affects Version/s: 3.0.0-alpha-1

Running HBase ({{master}} branch) in standalone mode, terminating the process results in the following stack traces logged at error. It appears we shutdown the zookeeper client out-of-order with {{shutdown}} of the thread pools.

{noformat}
2020-08-10 14:21:46,777 INFO  [RS:0;localhost:16020] zookeeper.ZooKeeper: Session: 0x100111361f20001 closed
2020-08-10 14:21:46,778 INFO  [RS:0;localhost:16020] regionserver.HRegionServer: Exiting; stopping=localhost,16020,1597094491257; zookeeper connection closed.
2020-08-10 14:21:46,778 ERROR [main-EventThread] zookeeper.ClientCnxn: Error while calling watcher 
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6e61af4b rejected from java.util.concurrent.ThreadPoolExecutor@6a5e365f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
        at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678)
        at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:559)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
2020-08-10 14:21:46,778 INFO  [shutdown-hook-0] regionserver.ShutdownHook: Starting fs shutdown hook thread.
2020-08-10 14:21:46,779 ERROR [main-EventThread] zookeeper.ClientCnxn: Error while calling watcher 
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7d41da91 rejected from java.util.concurrent.ThreadPoolExecutor@6a5e365f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
        at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678)
        at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:559)
        at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:40)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
2020-08-10 14:21:46,780 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x100111361f20001
2020-08-10 14:21:46,780 INFO  [shutdown-hook-0] regionserver.ShutdownHook: Shutdown hook finished.
{noformat}