Bug: MobFileCompactionChore is closing the master's shared cluster connection
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Oct/20 16:48
Affects Version/s: 3.0.0-alpha-1

Code which is doing so in MobFileCompactionChore
{code:java}
try (Connection conn = master.getConnection();
    Admin admin = conn.getAdmin();) { {code}
As master uses this connection to read the meta or other system tables, so none of the meta operations through master will work.

Symptoms in master logs:-
{code:java}
s, events=841, succcessCount=123, totalEvents=12824192, totalSuccessCount=1891300
2020-10-05 16:34:25,062 INFO org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor: Unable to get remote Address
2020-10-05 16:34:25,062 ERROR org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore: Failed to normalize regions.
java.io.IOException: connection is closed
        at org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:241)
        at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:797)
        at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:768)
        at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:727)
        at org.apache.hadoop.hbase.MetaTableAccessor.fullScanTables(MetaTableAccessor.java:215)
        at org.apache.hadoop.hbase.master.TableStateManager.getTablesInStates(TableStateManager.java:189)
        at org.apache.hadoop.hbase.master.HMaster.normalizeRegions(HMaster.java:1821)
        at org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore.chore(RegionNormalizerChore.java:48)
        at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:188)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748) {code}
Symptoms at the client:-
{code:java}

 RpcRetryingCaller{globalStartTime=1602099132430, pause=100, maxAttempts=11}, java.io.IOException: java.io.IOException: connection is closed
       at org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:241)
       at org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:1116)
       at org.apache.hadoop.hbase.master.TableStateManager.readMetaState(TableStateManager.java:258)
       at org.apache.hadoop.hbase.master.TableStateManager.isTablePresent(TableStateManager.java:175)
       at org.apache.hadoop.hbase.master.HMaster.getTableDescriptors(HMaster.java:3277)
       at org.apache.hadoop.hbase.master.HMaster.listTableDescriptors(HMaster.java:3221)
       at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1064)
       at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
       at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:418)
       at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
       at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
       at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318{code}