Bug: Can't get connection to ZooKeeper
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 05/Jun/23 14:52
Affects Version/s: 1.4.13

I am using Hbase cluster along with apache kylin, the connection between Edge node and the Hbase cluster is good.
following are the logs from Kylin side which shows Error exception:


{code:java}
java.net.SocketTimeoutException: callTimeout=1200000, callDuration=1275361: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase 
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:178)
    at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4551)
    at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:561)
    at org.apache.hadoop.hbase.client.HTable.getTableDescriptor(HTable.java:585)
    at org.apache.kylin.storage.hbase.steps.HFileOutputFormat3.configureIncrementalLoad(HFileOutputFormat3.java:328)
    at org.apache.kylin.storage.hbase.steps.CubeHFileJob.run(CubeHFileJob.java:101)
    at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:144)
    at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:179)
    at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
    at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:179)
    at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1618)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1638)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1795)
    at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:140)
    ... 13 more
Caused by: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:971)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.access$400(ConnectionManager.java:566)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1567)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1609)
    ... 17 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:220)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:425)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:960)
    ... 20 more {code}
Following are the logs from Hbase cluster master NOde which accepts the connection from Edge NOde(Kylin):
{code:java}
2023-06-05 10:00:30,336 [myid:0] - INFO [CommitProcessor:0:NIOServerCnxn@1056] - Closed socket connection for client /10.127.2.201:37328 which had sessionid 0x7311c000c
2023-06-05 13:14:48,346 [myid:0] - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task started.
2023-06-05 13:14:48,346 [myid:0] - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed.
2023-06-05 13:22:39,872 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@222] - Accepted socket connection from /10.127.2.233:42364
2023-06-05 13:22:39,873 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@949] - Client attempting to establish new session at /10.127.2.233:42364
2023-06-05 13:22:39,874 [myid:0] - INFO [CommitProcessor:0:ZooKeeperServer@694] - Established session 0x7311c0022 with negotiated timeout 40000 for client /10.127.2.233:42364 {code}

have check all the permission in HDFS and S3. Any leads would be really appreciated.