Bug: Issue with ZooKeeper configuration on CDP5
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 22/May/22 12:58

Hello everyone, I am new here and in HBase as well, so sorry for my lack of knowledge, I tried to read but the documentation and the environment that I got as a test are confusing me. The explanation is here, I have got Cloudera 5 with Hbase with Kerberos, 2 master nodes, and 3 worker nodes on Ubuntu machines, we have to connect DataStore with special application with checks krb5.conf file and keytab after authentication goes and connect to DS and check the tables the metadata there. When I am trying to push the application to connect the master node zookeeper is failing, when I am trying to connect zookeper node from the same application I can scan and do what I need, but every time the connection goes to 2181 localhost, how to check where is the issue of configuration that masters done will give me access to the data stores?

 

Here is log from master node: 

[2022-05-22 12:26:31.676Z] [WARN] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  [MasterNode]- Possibly transient ZooKeeper, quorum=10.2.0.30:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
[2022-05-22 12:26:31.676Z] [INFO] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.util.RetryCounter  [MasterNode]- Sleeping 1000ms before retry #0...
[2022-05-22 12:26:31.676Z] [WARN] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  [MasterNode]- Possibly transient ZooKeeper, quorum=10.2.0.30:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
[2022-05-22 12:26:31.676Z] [INFO] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.util.RetryCounter  [MasterNode]- Sleeping 2000ms before retry #1...
[2022-05-22 12:26:31.676Z] [WARN] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  [MasterNode]- Possibly transient ZooKeeper, quorum=10.2.0.30:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
[2022-05-22 12:26:31.676Z] [ERROR] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  [MasterNode]- ZooKeeper exists failed after 2 attempts
[2022-05-22 12:26:31.677Z] [WARN] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.zookeeper.ZKUtil  [MasterNode]- hconnection-0x46acd8780x0, quorum=10.2.0.30:2181, baseZNode=/hbase Unable to set watcher on znode (/hbase/hbaseid)
 org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:222)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
    at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
    at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:833)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:623)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
    at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
    at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
    at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
    at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
    at com.bigid.scanner.hbase.HBaseScanner.getDataSourceObjects(HBaseScanner.java:60)
    at com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper$4.run(HadoopUGIConnectorWrapper.java:108)
    at com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper$4.run(HadoopUGIConnectorWrapper.java:105)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/javax.security.auth.Subject.doAs(Unknown Source)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
    at com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper.getDataSourceObjects(HadoopUGIConnectorWrapper.java:105)
    at com.bigid.scanner.connectors.generic.DataSourceConnectorAdapter.testConnection(DataSourceConnectorAdapter.java:289)
    at com.bigid.scanner.core.tasks.legacy.LegacyTestConnectionTask.execute(LegacyTestConnectionTask.java:43)
    at com.bigid.scanner.core.TestConnectionTask.execute(TestConnectionTask.java:36)
    at com.bigid.scanner.core.TaskExecution.execute(TaskExecution.java:36)
    at com.bigid.scanner.scannerboot.common.services.TestConnectionService.doService(TestConnectionService.java:57)
    at com.bigid.scanner.scannerboot.common.services.TestConnectionService.doService(TestConnectionService.java:20)
    at com.bigid.scanner.scannerboot.common.http.managers.TestConnectionSyncManager.lambda$runService$0(TestConnectionSyncManager.java:55)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)

[2022-05-22 12:26:31.678Z] [ERROR] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher  [MasterNode]- hconnection-0x46acd8780x0, quorum=10.2.0.30:2181, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception
 org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:222)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
    at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
    at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:833)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:623)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
    at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
    at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
    at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
    at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
    at com.bigid.scanner.hbase.HBaseScanner.getDataSourceObjects(HBaseScanner.java:60)
    at com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper$4.run(HadoopUGIConnectorWrapper.java:108)
    at com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper$4.run(HadoopUGIConnectorWrapper.java:105)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/javax.security.auth.Subject.doAs(Unknown Source)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
    at com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper.getDataSourceObjects(HadoopUGIConnectorWrapper.java:105)
    at com.bigid.scanner.connectors.generic.DataSourceConnectorAdapter.testConnection(DataSourceConnectorAdapter.java:289)
    at com.bigid.scanner.core.tasks.legacy.LegacyTestConnectionTask.execute(LegacyTestConnectionTask.java:43)
    at com.bigid.scanner.core.TestConnectionTask.execute(TestConnectionTask.java:36)
    at com.bigid.scanner.core.TaskExecution.execute(TaskExecution.java:36)
    at com.bigid.scanner.scannerboot.common.services.TestConnectionService.doService(TestConnectionService.java:57)
    at com.bigid.scanner.scannerboot.common.services.TestConnectionService.doService(TestConnectionService.java:20)
    at com.bigid.scanner.scannerboot.common.http.managers.TestConnectionSyncManager.lambda$runService$0(TestConnectionSyncManager.java:55)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)

[2022-05-22 12:26:31.678Z] [WARN] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.client.ZooKeeperRegistry  [MasterNode]- Can't retrieve clusterId from Zookeeper
 org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:222)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
    at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
    at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:833)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:623)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
    at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
    at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
    at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
    at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
    at com.bigid.scanner.hbase.HBaseScanner.getDataSourceObjects(HBaseScanner.java:60)
    at com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper$4.run(HadoopUGIConnectorWrapper.java:108)
    at com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper$4.run(HadoopUGIConnectorWrapper.java:105)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/javax.security.auth.Subject.doAs(Unknown Source)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
    at com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper.getDataSourceObjects(HadoopUGIConnectorWrapper.java:105)
    at com.bigid.scanner.connectors.generic.DataSourceConnectorAdapter.testConnection(DataSourceConnectorAdapter.java:289)
    at com.bigid.scanner.core.tasks.legacy.LegacyTestConnectionTask.execute(LegacyTestConnectionTask.java:43)
    at com.bigid.scanner.core.TestConnectionTask.execute(TestConnectionTask.java:36)
    at com.bigid.scanner.core.TaskExecution.execute(TaskExecution.java:36)
    at com.bigid.scanner.scannerboot.common.services.TestConnectionService.doService(TestConnectionService.java:57)
    at com.bigid.scanner.scannerboot.common.services.TestConnectionService.doService(TestConnectionService.java:20)
    at com.bigid.scanner.scannerboot.common.http.managers.TestConnectionSyncManager.lambda$runService$0(TestConnectionSyncManager.java:55)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)

[2022-05-22 12:26:36.337Z] [INFO] scanner_file - 7a342cd4-cd4f-4a08-ab9f-8214b4f62f09 [poolScheduler-2] com.bigid.scanner.scannerboot.common.services.context.threads.ThreadPoolUpdatesService - Fetching thread pool updates
[2022-05-22 12:26:41.675Z] [WARN] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  [MasterNode]- Possibly transient ZooKeeper, quorum=10.2.0.30:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
[2022-05-22 12:26:41.676Z] [INFO] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.util.RetryCounter  [MasterNode]- Sleeping 1000ms before retry #0...
[2022-05-22 12:26:41.676Z] [WARN] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  [MasterNode]- Possibly transient ZooKeeper, quorum=10.2.0.30:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
[2022-05-22 12:26:41.676Z] [INFO] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.util.RetryCounter  [MasterNode]- Sleeping 2000ms before retry #1...
[2022-05-22 12:26:41.676Z] [WARN] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  [MasterNode]- Possibly transient ZooKeeper, quorum=10.2.0.30:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
[2022-05-22 12:26:41.676Z] [ERROR] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  [MasterNode]- ZooKeeper exists failed after 2 attempts
[2022-05-22 12:26:41.676Z] [WARN] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.zookeeper.ZKUtil  [MasterNode]- hconnection-0x46acd8780x0, quorum=10.2.0.30:2181, baseZNode=/hbase Unable to set watcher on znode (/hbase)
 org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:222)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:849)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.access$400(ConnectionManager.java:535)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1435)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1476)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1505)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1711)
    at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124)
    at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3601)
    at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:312)
    at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:296)
    at com.bigid.scanner.hbase.HBaseScanner.getDataSourceObjects(HBaseScanner.java:62)
    at com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper$4.run(HadoopUGIConnectorWrapper.java:108)
    at com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper$4.run(HadoopUGIConnectorWrapper.java:105)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/javax.security.auth.Subject.doAs(Unknown Source)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
    at com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper.getDataSourceObjects(HadoopUGIConnectorWrapper.java:105)
    at com.bigid.scanner.connectors.generic.DataSourceConnectorAdapter.testConnection(DataSourceConnectorAdapter.java:289)
    at com.bigid.scanner.core.tasks.legacy.LegacyTestConnectionTask.execute(LegacyTestConnectionTask.java:43)
    at com.bigid.scanner.core.TestConnectionTask.execute(TestConnectionTask.java:36)
    at com.bigid.scanner.core.TaskExecution.execute(TaskExecution.java:36)
    at com.bigid.scanner.scannerboot.common.services.TestConnectionService.doService(TestConnectionService.java:57)
    at com.bigid.scanner.scannerboot.common.services.TestConnectionService.doService(TestConnectionService.java:20)
    at com.bigid.scanner.scannerboot.common.http.managers.TestConnectionSyncManager.lambda$runService$0(TestConnectionSyncManager.java:55)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)

[2022-05-22 12:26:41.676Z] [ERROR] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher  [MasterNode]- hconnection-0x46acd8780x0, quorum=10.2.0.30:2181, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception
 org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:222)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:849)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.access$400(ConnectionManager.java:535)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1435)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1476)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1505)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1711)
    at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124)
    at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3601)
    at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:312)
    at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:296)
    at com.bigid.scanner.hbase.HBaseScanner.getDataSourceObjects(HBaseScanner.java:62)
    at com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper$4.run(HadoopUGIConnectorWrapper.java:108)
    at com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper$4.run(HadoopUGIConnectorWrapper.java:105)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/javax.security.auth.Subject.doAs(Unknown Source)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
    at com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper.getDataSourceObjects(HadoopUGIConnectorWrapper.java:105)
    at com.bigid.scanner.connectors.generic.DataSourceConnectorAdapter.testConnection(DataSourceConnectorAdapter.java:289)
    at com.bigid.scanner.core.tasks.legacy.LegacyTestConnectionTask.execute(LegacyTestConnectionTask.java:43)
    at com.bigid.scanner.core.TestConnectionTask.execute(TestConnectionTask.java:36)
    at com.bigid.scanner.core.TaskExecution.execute(TaskExecution.java:36)
    at com.bigid.scanner.scannerboot.common.services.TestConnectionService.doService(TestConnectionService.java:57)
    at com.bigid.scanner.scannerboot.common.services.TestConnectionService.doService(TestConnectionService.java:20)
    at com.bigid.scanner.scannerboot.common.http.managers.TestConnectionSyncManager.lambda$runService$0(TestConnectionSyncManager.java:55)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)

[2022-05-22 12:26:41.676Z] [ERROR] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation  [MasterNode]- Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase
[2022-05-22 12:26:41.676Z] [WARN] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  [MasterNode]- Possibly transient ZooKeeper, quorum=10.2.0.30:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
[2022-05-22 12:26:41.676Z] [INFO] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.util.RetryCounter  [MasterNode]- Sleeping 1000ms before retry #0...
[2022-05-22 12:26:41.676Z] [INFO] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [poolScheduler-7] com.bigid.scanner.scannerboot.common.services.context.threads.ThreadPoolUpdatesService - Fetching thread pool updates
[2022-05-22 12:26:41.676Z] [WARN] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  [MasterNode]- Possibly transient ZooKeeper, quorum=10.2.0.30:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
[2022-05-22 12:26:41.676Z] [INFO] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.util.RetryCounter  [MasterNode]- Sleeping 2000ms before retry #1...
[2022-05-22 12:26:41.676Z] [WARN] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  [MasterNode]- Possibly transient ZooKeeper, quorum=10.2.0.30:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
[2022-05-22 12:26:41.676Z] [ERROR] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  [MasterNode]- ZooKeeper exists failed after 2 attempts
[2022-05-22 12:26:41.676Z] [WARN] scanner_file - 12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312] org.apache.hadoop.hbase.zookeeper.ZKUtil  [MasterNode]- hconnection-0x46acd8780x0, quorum=10.2.0.30:2181, baseZNode=/hbase Unable to set watcher on znode (/hbase)
 org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:222)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:849)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.access$400(ConnectionManager.java:535)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1435)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1476)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1505)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1711)
    at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124)
    at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3601)
    at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:312)
    at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:296)
    at com.bigid.scanner.hbase.HBaseScanner.getDataSourceObjects(HBaseScanner.java:62)
    at com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper$4.run(HadoopUGIConnectorWrapper.java:108)
    at com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper$4.run(HadoopUGIConnectorWrapper.java:105)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/javax.security.auth.Subject.doAs(Unknown Source)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
    at com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper.getDataSourceObjects(HadoopUGIConnectorWrapper.java:105)
    at com.bigid.scanner.connectors.generic.DataSourceConnectorAdapter.testConnection(DataSourceConnectorAdapter.java:289)
    at com.bigid.scanner.core.tasks.legacy.LegacyTestConnectionTask.execute(LegacyTestConnectionTask.java:43)
    at com.bigid.scanner.core.TestConnectionTask.execute(TestConnectionTask.java:36)
    at com.bigid.scanner.core.TaskExecution.execute(TaskExecution.java:36)
    at com.bigid.scanner.scannerboot.common.services.TestConnectionService.doService(TestConnectionService.java:57)
    at com.bigid.scanner.scannerboot.common.services.TestConnectionService.doService(TestConnectionService.java:20)
    at com.bigid.scanner.scannerboot.common.http.managers.TestConnectionSyncManager.lambda$runService$0(TestConnectionSyncManager.java:55)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)