facebook/tsdash — Issue #8: No data provider available.
Author: phobos182
Created: 2012-03-05 22:34:35+00:00
State: open

12/03/05 22:32:31 INFO tsdb.services: HBase configuration: using tables 'tsdb', 'tsdb-uid', quorum 'hbtest1-zoo1:2282':/hbase
12/03/05 22:32:31 INFO tsdb.services: URL pattern: http://%h:%p/plots/%f
12/03/05 22:32:31 INFO tsdb.services: Plots are being written to: /var/www/tsdash/plots
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:155)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:990)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:303)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.(HConnectionManager.java:294)
at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:156)
at org.apache.hadoop.hbase.client.HTable.(HTable.java:168)
at org.apache.hadoop.hbase.client.HTable.(HTable.java:146)
at com.facebook.tsdb.tsdash.server.data.hbase.HBaseConnection.getDataTableConn(Unknown Source)
at com.facebook.tsdb.tsdash.server.data.hbase.HBaseDataProvider.(Unknown Source)
at com.facebook.tsdb.tsdash.server.data.TsdbDataProviderFactory.get(Unknown Source)
at com.facebook.tsdb.tsdash.server.MetricsEndpoint.doGet(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:837)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:903)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:133)
... 24 more

---
Using HBase 0.92 on CDH3 Update 3. Was not able to identify any issue with our HBase cluster or connection settings. We have 60 set for 'hbase.zookeeper.property.maxClientCnxns', and only 10 connections registered in zkDump for this test cluster.