nathanmarz/storm — Issue #68: Storm doesn't handle zookeeper recoverable errors properly
Author: killme2008
Created: 2011-12-12 15:37:11+00:00
State: open

I and my friend have met such exception occured when creating workers:

org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /storm
at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:837)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:90)
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
at backtype.storm.zookeeper$exists_node_QMARK_.invoke(zookeeper.clj:60)
at backtype.storm.zookeeper$mkdirs.invoke(zookeeper.clj:67)
at backtype.storm.cluster$mk_distributed_cluster_state.invoke(cluster.clj:23)
at backtype.storm.daemon.worker$fn__3181$exec_fn__945__auto____3182.invoke(worker.clj:88)
at clojure.lang.AFn.applyToHelper(AFn.java:187)
at clojure.lang.AFn.applyTo(AFn.java:151)
at clojure.core$apply.invoke(core.clj:540)
at backtype.storm.daemon.worker$fn__3181$mk_worker__3323.doInvoke(worker.clj:84)
at clojure.lang.RestFn.invoke(RestFn.java:513)
at backtype.storm.daemon.worker$_main.invoke(worker.clj:269)
at clojure.lang.AFn.applyToHelper(AFn.java:174)
at clojure.lang.AFn.applyTo(AFn.java:151)

It seems storm doesn't check the KeeperException code if the error was recoverable and try to re-process it again.The connection loss exception is a recoverable error which the ZooKeeper handle is still valid and future operations will succeed once the ZooKeeper library can reestablish its connection to ZooKeeper.

Zookeeper error handling please check:

http://wiki.apache.org/hadoop/ZooKeeper/ErrorHandling

I recommend to use ZkClient:
https://github.com/sgroschupf/zkclient

It supports IZkStateListener to listen events of zk session state changed.