Project: springxd
Issue XD-1739: Container reconnection to ZK fails intermittently
As reported by Matt Stine: After closing and reopening a laptop, the following stack trace appears in the container log: {noformat} 00:47:28,226 INFO main-EventThread state.ConnectionStateManager:194 - State change: RECONNECTED 00:47:28,226 INFO ConnectionStateManager-0 zookeeper.ZooKeeperConnection:255 - >>> Curator connected event: RECONNECTED 00:47:28,322 ERROR ConnectionStateManager-0 listen.ListenerContainer:96 - Listener (org.springframework.xd.dirt.zookeeper.ZooKeeperConnection$DelegatingConnectionStateListener@6abf4158) threw an exception java.lang.RuntimeException: java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /xd/containers/5a8deb7b-fd93-42a7-a393-2f15023e007a at org.springframework.xd.dirt.server.ContainerRegistrar.registerWithZooKeeper(ContainerRegistrar.java:301) at org.springframework.xd.dirt.server.ContainerRegistrar.access$100(ContainerRegistrar.java:93) at org.springframework.xd.dirt.server.ContainerRegistrar$ContainerAttributesRegisteringZooKeeperConnectionListener.onConnect(ContainerRegistrar.java:316) at org.springframework.xd.dirt.zookeeper.ZooKeeperConnection$DelegatingConnectionStateListener.stateChanged(ZooKeeperConnection.java:257) at org.apache.curator.framework.state.ConnectionStateManager$2.apply(ConnectionStateManager.java:222) at org.apache.curator.framework.state.ConnectionStateManager$2.apply(ConnectionStateManager.java:218) at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293) at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:215) at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:42) at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:110) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /xd/containers/5a8deb7b-fd93-42a7-a393-2f15023e007a at org.springframework.xd.dirt.container.store.ZooKeeperContainerAttributesRepository.save(ZooKeeperContainerAttributesRepository.java:75) at org.springframework.xd.dirt.container.store.ZooKeeperContainerAttributesRepository.save(ZooKeeperContainerAttributesRepository.java:42) at org.springframework.xd.dirt.server.ContainerRegistrar.registerWithZooKeeper(ContainerRegistrar.java:295) ... 15 more Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /xd/containers/5a8deb7b-fd93-42a7-a393-2f15023e007a at org.apache.zookeeper.KeeperException.create(KeeperException.java:119) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:676) at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:660) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:656) at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:441) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:431) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44) at org.springframework.xd.dirt.container.store.ZooKeeperContainerAttributesRepository.save(ZooKeeperContainerAttributesRepository.java:69) ... 17 more {noformat} This can occur if ZK does not remove the ephemeral node before the container creates a new one. This can be fixed in the following ways: * Remove the existing ephemeral node if it already exists * Register containers with a new UUID upon every new connection For now I'll implement the first solution.
Story points: 2