Project: springxd
Issue XD-2241: NoNode Exception in SpringXD Admin
Sorry to set it "Blocker"; but the problem makes SpringXD unusable. We are getting this weird, NoNode exception on the status ZNode. Example and Log given below. Once this happens, both streams and jobs cannot be deployed. For whatever reason the "status" znode goes missing. The only way for us to get the cluster back to working state, is to clear the zk znode /xd tree and restart spring-xd. At which point, we have to recreate all our streams and jobs back again.. /xd/deployments/streams/testCreateStream_SrcHttp_SinkFile1413234903170/status NoNode Exception: 13 Oct 2014 14:16:16,044 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/streams/testDestroyStream1413234903170, type: CHILD_REMOVED 13 Oct 2014 14:16:16,705 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/streams/testListStreams1413234903170, type: CHILD_ADDED 13 Oct 2014 14:16:22,818 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/streams/testListStreams1413234903170, type: CHILD_REMOVED 13 Oct 2014 14:16:23,585 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/streams/testStreamSrcHttpTimeseriesSink1413234903170, type: CHILD_ADDED 13 Oct 2014 14:16:37,694 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/jobs/filetsjob-newjob001, type: CHILD_ADDED 13 Oct 2014 14:16:49,950 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/streams/testStreamSrcHttpTimeseriesSink1413234903170, type: CHILD_REMOVED 13 Oct 2014 14:16:54,490 INFO Deployer server.StreamDeploymentListener - Deployment status for stream 'testCreateStream_SrcHttp_SinkFile1413234903170': DeploymentStatus{state=failed,error(s)=Deployment of module 'ModuleDeploymentKey{stream='testCreateStream_SrcHttp_SinkFile1413234903170', type=sink, label='file'}' to container 'd03bccd6-524b-4ff8-84d2-88f3f6daac42' timed out after 30000 ms; Deployment of module 'ModuleDeploymentKey{stream='testCreateStream_SrcHttp_SinkFile1413234903170', type=source, label='http'}' to container '52abf1c8-ba45-4994-8324-6079b03c670c' timed out after 30000 ms} 13 Oct 2014 14:16:54,493 ERROR Deployer server.InitialDeploymentListener - Exception caught while handling event org.springframework.xd.dirt.zookeeper.ZooKeeperAccessException: KeeperErrorCode = NoNode for /xd/deployments/streams/testCreateStream_SrcHttp_SinkFile1413234903170/status at org.springframework.xd.dirt.zookeeper.ZooKeeperUtils.wrapThrowable(ZooKeeperUtils.java:111) at org.springframework.xd.dirt.zookeeper.ZooKeeperUtils.wrapThrowable(ZooKeeperUtils.java:95) at org.springframework.xd.dirt.server.StreamDeploymentListener.deployStream(StreamDeploymentListener.java:185) at org.springframework.xd.dirt.server.StreamDeploymentListener.onChildAdded(StreamDeploymentListener.java:100) at org.springframework.xd.dirt.server.InitialDeploymentListener$EventHandler.call(InitialDeploymentListener.java:217) at org.springframework.xd.dirt.server.InitialDeploymentListener$EventHandler.call(InitialDeploymentListener.java:186) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /xd/deployments/streams/testCreateStream_SrcHttp_SinkFile1413234903170/status at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:274) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:270) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:266) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:253) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:41) at org.springframework.xd.dirt.server.StreamDeploymentListener.deployStream(StreamDeploymentListener.java:179) ... 7 more 13 Oct 2014 14:16:56,251 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/jobs/filetsjob-newjob001, type: CHILD_REMOVED 13 Oct 2014 14:17:08,179 INFO Deployer server.JobDeploymentListener - Deployment status for job 'filetsjob-newjob001': DeploymentStatus{state=failed,error(s)=Deployment of module 'ModuleDeploymentKey{stream='filetsjob-newjob001', type=job, label='filepolltimeseries'}' to container '244d5076-f69d-42a4-8110-3b046cea2667' timed out after 30000 ms} 13 Oct 2014 14:17:08,181 ERROR Deployer server.InitialDeploymentListener - Exception caught while handling event org.springframework.xd.dirt.zookeeper.ZooKeeperAccessException: KeeperErrorCode = NoNode for /xd/deployments/jobs/filetsjob-newjob001/status at org.springframework.xd.dirt.zookeeper.ZooKeeperUtils.wrapThrowable(ZooKeeperUtils.java:111) at org.springframework.xd.dirt.zookeeper.ZooKeeperUtils.wrapThrowable(ZooKeeperUtils.java:95) at org.springframework.xd.dirt.server.JobDeploymentListener.deployJob(JobDeploymentListener.java:175) at org.springframework.xd.dirt.server.JobDeploymentListener.onChildAdded(JobDeploymentListener.java:99) at org.springframework.xd.dirt.server.InitialDeploymentListener$EventHandler.call(InitialDeploymentListener.java:217) at org.springframework.xd.dirt.server.InitialDeploymentListener$EventHandler.call(InitialDeploymentListener.java:186) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /xd/deployments/jobs/filetsjob-newjob001/status at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:274) at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:270) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:266) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:253) at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:41) at org.springframework.xd.dirt.server.JobDeploymentListener.deployJob(JobDeploymentListener.java:165) ... 7 more 13 Oct 2014 14:17:10,553 INFO DeploymentSupervisorCacheListener-0 server.InitialDeploymentListener - Path cache event: /deployments/jobs/filetsjob-newjob001, type: CHILD_ADDED
Story points: 1