Project: springxd
Issue XD-1783: Make failed deployment cleanup more robust
When a deployment fails, the supervisor will clean up failed deployment attempts. If the deployment path is removed while the supervisor is waiting (for instance if the target container departs the cluster) then a NoNodeException will be thrown: {code} 17:22:59,702 WARN ContainersPathChildrenCache-0 server.ModuleDeploymentWriter:361 - Error while cleaning up failed deployment /deployments/modules/40494bdb-0d8c-4b5d-a895-bf94432d9d3b/s.sink.log-1 org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /xd/deployments/modules/40494bdb-0d8c-4b5d-a895-bf94432d9d3b/s.sink.log-1 org.apache.zookeeper.KeeperException.create(KeeperException.java:111) org.apache.zookeeper.KeeperException.create(KeeperException.java:51) org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:239) org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:234) org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:230) org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:215) org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:42) org.springframework.xd.dirt.server.ModuleDeploymentWriter.processResults(ModuleDeploymentWriter.java:355) org.springframework.xd.dirt.server.ModuleDeploymentWriter.writeDeployment(ModuleDeploymentWriter.java:325) org.springframework.xd.dirt.server.ModuleDeploymentWriter.writeDeployment(ModuleDeploymentWriter.java:247) org.springframework.xd.dirt.server.ContainerListener.redeployStreamModule(ContainerListener.java:432) org.springframework.xd.dirt.server.ContainerListener.onChildLeft(ContainerListener.java:403) org.springframework.xd.dirt.server.ContainerListener.childEvent(ContainerListener.java:159) org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:509) org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:503) org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:500) org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) org.apache.curator.framework.recipes.cache.PathChildrenCache$10.run(PathChildrenCache.java:762) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask.run(FutureTask.java:262) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask.run(FutureTask.java:262) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:744) 17:22:59,718 ERROR ContainersPathChildrenCache-0 cache.PathChildrenCache:557 - java.lang.IllegalStateException: Container 40494bdb-0d8c-4b5d-a895-bf94432d9d3b experienced the following error deploying module log-1 of type sink: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode org.springframework.xd.dirt.server.ModuleDeploymentWriter.validateResults(ModuleDeploymentWriter.java:523) org.springframework.xd.dirt.server.ModuleDeploymentWriter.validateResult(ModuleDeploymentWriter.java:474) org.springframework.xd.dirt.server.ContainerListener.redeployStreamModule(ContainerListener.java:436) org.springframework.xd.dirt.server.ContainerListener.onChildLeft(ContainerListener.java:403) org.springframework.xd.dirt.server.ContainerListener.childEvent(ContainerListener.java:159) org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:509) org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:503) org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:500) org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) org.apache.curator.framework.recipes.cache.PathChildrenCache$10.run(PathChildrenCache.java:762) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask.run(FutureTask.java:262) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask.run(FutureTask.java:262) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:744) {code} Additionally, if the supervisor tries to remove a failed deployment after the container has written to it, the following appears in the log: {code} WARN ContainersPathChildrenCache-0 server.ModuleDeploymentWriter:361 - Error while cleaning up failed deployment /deployments/modules/18c7b4d7-991c-487b-a6e3-006b2dbf87fa/s.source.http-0 org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /xd/deployments/modules/18c7b4d7-991c-487b-a6e3-006b2dbf87fa/s.source.http-0 {code} The supervisor has to force the removal of the node, including children.
Story points: 2