Project: springxd
Issue XD-2080: Modules do not redeploy properly when Zookeeper node is lost.
* SHA baddfc24b08286a78392d5f565742c9bab5adfea * EC2 Environment ** Look at Zookeeper Ec2 Deployment Test Topology.png for a view of the topology h2. The test scenario # Bring up a up a 5 container 2 admin XD Cluster up using 3 ZK Server ensemble. # Create ticktock stream "time|log" # Deploy with --properties "module.log.count=5" # Kill one of the ZK Servers in the ensemble h2. Observed Behavior. # In this particular scenario 3 containers were affected by killing (sudo kill ) Zookeeper 2 # 2 Containers did not come back online even though they did show up in the runtime containers h2. Timeline # 14:08:21 deployed stream # 14:09:10 kill server in ZK Ensemble # After waiting a few seconds ran runtime Modules (*Note:* log2 is undeployed and log5 is then deployed) : xd:>runtime modules Module Container Id Options Deployment Properties ----------------- ------------------------------------ ------------------------------------------ --------------------- foo.sink.log.1 98a32c62-302a-484b-af9c-d670f2a3cfc2 {name=foo, expression=payload, level=INFO} {count=5, sequence=1} foo.sink.log.2 9a3a1846-bac4-4504-81fd-151665d851dc {name=foo, expression=payload, level=INFO} {count=5, sequence=2} foo.source.time.1 9a3a1846-bac4-4504-81fd-151665d851dc {fixedDelay=1, format=yyyy-MM-dd HH:mm:ss} {count=1, sequence=1} xd:>runtime modules Module Container Id Options Deployment Properties ----------------- ------------------------------------ ------------------------------------------ --------------------- foo.sink.log.1 98a32c62-302a-484b-af9c-d670f2a3cfc2 {name=foo, expression=payload, level=INFO} {count=5, sequence=1} foo.sink.log.5 5c454a39-fc4c-4bd3-b828-08cd837dc4ba {name=foo, expression=payload, level=INFO} {count=5, sequence=5} foo.source.time.1 9a3a1846-bac4-4504-81fd-151665d851dc {fixedDelay=1, format=yyyy-MM-dd HH:mm:ss} {count=1, sequence=1} xd:>runtime modules Module Container Id Options Deployment Properties ----------------- ------------------------------------ ------------------------------------------ --------------------- foo.sink.log.1 98a32c62-302a-484b-af9c-d670f2a3cfc2 {name=foo, expression=payload, level=INFO} {count=5, sequence=1} foo.sink.log.5 5c454a39-fc4c-4bd3-b828-08cd837dc4ba {name=foo, expression=payload, level=INFO} {count=5, sequence=5} foo.source.time.1 9a3a1846-bac4-4504-81fd-151665d851dc {fixedDelay=1, format=yyyy-MM-dd HH:mm:ss} {count=1, sequence=1} xd:>runtime containers Container Id Host IP Address PID Groups Custom Attributes ------------------------------------ ---------------- ------------- ---- ------ ----------------- 0ba5e6ce-aedf-429c-b846-1cd4e32836c7 ip-10-2-209-174 10.2.209.174 1045 5c454a39-fc4c-4bd3-b828-08cd837dc4ba ip-10-70-9-57 10.70.9.57 1099 707a968b-15a5-451f-9034-1e7f05cdcf97 ip-10-70-11-185 10.70.11.185 1055 98a32c62-302a-484b-af9c-d670f2a3cfc2 ip-10-110-186-48 10.110.186.48 1056 GROUPA 9a3a1846-bac4-4504-81fd-151665d851dc ip-10-70-9-153 10.70.9.153 1020 GROUP0 h2. Undeploy and redeploy stream # 14:16:42 Undeploy and redploy with module.log.count=5 xd:>runtime modules Module Container Id Options Deployment Properties ----------------- ------------------------------------ ------------------------------------------ --------------------- foo.sink.log.1 98a32c62-302a-484b-af9c-d670f2a3cfc2 {name=foo, expression=payload, level=INFO} {count=5, sequence=1} foo.sink.log.2 5c454a39-fc4c-4bd3-b828-08cd837dc4ba {name=foo, expression=payload, level=INFO} {count=5, sequence=2} foo.sink.log.5 9a3a1846-bac4-4504-81fd-151665d851dc {name=foo, expression=payload, level=INFO} {count=5, sequence=5} foo.source.time.1 98a32c62-302a-484b-af9c-d670f2a3cfc2 {fixedDelay=1, format=yyyy-MM-dd HH:mm:ss} {count=1, sequence=1} xd:>runtime containers Container Id Host IP Address PID Groups Custom Attributes ------------------------------------ ---------------- ------------- ---- ------ ----------------- 0ba5e6ce-aedf-429c-b846-1cd4e32836c7 ip-10-2-209-174 10.2.209.174 1045 5c454a39-fc4c-4bd3-b828-08cd837dc4ba ip-10-70-9-57 10.70.9.57 1099 707a968b-15a5-451f-9034-1e7f05cdcf97 ip-10-70-11-185 10.70.11.185 1055 98a32c62-302a-484b-af9c-d670f2a3cfc2 ip-10-110-186-48 10.110.186.48 1056 GROUPA 9a3a1846-bac4-4504-81fd-151665d851dc ip-10-70-9-153 10.70.9.153 1020 GROUP0 # 14:21:06 undeploy foo
Story points: 5