Change:
Two cases of Reboot e2e are flaky on Jenkins: unclean reboot (echo b sudo tee /proc/sysrq-trigger). They reboot only one node, while reboot of the second doesn't happen and causes test timeout:
CC
Master tracking bug
Re-opening. Current status as follows: each node by dropping all outbound packages for a while and ensure they function afterwards (Failed 8 times in the last 30 runs. Stability: 73 %) each node by dropping all inbound packages for a while and ensure they function afterwards (Failed 5 times in the last 30 runs. Stability: 83 %) each node by ordering unclean reboot and ensure they function upon restart (Failed 3 times in the last 30 runs. Stability: 90 %) each node by triggering kernel panic and ensure they function upon restart (Failed 3 times in the last 30 runs. Stability: 90 %) each node by switching off the network interface and ensure they function upon switch on (Failed 3 times in the last 30 runs. Stability: 90 %) each node by ordering clean reboot and ensure they function upon restart (Failed 3 times in the last 30 runs. Stability: 90 %)
Any progress on this ? cc in case Jerzey is OOTO.
Sorry for replying so late, but I was on holidays. The flakiness was caused by liveness probes of some pods (e.g.: kube-dns).
Also - implemented a fix for it - see
For some reason seems to have fixed most but not all reboot tests. For example: Reboot each node by dropping all outbound packets for a while and ensure they function afterwards: kubernetes-e2e-gce-reboot (Failed 8 times in the last 30 runs. Stability: 73 %) Need to investigate and fix.
perhaps you could take a look in Poland while the U.S. is celebrating July 2nd, 3rd, 4th? :-)
There was a problem with monit restarting kubelet because of blocking localhost communication on firewall. Fixed by .
- since is now merged, can we close this issue? Or is there anything remaining here?
Yes, I think that we can close this now. All reboot tests are pretty stable (1 failure in 180 runs, in total): each node by dropping all inbound packets for a while and ensure they function afterwards (No known failures. Stability 100 %) 4 min 5 sec Passed each node by dropping all outbound packets for a while and ensure they function afterwards (No known failures. Stability 100 %) 4 min 6 sec Passed each node by ordering clean reboot and ensure they function upon restart (Failed 1 times in the last 30 runs. Stability: 96 %) 1 min 41 sec Passed each node by ordering unclean reboot and ensure they function upon restart (No known 1. failures. Stability 100 %) 1 min 25 sec Passed each node by switching off the network interface and ensure they function upon switch on (No known failures. Stability 100 %) 4 min 0 sec Passed each node by triggering kernel panic and ensure they function upon restart (No known failures. Stability 100 %)

Discussion:
It("each node by ordering clean reboot and ensure they function upon restart", func() { // clean shutdown and restart <del> testReboot(c, "sudo reboot") </del> <ins> // We sleep 10 seconds to give some time for ssh command to cleanly finish before the node is rebooted. testReboot(c, "nohup sh -c 'sleep 10 && sudo reboot' >/dev/null 2>&1 &") </ins> }) It("each node by ordering unclean reboot and ensure they function upon restart", func() { // unclean shutdown and restart <del> testReboot(c, "echo b | sudo tee /proc/sysrq-trigger") </del> <ins> // We sleep 10 seconds to give some time for ssh command to cleanly finish before the node is shutdown. testReboot(c, "nohup sh -c 'sleep 10 && echo b | sudo tee /proc/sysrq-trigger' >/dev/null 2>&1 &") </ins> }) It("each node by triggering kernel panic and ensure they function upon restart", func() { // kernel panic <del> testReboot(c, "echo c | sudo tee /proc/sysrq-trigger") </del> <ins> // We sleep 10 seconds to give some time for ssh command to cleanly finish before kernel panic is triggered. testReboot(c, "nohup sh -c 'sleep 10 && echo c | sudo tee /proc/sysrq-trigger' >/dev/null 2>&1 &") </ins> }) It("each node by switching off the network interface and ensure they function upon switch on", func() { // switch the network interface off for a while to simulate a network outage <del> testReboot(c, "sudo ifdown eth0 && sleep 120 && sudo ifup eth0") </del> <ins> // We sleep 10 seconds to give some time for ssh command to cleanly finish before network is down. testReboot(c, "nohup sh -c 'sleep 10 && sudo ifdown eth0 && sleep 120 && sudo ifup eth0' >/dev/null 2>&1 &") </ins> }) <del> It("each node by dropping all inbound packages for a while and ensure they function afterwards", func() { </del> <ins> It("each node by dropping all inbound packets for a while and ensure they function afterwards", func() { </ins> // tell the firewall to drop all inbound packets for a while <del> testReboot(c, "sudo iptables -A INPUT -j DROP && sleep 120 && sudo iptables -D INPUT -j DROP") </del> <ins> // We sleep 10 seconds to give some time for ssh command to cleanly finish before starting dropping inbound packets. testReboot(c, "nohup sh -c 'sleep 10 && sudo iptables -A INPUT -j DROP && sleep 120 && sudo iptables -D INPUT -j DROP' >/dev/null 2>&1 &") </ins> }) <del> It("each node by dropping all outbound packages for a while and ensure they function afterwards", func() { </del> <ins> It("each node by dropping all outbound packets for a while and ensure they function afterwards", func() { </ins> // tell the firewall to drop all outbound packets for a while <del> testReboot(c, "sudo iptables -A OUTPUT -j DROP && sleep 120 && sudo iptables -D OUTPUT -j DROP") </del> <ins> // We sleep 10 seconds to give some time for ssh command to cleanly finish before starting dropping outbound packets. testReboot(c, "nohup sh -c 'sleep 10 && sudo iptables -A OUTPUT -j DROP && sleep 120 && sudo iptables -D OUTPUT -j DROP' >/dev/null 2>&1 &") </ins> }) })

// Reboot the node. if err = issueSSHCommand(node, provider, rebootCmd); err != nil { <del> // Just log the error as reboot may cause unclean termination of ssh session, which is expected. </del> Logf("Error while issuing ssh command: %v", err) <ins> result <- false return </ins> } // Wait for some kind of "not ready" status.