Code change:
Which jobs are failing: Which test(s) are failing: Testgrid link: Reason for failure: check Anything else we need to know: /cc /milestone v1.18 /sig cloud-provider-gcp
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Which jobs are failing: Which test(s) are failing: Testgrid link: check /cc /milestone v1.18 /sig cloud-provider-gcp </details
/sig cloud-provider
/area provider/gcp
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /area provider/gke </details
Probably related to
shows the merge shows the errant merge that inadvertantly disabled the test This test was probably failing since forever.
Any news?
this one is on cloud provider GCP.
Can you please take a look?
Follow up from today's release meeting: If the test has been broken for as long as we know it, theres value in just removing it. If the test owners don't have capacity to look into it, CI Signal can investigate.
Any advice on which team is responsible? It seems there isn't any maintainer for this job.
try talking to please
/assign
Turning to the question of whether these tests are actually useful... One of the tests that has been continuously failing is this one Which essentially does because From the logs (example ) it waits for 2 minutes (120 seconds) for each node to not be ready and then for the next couple minutes it just checks this condition until it fails So it looks like the e2e tests are not failing because of inherent infra issues (i.e., kube-up decided to do some weird thing) but the issue seems to be with the time to reach a condition. Again if we look at the logs we can see that switching the network interface does have an effect and leads to the nodes reaching a non-ready status but ultimately fail to come back up within the desired 5m time mark - this bit of the tests is where it all goes bad . The question then turns to turning off the network interface for a node in such a way () and then expecting the node to come back to live within 5 minutes of turning the interfac back on something reasonable? If it is, then either we have to look at why it takes longer than 5 minutes or we can increase the timeouts If it isn't, then how can we test the ability of nodes to be rebooted?
/assign
Wanted to triple check on the test looking for nodes to be back up after being rebooted: I dont think 5 minutes is a hard deadline - only SLOs I could find were for scalability

Related discussion:
text: ginkgo.It("each node by switching off the network interface and ensure they function upon switch on", func() { // switch the network interface off for a while to simulate a network outage // We sleep 10 seconds to give some time for ssh command to cleanly finish before network is down. <del> testReboot(f.ClientSet, "nohup sh -c 'sleep 10 && sudo ip link set eth0 down && sleep 120 && sudo ip link set eth0 up && (sudo dhclient || true)' >/dev/null 2>&1 &", nil) </del> <ins> cmd := "nohup sh -c '" + "sleep 10; " + "echo Shutting down eth0 | sudo tee /dev/kmsg; " + "sudo ip link set eth0 down | sudo tee /dev/kmsg; " + "sleep 120; " + "echo Starting up eth0 | sudo tee /dev/kmsg; " + "sudo ip link set eth0 up | sudo tee /dev/kmsg; " + "sleep 10; " + "echo Retrying starting up eth0 | sudo tee /dev/kmsg; " + "sudo ip link set eth0 up | sudo tee /dev/kmsg; " + "echo Running dhclient | sudo tee /dev/kmsg; " + "sudo dhclient | sudo tee /dev/kmsg; " + "echo Starting systemd-networkd | sudo tee /dev/kmsg; " + "sudo systemctl restart systemd-networkd | sudo tee /dev/kmsg" + "' >/dev/null 2>&1 &" testReboot(f.ClientSet, cmd, nil) </ins> }) ginkgo.It("each node by dropping all inbound packets for a while and ensure they function afterwards", func() {