Code change:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --ci-kubernetes-e2e-ubuntu-gce [sig-storage] Subpath Container restart should verify that container can restart successfully after configmaps modified links to appreciated links to specific failures in spyglass appreciated Triage shows that this has been flaking quite frequently: It has flaked in 5 of the last 10 runs: /sig storage /priority important-soon <!-- Please see the deflaking doc () for more guidance! --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Noticed some more flakes on this one
Yep, looks like this is still flaking quite frequently: Going to re-ping SIG Storage.
/assign
Test is flaking maybe half of the time. ~This test is failing in gce-ubuntu-master-default - not sure if it's failing in any other dashboards, nor whether this is run anywhere else.~ edit: This is failing on Ubuntu only. Not sure if this is limited to master branch. Fix to the original issue was in since 1.19. This is a subpath test which a configmap a container that mounts the configmap a liveness probe and waits for the container to restart. liveness probe and waits for the container to stop restarting. The test failed at the last step - the container never stopped restarting. Timeout is 2min. The liveness probe continuously checks whether a "probeFile" exists in (an emptyDir), and if it doesn't exist, whether the configmap file contains the modified value. The probeFile essentially acts as a condition which guards the probe from checking the configmap file. The test fails the liveness probe by removing the probeFile without updating the configmap just yet. It then fixes the liveness probe by updating the configmap to the modified value. So if the container keeps restarting, the liveness probe could've been failing continuously within 2min, due to one of the following: (? not sure about this one) Command can't be executed (for example if container can't start) configmap file was never updated. Within 2min, the pod only restarted 5 times. The liveness probe's period is 2s, initial delay is 1s, and failure threshold is 1, so a restart due to liveness probe failure is triggered every 3s. The image is busybox. The command is just , so it's unlikely for the container to have crashed. There were no liveness probe unhealthy events after the test fixes the probe, and yet it continued to restart. No more events after probe is fixed.
Could we check if the configmap value was updated for sure to rule out that scenario ? I have not tried to run an end to end test locally :) so am not sure how to debug it
Will do! I ran the test in a GKE Ubuntu cluster running 1.19.6 and couldn't reproduce the failure. I'm now trying to run a cluster on GCE with Ubuntu nodes running k8s master, but haven't been successful after trying a number of techniques. If anyone happens to know how to do this let me know; I'll keep asking around otherwise.
I got a repro on a GKE cluster backed by COS. After the test failed the container ends up in Running state eventually, and the configmap file outputs the expected modified value. The fix is to increase the test timeout. PR coming up.

Related discussion:
text: ginkgo.By("Waiting for container to stop restarting") stableCount := int(0) stableThreshold := int(time.Minute / framework.Poll) <del> err = wait.PollImmediate(framework.Poll, 2*time.Minute, func() (bool, error) { </del> <ins> err = wait.PollImmediate(framework.Poll, framework.PodStartTimeout, func() (bool, error) { </ins> pod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Get(context.TODO(), pod.Name, metav1.GetOptions{}) if err != nil { return false, err