Change:
if ns != "" { nsArg = fmt.Sprintf("--namespace=%s", ns) } <del> for _, selector := range selectors { resources := RunKubectlOrDie("get", "rc,svc", "-l", selector, "--no-headers", nsArg) if resources != "" { Failf("Resources left running after stop:n%s", resources) } pods := RunKubectlOrDie("get", "pods", "-l", selector, nsArg, "-o", "go-template={{ range .items }}{{ if not .metadata.deletionTimestamp }}{{ .metadata.name }}{{ "n" }}{{ end }}{{ end }}") if pods != "" { Failf("Pods left unterminated after stop:n%s", pods) </del> <ins> backoff := wait.Backoff{ Duration: 5 * time.Second, Factor:   2, Steps:    3, } var e error verifyCleanupFunc := func() (bool, error) { e = nil for _, selector := range selectors { resources := RunKubectlOrDie("get", "rc,svc", "-l", selector, "--no-headers", nsArg) if resources != "" { e = fmt.Errorf("Resources left running after stop:n%s", resources) return false, nil } pods := RunKubectlOrDie("get", "pods", "-l", selector, nsArg, "-o", "go-template={{ range .items }}{{ if not .metadata.deletionTimestamp }}{{ .metadata.name }}{{ "n" }}{{ end }}{{ end }}") if pods != "" { e = fmt.Errorf("Pods left unterminated after stop:n%s", pods) return false, nil } </ins> } <ins> return true, nil } err := wait.ExponentialBackoff(backoff, verifyCleanupFunc) if err != nil { Failf(e.Error()) </ins> } }

Discussion:
/priority failing-test /priority important-soon /sig cli /kind flake
Reiterating the mentions to trigger a notification: <detailsIn response to : ### Flaky Job: pr:pull-kubernetes-e2e-gce /priority important-soon /sig cli /kind flake Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Here's the TestGrid view to dig into the flakes and get more detailed logs https://k8s-
/assign
GitHub didn't allow me to assign the following users: Liujingfang1. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The failure is that after the replication controller is cleaned up, the pods are not terminated. I investigated the failure. It is from function after . does two things: make sure the resource is deleted and make sure the pods are deleted. If the pods haven't started termination during the running, we will see this failure. The solution is to add several retry, give the pods more time to start termination.
Liujingfang1@ I see you have a PR out for the fix, thanks. Any ETA on when this will be merged? I can monitor test status once its in before pinging you to close the issue.
ETA is this week. Maybe tomorrow :)
can we plz reopen this issue and close it once we see the flake is actually fixed? Thanks
/reopen
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/reopen
what's wrong with having it closed and re-open if the flake hits back?
there is no harm in doing it either way. The only reason I prefer keeping it open is because it filters in my customized query tracking all open issues created by me. Its easier to verify the fix that way versus going after .

closed issues that fell off the list.
sgtm as long as you'll don't forget to close it when confirmed :wink: The reason I usually close them is so that I don't have to remember.
I will definitely keep an eye out and close it :) I usually check on these before my weekly flake report.
I do see a few flakes with this test still https://k8s- Both failures are the same: Apr 2 21:35:55.396: Pods left unterminated after stop: update-demo-nautilus-dggjz update-demo-nautilus-zb8pv Gubernator results: https://k8s- https://k8s-
I've opened and increased the timeout to 1 minute similarly to the time GC tests are waiting for cleanups. ptal
it looks like the last linked PR fixed the problem and from looking at the testgrid it hasn't appeared, mind closing?
Thanks for the fix. The test seems to pass fine in the GCE job. However I do see a couple fo flakes in the GKE job. https://k8s- https://k8s- Let me know if this needs a different fix/investigation. If so I am happy to close this and open a new issue.
There are only 2 test failure happened from 03/27 to 04/10. And these 2 failures are high likely to be related to some cross-cutting infra failure, e.g. https://k8s- I believe we can close this issue.
Sounds good. I am closing this issue now. Will reopen if it surfaces as top flake again. Thanks for your efforts here.
/close