Change:
/assign The test flakes presubmit testing in PRs six times per week The site can focus on just this test: Which shows that 83 out of 85 errors match this pattern:
Previous attempt to address this flake:
CC
Took a quick look at a failed test: https://k8s- The indicates: . or and can you help investigate.
CC
The failure in the GKE multizone test suite is probably due to the GKE issue I fixed last week.
We'll dig into this and see what's up.
Expectations: This work should interrupt and take priority over other activity while it is assigned to you. Expectation for sig leads is to identify someone to fix the issue, them and yourself. Expectation for assignees are to investigate the issue and either a) resolve it or b) it to a more appropriate person and b) yourself.
I repeatedly ran this test on my e2e cluster since 7pm last night, and it has not failed yet. Seems like it fails more frequently in the PR env instead of the CI env, so may need to look into differences between the two. The test is failing during the deletion of the namespace. It seems like there are still some resources not getting deleted. I'm trying to dig deeper in the logs to see if I can find what resources are not getting cleaned up.
Here's one failed : From the API server initial namespace deletion request from the e2e test: I0427 23:23:29. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-cx3bh: (4.744775ms) 200 [[ (linux/amd64) kubernetes/] 104.197.101.155:] controller triggers deletion on all the objects in the namespace: I0427 23:24:37. 5 ] DELETE /apis/apps/v1beta1/namespaces/e2e-tests-pv-cx3bh/statefulsets: (91.176059ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccount:kube-system:namespace-controller] [::1]:] I0427 23:24:37. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-cx3bh/resourcequotas: (7.608267ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccount:kube-system:namespace-controller] [::1]:] etc ... gets finally deleted by controller I0427 23:26:30. 5 ] DELETE /apis/apps/v1beta1/namespaces/e2e-tests-pv-cx3bh/statefulsets: (87.184389ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serv iceaccount:kube-system:namespace-controller] [::1]:] I0427 23:26:30. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-cx3bh/resourcequotas: (21.517822ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccoun t:kube-system:namespace-controller] [::1]:] etc ... deletion finished I0427 23:26:31. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-cx3bh: (16.58598ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccount:kube-system:namespace-controller] [::1]:] 2E test cleanup tries to delete it again: I0427 23:26:31. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-cx3bh: (9.104808ms) 404 [[ (linux/amd64) kubernetes/] 104.197.101.155:] <= This is the e2e test failure cleanup Total time to delete 3 mins Here's a successful : : deletion request I0428 00:45:14. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-c4t8j: (11.208829ms) 200 [[ (linux/amd64) kubernetes/] 104.197.153.246:] starts object deletions I0428 00:45:46. 5 ] DELETE /apis/apps/v1beta1/namespaces/e2e-tests-pv-c4t8j/deployments: (33.104152ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:servi ceaccount:kube-system:namespace-controller] [::1]:] I0428 00:45:46. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-c4t8j/podtemplates: (3.447937ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccount:k ube-system:namespace-controller] [::1]:] etc ... gets finally deleted by controller I0428 00:46:17. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-c4t8j/podtemplates: (10.679029ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccount: kube-system:namespace-controller] [::1]:] I0428 00:46:20. 5 ] DELETE /apis/extensions/v1beta1/namespaces/e2e-tests-pv-c4t8j/deployments: (6.130617ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system: serviceaccount:kube-system:namespace-controller] [::1]:] etc ... deletion is finished I0428 00:46:21. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-c4t8j: (21.10359ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccount:kube-system:na mespace-controller] [::1]:] 2E test cleanup I0428 00:46:32. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-c4t8j: (1.436939ms) 404 [[ (linux/amd64) kubernetes/] 104.197.153.246:] Total time to delete ~1 minute Comparing the bad and good runs, these were some supicious areas: the bad run, the period between 1) and 2) was 1 minute, whereas in the good run it was 30 seconds the bad run, the period between 2) and 3) was ~2 minutes, whereas in the good run it was ~20 seconds. The slowness in the bad run seems systematic, I did not find any one object that was taking a particularly long time to delete, all of them seemed to be taking longer compared to the good run. I'm not sure where to dig down further to find the cause of the general slowness. I don't know if this is expected, and we should just increase the timeout? The test framework seems to have a default timeout of 5 minutes, whereas this volume test timeout is only 3 minutes.
Great analysis! Since no singular event stands out as taking too much time how about, in the name of expediency, just changing the timeout to 5m and see how the tests behave?
The delay of ns deleteion might related to this PR and . is still open.
Those two issues do not seem to impact the deletion of the namespace itself, so I don't think fixing those will solve this issue.
But I think delaying the deletion of a pod will affect the time to delete the namespace.

Discussion:
) const ( <del> maxKubectlExecRetries = 5 </del> <ins> maxKubectlExecRetries           = 5 DefaultNamespaceDeletionTimeout = 5 * time.Minute </ins> ) // Framework supports common operations used by e2e tests; it will keep a client & a namespace for you.

if TestContext.DeleteNamespace && (TestContext.DeleteNamespaceOnFailure || !CurrentGinkgoTestDescription().Failed) { for _, ns := range f.namespacesToDelete { By(fmt.Sprintf("Destroying namespace %q for this suite.", ns.Name)) <del> timeout := 5 * time.Minute </del> <ins> timeout := DefaultNamespaceDeletionTimeout </ins> if f.NamespaceDeletionTimeout != 0 { timeout = f.NamespaceDeletionTimeout }

package storage import ( <del> "time" </del> . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

err := c.CoreV1().Namespaces().Delete(ns, nil) Expect(err).NotTo(HaveOccurred()) <del> err = framework.WaitForNamespacesDeleted(c, []string{ns}, 3*time.Minute) </del> <ins> err = framework.WaitForNamespacesDeleted(c, []string{ns}, framework.DefaultNamespaceDeletionTimeout) </ins> Expect(err).NotTo(HaveOccurred()) By("Verifying Persistent Disk detaches")