Change:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce Which test(s) are flaking: [sig-storage] PersistentVolumes GCEPD should test that deleting the Namespace of a PVC and Pod causes the successful detach of Persistent Disk Testgrid link: Reason for failure: Anything else we need to know: it seems like this started flaking more frequently during 2019-12-02 triage link: examples of failures: /area platform/gcp /sig storage
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : <!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --- pull-kubernetes-e2e-gce [sig-storage] PersistentVolumes GCEPD should test that deleting the Namespace of a PVC and Pod causes the successful detach of Persistent Anything else we need to know: it seems like this started flaking more frequently during 2019-12-02 triage link: examples of failures: /sig storage Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area provider/gcp
/sig cloud-provider I question whether we should be running this test by default, since it seems like it's exercising a specific cloud provider's implementation of persistent volumes?
Another occurrence of the test failure:
interestingly this is not flaky at all against master (opening the last several dozen runs doesn't show any failures/retries of that test) and is exceptionally flaky in PR jobs: in the last 47 runs, there were 26 failures (and only one PR looked like it could have introduced relevant failures): (potentially PD storage-related PR) (x3) (x3)
/assign
also, the timeout error is on the namespace delete, not the PD detach
opened to mark flaky to unblock the queue, opened to add debugging around the timeout failure; will keep this issue open and in the milestone to ensure the test is fixed or removed before 1.18.0
Looking at this run: namespace deletion timeout in the test is 5 minutes, so looks like this took just a little longer. Looking into what is taking the pod so long to be deleted
It seems like kubelet tore down and deleted the Pod pretty quickly and it was registered by apiserver log. For some reason, the namespace controller doesn't see that deletion?
From in , a little earlier than the log posted: E1216 11:23:16. 1335 ] Operation for """ ("-c02f-4701-9708-")" failed. No retries permitted until 2019-12-16 11:23:16. +0000 UTC m=+1060. (durationBeforeRetry 500ms). Error: "recovered from panic "runtime error: invalid memory address or nil pointer dereference". (err=<nil) Call stack:ngoroutine [running]:(0xc0032cad50)n +0xb5npanic(0x3a27160, 0x6d2bdf0)n +0x1b2(0xc000c94a20, 0x81, 0x0, 0x0, 0x4979460, 0xc0021d0b40, 0x0, 0x0)n +0xbd(0xc001f82a80, 0x7f, 0x4a33500, 0xc002099790, 0xc0011a7ba8, 0x0, 0x7f)n +0x20cnpath/filepath.Walk(0xc001f82a80, 0x7f, 0xc001fdcba8, 0xc001f82a80, 0x7f)n +0xff(0x4a28680, 0xc00090d580, 0xc0016416c0, 0x3a, 0xc002b50dd0, 0xb, 0xc0016416c0, 0x3a)n +0x8ce(subpath).CleanSubPaths(0xc0004a08b0, 0xc0016416c0, 0x3a, 0xc002b50dd0, 0xb, 0x1, 0x0)n +0x5f(operationGenerator).GenerateUnmountVolumeFunc.func1(0xc0012f95e0, 0xc0032cad50, 0x2e, 0x1)n +0x15a(GeneratedOperations).Run(0xc001dd3200, 0x45c1e0, 0xc0016b6ba0, 0xc000e16c90, 0xc0019c02a1)n +0xd6(nestedPendingOperations).Run.func1(0xc001039580, 0xc0010a2e10, 0x48, 0xc0020085d0, 0x24, 0xc001dd3200, 0x0, 0x0, 0x0, 0x0)n +0x114ncreated by (*nestedPendingOperations).Runn +0x39fn" which is addressed over
I'm not sure the kubelet panic is related to this test flake. So far in this flake, the API server did register pod deletion, but the namespace controller still seems to think the pod is not deleted.
This problem also happened in kops.
I encountered this problem in kubernetes and Kops projects, which seems to be more serious.
test has been marked flaky, and can now be seen in root cause is an extremely slow namespace deletion controller; see analysis in
Opened to track the namespace controller issue. In the meantime, I would probably suggest rewriting this test to: the namespace delete deletecollection calls against the pvcs and pods, rather than waiting for the namespace controller for the absence of the pvc and pod objects, rather than the namespace
I think then the test case would be very similar to "[sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node", which 2 PVCs used by a single Pod the pod reschedules the pod to a different node and validates that it was successful (which indicates that it got detached and reattached successfully) Pod and PVC at the end and verifies PV deletion. I'm in favor of just removing this test (and having a separate non-storage test for namespace deletion) since we already have similar coverage in an existing test case.
Doing this inside a deleting namespace ensures nothing in the PVC in mount or pod teardown blocks on the inability to create events. I would keep that aspect of it and just trigger the pod/pvc deletions yourself.
Ok, then I would be more in favor of adding a new test case in our testsuite that explicitly does the namespace deletion. I want to start moving away from having cloud provider-specific tests, and as a bonus we'll get coverage for CSI drivers too.
I'm ok adding a non-storage test specifically around namespace deletion, but ensuring the PV/PVC infrastructure behaves well in a namespace that is being deleted seems valuable as well (and that test would not need to wait on the namespace to disappear)
Agree. When I said "testsuite" I meant the
/milestone v1.19
Hit by see

Discussion:
import ( "context" "github.com/onsi/ginkgo" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

}) // Test that a Pod and PVC attached to a GCEPD successfully unmounts and detaches when the encompassing Namespace is deleted. <del> ginkgo.It("should test that deleting the Namespace of a PVC and Pod causes the successful detach of Persistent Disk [Flaky]", func() { </del> <ins> ginkgo.It("should test that deleting the Namespace of a PVC and Pod causes the successful detach of Persistent Disk", func() { </ins> ginkgo.By("Deleting the Namespace") err := c.CoreV1().Namespaces().Delete(context.TODO(), ns, metav1.DeleteOptions{}) framework.ExpectNoError(err) <del> err = framework.WaitForNamespacesDeleted(c, []string{ns}, framework.DefaultNamespaceDeletionTimeout) framework.ExpectNoError(err) </del> <ins> // issue deletes for the client pod and claim, accelerating namespace controller actions e2epod.DeletePodOrFail(c, clientPod.Namespace, clientPod.Name) framework.ExpectNoError(e2epv.DeletePersistentVolumeClaim(c, pvc.Name, ns), "Unable to delete PVC ", pvc.Name) </ins> ginkgo.By("Verifying Persistent Disk detaches") framework.ExpectNoError(waitForPDDetach(diskName, node), "PD ", diskName, " did not detach")