Change:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: ci-kubernetes-e2e-ubuntu-gce-containerd Which test(s) are flaking: [sig-storage] In-tree Volumes [Driver: gluster] [Testpattern: Pre-provisioned PV (default fs)] subPath should support file as subpath [LinuxOnly] Testgrid link: Reason for failure: Will look at it shortly, why it is flaking. Anything else we need to know: links to appreciatedlinks to specific failures in spyglass appreciated
/sig storage
The namespace terminating during test setup is super strange but seems like a test-only issue, not an issue with the functionality being tested. I'd recommend looking at the test, API server, and controller logs to figure out why the test namespace () was deleted and who deleted it
does this help ?
feels like a test only problem like said. We don't even have gluster in our CI/prow to test against... cc Michelle, Xing, Can you please confirm that this is not a blocker for 1.19.0?
Knowing the timeframe for when the test started and when that delete happened would be interesting to know. Possibilities I can see: something in the test timed out, and the cleanup happened in parallel with something in the test attempting to run multiple tests are reusing the same test context and namespace name at the same time (I can never keep straight how ginkgo parallel tests work)
This does not look like a blocker for 1.19.0.
Thanks /milestone v1.20
looking at the logs it seems to be the latter "same test context and namespace name at the same time", but we can deal with this for 1.19.1
cc This looks like another instance of "the same test namespace is being reused". I believe had a mitigation for that to try to make the namespaces more unique.
(btw, the test brings up a containerized gluster server so we are exercising gluster functionality)
Yeah, looking at the build log found that it is the same issue as The issue is still around the double deletion of namespace, i.e. we delete the test namespace during the test execution, but later on the call we delete the namespace again to ensure it truly get cleaned up. But unfortunately, this namespace has been reused by another test case between this gap.
Some potential fix I can think of: deletion of the primary namespace twice is inevitable, maybe we should consider generate primary namespace based on a hashing of the test cases name? the 4 digits random namespace to 6, this should not solve the problem in nature but it will extremely decrease the chance it happens. using an in-memory namespace map to keep record of already generated namespaces for the whole test run and refuse to generate the same namespace again in the whole lifecycle of a prow job.
do you know if this gluster test is using your mitigation in ?
hash of the testcase name + random suffix (because tests can be run multiple times) seems like a good way to mitigate the chances of tests reusing the same namespace. I'm unsure if we know the test case name at the time of test namespace creation though. I think increasing the length of the random suffix is a quick and easy mitigation believe ginkgo to run in parallel, so this may not work
Yes it is. But my fix in can only reduce the chance of collision and probably improve readability of logs. But not to eliminate this issue in nature.
AFAICT does not affect intree code at all, because it only fixes namespace of driver. If we are seeing collision of primary namespace for intree code then it is a different problem.
on only deals with the collision between test namespace and driver namespace. And that fix is taking affect. failure is because a test namespace collides with another test namespace. And this is not related to in-tree code at all. Any test could potentially become flaky because their namespace can be deleted during test execution. It's just happens to be this intree test this time. The root cause for this problem is: We cleaned up the test namespace twice which gives it a small gap that another testcase can generate the same namespace and be affected by the second namespace deletion from the original testcase.
/assign I would like to give it a shot.
I've opened to address this issue.
/unassign /assign

Discussion:
} } <ins> // DeleteNamespace can be used to delete a namespace. Additionally it can be used to // dump namespace information so as it can be used as an alternative of framework // deleting the namespace towards the end. func (f *Framework) DeleteNamespace(name string) { defer func() { err := f.ClientSet.CoreV1().Namespaces().Delete(context.TODO(), name, metav1.DeleteOptions{}) if err != nil && !apierrors.IsNotFound(err) { Logf("error deleting namespace %s: %v", name, err) return } err = WaitForNamespacesDeleted(f.ClientSet, []string{name}, DefaultNamespaceDeletionTimeout) if err != nil { Logf("error deleting namespace %s: %v", name, err) return } // remove deleted namespace from namespacesToDelete map for i, ns := range f.namespacesToDelete { if ns == nil { continue } if ns.Name == name { f.namespacesToDelete = append(f.namespacesToDelete[:i], f.namespacesToDelete[i+1:]...) } } }() // if current test failed then we should dump namespace information if !f.SkipNamespaceCreation && ginkgo.CurrentGinkgoTestDescription().Failed && TestContext.DumpLogsOnFailure { DumpAllNamespaceInfo(f.ClientSet, name) } } </ins> // CreateNamespace creates a namespace for e2e testing. func (f *Framework) CreateNamespace(baseName string, labels map[string]string) (*v1.Namespace, error) { createTestingNS := TestContext.CreateTestingNS

ginkgo.By(fmt.Sprintf("deleting the test namespace: %s", ns1)) // Delete the primary namespace but its okay to fail here because this namespace will // also be deleted by framework.Aftereach hook <del> tryFunc(deleteNamespaceFunc(f.ClientSet, ns1, framework.DefaultNamespaceDeletionTimeout)) </del> <ins> tryFunc(func() { f.DeleteNamespace(ns1) }) </ins> ginkgo.By("uninstalling csi mock driver") tryFunc(func() {

tryFunc(cleanup) tryFunc(cancelLogging) ginkgo.By(fmt.Sprintf("deleting the driver namespace: %s", ns2)) <del> tryFunc(deleteNamespaceFunc(f.ClientSet, ns2, framework.DefaultNamespaceDeletionTimeout)) </del> <ins> tryFunc(func() { f.DeleteNamespace(ns2) }) </ins> // cleanup function has already ran and hence we don't need to run it again. // We do this as very last action because in-case defer(or AfterEach) races // with AfterSuite and test routine gets killed then this block still

ginkgo.By(fmt.Sprintf("deleting the test namespace: %s", ns1)) // Delete the primary namespace but its okay to fail here because this namespace will // also be deleted by framework.Aftereach hook <del> tryFunc(deleteNamespaceFunc(f.ClientSet, ns1, framework.DefaultNamespaceDeletionTimeout)) </del> <ins> tryFunc(func() { f.DeleteNamespace(ns1) }) </ins> ginkgo.By("uninstalling csi mock driver") tryFunc(cleanup) tryFunc(cancelLogging) ginkgo.By(fmt.Sprintf("deleting the driver namespace: %s", ns2)) <del> tryFunc(deleteNamespaceFunc(f.ClientSet, ns2, framework.DefaultNamespaceDeletionTimeout)) </del> <ins> tryFunc(func() { f.DeleteNamespace(ns2) }) </ins> // cleanup function has already ran and hence we don't need to run it again. // We do this as very last action because in-case defer(or AfterEach) races // with AfterSuite and test routine gets killed then this block still

return nil } <del> func deleteNamespaceFunc(cs clientset.Interface, ns string, timeout time.Duration) func() { return func() { err := cs.CoreV1().Namespaces().Delete(context.TODO(), ns, metav1.DeleteOptions{}) if err != nil && !apierrors.IsNotFound(err) { framework.Logf("error deleting namespace %s: %v", ns, err) } err = framework.WaitForNamespacesDeleted(cs, []string{ns}, timeout) if err != nil { framework.Logf("error deleting namespace %s: %v", ns, err) } } } </del> func tryFunc(f func()) error { var err error if f == nil {