Change:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: Testgrid link: Reason for failure: This appears to be an issue with the test cleaning up a namespace too early. Anything else we need to know: /cc /sig storage /milestone v1.19 /priority important-soon
I wonder if the generated namespace suffix is not unique enough. cc to take a look
We do check if the namespace already exists here:
For CSI tests, we generate a 2nd namespace for the driver. Does that use the same method? cc
Another look at the raw-log. It seems like we do deleted the namespace in another test... This namespace was in the testcase namespace in that test. And it is also the csi-driver namespace for this test. And after the first test terminated, that namespace get deleted. That's why the second test failed.
Okay, I found the issue. This happens because of the cleanup mechanism in CSI mock volume. So Let's say we have two testcases A and B. A is in namespace and it creates a namespace for its CSIDriver called B is in namespace and it creates a namespace for its CSIDriver called . theory, testcase B cannot create for its CSIDriver because that namespace has already been used. However, there is a potential race here. 's say Testcase A executes first, and after it is finished, it's namespaces are getting cleanup by in its CSIDriver. So now the namespace and was deleted. the Testcase B comes and it creates namespace for its CSIDriver and it succeed, because the namespace was already deleted and Testcase B is creating a new one. step 5, the cleanup function was called for testcase A: During this function, it deletes the namespaces again to ensure the namespace and was cleaned up. And then it detects that the still exists, the test framework think this is a leftover from testcaseA, however it is a newly created namespace from testcaseB. So the aftereach() call will delete the namespace. namespace was deleted, and boom! Testcase B failed in the middle of execution. This can happen only when testcaseA and B are pretty close to their execution time and also the namespace they generated are the same. Some potential fix thoughts: to delete the first namespace in cleanupFunc(), because the Aftereach func will help to delete that namespace anyway. longer random string, right now it's 4 digits. If we longer it to 6 or 8, the failure rate will decrease. different base name for the two namespace generated for csi mock driver. Right now they are both , we may need to change the CSI driver namespace to . This way we are also reducing the chance of collision.
We have to delete the first namespace (which includes test pods), otherwise if you delete them after deleting the driver pods - we will have pods stuck forever in terminating state. We could try naming driver namespace differently but it sounds like then this could still happen. Have we considered first 6 chars of SHA1 or something similar for namespaces? (I know shas can collide too)
Maybe try both 2) and 3) to reduce the changes of collision

Discussion:
// CreateDriverNamespace creates a namespace for CSI driver installation. // The namespace is still tracked and ensured that gets deleted when test terminates. func CreateDriverNamespace(f *framework.Framework) *v1.Namespace { <del> ginkgo.By(fmt.Sprintf("Building a driver namespace object, basename %s", f.BaseName)) namespace, err := f.CreateNamespace(f.BaseName, map[string]string{ "e2e-framework": f.BaseName, </del> <ins> ginkgo.By(fmt.Sprintf("Building a driver namespace object, basename %s", f.Namespace.Name)) // The driver namespace will be bound to the test namespace in the prefix namespace, err := f.CreateNamespace(f.Namespace.Name, map[string]string{ "e2e-framework":      f.BaseName, "e2e-test-namespace": f.Namespace.Name, </ins> }) framework.ExpectNoError(err)