Change:
master-informing gce-cos-master-serial Kubernetes e2e suite.[It] [sig-storage] PersistentVolumes-local Stress with local volumes [Serial] should be able to process many pods and reuse local volumes 2023-02-15 15:12:38 +0000 UTC { failed [FAILED] persistentvolumes "local-pvt2rz2" not found In [AfterEach] at: test/e2e/storage/persistent_volumes- @ 02/21/23 18:15:38.461 } Failed multiple times: /sig storage
I think the AfterEach is racing with the test's AfterEach cleanup: Test's PV recycling go routine: We're supposed to before proceeding with cleanup. do you know if this is still a valid assumption?
/triage accepted
The recent Ginkgo changes didn't change anything regarding Go routines. If a test relies on a Go routine to have stopped, it needs to wait for that explicitly. Note that the created for an node automatically gets cancelled when the callback returns. However, that does not necessarily mean that Go routines terminate immediately.
x-post: This has been analyzed as a test issue and not a product issue, so it shouldn't block tomorrow's v1.27.0-alpha.3 release
Just bumping that this test with this same error as OP is still a common flake as of today. <img width="2768" alt="Screenshot 2023-03-20 at 3 22 23 PM" src="https://user-"
FYI, these are still flaking. Here are some failures over the last few days: !
/help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
! Just bumping that this is still flaky and failed thrice in a row recently.
The test seems to add a DeferCleanup to stop the goroutine mentioned in From the docs Is guaranteed the order and the DeferCleanup will always run before the existing AfterEach block?
I believe the order is , then (first-in-first-out), (first-in-last-out).
yes is correct. for this test that goroutine certainly adds some complexity that makes things a bit harder to reason about. assuming it's necessary and/or if you'd prefer to avoid making too many changes i'd recommend breaking the up into some s I'd be happy to submit a PR that does just that.
then the order is wrong, because the goroutine should be stopped before AfterEach, that may explain the races please be my guest, tag me and/or patrick for review
Certainly we don't need to fix our tests for us?
I know, I assumed he wanted to do it just for fun or something ... is the problem of text, you can't read between lines :)
lol! i'm happy to do it and contribute.
LOL Thanks a ton for all that you do !
PR is here: - I'm not familiar with all the k8s backslash verbs and nouns but let me know if you need me to do anything on the PR to help move it along. (and/or teach me how this works by telling me what I should do next)

Discussion:
ginkgo.BeforeEach(func(ctx context.Context) { setupStorageClass(ctx, config, &waitMode) <ins> ginkgo.DeferCleanup(cleanupStorageClass, config) </ins> for i, node := range config.nodes { ginkgo.By(fmt.Sprintf("Setting up %d local volumes on node %q", volsPerNode, node.Name)) allLocalVolumes[node.Name] = setupLocalVolumes(ctx, config, volType, &config.nodes[i], volsPerNode)

framework.ExpectNoError(err) } } <ins> ginkgo.DeferCleanup(func(ctx context.Context) { ginkgo.By("Clean all PVs") for nodeName, localVolumes := range allLocalVolumes { ginkgo.By(fmt.Sprintf("Cleaning up %d local volumes on node %q", len(localVolumes), nodeName)) cleanupLocalVolumes(ctx, config, localVolumes) } }) </ins> ginkgo.By("Start a goroutine to recycle unbound PVs") backgroundCtx, cancel := context.WithCancel(context.Background()) var wg sync.WaitGroup

}() }) <del> ginkgo.AfterEach(func(ctx context.Context) { ginkgo.By("Clean all PVs") for nodeName, localVolumes := range allLocalVolumes { ginkgo.By(fmt.Sprintf("Cleaning up %d local volumes on node %q", len(localVolumes), nodeName)) cleanupLocalVolumes(ctx, config, localVolumes) } cleanupStorageClass(ctx, config) }) </del> ginkgo.It("should be able to process many pods and reuse local volumes", func(ctx context.Context) { var ( podsLock sync.Mutex