Change:
master-informing: gce-cos-master-serial ! 2023-06-08 20:01:55 +0000 UTC { failed [FAILED] PV Create API error: Post "https://35.247.13.135/api/v1/persistentvolumes": context canceled In [DeferCleanup (Each)] at: test/e2e/storage/persistentvolumes- @ 06/09/23 18:05:53.165 } No response_ /sig storage cc
I think we did something wrong in /triage accepted
/assign
do you have a link to a full log? It's not clear where comes from (will look closer on Monday), but as the test is shutting down, not allowing new PVs to be created may be normal.
Never mind, the description has links to failing jobs.
ok - this looks like an actual bug with the code as written. what's happening is the "Stop and wait.." at line 492 is calling of the passed to the "goroutine [that] recycle[s] unbound PVs". Ordinarily that signal triggers the and the goroutine exits. In this case (and it will be flaky) it triggers while the case is running and it causes the on line 533 to error with . I don't know what the desired behavior is - should a error be ignored since we're signaling that it's time to leave? or should we actually wait for the to complete and use a different (timeout??) context for that instead.
one potential (if complex) fix is to pass two contexts down to the goroutine. one, to signal “you should stop when you are next at the select statement” and another , that is passed down to CreatePV that signals “stop immediately”. The would then take a Ginkgo context with a timeout. when the first launches it cancels and then if the timeout expires it cancels there might well be a simpler way to track all this but the very nature of what this goroutine is doing seems to need some sort of two-signal approach. alternatively: we can say that within this goroutine is always safe to ignore. it’s simpler as long as there are no test pollution concerns about forcibly canceling the CreatePV.
Patrick is the most knowledgable person in this area, we should wait for his response
Sounds good. This might be a one off - or they may be a class of flaky tests that have a similar structure and so coming up with, and applying, a common pattern for signaling a background goroutine to exit gracefully could be useful.
I think ignoring the error is fine in this case. There is no reason for it to occur except when the test has already completed. -
sounds good - I suppose the only concern would be that forcibly aborting the might be an issue (e.g. the server's still working on things and we fail to clean up or race with it). but y'al lknow k8s far better than i do so maybe there's reasons to have confidence that it's a nonissue.
If the operation completes on the server side, then the other registered earlier will delete the PV.
last run was green let's see this is stable now, thanks for the help

Discussion:
import ( "context" <ins> "errors" </ins> "fmt" "math/rand" "path/filepath"

continue } pv, err = config.client.CoreV1().PersistentVolumes().Get(backgroundCtx, pv.Name, metav1.GetOptions{}) <del> if apierrors.IsNotFound(err) { </del> <ins> if apierrors.IsNotFound(err) || errors.Is(err, context.Canceled) { </ins> continue } // Delete and create a new PV for same local volume storage

continue } err = config.client.CoreV1().PersistentVolumes().Delete(backgroundCtx, pv.Name, metav1.DeleteOptions{}) <ins> if errors.Is(err, context.Canceled) { continue } </ins> framework.ExpectNoError(err) pvConfig := makeLocalPVConfig(config, localVolume) localVolume.pv, err = e2epv.CreatePV(backgroundCtx, config.client, f.Timeouts, e2epv.MakePersistentVolume(pvConfig)) <ins> if errors.Is(err, context.Canceled) { continue } </ins> framework.ExpectNoError(err) } }