Change:
The volume snapshot controller tests can result in an empty VolumeSnapshot being taken if NodeUnstageVolume occurs after the VolumeSnapshot is taken. Previously there was a change made in Windows to ensure that NodeUnstageVolume finishes before continuing with the test; however, this needs to also to be included in other distros as well. NodeUnpublish, which removes the bind mount, does not force the caches to flush. Tests - The CI jobs should pass when running with a non-hostpath driver. I'm uncertain on why we've seen an increase in these failures, but running the following tests with a non-hostpath driver appear to flake regularly: [Testpattern: Pre-provisioned Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion [Testpattern: Dynamic Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion [Testpattern: Dynamic Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion In these tests we've seen the following with the AWS EBS CSI Driver. Driver logs on the node: In the driver logs (in the controller pod) I can see:
/sig storage
/triage accepted /priority important-soon all snapshot tests with real CSI drivers are quite flaky, I saw it at least in AWS, Azure and GCE in a downstream CI job

Discussion:
// - a pod was created with a PV that's supposed to have data // // However there's a caching issue that @jinxu97 explained and it's related with the pod & volume <del> // lifecycle in windows, to understand it we first analyze what the volumemanager does: </del> <ins> // lifecycle, to understand it we first analyze what the volumemanager does: </ins> // - when a pod is delete the volumemanager will try to cleanup the volume mounts //   - NodeUnpublishVolume: unbinds the bind mount from the container <del> //     - Linux: the data is flushed to disk </del> <ins> //     - Linux: the bind mount is removed, which does not flush any cache </ins> //     - Windows: we delete a symlink, data's not flushed yet to disk //   - NodeUnstageVolume: unmount the global mount <del> //     - Linux: disk is detached </del> <ins> //     - Linux: disk is unmounted and all caches flushed. </ins> //     - Windows: data is flushed to disk and the disk is detached // // Pod deletion might not guarantee a data flush to disk, however NodeUnstageVolume adds the logic <del> // to flush the data to disk (see #81690 for details). </del> <ins> // to flush the data to disk (see #81690 for details). We need to wait for NodeUnstageVolume, as // NodeUnpublishVolume only removes the bind mount, which doesn't force the caches to flush. // It's possible to create empty snapshots if we don't wait (see #101279 for details). </ins> // // In the following code by checking if the PV is not in the node.Status.VolumesInUse field we // ensure that the volume is not used by the node anymore (an indicator that NodeUnstageVolume has // already finished) <del> if framework.NodeOSDistroIs("windows") { nodeName := pod.Spec.NodeName gomega.Expect(nodeName).NotTo(gomega.BeEmpty(), "pod.Spec.NodeName must not be empty") </del> <ins> nodeName := pod.Spec.NodeName gomega.Expect(nodeName).NotTo(gomega.BeEmpty(), "pod.Spec.NodeName must not be empty") </ins> <del> ginkgo.By(fmt.Sprintf("[init] waiting until the node=%s is not using the volume=%s", nodeName, pv.Name)) success := storageutils.WaitUntil(framework.Poll, f.Timeouts.PVDelete, func() bool { node, err := cs.CoreV1().Nodes().Get(context.TODO(), nodeName, metav1.GetOptions{}) framework.ExpectNoError(err) volumesInUse := node.Status.VolumesInUse framework.Logf("current volumes in use: %+v", volumesInUse) for i := 0; i < len(volumesInUse); i++ { if strings.HasSuffix(string(volumesInUse[i]), pv.Name) { return false } </del> <ins> ginkgo.By(fmt.Sprintf("[init] waiting until the node=%s is not using the volume=%s", nodeName, pv.Name)) success := storageutils.WaitUntil(framework.Poll, f.Timeouts.PVDelete, func() bool { node, err := cs.CoreV1().Nodes().Get(context.TODO(), nodeName, metav1.GetOptions{}) framework.ExpectNoError(err) volumesInUse := node.Status.VolumesInUse framework.Logf("current volumes in use: %+v", volumesInUse) for i := 0; i < len(volumesInUse); i++ { if strings.HasSuffix(string(volumesInUse[i]), pv.Name) { return false </ins> } <del> return true }) framework.ExpectEqual(success, true) } </del> <ins> } return true }) framework.ExpectEqual(success, true) </ins> } cleanup := func() {