Change:
}, } config := convertTestConfig(l.config) <del> framework.InjectHtml(f.ClientSet, config, tests[0].Volume, tests[0].ExpectedContent) </del> var fsGroup *int64 if dInfo.Capabilities[CapFsGroup] { fsGroupVal := int64(1234) fsGroup = &fsGroupVal } <ins> // We set same fsGroup for both pods, because for same volumes (e.g. // local), plugin skips setting fsGroup if volume is already mounted // and we don't have reliable way to detect volumes are unmounted or // not before starting the second pod. framework.InjectHtml(f.ClientSet, config, fsGroup, tests[0].Volume, tests[0].ExpectedContent) </ins> framework.TestVolumeClient(f.ClientSet, config, fsGroup, pattern.FsType, tests) })

Discussion:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: block] [Testpattern: Pre-provisioned PV (default fs)] volumes should be mountable Since when has it been failing: Testgrid link: Reason for failure: Volume of previous pod is not unmounted before the second pod is trying to mount its volume. Anything else we need to know:
/sig storage /assign
The problem is when client is deleting first pod, it is deleted immediately. This is because when pod status phase is , . But pod is set to before its volumes are unmounted and detached. pod object may be deleted from apiserver before its volumes are unmounted, so it may have conflicts in setting fsGroup for second pod `in-tree-client. If we want to wait for the pod to be deleted by kubelet instead of apiserver, we need to make it a long-running pod. Alternatively, we can make the two pods use the same fsgroup . cc What do you think?
cc is this expected? I thought kubelet is supposed to delete the pod object only after the volumes are unmounted. This also impacts the StorageProtection feature, which assumes that if the Pod object is deleted, then the volume has been unmounted. For a storage backend like nfs, it may result in unmount hanging. cc to fix the flakiness, the simplest thing may be to make the two pods use the same fsgroup. I don't see a strong use case for two pods using different fsgroups.
That is only during graceful deletion. If a pod is Succeeded, do we begin unmounting volumes? Or do we wait until the deletion timestamp is set? Would it be good enough to start unmounting when the pod is failed/succeeded? If we want to be 100% sure this isn't a problem, we would need to always do graceful deletion, even when the pod is failed/succeeded.
Yes, we do start unmounting when pod is . The API is setting grace period to 0 in the case of pod success or failed. Maybe that needs to be removed if we want to use Pod deletion as a signal that volumes are unmounted.

To always do graceful deletion even when the pod is failed/succeeded is simple, then we can assume that normally when pod objects are deleted (except forcefully), volumes are already unmounted by kubelet. But , and it helps to improve performance in deleting jobs in batch. For this flaky test only, what I need is to make sure volumes are unmounted before the second pod is going to mount them, but local volumes are not reported in or other fields of node status, I can't make sure of this by checking node status either. Possible workarounds: make the first a long-running pod (too hacky) wait for a while before starting the second pod (short period may be flaky too, long period increases test time) use same fsGroup for both pods (do not test that fsGroup can be changed) I'm going to use same fsGroup for both pods until we have a reliable way to detect volumes are unmounted or not.