Change:
What would you like to be : PV controller should not try to delete volumes that are attached. Right now, it tries to delete a volume as soon as PVC is deleted, which is too early. When user deletes a namespace, a pod and PVC is deleted at the same time, causing unnecessary delete attempts when the volume is still attached to a node. VolumeInUseProtection won't help us, as a Pod object is deleted before its volumes are detached. We've seen this in CI both on AWS and on GCE, I assume other clouds behave similarly. Why is this needed: It will reduce pressure on cloud API, as we try to delete the volume quite often (at least few initial retries are fast, until exp. backoff is high enough). On AWS, it would reduce our DescribeVolumes usage. It will reduce event noise / PVC being in Failed state. It will reduce logging noise. A simple implementation could pass Node informer into PV controller and PV controller could check before deleting a volume. The largest change would be actually in the unit test framework. /sig storage what do you think? Any volunteer for implementation? :-)
What if you put it in protection controller instead so we avoid having to make significant changes to PV controller?
Agree. Another idea I was thinking is now that we have volumeAttachment objects we could use those for synchronization between attach/detach controller and pv controller, but actually using is much simpler and works for all plugins, not just csi. As for putting it in the protection controller, I think it is achievable but it will be in a roundabout way. If we use protection controller, the logic in PV controller for when to delete a backing volume will be the same: if && ReclaimPolicy==Delete. I guess protection controller would somehow prevent from getting to Released until the volume is detached? If we use PV controller, the logic will change to be: if && ReclaimPolicy==Delete && PV is not present in any I don't think this is too scary/big of a change compared to changing protection controller. To the point "It will reduce logging noise:" we had a customer who saw these many "deletion failed because volume is still attached" and thought they were a "real" error. Tried to give a technical explanation and they (and some other folks) were confused why pv controller and a/d controller don't communicate with each other...
Protection controller would need to keep the finalizer on until the volume is gone from volumesAttached. The way I see it, PV controller and attach/detach controller have independent concerns. Protection controller is actually coordinating operations between all these other controllers (ie kubelet and pv controller, AD and pv controller, etc)
OK. PV controller still needs to be aware of the finalizer somewhere before it calls . But I see your point about offloading the logic to protection controller.
The protection controller today already controls when PV controller goes into reclaim with the existing finalizer.
Is is it like: finalizer on the PVC prevents PVC from being deleted which controls whether PV goes into Released. Anyway, I can read the protection controller code on my own time... (Since I don't understand it I think that disqualifies me from implementing this? :))
Yes that's the gist of it :-) If you're interested, here's some leisure reading:
Is it possible that a volume may fail or take some time to detach from the host? If it has any chance, it might not a good idea to delay pvc deletion in this case. This will prevent users from using the pvc with the same name to claim a new volume or slow down this process. When using statefulset, pods have stable storage identities. If associated volume has failures or a new volume is required, new pvc can only be created after the old volume is detached.
I think this is not an issue for cloud volumes, which protect against deletion in its control plane, but for other plugins that don't have this kind of protection, would deleting the volume first cause detach to permanently fail?
Would need metrics to prove this, but my suspicion is that implementing this would actually improve pvc deletion time. Currently, deletion fails repeatedly while the volume is still attached and it exponentially backs off. Then the next step after the volume is detached succeeds. With this implemented, deletion should succeed on the first try almost immediately after the volume is detached.
Right now pvc can be deleted immediately if the pod is gone (pvc protection controller). It’s not possible to improve pvc deletion time. I might be wrong. Volume is going to be released and reclaimed (Delete reclaim policy) after pvc is deleted. If we are going to solve this issue in protection controller, pvc deletion should be delayed until volume is detached. My concern is this might slow down or fail pvc deletion if the volume has some issues to be detached from the host. And I think the problem of old volume should not prevent users from requesting a new volume.
For cloud volumes, deletion will be triggered, but will fail repeatedly until volume is detached because the cloud control planes prevent disks from being deleted while attached.
I see. Sorry I don’t know much about cloud volumes.
I agree this problem should be solved but don’t think it is a good idea to delay pvc deletion to delay volume deletion. I remember there are some issues to detach volumes from unresponsive nodes (because we don’t know the node is shut down or only inaccessible from apiserver). I prefer implementing it in pv controller.
IMO, it's only clouds who implement 3rd party attach - because there must be some control over the hosts (real HW, not necessarily VMs) to add / remove devices there. And those typically have a protection. The reason we have PVC protection is that users tend to delete PVCs while they still have running pods. They then get surprised that their PV is deleted and their pod is not happy that underlying storage is gone (e.g. iSCSI starts returning I/O errors). IMO it's a good idea to extend the protection from "being used by a pod (=being mounted)" to "being attached". Because the volume should not be deleted anyway. (On related note, I've seen AWS API taking detach for more than 2 minutes when detaching thousands of volumes at once. Damn API throttling).
PV can wait for its underlying storage to be detached from the host but IMO pvc does not need to. Otherwise a pvc cannot claim (recreate with the same name) a new PV until previous PV is detached from the host.
We encountered the same issue in our production environment. In our env, we may immediately delete PVC once the pod is deletedand the following two workflow will happen: delete event -attach detach controller -(VolumeAttachment)-csi attacher controller -csi plugin -cloud volume detach open api, and it will take about 3~5s normally delete event -pv controller set PV into status -csi provisioner controller -csi plugin -cloud volume delete open api, and it will take less than 3s normally So the operation of deleting volume is very likely to be executed before volume is detached successfully when pod's and pvc's deletion events are triggered at the same time. How did we avoid it? We do check if the volume is fully detached at each deletion operation in csi plugin, and do real deletion after detachment, but which led to many retries for volume deletion operation.
IMO, firstly, pvc&pv protection controllers is used for preventing that pvc&pv resource object themselves are deleted from etcd when they are used or referenced, but don't care the under operation sequence of volume, secondly, it's not a good idea about judging if volume has been detached via , because AD controller updates before triggering real volume detachment action in .
And my idea to fix or improve this issue is that AD controller add new finalizer to PVC after bound PV is attached to one node then removes it after bound PV is detached from all node. This new finalizer can act as the communication medium between AD controller and PV controller.
Will this cause any surprises for users who previously managed to delete PVs from unresponsive nodes and now they can't?
instead of trying to solve this for all volume types (considering that most of the attachable ones are getting migrated to csi), what if we look into solving this just for CSI, and use VolumeAttachment object?
Actually external-attacher already seems to until all VolumeAttachment objects are gone. I guess this doesn't work because external-provisioner ignores it?
External-attacher finalizers are not evaluated in the provisioner. Maybe they could. It introduces dependency between the attacher and provisioner, but I don't see a big issue there. The provisioner could even check if there is a VolumeAttachment object for a PVC before deleting it.
/assign
GitHub didn't allow me to assign the following users: RaunakShah. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is interested in working on this.
Thanks /assign are we okay with either of these?
I would prefer to check for existence, it looks cleaner from API point of view.
Hm that means external-provisioner needs to watch and lookup VolumeAttachment objects?
Checking PV finalizers will work, but basing API on finalizers looks quite odd.
I see that external-provisioner only has a PersistentVolumeClaimLister, but not a PersistentVolumeLister. So we'll need to add a Lister to watch PersistentVolume objects too if we check finalizer on PV's?
For a multi-attach volume, you can have multiple VolumeAttachment objects for the same PV on different nodes. Would external-provisioner be able to properly serialize without races in this case?
I prototyped adding a PersistentVolumeLister and checking for external-attacher Finalizer before calling here - Albeit with fairly primitive testing, it seems that this creates a loop of dependancies between external-attacher and external-provisioner: -provisioner will not call until the Finalizer prefixed with "external-attacher/" is removed. -attacher does not remove its Finalizer until is set on the PV.
I think so. At the time is called, PVC was deleted, so no new VolumeAttachment can be created (unless someone creates it manually). No, call from the provisioner lib gets PV: It gets an older instance of the PV (, from informer), while the provisioner has already a newer instance (, from GET), but that should not hurt - in the worst case, gets a PV with finalizer and returns an error and new will follow after exp. backoff. And the provisioner lib can be fixed easily. Good catch! I totally forgot about this. Waiting for prevents the attacher from races between removing the finalizer and at the same time a new VolumeAttachment being created.
Would appreciate your reviews at Based on the discussions above, i've gone with the volumeattachment approach. If you feel that there's a potential race condition that needs addressing too, please let me know!
Any new updates about this issue?
PR is being reviewed here:
We have a customer running into detach/delete race with the in-tree plugin. The fix is only available for CSI customers. Can we fix this issue for the in-tree volume plugins as well? Customer is not yet ready to move the production workload to CSI. cc:
cc

Discussion:
fi fi <del> KUBE_REPO_REL_ROOT="$(dirname ${BASH_SOURCE:-$0})/.." KUBE_REPO_ROOT="$(readlink -f ${KUBE_REPO_REL_ROOT})" </del> <ins> if [[ "$OSTYPE" == *darwin* ]]; then KUBE_REPO_ROOT="${PWD}" else KUBE_REPO_REL_ROOT="$(dirname ${BASH_SOURCE:-$0})/.." KUBE_REPO_ROOT="$(readlink -f ${KUBE_REPO_REL_ROOT})" fi </ins> KUBE_TARGET="${KUBE_REPO_ROOT}/output/go" mkdir -p "${KUBE_TARGET}"