Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Volumeattachments are not being removed when a pod is deleted. This happens for some random storage classes based PVC while the remaining pods are having volumeattachments removed properly. What you expected to happen: Kubelet has to remove the volumeattachment which will make the external attacher(v3.0.0) to call ControllerUnpublish How to reproduce it (as minimally and precisely as possible): Install CSI Unity v1.3/v1.4 on k8s v1.19.x, there would be 3 storage classes would be generated during the driver installation. Create pvc and pods using all 3 storage classes. Now delete all pods one by one and one of the pods will have the volume attachment left out without being removed. The ATTACHED status of the volumeattachment stays true. This issue is intermittent and happens with one specific protocol based storageclass. Anything else we need to know?: After deleting the pod, CSI methods Node Unpublish volume and Node Unstage volume have been executed successfully and proper response is returned by the driver. I have extracted the below Controller managar logs from kube-system which says "DetachVolume.Detach succeeded for volume..." but still volumeattachment is left in the cluster. I1211 04:54:08. 1 ] attacherDetacher.DetachVolume started for volume "csi-vol-f-" (UniqueName: "^csi-vol-f--FC-apm0020-sv4501") on node "" I1211 04:54:08. 1 ] Verified volume is safe to detach for volume "csi-vol-f-" (UniqueName: "^csi-vol-f--FC-apm0020-sv4501") on node "" I1211 04:54:08. 1 ] DetachVolume.Detach succeeded for volume "csi-vol-f-" (UniqueName: "^csi-vol-f--FC-apm0020*-sv4501") on node "" Environment: Kubernetes version (use ): v1.1.9.x Cloud provider or hardware configuration: CSI Unity 1.3/1.4 OS (e.g: ):Centos 7.6, RHEL 7.6, 7.8 Kernel (e.g. ): Linux 3.10.0-957.27.2.el7.x8664 SMP Mon Jul 29 17:46:05 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others: /wg storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
Did you get a chance to try on latest k8s version that has this issue fixed
Yes, tested in k8s - v1.20.2 and still, volume attachments are not getting deleted after pod deletion.
On the KCM side, looking at waitForVolumeAttachDetachStatus, couldn't find a scenario where it may incorrectly report success during detach. When this issue happens, it would probably be worth to review the external-attacher
These are the attacher logs for a volume attachment that is not deleted even after its pod deletion. From these logs, it can be seen that the detach operation is never initiated for this volume attachment. One more observation is: when a pod is deleted then the corresponding volume attachment is with a deletion timestamp but here for this volume attachment, deletion timestamp is not . Attacher Logs: I0118 06:26:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:03. 1 ] Attaching "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:03. 1 ] Starting attach operation for "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:03. 1 ] VA finalizer to "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:03. 1 ] NodeID annotation to "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:18. 1 ] Saving attach error to "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:18. 1 ] Saved attach error to "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:18. 1 ] Error processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68": failed to attach: rpc error: code = DeadlineExceeded desc = context deadline exceeded I0118 06:26:18. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:18. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:18. 1 ] Attaching "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:18. 1 ] Starting attach operation for "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:18. 1 ] VA finalizer is already set on "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:18. 1 ] NodeID annotation is already set on "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:25. 1 ] Saving attach error to "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:25. 1 ] Saved attach error to "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:25. 1 ] Error processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68": failed to attach: rpc error: code = Unknown desc = runid=57 Export Volume Failed [{The requested operation has failed because the LUN has already been modified by another request. Please retry. (Error Code:0x6701500)}] I0118 06:26:25. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:25. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:25. 1 ] Attaching "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:25. 1 ] Starting attach operation for "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:25. 1 ] Ignoring VolumeAttachment "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" change I0118 06:26:25. 1 ] VA finalizer is already set on "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:25. 1 ] NodeID annotation is already set on "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:27. 1 ] Attached "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:27. 1 ] Marking as attached "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:27. 1 ] Marked as attached "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:27. 1 ] Fully attached "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:27. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:27. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:27. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:27. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 06:26:27. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:27. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:27. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:26:27. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 06:26:27. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:33:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:33:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:33:03. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 06:33:03. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:43:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:43:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:43:03. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 06:43:03. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:53:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:53:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 06:53:03. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 06:53:03. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 07:03:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 07:03:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 07:03:03. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 07:03:03. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 07:13:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 07:13:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 07:13:03. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 07:13:03. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 07:23:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 07:23:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 07:23:03. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 07:23:03. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 07:33:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 07:33:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 07:33:03. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 07:33:03. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 07:43:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 07:43:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 07:43:03. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 07:43:03. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 07:53:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 07:53:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 07:53:03. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 07:53:03. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 08:03:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 08:03:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 08:03:03. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 08:03:03. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 08:13:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 08:13:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 08:13:03. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 08:13:03. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 08:23:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 08:23:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 08:23:03. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 08:23:03. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 08:33:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 08:33:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 08:33:03. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 08:33:03. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 08:43:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 08:43:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 08:43:03. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 08:43:03. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 08:53:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 08:53:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 08:53:03. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 08:53:03. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 09:03:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 09:03:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 09:03:03. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 09:03:03. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 09:13:11. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 09:13:11. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 09:13:11. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 09:13:11. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 09:23:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 09:23:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 09:23:03. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 09:23:03. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 09:33:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 09:33:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 09:33:03. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 09:33:03. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 09:43:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 09:43:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 09:43:03. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 09:43:03. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 09:53:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 09:53:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 09:53:03. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 09:53:03. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 10:03:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 10:03:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 10:03:03. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 10:03:03. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 10:13:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 10:13:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 10:13:03. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 10:13:03. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 10:23:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 10:23:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 10:23:03. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 10:23:03. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 10:33:03. 1 ] Started VA processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 10:33:03. 1 ] CSIHandler: processing VA "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" I0118 10:33:03. 1 ] "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" is already attached I0118 10:33:03. 1 ] CSIHandler: finished processing "csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68" kubectl describe volumeattachment csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68 Name: csi-192ab83086c9f8a2188d331601dd8882d8be3c12c496b9350793d009d687ae68 Namespace: Labels: &lt;noneAnnotations: lg*9 API Version: Kind: VolumeAttachment Metadata: Creation Timestamp: 2021-01-18T06:25:55Z Finalizers: external-attacher/csi-unity-dellemc-com Managed Fields: API Version: Fields Type: FieldsV1 fieldsV1: f:spec: f:attacher: f:nodeName: f:source: f:persistentVolumeName: Manager: kube-controller-manager Operation: Update Time: 2021-01-18T06:25:55Z API Version: Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:finalizers: .: v:"external-attacher/csi-unity-dellemc-com": f:status: f:attached: f:attachmentMetadata: .: f:arrayId: f:host: f:lun: f:volumeContextId: Manager: csi-attacher Operation: Update Time: 2021-01-18T06:26:27Z Resource Version: 3633 UID: -d73e-4b08-9004- Spec: Attacher: csi- Node Name: lg9 Source: Persistent Volume Name: csivol- Status: Attached: true Attachment Metadata: Array Id: ap*1 Host: lg9 Lun: sv6634 Volume Context Id: csivol--iSCSI-ap*1-sv_6634
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale still relevant
We are still facing this issue. Logs are attached in the above comments.
I found this bug in my cluster, too. And then, I change the of to the higher level, actually the level in the OpenShift Cluster. There is some wierd message as below: With the function in the , when a detach request be sent to controller-manager, it would check the is a real or a in the CSI Plugin. But the check method is too simple to separate all of them, like below:
In this case, we tried a workaround and succeeded to fix this problem perfectly. Change your more longer in the CRD First, there are something we need to know: for a CSI volume is combined with and , or called . is combined with , , and . is generated automatically with the prefix what we defined in the sidecar of CRD . Here is two mistake to hit this bug: has a prefix for the driver name, like . This is really a mistake with , but there is nothing we could do. The length of is acctually 68 bytes. In this case, changing the and is not possible, and changing and could affect other volume in the same external storage. But we still be able to change the length of prefix of to promise total length is more than 68 bytes. Increasing the length is better than decreasing, because the is incremental automatically. (Notice: You should backup all data in the origin volume, and provision new volume by re-creating the to prevent this problem again) Here is part of my CRD , see the argument as below:
Please try this workaround, thanks.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten We are facing the same issue
/remove-lifecycle stale We are facing the same issue
Have you try the workaround above?
Please make sure the length of is more than 68 byte. Here is part of my CRD , see the argument as below:
Sorry I didn't elaborate, we are experiencing the same issue with ebs controller. which does not have the same CRD

Discussion:
return errors.New("missing expected parameter volumeName") } <del> if isAttachmentName(volumeName) { // Detach can also be called with the attach ID as the `volumeName`. This codepath is // hit only when we have migrated an in-tree volume to CSI and the A/D Controller is shut // down, the pod with the volume is deleted, and the A/D Controller starts back up in that // order. attachID = volumeName // Vol ID should be the volume handle, except that is not available here. // It is only used in log messages so in the event that this happens log messages will be // printing out the attachID instead of the volume handle. volID = volumeName } else { // volumeName in format driverName<SEP>volumeHandle generated by plugin.GetVolumeName() parts := strings.Split(volumeName, volNameSep) if len(parts) != 2 { klog.Error(log("detacher.Detach insufficient info encoded in volumeName")) return errors.New("volumeName missing expected data") } driverName := parts[0] volID = parts[1] attachID = getAttachmentName(volID, driverName, string(nodeName)) </del> <ins> // volumeName in format driverName<SEP>volumeHandle generated by plugin.GetVolumeName() parts := strings.Split(volumeName, volNameSep) if len(parts) != 2 { klog.Error(log("detacher.Detach insufficient info encoded in volumeName")) return errors.New("volumeName missing expected data") </ins> } <ins> driverName := parts[0] volID = parts[1] attachID = getAttachmentName(volID, driverName, string(nodeName)) </ins> if err := c.k8s.StorageV1().VolumeAttachments().Delete(context.TODO(), attachID, metav1.DeleteOptions{}); err != nil { if apierrors.IsNotFound(err) { // object deleted or never existed, done

return fmt.Sprintf("csi-%x", result) } <del> // isAttachmentName returns true if the string given is of the form of an Attach ID // and false otherwise func isAttachmentName(unknownString string) bool { // 68 == "csi-" + len(sha256hash) return strings.HasPrefix(unknownString, "csi-") && len(unknownString) == 68 } </del> func makeDeviceMountPath(plugin *csiPlugin, spec *volume.Spec) (string, error) { if spec == nil { return "", errors.New(log("makeDeviceMountPath failed, spec is nil"))