Change:
Hello, I've recently been testing the reliability of storage for k8s v1.25.0. A strange phenomenon occurs occasionally during the test: the volume cannot be detached after the PVC and pod are deleted(the reclaimPolicy of pv is delete. So I first went to check the corresponding and objects and found that they both existed. Then I checked the kube-controllerManager and kubelet logs and found that kubelet logs kept printing: But I found that at the beginning, the volume was unmountDevice succeed, as shown in the following kubelet log: it is so wired ! Then I some log variables to the unmountDetachDevices method: The results show that And I've drawn up a time-series chart of the contents of the log. ! Let's go back to the unmountDetachDevices method. According to the above-mentioned log content, the first operation successfully changes of the corresponding attachedVolume to , but the function in the subsequent operation still uses stale DeviceMountState. that the unmountDevice operation is not performed after the first unmountDevice operation succeeds. multiple unmountDevice operations are allowed, ensure that the correct pvname can be obtained in the method. the volume is detached after the pod and PVC objects are deleted This problem can be reproduced by repeatedly deleting and recreating pods that require PVC in k8s 1.25.0 cluster No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig storage
we can't safely assume that the json is always there.
ping
Thanks for the detailed analysis . I'm still sorting through this, do you think that is involved with this?
Yes. look at this log content: The global mount path is changed in the As a result, the method obtains incorrect pvName(e6104b10fea107e004e3307b0444fd7b775fedd804ab47aa997118b7927c32e0). The unmountDevice operation may be performed multiple times in [1.19 - 1.25] kubernetes versions. This problem does not occur before because can still obtain the correct pvName from the global mount path. However, this problem occurs after is merged. https://user- <br</brAfter the first unmountDevice operation is performed successfully, the file is deleted. So the subsequent unmountDevice operation will perform method because file doesn't exist.
Awesome, thanks. Let me find someone to work on this.
I ll take a look at the issue, since its related to the change I made Note: this change needs the node to drained before upgrading to a version with this change . Please check the release notes
Thanks a lot !
yep, I always reproduce this problem in a newly created k8s 1.25.0 cluster.
Hihas there been any progress on this issue?
/priority important-soon
/triage accepted
Hihas there been any progress?
I also hit this error after pvc and pod deleted on k8s 1.25: and then these two volumes cannot be unmounted on the node forever: I tried to delete pv, it's stuck in terminating state forever
restart the kubelet on the node in problem could mitigate this issue, but we really need to fix this issue ASAP, I could repro this issue easily when running 1000 disk attach/detach test
I think this is again a TOCTOU bug in unmountDetachDevices.
The Unmount operation is , thus the state may not be set as before next reconcile, so it is possible that unmountDevice operation performed multiple times. So the root cause of this issue is that got wrong PV name, since changed global mount path.
I have found a similar issue in our environment, which is based on Kubernetes 1.25.3. If I install a Pod with a PVC, and I force move the Pod with NodeAffinity around the cluster, around 10-20 retries the Pod does not come up: the VolumeAttachment is bound to the old node and is not moved by the system anymore. If this is the same issue as reported here I do not know, but the logs are similar: From the Worker's kubelet where the volumeattachment is bound to: Also I see lots of these: But, if this is the same, I would like to know that the referenced PR was merged into exactly which Kubernetes version, as I have found in the RELEASE NOTES, that it was in 1.24.0-alpha: But we could not reproduce the issue on Kubernetes 1.24.2. How come?
As per the logs, it's the same issue. Kubelet operates in a reconcile loop, and is an async operation. This bug only happened when is operated for multiple times, so it cannot be 100% reproduced. However, if the possibility of reproduction in 1.25 is much higher than the cluster lower than 1.25, we should figure out whether there are some other changes that makes this bug happened more frequently.
I have 1K disk attach/detach to 40 nodes test, if it runs multiple times, it could 100% repro, we could use this patch for testing, if 100 test runs pass, then it could prove the fix really works.
Sure. I mean not every single Mount/Unmount operation will lead to this error, but if we do it enough times, then it could be 100% to reproduce.
May I ask what could be a workaround in production systems or even CI systems on this issue? Our CI is struggling with this and other random issues, to the point we would like to stabilize it by kicking the system in a way to fix things... I was trying a kubelet restart on the Worker where the Mount is stuck... but many cases the kubelet restart itself hangs or does not produce a correct restart. So what could be a workaround if we find out that there is a hung volumeattachment?
restart kubelet, and if restart kubelet hangs, reboot the node.
Volumeattachment failed to detach from old node When statefulset application with pvc scheduled on worker1 during deployment its volumeattachment bounds to worker1 when same application upgraded scheduled on another worker node say worker2 and struck in ContainerCreating state and volumeattachment failed to detach from old node . when pod described: Events: Warning FailedMount 23m (x820 over 6d2h) kubelet Unable to attach or mount volumes: unmounted volumes=[datadir], unattached volumes=[ datadir]: timed out waiting for the condition Warning FailedMount 5m22s (x3069 over 6d3h) kubelet Unable to attach or mount volumes: unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition Environment: NAME="SLES" VERSION="15-SP4" VERSIONID="15.4" PRETTYNAME="SUSE Linux Enterprise Server 15 SP4" ID="sles" IDLIKE="suse" ANSICOLOR="0;32" CPENAME="cpe:/o:suse:sles:15:sp4" DOCUMENTATIONURL=""' Kubernetes version: v1.25.3 kernel: 5.14.21-.24.41-default Storage: Rook: v1.9.12 ceph: 16.2.10 logs of kubelet: Mar 21 11:35:55 kubelet[]: E0321 11:35:55. ] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" Mar 21 11:35:55 kubelet[]: E0321 11:35:55. ] "Error syncing pod, skipping" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" podUID=-1432-473d-af85- Mar 21 11:38:13 kubelet[]: E0321 11:38:13. ] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" Mar 21 11:38:13 kubelet[]: E0321 11:38:13. ] "Error syncing pod, skipping" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" podUID=-1432-473d-af85- Mar 21 11:40:29 kubelet[]: E0321 11:40:29. ] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" Mar 21 11:40:29 kubelet[]: E0321 11:40:29. ] "Error syncing pod, skipping" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" podUID=-1432-473d-af85-
FYI. this issue is fixed in 1.24.14, 1.25.10, 1.26.5, and has been fixed in v1.27.0
Hello , can you have a look at this reconciler run's operationExecutor fails to umount volume due to missing vol- It appears to be Kube v1.22 not .25 to boot is this the same issue?
I think they are the same issue and I have left some comments in
Issue reoccurs in kubelet version

Discussion:
driverName = data[volDataKey.driverName] volID = data[volDataKey.volHandle] } else { <del> klog.Error(log("UnmountDevice failed to load volume data file [%s]: %v", dataDir, err)) // The volume might have been mounted by old CSI volume plugin. Fall back to the old behavior: read PV from API server driverName, volID, err = getDriverAndVolNameFromDeviceMountPath(c.k8s, deviceMountPath) if err != nil { klog.Errorf(log("attacher.UnmountDevice failed to get driver and volume name from device mount path: %v", err)) return err </del> <ins> if errors.Is(err, os.ErrNotExist) { klog.V(4).Info(log("attacher.UnmountDevice skipped because volume data file [%s] does not exist", dataDir)) return nil </ins> } <ins> klog.Errorf(log("attacher.UnmountDevice failed to get driver and volume name from device mount path: %v", err)) return err </ins> } if c.csiClient == nil {

return filepath.Join(plugin.host.GetPluginDir(plugin.GetPluginName()), driver, volSha, globalMountInGlobalPath), nil } <del> func getDriverAndVolNameFromDeviceMountPath(k8s kubernetes.Interface, deviceMountPath string) (string, string, error) { // deviceMountPath structure: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/{pvname}/globalmount dir := filepath.Dir(deviceMountPath) if file := filepath.Base(deviceMountPath); file != globalMountInGlobalPath { return "", "", errors.New(log("getDriverAndVolNameFromDeviceMountPath failed, path did not end in %s", globalMountInGlobalPath)) } // dir is now /var/lib/kubelet/plugins/kubernetes.io/csi/pv/{pvname} pvName := filepath.Base(dir) // Get PV and check for errors pv, err := k8s.CoreV1().PersistentVolumes().Get(context.TODO(), pvName, metav1.GetOptions{}) if err != nil { return "", "", err } if pv == nil || pv.Spec.CSI == nil { return "", "", errors.New(log("getDriverAndVolNameFromDeviceMountPath could not find CSI Persistent Volume Source for pv: %s", pvName)) } // Get VolumeHandle and PluginName from pv csiSource := pv.Spec.CSI if csiSource.Driver == "" { return "", "", errors.New(log("getDriverAndVolNameFromDeviceMountPath failed, driver name empty")) } if csiSource.VolumeHandle == "" { return "", "", errors.New(log("getDriverAndVolNameFromDeviceMountPath failed, VolumeHandle empty")) } return csiSource.Driver, csiSource.VolumeHandle, nil } </del> func verifyAttachmentStatus(attachment *storage.VolumeAttachment, volumeHandle string) (bool, error) { // when we received a deleted event during attachment, fail fast if attachment == nil {

}, // PV agnostic path negative test cases { <del> testName:        "fail: missing json, fail to retrieve driver and volumeID from globalpath", volID:           "project/zone/test-vol1", </del> <ins> testName:        "success: json file doesn't exist, unmount device is skipped", </ins> deviceMountPath: "plugins/csi/" + generateSha("project/zone/test-vol1") + "/globalmount", jsonFile:        "", stageUnstageSet: true, <del> shouldFail:      true, </del> <ins> createPV:        true, </ins> }, { testName:        "fail: invalid json, fail to retrieve driver and volumeID from globalpath",

stageUnstageSet: true, shouldFail:      true, }, <del> // Old style PV based path positive test cases { testName:        "success: json file doesn't exist, old style pv based global path -> use PV", volID:           "project/zone/test-vol1", deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", jsonFile:        "", stageUnstageSet: true, createPV:        true, }, { testName:        "success: invalid json file, old style pv based global path -> use PV", volID:           "project/zone/test-vol1", deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", jsonFile:        `{"driverName"}}`, stageUnstageSet: true, createPV:        true, }, { testName:        "stage_unstage not set, PV based path, unmount device is skipped", deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", jsonFile:        `{"driverName":"test-driver","volumeHandle":"test-vol1"}`, stageUnstageSet: false, }, // Old style PV based path negative test cases { testName:        "fail: json file doesn't exist, old style pv based device path, missing PV", volID:           "project/zone/test-vol1", deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", jsonFile:        "", stageUnstageSet: true, shouldFail:      true, }, { testName:        "fail: no json, no PV.volID, old style pv based global path", volID:           "", deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", jsonFile:        "", shouldFail:      true, }, </del> } for _, tc := range testCases {

// Verify call goes through all the way expectedSet := 0 <del> if !tc.stageUnstageSet { </del> <ins> if !tc.stageUnstageSet || tc.volID == "" { </ins> expectedSet = 1 } staged := cdc.nodeClient.GetNodeStagedVolumes()

} _, ok := staged[tc.volID] <del> if ok && tc.stageUnstageSet { </del> <ins> if ok && tc.stageUnstageSet && tc.volID != "" { </ins> t.Errorf("found unexpected staged volume: %s", tc.volID) } else if !ok && !tc.stageUnstageSet { t.Errorf("could not find expected staged volume: %s", tc.volID)

file, err := os.Open(dataFileName) if err != nil { <del> return nil, errors.New(log("failed to open volume data file [%s]: %v", dataFileName, err)) </del> <ins> return nil, fmt.Errorf("%s: %w", log("failed to open volume data file [%s]", dataFileName), err) </ins> } defer file.Close() data := map[string]string{}