Code change:
A pod uses a PVC with . The block volume is backed by a block device on the node (which is created by some CSI storage providers). If the underlying block device corresponding to the volume is accidentally disconnected from the node (due to network, hardware problem, etc...), the user cannot delete the pod. The pod would be stuck in terminating forever since Kubelet refuses to unmount the block volume. This prevents the user from cleaning up the pod and spinning up a new replacement pod thus leading to a long service degradation. For example, if the pod is part of a statefulset, the replacement pod cannot come up due to the old pod being stuck terminating. Kubelet should be able to finish terminating a pod when the corresponding block device is already removed from the node. a CSI storage provider that supports . For example, a statefulset that has volumeClaimTemplates with . For example: <details close<summaryclick to open yaml manifest</summary</detailsa network/hardware problem in which the backing block device is removed from the host unexpectedly. In our example, we delete a Longhorn engine instance manager which leads to the underlying block device being removed from the host. that the pod cannot access the block volume, try to delete the pod. pod is stuck in terminating state forever. Logs from kubelet: <details close<summaryClick to open logs</summary<br</detailsPlease pay attention to the log . Kubelet is refusing to terminate the pod because the underlying block device is removed from the host. If we run on the host, we can see that the backing file of the corresponding loop device is marked as : In our humble opinion, we think that Kubelet should process to remove the loop device corresponding to the deleted block device so that pod can finish terminating as detailed in this PR: We have tested that PR and can confirm that it fixes the issue. We have prepared a PR at : We have tested that PR and can confirm that it fixes the issue. <details</details<detailsRancher RKE2 </details<details</details<detailsRancher RKE2 </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
Ref:
/cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle rotten
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
kubelet log 13:31:57 wanguo-72 kubelet[1131]: E0428 13:31:57. 1131 ] Operation for "{^-54fa-4068-bbad- podName:-9521-4aed-9bfe- nodeName:}" failed. No retries permitted until 2023-04-28 13:33:59. +0800 CST m=+8838. (durationBeforeRetry 2m2s). Error: UnmapVolume.UnmapBlockVolume failed for volume "pvc--a4e4-48a7-9057-" (UniqueName: "^-54fa-4068-bbad-") pod "-9521-4aed-9bfe-" (UID: "-9521-4aed-9bfe-") : blkUtil.DetachFileDevice failed. globalUnmapPath:, podUID: -9521-4aed-9bfe-, bindMount: true: failed to unmap device from map path. mapPath is empty node poweroff restart , pod use block mode volume, and pod stuck in terminating
so it is a new issue, and I encounter it as well with XSKY CSI after node restart. My kubernetes version is 1.25.8
which Kubernetes version is it?
Hi, I also see the issue reported by I'm using v1.25.10. I assume it happens when using containerized kubelet. In my environment, losetup reports weird backing file paths. This can be reproduced as follows. Until this moment, the correct path is visible from the host. But, after terminating the container, I get the wrong results. I consider this behavior causes failing unmap. Any thoughts?

Related discussion:
text: <ins> /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package volumepathhandler import ( "fmt" "testing" ) func pathWithSuffix(suffix string) string { return fmt.Sprintf("%s%s", "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-1d205234-06cd-4fe4-a7ea-0e8f3e2faf5f/dev/e196ebd3-2ab1-4185-bed4-b997ba38d1dc", suffix) } func TestCleanBackingFilePath(t *testing.T) { const defaultPath = "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-1d205234-06cd-4fe4-a7ea-0e8f3e2faf5f/dev/e196ebd3-2ab1-4185-bed4-b997ba38d1dc" testCases := []struct { name          string input         string expectedOuput string }{ { name:          "regular path", input:         defaultPath, expectedOuput: defaultPath, }, { name:          "path is suffixed with whitespaces", input:         fmt.Sprintf("%srtn ", defaultPath), expectedOuput: defaultPath, }, { name:          "path is suffixed with "(deleted)"", input:         pathWithSuffix("(deleted)"), expectedOuput: defaultPath, }, { name:          "path is suffixed with "(deleted)" and whitespaces", input:         pathWithSuffix(" (deleted)t"), expectedOuput: defaultPath, }, } for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { output := cleanBackingFilePath(tc.input) if output != tc.expectedOuput { t.Fatalf("expected %q, got %q", tc.expectedOuput, output) } }) } } </ins>