Change:
My deployment mount the nfs volume and use subpath to mount the different path in the container. The mount Setting like this But when I delete the pod, the pod will stack in Terminating status. In the kubelet log, I found the error according to the umount codethe unmount function returns nil the subpath umount successfully and the pod can be deleted just like the description in and the subpath in nfs server mounted with --bind flag No response <details</details<details</details<details</details<details</details<details</details<detailsnfs server system version: NAME="Ubuntu" VERSION="18.04.4 LTS (Bionic Beaver)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 18.04.4 LTS" VERSIONID="18.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" PRIVACYPOLICYURL="" VERSIONCODENAME=bionic UBUNTU_CODENAME=bionic nfs version dpkg -l |grep nfs ii libnfsidmap2:amd64 0.25-5.1 amd64 NFS idmapping library ii nfs-common 1:1.3.4-2.1ubuntu5.2 amd64 NFS support files common to client and server ii nfs-kernel-server 1:1.3.4-2.1ubuntu5.2 amd64 support for NFS kernel server </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
what's the probability for your to reproduce the problem? if possible, you can set the kubelet log level to 4, and feedback the log
Even I am encountering this issue. We have a cronjob that runs on a scheduled basis. When it completes doing its job, Kubelet is trying to unmount the mounted NFS. Kubelet Logs are getting piled up. And then Kubelet is watching more files and making inotify watch count increase. Eventually making Node to get Slowed Down. Once the Inotify watch count Increased to the limit. Kubelet is able to Schedule the Pod/CronJob, But it is forever in Pending State.
I increased our kubelet verbosity up to 4. Unfortunately, I can't easily isolate this job from other jobs so it's hard to get the debug - but here is my best shot using grep with context. <preJul 1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34. 7422 ] "SyncLoop (housekeeping) end" Jul 1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34. 7422 ] "HTTP" verb="GET" URI="/metrics/cadvisor" latency="85.194498ms" userAgent="Prometheus/2.28.1" audit-ID="" srcIP="172.31.32.247:" resp=200 Jul 1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/124aee32274342c597bb75ef7656af76eec64c1d42dfe09489f382841c5242e2/merged Jul 1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-2ac4-44f9-b9e3-/volume-subpaths/depot-pv-smoke/bubblestatus/2 Jul 1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-0964-48fc-a4fb- Jul 1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-0964-48fc-a4fb- Jul 1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-2ac4-44f9-b9e3- Jul 1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-2ac4-44f9-b9e3- Jul 1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-2ac4-44f9-b9e3- Jul 1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/a111cfc33a717069e798806bf7f6c1f1e6b90ba8d3c86a93e753b401607577f0/merged Jul 1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/ffb20b27fed75b16ba288838dc18b24300ad527068d76350bf031014ba8ed8df/merged Jul 1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-2ac4-44f9-b9e3- Jul 1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/db051adec95bca7d895c26bd282bcef0061d7f1f2e1be7d9442b6c9d1e799354/merged Jul 1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-0964-48fc-a4fb-/volume-subpaths/depot-pv-schirato/bubblestatus/2 Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/ffb20b27fed75b16ba288838dc18b24300ad527068d76350bf031014ba8ed8df/merged Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-2ac4-44f9-b9e3- Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/db051adec95bca7d895c26bd282bcef0061d7f1f2e1be7d9442b6c9d1e799354/merged Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-0964-48fc-a4fb-/volume-subpaths/depot-pv-schirato/bubblestatus/2 Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/124aee32274342c597bb75ef7656af76eec64c1d42dfe09489f382841c5242e2/merged Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-0964-48fc-a4fb- Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-2ac4-44f9-b9e3-/volume-subpaths/depot-pv-smoke/bubblestatus/2 Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-2ac4-44f9-b9e3- Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-0964-48fc-a4fb- Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-2ac4-44f9-b9e3- Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-2ac4-44f9-b9e3- Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/a111cfc33a717069e798806bf7f6c1f1e6b90ba8d3c86a93e753b401607577f0/merged Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] "operationExecutor.UnmountVolume started for volume "infra" (UniqueName: "") pod "-a775-406b-b461-" (UID: "-a775-406b-b461-") " Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] Cleaning up subpath mounts for /var/lib/kubelet/pods/-a775-406b-b461-/volume-subpaths/infra-pv-schirato Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] Cleaning up subpath mounts for container bubblestatus Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] Cleaning up subpath mounts for subpath 6 Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] "/var/lib/kubelet/pods/-a775-406b-b461-/volume-subpaths/infra-pv-schirato/bubblestatus/6" is a mountpoint, unmounting Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] Unmounting /var/lib/kubelet/pods/-a775-406b-b461-/volume-subpaths/infra-pv-schirato/bubblestatus/6 Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: E0701 13:52:35. 7422 ] Operation for "{ podName:-a775-406b-b461- nodeName:}" failed. No retries permitted until 2022-07-01 13:53:39. -0400 EDT m=+640. (durationBeforeRetry 1m4s). Error: error cleaning subPath mounts for volume "infra" (UniqueName: "") pod "-a775-406b-b461-" (UID: "-a775-406b-b461-") : error processing /var/lib/kubelet/pods/-a775-406b-b461-/volume-subpaths/infra-pv-schirato/bubblestatus: error cleaning subpath mount /var/lib/kubelet/pods/-a775-406b-b461-/volume-subpaths/infra-pv-schirato/bubblestatus/6: Failed to unmount path /var/lib/kubelet/pods/-a775-406b-b461-/volume-subpaths/infra-pv-schirato/bubblestatus/6 Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-2ac4-44f9-b9e3-/volume-subpaths/depot-pv-smoke/bubblestatus/2 Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-0964-48fc-a4fb- Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-2ac4-44f9-b9e3- Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-0964-48fc-a4fb- Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-2ac4-44f9-b9e3- Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/a111cfc33a717069e798806bf7f6c1f1e6b90ba8d3c86a93e753b401607577f0/merged Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-2ac4-44f9-b9e3- Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/ffb20b27fed75b16ba288838dc18b24300ad527068d76350bf031014ba8ed8df/merged Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-0964-48fc-a4fb-/volume-subpaths/depot-pv-schirato/bubblestatus/2 Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-2ac4-44f9-b9e3- Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/db051adec95bca7d895c26bd282bcef0061d7f1f2e1be7d9442b6c9d1e799354/merged Jul 1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/124aee32274342c597bb75ef7656af76eec64c1d42dfe09489f382841c5242e2/merged Jul 1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-2ac4-44f9-b9e3-/volume-subpaths/depot-pv-smoke/bubblestatus/2 Jul 1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-0964-48fc-a4fb- Jul 1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-2ac4-44f9-b9e3- Jul 1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-0964-48fc-a4fb- Jul 1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-2ac4-44f9-b9e3- Jul 1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-2ac4-44f9-b9e3- Jul 1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/a111cfc33a717069e798806bf7f6c1f1e6b90ba8d3c86a93e753b401607577f0/merged Jul 1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/ffb20b27fed75b16ba288838dc18b24300ad527068d76350bf031014ba8ed8df/merged Jul 1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-2ac4-44f9-b9e3- Jul 1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/db051adec95bca7d895c26bd282bcef0061d7f1f2e1be7d9442b6c9d1e799354/merged Jul 1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/-0964-48fc-a4fb-/volume-subpaths/depot-pv-schirato/bubblestatus/2 Jul 1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36. 7422 ] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/124aee32274342c597bb75ef7656af76eec64c1d42dfe09489f382841c5242e2/merged Jul 1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36. 7422 ] "SyncLoop (housekeeping)" Jul 1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36. 7422 ] "Clean up pod workers for terminated pods" Jul 1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36. 7422 ] "Clean up probes for terminating and terminated pods" Jul 1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36. 7422 ] "Clean up orphaned pod statuses" Jul 1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36. 7422 ] "Clean up orphaned pod directories" Jul 1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36. 7422 ] "Orphaned pod found, but volumes are not cleaned up" podUID=-b0be-42b8-8796- </preLet me know if that doesn't include what you are looking for. We end up with a orphaned mount everytime this cron runs. If we remove the mount manually - kubelet detects it and cleans up fine. I rolled us back from 1.23.8 to 1.22.11 and were we seeing this in that release too. We also see it in 1.21.1. We only see the pods get stuck Terminating if we delete the pod while it is runnning. It gets to a completed state if we leave it alone, but we end up with all these orphaned mounts. We didn't notice we were running out of inotifies until we upgraded to 1.23.6 (also 1.23.8) but it looks like in 1.23 we are leaking faster and that is why we saw it.
if you encountered this error, did the mountpoint still exsit on the path ? if so, there maybe something wrong with the umount func, otherwise it would have some bugs on checkMountpoint. you can check this manually
Yes. That path exists each time it fails and for all the retries. If we manually remove it from the host, kubelet detects it and rectifies the pod that was trying to be cleaned up. Is there something additional we can do here to help with root cause?
yes, the mountpoint is still existsI have to manually umount it and after sometime , kubelet will remove the pod. I notice that the mount function in release-v1.14 is But in release-1.20 is And I try to modify the code in branch release-1.20 to use the function or It works well The difference between and is, will use the mount option but not
I modify the umount function add to the umount command line and log the strace output At the end of the strace infoIt finally return 0but still and I check the mountpoint using the mountpoint is still exists.
That means: if using , the code does , the umount command exits 0, with no error, but the mountpoint still exists you execute command manually again, it can delete the mountpoint successfully if removing --no-canonicalize flag, umonut always successes can you test again that if using --no-canonicalize flag with command, whether the bug still exists?
I've tried to modify the code to add umount flag ``, and the bug staill exists
/cc
I found that if I use the bind mount point in nfs-server as the volume subpath in the deployment, the problem reproduced the subpath in the nfs-server mounted with --bind flag and set the volumes and volumeMounts in the deployment, And the failed log:
This issue is caused by embedded child filesystem: Run these commands as root user: below simulates the behaviour of procfs trick in , but it wrongly mounts source directory "b2" on "/a/test" which is "test" file system to target "/root/mnt2/b2", then simulates the behaviour of () in , this triggers the mounting of source directory "/a/test/b2" which is "test/b2" file system to target "/root/mnt2/b2". So "/root/mnt2/b2" is mounted twice(notice their device numbers are different!), tries to unmount once, then finds "/root/mnt2/b2" is still a mountpoint in . Source code for :
awesome! I have modify the code according to commit and It works well ! mounting the subpath from configmap also works

Discussion:
// Follow the segments one by one using openat() to make // sure the user cannot change already existing directories into symlinks. for _, seg := range segments { <ins> var deviceStat unix.Stat_t </ins> currentPath = filepath.Join(currentPath, seg) if !mount.PathWithinBase(currentPath, base) { return -1, fmt.Errorf("path %s is outside of allowed base %s", currentPath, base) } <ins> // Trigger auto mount if it's an auto-mounted directory, ignore error if not a directory. // Notice the trailing slash is mandatory, see "automount" in openat(2) and open_by_handle_at(2). unix.Fstatat(parentFD, seg+"/", &deviceStat, unix.AT_SYMLINK_NOFOLLOW) </ins> klog.V(5).Infof("Opening path %s", currentPath) childFD, err = syscall.Openat(parentFD, seg, openFDFlags|unix.O_CLOEXEC, 0) if err != nil { return -1, fmt.Errorf("cannot open %s: %s", currentPath, err) } <del> var deviceStat unix.Stat_t </del> err := unix.Fstat(childFD, &deviceStat) if err != nil { return -1, fmt.Errorf("error running fstat on %s with %v", currentPath, err)