Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: BUG What happened: Kubelet is periodically going into an error state and causing errors with our storage layer (ceph, shared filesystem). Upon cleaning out the orphaned pod directory things eventually right themselves. Workaround: What you expected to happen: Kubelet should intelligently deal with orphaned pods. Cleaning a stale directory manually should not be required. How to reproduce it (as minimally and precisely as possible): Using rook-0.7.0 (this isn't a rook problem as far as I can tell but this is how we're reproducing): kubectl create -f rook- kubectl create -f rook- kubectl create -f rook- Mount/write to the shared filesystem and monitor /var/log/messages for the following: kubelet: E0309 16:46:30. 3112 ] Orphaned pod "-219b-11e8-8a2a-" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them. Anything else we need to know?: This looks identical to the following: but for a different plugin. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-07T12:22:21Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-07T11:55:20Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Bare-metal private cloud OS (e.g. from /etc/os-release): Red Hat Enterprise Linux Server release 7.4 (Maipo) Kernel (e.g. ): Linux 4.4.115- SMP Sat Feb 3 20:11:41 EST 2018 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm
/sig storage
I have the same issue, except that I can't even remove that directory:
When we've run into that issue the only recourse is to unfortunately reboot the host. Once it comes up things seem to be in a clean state.
i also experience this issue using kubernetes v1.10.1. Manually deleting the directory solves the problem. But yes, kubelet should intelligently deal with orphaned pods maybe you should try umounting the directory from tmpfs
Same here with minikube : That looks like an internal pod. I guess there is nothing to remove.
We are seeing a similar issue with our own custom flexvolume in a Kubernetes 1.8.9 cluster. Is there any way to resolve this without restarting the host until there is an actual solution?
that did help with removing them, but then new volume mounts failed as the host was stuck in uninterruptable I/O. I had to cold cycle to nodes in the end.
I'm having the same issue with kubernetes 1.8.5, rancher 1.6.13, docker-ce17.03.02. I think the kubelet should be able to acknowledge this problem, which obviously doesn't not happen..
We're also having this issue with Kubernetes 1.9.6, Docker 17.03.1-ce and vSphere Cloud Provider for persistent storage.
Having the same issue with Kubernetes 1.10.2, Docker 18.06.0-ce
Having the same issue with Kubernetes 1.11, Docker 18.06.0-ce and ceph 13.2.1
Having the same issue with Kubernetes v1.9.0 Docker 1.12.6 and rook master
those who affected, do you see anything interesting when you run kubelet with ?
I ran with --v=10. Haven't found too much info though.
I ran also with version 11 and vSphere dynamic storage provider
Many cloudproviders call generic util helper to unmount and delete directory, it would explain why multiple providers show same symptoms. I assume disks are unmounted , but just a directory is missed behind, right? Directory deletion code is following: and it got to be producing some error or info v4 message immediately after is printed. In a working case it should be printing or an error. Only way it can decide not to delete directory is if thinks it is still mounted. Maybe it is containerized kubelet which confuses it , or some in upper directories.
I tried to rerpoduce it by creating/deleting following stateful with single PVC with 50 pods in it all node allocated to a single node, but no luck. Kubelet 1.11.2 , vSphere cloud provider. Pretty much same setup except for kubernetes version was regularly reporting orphaned pods with kubelet 1.10.5
k8s 1.6.4 docker 1.12.6 E0816 16:22:24. ] Orphaned pod "-a12d-11e8-b849-" found, but volume paths are still present on disk.
confirmed here also, had to clean the files manually a reboot did not solve this issue.
I observe lot of those errors logged by kubelet in my clusters , this bugs seems to be releated to pods that were using a custom bash-flexvolume plugin ( which mount cifs volumes ), anyway this is very annoing I about 90% of the kubelet logs are those printed every two seconds.. fixing require a manual operation ( or automate a risky rm -Rf operation based on a log line parser ) but this is a poor-man workaround.. while kubelet could/should/must! handle that problem itself I see discussion around this bug since more than one year.. it's possible that no one consider this to be fixed? if no one want to fix it I suggest to decrease the error-level ( outputting "ERROR" in 90% of my logs lines when you don't consider this as a serious bug is wrong )
Having the same issue:[root /var/log]$ kubectl version Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-04-27T09:22:21Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-04-27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}[root ~]$ calicoctl version -c Client Version: v1.6.3 Build date: 2017-12-20T22:32:36+0000 Git commit: Cluster Version: v2.6.6 Cluster Type: unknownAt least version 1.11.2 doesn't have this issue, we stopped seeing this error after an upgrade.
the mountpoint is not umount while remove orphan pod, and RemoveAllOneFilesystem not be executed. the reason why mountpoint not umounted? maybe volume manager have a error process while umount. maybe user service is using directory of mountpoint while umount. or some other reason? I think we can add the umount process while checking with mountpoint left. PR:
Faced the same problem with datera array and K8s. Rebooting nodes cleared up hung pods.
I am setting the kubelet to talk to kub-apiserver manually on localhost and seeing similar isssue with following lines being repeated over and over. kubelet version: 1.10.3 kube-apiserver verison: 1.10.3 Removing the orphan pod under resolve my issue.
/sig node
I have the same issue with k8s v1.11.1 and Docker 1.13.1 on Ubuntu 16.04. Does not occur k8s v1.11.2?
I have the same issue with k8s v1.11.2 and Docker 1.13.1 on CentOS Linux release 7.2.1511 (Core) . this is log: Oct 12 11:27:29 test-slave-114 kubelet[]: E1012 11:27:29. ] Orphaned pod "-a053-11e8-a528-" found, but volume paths are still present on disk : There were a total of 3 errors similar to this. Turn up verbosity to see them. Oct 12 11:27:31 test-slave-114 kubelet[]: E1012 11:27:31. ] Orphaned pod "-a053-11e8-a528-" found, but volume paths are still present on disk : There were a total of 3 errors similar to this. Turn up verbosity to see them. Oct 12 11:27:33 test-slave-114 kubelet[]: E1012 11:27:33. ] Orphaned pod "-a053-11e8-a528-" found, but volume paths are still present on disk : There were a total of 3 errors similar to this. Turn up verbosity to see them. Oct 12 11:27:35 test-slave-114 kubelet[]: E1012 11:27:35. ] Orphaned pod "-a053-11e8-a528-" found, but volume paths are still present on disk : There were a total of 3 errors similar to this. Turn up verbosity to see them. Oct 12 11:27:37 test-slave-114 kubelet[]: E1012 11:27:37. ] Orphaned pod "-a053-11e8-a528-" found, but volume paths are still present on disk : There were a total of 3 errors similar to this. Turn up verbosity to see them. Oct 12 11:27:39 test-slave-114 kubelet[]: E1012 11:27:39. ] Orphaned pod "-a053-11e8-a528-" found, but volume paths are still present on disk : There were a total of 3 errors similar to this. Turn up verbosity to see them. Oct 12 11:27:41 test-slave-114 kubelet[]: E1012 11:27:41. ] Orphaned pod "-a053-11e8-a528-" found, but volume paths are still present on disk : There were a total of 3 errors similar to this. Turn up verbosity to see them. Oct 12 11:27:43 test-slave-114 kubelet[]: E1012 11:27:43. ] Orphaned pod "-a053-11e8-a528-" found, but volume paths are still present on disk : There were a total of 3 errors similar to this. Turn up verbosity to see them. Oct 12 11:27:45 test-slave-114 kubelet[]: E1012 11:27:45. ] Orphaned pod "-a053-11e8-a528-" found, but volume paths are still present on disk : There were a total of 3 errors similar to this. Turn up verbosity to see them. Oct 12 11:27:48 test-slave-114 kubelet[]: E1012 11:27:48. ] Orphaned pod "-a053-11e8-a528-" found, but volume paths are still present on disk : There were a total of 3 errors similar to this. Turn up verbosity to see them. Oct 12 11:27:49 test-slave-114 kubelet[]: E1012 11:27:49. ] Orphaned pod "-a053-11e8-a528-" found, but volume paths are still present on disk : There were a total of 3 errors similar to this. Turn up verbosity to see them.
remove Orphaned pod, and restart kubelet if needed. the kubelet logical, PR: the script on the host, watch orphaned pod and remove them in runtime.
running the script on the host, watch orphaned pod and remove them in runtime. where is the script?
same problem here Ubuntu 16.04. LTS Docker 17.03.3.-ce using HostPath and iSCSI as volumes Kernel:
is there any update on this, this is causing us alot of grief.
same issue with v1.10
/sig node
See the same with v1.12.2 Docker 18.06.1-ce Red Hat Enterprise Linux Server release 7.5 (Maipo) 3.10.0-862.el7.x86_64 Running AWS ami image
same error with v1.12.3 docker 17.09.1-ce centos 7.4
same error with v1.11.6-gke.0
same error with v1.13.2 docker 18.06.1-ce, build ubuntu 16.0.4 x64
+1 v1.13.2 Docker 18.06.1-ce, build Ubuntu 18.04.1 LTS
k8s version 1.11.2 docker version 18.03.0-ce CentOS Linux release 7.5.1804 storage: portworx still have the issue
same here with v1.13.3 Ubuntu 16.04 Docker 18.06.2-ce
Also seeing this with NFS backing Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/arm64"} nfs-provisioner: Image: CentOS 7.6.1810 4.20.0-1090-ayufan-gd1277c20e10d Server: Docker Engine - Community Engine: Version: 18.09.2 API version: 1.39 (minimum version 1.12) Go version: go1.10.6 Git commit: Built: Sun Feb 10 03:55:10 2019 OS/Arch: linux/arm64 Experimental: false Workaround for me: Mar 2 13:14:30 kube3 kubelet: E0302 13:14:30. ] Orphaned pod "-3c82-11e9-8bf4-" found, but volume subpaths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them. kubectl drain kube3 # save it, just in case mv /var/lib/kubelet/pods/-3c82-11e9-8bf4- /tmp kubectl uncordon kube3
the path in the orphaned pod. orphaned pod directory the script may be useful.
same as above Kubernetes v1.13.3 Docker 18.06.1-ce CentOS Linux 7 (Core)
Same with: Kubernetes v1.13.3 Docker 18.3.1 Ubuntu 16.04.5
remove orphaned pod saved the problem
Everybody reporting that it happens to them too is not really helping. The cause is known, and I've PRed a fix () over a month ago, but nobody seems to review it. I'd appreciate people testing my branch or getting it to the attention of sig-storage/sig-node members.
Thanks for that information - I think for non-developers in Go the PR was just overlooked. At least from my side. Hoping this will be fixed/merged soon.
After deleting the directory I've got another error: I can't prove it but for me the issue is also related to There are also messages like: I found and removed all exited Docker containers with the command: The ID was in four rows in the list of the exited containers. PLEG returned back, but the is still there. Plus one more: It looks like the only way to handle the problem completely is to restart docker or even terminate the node. It's a nightmare in production. At the and I run: And PLEG died again. So I had to clean exited containers once again. This is scaring me. For the history: Up: and finally after it has completely died. Facepalm...
+1 v1.13.2 log info:
it will be cause other problem,like I find if I just 'rm /var/lib/kubelet/pods/{podUid}{volumeID}', the orphaned pod will auto cleaned by kubelet
So the script should be tunned to be more accurate and delete not the entire pods directory but only a volume directory, right? How to get the volume id automatically? Could you help to make this babysitting script better?
I think the script write by may work
Just umount the directory under orphaned pod(if exist), and remove the mountpoint, kubelet "cleanup" will do the pod cleaning process.
We have the same issue happening on: K8s 1.13.2 Glusterfs 3.12.15 endpoints Ubuntu LTS (16.04) Docker 18.09.3 ... not sure how long back we've had it though.
We have the same issue happening on: k8s v1.9.0 cephfs centos 3.10.0-693.21.1.el7.x86_64 docker version: 1.13.1 error message:
k8s v1.13.3 fstab/cifs custom flexvolume plugin Ubuntu 18.04 LTS docker version: 18.06.2-ce This script helps me to find all directories inside /var/lib/kubelet/pods that not mounted, and move them to /tmp. In this case script moved 152 directories left by old pods.
Is there any progress on this issue ?
This is a really bad idea since sometimes the volumes are still mounted and this will wipe the volumes. On Fri, 9 Aug 2019 at 18:11, cpu100 &lt;:
This pr can fix which using flexvolume
on k8s v1.13.5 : Following worked stop kubelet docker the pod id -journalctl -fu kubelet | awk -F'"' '{ print $2}' /var/lib/kubelet/pods -rf &lt;pod id#
it may have worked, but it may also delete files from mounted volumes as sometimes there are still mounted volumes under that folder - not a good idea to rm -rf them
I hit the same issue on Kubernetes 1.15.4.
Issues observed in kubernetes 1.14.3 [root ~]# kubelet --version Kubernetes v1.14.3 var/log/messages logs: Oct 15 14:48:41 cluster2cwe1 start- 2019-10-15 14:48:41. I | embed: rejected connection from "10.200.211.102:" (error "EOF", ServerName "") Oct 15 14:48:41 cluster2cwe1 kubelet: E1015 14:48:41. ] Orphaned pod "-ebfe-11e9-a2b6-" found, but volume paths are still present on disk : There were a total of 5 errors similar to this. Turn up verbosity to see them. Oct 15 14:48:43 cluster2cwe1 kubelet: E1015 14:48:43. ] Orphaned pod "-ebfe-11e9-a2b6-" found, but volume paths are still present on disk : There were a total of 5 errors similar to this. Turn up verbosity to see them. Oct 15 14:48:45 cluster2cwe1 kubelet: E1015 14:48:45. ] Orphaned pod "-ebfe-11e9-a2b6-" found, but volume paths are still present on disk : There were a total of 5 errors similar to this. Turn up verbosity to see them. Oct 15 14:48:47 cluster2cwe1 kubelet: E1015 14:48:47. ] Orphaned pod "-ebfe-11e9-a2b6-" found, but volume paths are still present on disk : There were a total of 5 errors similar to this. Turn up verbosity to see them. Oct 15 14:48:48 cluster2cwe1 kubelet: W1015 14:48:48. ] object-"default"/"fluent-bit-config-eps-eventdb": watch of *v1.ConfigMap ended with: too old resource version: ()
same on kubelet --version Kubernetes v1.16.1 no cloud provider, bare-metal
same on kubelet --version Kubernetes v1.16.0 no cloud provider, bare-metal &lt;preNov 27 16:20:21 localhost kubelet[]: E1127 16:20:21. ] orphaned pod "-0fb8-4cce-8126-" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them. Nov 27 16:20:23 localhost kubelet[]: E1127 16:20:23. ] orphaned pod "-0fb8-4cce-8126-" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them. &lt;/pre
I am having the same issue on Kubernetes v1.13.2 Anyone have a solution? this just randomly started happening last week and I have no idea why
Same on HW deployment with 1.15.5
ran into this issue today after a host started misbehaving, k8s v1.16.1 on baremetal and storage backend.
Same here
same on baremetal rook and Kubernetes v1.15.6
And on 1.15.5 on AKS
Got same issue since 2 days on k8s v1.17.2 docker://19.3.5 . Came right after updating from 1.16.2 . Sadly this "orphaned" pod was an active rook-ceph one and removing it's data crashed my CephFS. ceph version 14.2.4 stable. I'm using csi-cephfsplugin cephcsi:v1.2.1
i use CSI plugin, and had two pods in two separate namespaces, with two separate CephFSs. Both pods were not able to see their cephfs mounts. After manually updating rook-ceph to 1.2.3, ONE of the pods was able to mount it's cephfs with all data. Second one is still unable. I can't see any errors in ceph itself as well.
getting spammed by this on EKS, cloudwatch ain't free
, but K8s is free, right? ;)
same issue on k8s 1.16.3 with CRI-O 1.15.2-1.el7
same issue found on k8s 1.15.5 Docker: 19.03.5 using nfs persistent storage no cloud provider, bare-metal
anyone has sorted this out? My logs are still flooded by "orphan pod, but volume.. mimimi"
just reinstalled everything for kubernetes and it was happy...
reinstalled rook ?
kubelet logs in one of the worker still outputs the orphaned pod xxxxx found error.
The issue seems to be result of NodeUnpublishVolume failures from the custom driver. Upon error, K8S does not cleanup the mount directories of the pod.
What do you think about the above change ?
Got same issue on an new cluster k8s v1.17.2 docker://19.3.6 ceph version 14.2.6 stable; cephcsi:v1.2.2
Related to your code reference, I would like to mention that we are facing the problem while using Rook / Ceph with flex drivers, not CSI drivers.
I do not have much idea about the semantics of the Flex drivers but the behavior seems to be same i.e, exits on unmount error from the driver without executing the cleanup of mounts inside K8S.
I have updated to cover Thanks
We have same issue. Kubernetes version 1.12.1 on bare metal. Not using any persistent volumes. Only emptyDir.
And on 1.15.11
I'm seeing this on 1.18.0 using the cephfs CSI plugin as well as pods using volumes directly in the manifest (e.g. not through a PV/PVC), although not all of my pods use persistent storage of any sort. To that point, not all of the orphaned pod directories in /var/lib/kubelet/pods contain a or directory under volumes; some of them only have a and/or and/or .
did you see this after a node reboot? Are you able to provide kubelet logs? The orphaned pod messages indicate that the volume plugin was unable to cleanup the mount properly.
I've seen this a lot too. We also are using ceph-csi plugins.
how can fix this problem?
We are also getting this problem, appreciate any help May 4 14:39:52 pool-tp0nchqvd-5k2s E0504 18:39:52. 1717 ] orphaned pod "-c42a-4840-a596-" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them. May 4 14:39:52 pool-tp0nchqvd-5k2s syslog E0504 18:39:52. 1717 ] orphaned pod "-c42a-4840-a596-" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
Same problem with + k8s 1.15.5
still see the issue on k8s 1.18.2 with ceph 14.2.6 and two storage classes rook- rook-
I got this issue too...
When a Pod which has used PVC is deleted from kubuernetes cluster we can get a error message repeatedly(Orphaned pod "-...-4b38" found, but volume paths are still present on disk) from kubelet log. this error message is caused by kubelet because it cannot clean the parent path of targetPath when Pod is deleted from cluster (targetPath like: /var/lib/kubelet/pods/-7fb3-11ea-80b3- ). If I remove the parent path of targetPath(/var/lib/kubelet/pods/-7fb3-11ea-80b3-), kubelet stop output this error message. CSI plugin(like: ) can remove the parent path of targetPath in NodeUnpublishVolume method. I think that it is a better solution that fix this issue in kubelet. And other CSI Plugin can aviod this issue first using this solution before fixed kubelet.
Kubelet ends up hogging resources ,nodes become not ready
16.10 still same issue
We just came up with a workaround (cleanup) script - ... about to drop it into ... any thoughts/feedback?
In my very similar but different script I instead only cleanup volume directories, not the whole pod directory. That way I try to let kubelet do its job and not interfere much in its cleanup logic.
Interesting, that is probably a good idea. Do you clean the "volumes" dir or the directories inside "volumes"?
I remove directories inside as long as they are empty. And if not - I exit with status 1 and output the contents. In that case I give myself a chance to fine tune the script to know which directories can be .
could you share your script? :heart:
At the moment it's but it's heavily under . PS: I never could learn to bash-program properly, so may look funny or even be wrong, sorry in that case :-) PPS: some ideas are borrowed from
You really don't want to be doing on the mount dir - you just need If the vol is still mounted there - couldn't unmount for whatever reason - you will wipe out any data that is in the volume. If the rmdir fails, then I suggest your script should fail too.
I guess that is not a case that we considered, every one we found was just left there, there were no mounts, but that is (logically) probably why it was left behind. I guess we better look for mounts too.
I can attest that’s what will happen as I had not considered that when I ran the script. The volumes didn’t unmount, and it did indeed wipe the data. Fortunately I had a backup. Hope at some point we don’t have to manually cleanup stale pods.
Same issue in Kubernetes v1.14.3
Same issue in kubernetes v1.14.8 list pod folder data below:
Same issue in kubernetes v1.14.2 Checked the orphaned pod still holds the configmap under volume-subpaths/ After removing the pod folder manually, the log goes fine.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
kubernetes version 1.16.9 docker version: 18.09.2
same issue kubernetes 1.18
same issue any progress on this issue?
same issue Kubernetes 1.19.4 Rook 1.5.4 Kernel: 3.10.0-1160.11.1.el7.x86_64
For the past 2 to reproduce: run k8s for a while. Ungracefully reboot the server. It will come back up with 1 to a dozen of these, spewing out every second or two: enough to dominate all entries sent from a production cluster to a centralized syslog.
same issue Kubernetes v1.18.14 Docker 19.03.8 Kernel: 3.10.0-1127.13.1.el7.x86_64
Same issue in kubernetes v1.20.2
why this issue is closed?
/reopen
Reopened this issue. &lt;detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. &lt;/details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. &lt;detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. &lt;/details
seems like there is a merge to fix that:
thanks! close it. /close
Closing this issue. &lt;detailsIn response to : thanks! close it. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. &lt;/details
/close
For those of you (like me) still using an older version &amp; facing this problem. You tried but that failed with . You can try to umount the path by Identify the mounted path. I used to get the exact path those paths
A warning to anyone with those older versions still running: avoid invoking in any affected subdirectory. There could be live data still mounted under those pods: and you will lose it, even if for example it's a remote volume on NFS / EFS or the like. Instead, to stop such orphaned-pod warning logs, do an command to move the affected directory out from under .
fyi, the fix in only removes empty directories. This is to address some use cases where the node rebooted and the mounts are gone but the directories remain. It doesn't fix other scenarios that others are mentioning where the directory is still mounted. In those cases, you will need to work with the affected volume plugin owners to figure out what is going wrong that is preventing the volume from getting unmounted.
We had that from below ( a big pile of them i must say ) in our kubelet logs just today, our k8s v1.18.10 seems little outdated but we are stuck at that version for nowE1204 15:46:01. 1556 ] orphaned pod "-7ad7-4bbe-b83c-" found, but volume paths are still present on disk : There were a total of 40 errors similar to this. Turn up verbosity to see them. ` is there any script to find those 40 orphaned pods we have? change verbosity is not possible just now for us
Just leaving this here for further use.. maybe ;)
the same with:
Since we can not solve this problem once and for all. Why are't we report it as an event. Let the user to choose their way to fix this problem in their scene.
FINALLY FOUND THE ISSUE: Output: mount: mount(2) system call failed: File exists. This part of the error is important. So, i spun up a pod to use a pvc to test the allocation of the volume. and then i deleted the pvc. it is in terminating stage. I never gave much thought to it. Then after getting stressed over it for 10hrs. and restarting it after 5hrs helped me to think calm. so i saw that a pod is still running that is using the ebs claim, I deleted the pod and then the pvc. immediately the volume got allocated to redis pod and the redis pods are running.

Discussion:
return mountedVolumes, nil } <del> // podVolumesSubpathsDirExists returns true if the pod volume-subpaths directory for // a given pod exists func (kl *Kubelet) podVolumeSubpathsDirExists(podUID types.UID) (bool, error) { podVolDir := kl.getPodVolumeSubpathsDir(podUID) </del> <ins> // getPodVolumeSubpathListFromDisk returns a list of the volume-subpath paths by reading the // subpath directories for the given pod from the disk. func (kl *Kubelet) getPodVolumeSubpathListFromDisk(podUID types.UID) ([]string, error) { volumes := []string{} podSubpathsDir := kl.getPodVolumeSubpathsDir(podUID) </ins> <del> if pathExists, pathErr := mount.PathExists(podVolDir); pathErr != nil { return true, fmt.Errorf("error checking if path %q exists: %v", podVolDir, pathErr) </del> <ins> if pathExists, pathErr := mount.PathExists(podSubpathsDir); pathErr != nil { return nil, fmt.Errorf("error checking if path %q exists: %v", podSubpathsDir, pathErr) </ins> } else if !pathExists { <del> return false, nil </del> <ins> return volumes, nil </ins> } <del> return true, nil </del> <ins> // Explicitly walks /<volume>/<container name>/<subPathIndex> volumePluginDirs, err := ioutil.ReadDir(podSubpathsDir) if err != nil { klog.Errorf("Could not read directory %s: %v", podSubpathsDir, err) return volumes, err } for _, volumePluginDir := range volumePluginDirs { volumePluginName := volumePluginDir.Name() volumePluginPath := filepath.Join(podSubpathsDir, volumePluginName) containerDirs, err := ioutil.ReadDir(volumePluginPath) if err != nil { return volumes, fmt.Errorf("could not read directory %s: %v", volumePluginPath, err) } for _, containerDir := range containerDirs { containerName := containerDir.Name() containerPath := filepath.Join(volumePluginPath, containerName) // Switch to ReadDirNoStat at the subPathIndex level to prevent issues with stat'ing // mount points that may not be responsive subPaths, err := utilpath.ReadDirNoStat(containerPath) if err != nil { return volumes, fmt.Errorf("could not read directory %s: %v", containerPath, err) } for _, subPathDir := range subPaths { volumes = append(volumes, filepath.Join(containerPath, subPathDir)) } } } return volumes, nil </ins> } // GetRequestedContainersInfo returns container info.

import ( "fmt" <ins> "syscall" </ins> v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/types"

klog.V(3).Infof("Orphaned pod %q found, but volumes are not cleaned up", uid) continue } <del> // If there are still volume directories, do not delete directory </del> <ins> allVolumesCleanedUp := true // If there are still volume directories, attempt to rmdir them </ins> volumePaths, err := kl.getPodVolumePathListFromDisk(uid) if err != nil { orphanVolumeErrors = append(orphanVolumeErrors, fmt.Errorf("orphaned pod %q found, but error %v occurred during reading volume dir from disk", uid, err)) continue } if len(volumePaths) > 0 { <del> orphanVolumeErrors = append(orphanVolumeErrors, fmt.Errorf("orphaned pod %q found, but volume paths are still present on disk", uid)) continue </del> <ins> for _, volumePath := range volumePaths { if err := syscall.Rmdir(volumePath); err != nil { orphanVolumeErrors = append(orphanVolumeErrors, fmt.Errorf("orphaned pod %q found, but failed to rmdir() volume at path %v: %v", uid, volumePath, err)) allVolumesCleanedUp = false } else { klog.Warningf("Cleaned up orphaned volume from pod %q at %s", uid, volumePath) } } </ins> } <del> // If there are any volume-subpaths, do not cleanup directories volumeSubpathExists, err := kl.podVolumeSubpathsDirExists(uid) </del> <ins> // If there are any volume-subpaths, attempt to rmdir them subpathVolumePaths, err := kl.getPodVolumeSubpathListFromDisk(uid) </ins> if err != nil { orphanVolumeErrors = append(orphanVolumeErrors, fmt.Errorf("orphaned pod %q found, but error %v occurred during reading of volume-subpaths dir from disk", uid, err)) continue } <del> if volumeSubpathExists { orphanVolumeErrors = append(orphanVolumeErrors, fmt.Errorf("orphaned pod %q found, but volume subpaths are still present on disk", uid)) </del> <ins> if len(subpathVolumePaths) > 0 { for _, subpathVolumePath := range subpathVolumePaths { if err := syscall.Rmdir(subpathVolumePath); err != nil { orphanVolumeErrors = append(orphanVolumeErrors, fmt.Errorf("orphaned pod %q found, but failed to rmdir() subpath at path %v: %v", uid, subpathVolumePath, err)) allVolumesCleanedUp = false } else { klog.Warningf("Cleaned up orphaned volume subpath from pod %q at %s", uid, subpathVolumePath) } } } if !allVolumesCleanedUp { // Not all volumes were removed, so don't clean up the pod directory yet. It is likely // that there are still mountpoints left which could stall RemoveAllOneFilesystem which // would otherwise be called below. // Errors for all removal operations have already been recorded, so don't add another // one here. </ins> continue }

}, validateFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir("pod1uid") <del> return validateDirExists(filepath.Join(podDir, "volumes/plugin/name")) </del> <ins> return validateDirNotExists(podDir) </ins> }, }, "pod-doesnot-exist-with-subpath": {

}, validateFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir("pod1uid") <del> return validateDirExists(filepath.Join(podDir, "volume-subpaths/volume/container/index")) </del> <ins> return validateDirNotExists(podDir) </ins> }, }, "pod-doesnot-exist-with-subpath-top": {

}, validateFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir("pod1uid") <del> return validateDirExists(filepath.Join(podDir, "volume-subpaths")) </del> <ins> return validateDirNotExists(podDir) }, }, "pod-doesnot-exists-with-populated-volume": { prepareFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir("pod1uid") volumePath := filepath.Join(podDir, "volumes/plugin/name") if err := os.MkdirAll(volumePath, 0750); err != nil { return err } return ioutil.WriteFile(filepath.Join(volumePath, "test.txt"), []byte("test1"), 0640) }, validateFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir("pod1uid") return validateDirExists(filepath.Join(podDir, "volumes/plugin/name")) }, }, "pod-doesnot-exists-with-populated-subpath": { prepareFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir("pod1uid") subPath := filepath.Join(podDir, "volume-subpaths/volume/container/index") if err := os.MkdirAll(subPath, 0750); err != nil { return err } return ioutil.WriteFile(filepath.Join(subPath, "test.txt"), []byte("test1"), 0640) }, validateFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir("pod1uid") return validateDirExists(filepath.Join(podDir, "volume-subpaths/volume/container/index")) </ins> }, }, // TODO: test volume in volume-manager