Change:
glog.V(5).Infof("findmnt: directory %s does not exist", file) return true, err } <del> args := []string{"--mount=/rootfs/proc/1/ns/mnt", "--", n.absHostPath("findmnt"), "-o", "target", "--noheadings", "--target", file} </del> <ins> // Add --first-only option: since we are testing for the absense of a mountpoint, it is sufficient to get only // the first of multiple possible mountpoints using --first-only. // Also add fstype output to make sure that the output of target file will give the full path // TODO: Need more refactoring for this function. Track the solution with issue #26996 args := []string{"--mount=/rootfs/proc/1/ns/mnt", "--", n.absHostPath("findmnt"), "-o", "target,fstype", "--noheadings", "--first-only", "--target", file} </ins> glog.V(5).Infof("findmnt command: %v %v", nsenterPath, args) exec := exec.New()

Discussion:
Kubelet (v1.2.4) can consume a lot cycles and when it keeps trying to tear orphaned volumes down:
See also cc
Assigning (all seem related--secret volume tear down issues with hyperkube) to for investigation.
/cc

I'm running Kubernetes v1.2.3 using the single-node instructions here: with Docker v1.10. Everything worked as expected for several hours, but now pods are not being scheduled or removed. I've attempted to update a deployment several times over the last hour. Here is for that deployment: The ContainerCreating pod's last event from is: When I delete a pod, the scheduler does seem to start the new pod almost immediately, and the kubelet logs: The kubelet then begins logging a readiness check failure every 10s: but I can't see any evidence that it actually tries to run the container. Docker doesn't emit any logs. The Docker daemon is responsive, and the image is definitely present. When I run the image using it works fine. Anything I can check? Thanks!
At a glance, looks like some bug in state transitions at the kubelet.
is there any message in the kubelet log related to image pulling? Without further information, it seems like kubelet could be stuck at pulling the images. It'd be helpful to see if there are related information.
Can you run docker ps -a to check if docker failed to create / run containers? Could you please upload related Kubelet log somewhere so that we can access. Based on the initial description, it looks like the image is successfully pulled.
I was just making a wild guess because image pulling is usually where the pod workers get stuck and become unresponsive for a long time :) From the description, the "cluster" was working for a few hours, but the pods were newly created.
and , does not show any attempted containers of the relevant image. Here are the logs of the kubelet. I d the existing pod in the deployment, which was in the state; and a new pod was immediately scheduled. These logs cover those events. The Readiness probes continue at 10s intervals. I sent you the full logs of the kubelet in a Slack pm.
just an update, the kubelet is still in the same state: Some context that may be helpful is this happened after I had been experimenting with deploying an NFS server using the current Kubernetes example, , and . I changed the replication controller at to a deployment, and updated it several times.

I was changing the code inside the container, so some of those attempts may have contained incorrect NFS configuration and/or failed to clean up completely before the container exited. I haven't seen this particular kubelet state before. I have seen slow pod state changes, , but have never seen them halt completely. This state is also different because the kubelet is not logging anything. Previously when I have seen slow pod state changes, it has logged repeatedly about orphaned volumes. Is it plausible that this issue has something to do with the state of the kernel after my experimentation with NFS?
/cc
If this is related to stuck image pulling, what you'll see in the docker daemon's log is something like this: If pulling is stuck, this is all that you'll see. If pulling isn't stuck, then you should see something like the following: And in the kubelet's log, this is the last entry you should expect to see for a pod that is stuck Pending / ContainerCreating if pulling is indeed stuck:
the image is definitely present on the node. I've verified by hand: and also the kubelet doesn't log about creating an image. The logs here: are all it says after a pod is scheduled. Note also that several pods have been in the 'Terminating' state for three days.
can you ssh to the node and curl the kubelet port to get a stack trace? e.g.
Stack trace attached.
goroutine 192 looks
What does it mean?
I think it's trying to mount the NFS directory. You said you had an incorrect configuration at one point?
There are several goroutines in this state
It's safe to assume I had an incorrect configuration at one point, yeah.
Wrong IP address for the NFS server, or ? , Anand Patil wrote:
To clarify, that deployment doesn't consume any NFS persistent volumes, and none of my versions did. It consumes a host volume and runs an NFS server. The pod spec is attached.
So it can't be hanging trying to mount an NFS volume, it's trying to create an NFS volume.
By "incorrect configuration", I meant that my version of this: could have been wrong.
Oh ok. Here's one of several hung pod workers: I don't know that this pod worker is necessarily associated with the nfs server pod. Regardless, it's trying to run And hanging.

You probably need to get some help from when they have time.
OK, I appreciate your attention to this issue, I look forward to hearing from them.
do you have installed on the Docker host?
xref
Hi unfortunately the machine where this issue was observed has been shut down. I saw something similar since then, though, and have a hunch about what was going on. A running pod had an NFS volume source, then the NFS server was shut down. The result was that there were some folders in in which recursive walkers like and would hang as described here: . I can try to create a simple repro case, if that would be helpful.
Assigning (all seem related--secret volume tear down issues with hyperkube) to for investigation.

Kubelet continuously throws lots of errors: I'm starting the cluster this way:
To reproduce this behavior I cleaned up everything related to kubernetes (containers, /var/lib/kubernetes contents, etc), started hyperkube, then started a bunch of containers - everything is good up to this point. Then I delete a pod with (kubectl delete pod frontend), and this happens:
More observations. Here is my default token: Here is error message I get: This path is changing all the time: (Note the subdirectory names are different) Pod deleteion code was recently touched by authentication tokens (whose directories can't be removed) by Any ideas what it could be? I would ignore this problem if it didn't consume 15% of CPU in a very tight orphan removal loop.
Those tokens are just secrets as far as the kubelet is concerned. Not sure how secret volume cleanup differs in hyperkube. cc
We have same problem on nodes with CoreOS alpha 991.0.0 and hyperkube 1.2 a lot of messages like:
same issue here.
Same on Ubuntu 14.04, hyperkube v1.2.3, docker 1.9.1 with aufs backend, kernel 3.13.0-48-generic. I also see pod state transitions taking a long time; they hang in "Pending" and "Terminating" for minutes. Does anyone else see that in conjunction with this issue?
For context, this issue doesn't show up immediately in my hyperkube setup. It turns up after a few days. During that time, I update 6 or 7 deployments simultaneously, several times per day.
Sorry I missed the ping on this. Assigned to self, will try to triage by the end of the week.
cc
got the same problem on centos
I am wondering if this depends on Docker version, and not so much the graph driver.
xref
Same problem here, any ideas?
could you please re-triage this bug. I'm not sure "team/ux" is the right tag for it - it doesn't have anything common with UX. It looks like a problem with some low-level container machinery. And given the number of +1's, maybe raise the priority? Thanks.
What problem does this cause for you?
!

The error msg will be a confuse to log/alert monitor infrastructures and really hope it can be fixed.
As I wrote before: I have a 8-core laptop that can work ~ 4 hours on battery (enough for a mid-range flight). When I run Kubernetes cluster via hyperkube, it loads 15% of my CPU resources (more than 1 core!) in a tight loop draining my battery less than in hour.
Assigning (all seem related--secret volume tear down issues with hyperkube) to for investigation.
After setting up a hypekube and digging the logs, I think the following happened and caused the issue. Hyperkube automatically creates a few pods for dns, dashboard, proxy services etc. But there is something wrong when setting up and mounting the volumes. The following shows some logs from one such pod. (1) 21:20:03. ] SyncLoop (ADD, "api"): "kubernetes-dashboard-fnivxkube-system(-29d0-11e6-bafc-)" (2) 21:20:03. ] Mount command: nsenter [--mount=/rootfs/proc/1/ns/mnt -- /bin/mount -t tmpfs tmpfs /var/lib/kubelet/pods/-29d0-11e6-bafc-]  These two steps show that volume is mounted (3) 21:20:07. ] IsLikelyNotMountPoint findmnt output for path /var/lib/kubelet/pods/-29d0-11e6-bafc- /var/lib/kubelet/pods/-29d0-11e6-bafc-  Very strange thing here is when SychPod check the mount path, the last letter "7" of the path is missing. Now the step (4) show that the volume is mounted again (4) 21:20:07. ] Mount command: nsenter [--mount=/rootfs/proc/1/ns/mnt -- /bin/mount -t tmpfs tmpfs /var/lib/kubelet/pods/-29d0-11e6-bafc-] From this point, each time SyncPod check the volume, it mounts again the again which is proved by the findmnt output in the log. After a few hours running, there are over hundreds of mounts.

I am working on finding the reason why the last letter of the returned mount point is missing. I manually tested the findmnt command inside of the container and it returned the correct result. There is also an issue for umount. The umount code will first move the mount dir to a temporary dir, but failed with "device or resource busy" error. I also tried to mount tmpfs in the container with command mount -t tmpfs tmpfs userdir Then tried to mv userdir also failed with the same error. Will continue working on solving this issue. Please let me know if you any comments
Excellent debugging Looking forward to seeing what the root cause is for both issues.
Some updates for this bug issue one that path name missing the last character: I tried to run the "findmnt" command directly on terminal, the output is fine. But when executing the command though exec.command.Output(), the target path name's last character always misses. I tried out different findmnt options, it does not work. Only if the mounting path name is relatively short (looks like under 90~100 characters), there is no missing character. Also no matter how long the path is, only the last character is missing. Our current code in function nsenter_mount.IsLikelyNotMountPoint() uses exact string match to detect whether the path is already mounted or not, which is not robust. We can change to substring match instead. umount issue. The tearDown also invokes IsLikelyNotMountPoint(). If it returns error message, GetMountMedium() function will regard the mount medium as default, but instead it should be memory. Then teaddown will invokes tearDownDefault and tries to rename directories and delete them, which in turn results in "Device or resource busy" error. Also it does not handle the case if path is mounted multiple times (this case should not happen in normal situation) I will try to modify the IsLikeLyNotMountPoint() to make it more robust. Please let me know if you have any comments or suggestions. Thanks!
PR and PR both modified the behavior of to fix issues and evidently introduced different issues. So watch out for regressions in any potential fix. CC Also, have you observed the missing character issue in a non-hyperkube environment, or is there something unique about running locally inside a docker container that causes this to happen?

I see isLikelyNotMountPoint() function is implmented in two places: and My understanding is nsenterMounter is only used when kubelet is running inside of container. nscentermount uses findmnt command result to check the mount point. In , it uses a different approach: It checks whether the directory has a different device as parent, then it is a mountpoint I also wrote a small go program to test the "findmnt" command in my local environment, it also truncate the last letter if the mounting path is relatively long. So this "findmnt" behavior is not related to hyperkube environment I think.
Thanks for the clarification
Thanks for debugging this. May I set the milestone, as I think we should fix this before release?
Sure. THanks! Jing , Lucas Käldström wrote:
Please note that findmnt command does not exist on docker host vm for virtualbox/windows setup (i.e. 'boot2docker'). I'm pretty sure this will be the case for Mac too. The errors that I get are: ] Failed findmnt command for path /var/lib/kubelet/pods/-31ba-11e6-bdbe- exit status 1 and these don't seem to get cleaned up
This problem is still there. Right now I'm observing it on my GKE instances (managed VM, Kubernetes 1.3.5): I noticed it after CPU load jumped from typical 15% to crazy 100% on both my nodes: !
Thank you for providing this information. After checking the log offline, the problem should be fixed in v1.3.6. Please upgrade to v1.3.6 and let us know if you have any problem. Thanks!