Change:
Kubelet continuously throws lots of errors: I'm starting the cluster this way:
To reproduce this behavior I cleaned up everything related to kubernetes (containers, /var/lib/kubernetes contents, etc), started hyperkube, then started a bunch of containers - everything is good up to this point. Then I delete a pod with (kubectl delete pod frontend), and this happens:
More observations. Here is my default token: Here is error message I get: This path is changing all the time: (Note the subdirectory names are different) Pod deleteion code was recently touched by authentication tokens (whose directories can't be removed) by Any ideas what it could be? I would ignore this problem if it didn't consume 15% of CPU in a very tight orphan removal loop.
Those tokens are just secrets as far as the kubelet is concerned. Not sure how secret volume cleanup differs in hyperkube. cc
We have same problem on nodes with CoreOS alpha 991.0.0 and hyperkube 1.2 a lot of messages like:
same issue here.
Same on Ubuntu 14.04, hyperkube v1.2.3, docker 1.9.1 with aufs backend, kernel 3.13.0-48-generic. I also see pod state transitions taking a long time; they hang in "Pending" and "Terminating" for minutes. Does anyone else see that in conjunction with this issue?
For context, this issue doesn't show up immediately in my hyperkube setup. It turns up after a few days. During that time, I update 6 or 7 deployments simultaneously, several times per day.
Sorry I missed the ping on this. Assigned to self, will try to triage by the end of the week.
cc
got the same problem on centos
I am wondering if this depends on Docker version, and not so much the graph driver.
xref
Same problem here, any ideas?
could you please re-triage this bug. I'm not sure "team/ux" is the right tag for it - it doesn't have anything common with UX. It looks like a problem with some low-level container machinery. And given the number of +1's, maybe raise the priority? Thanks.
What problem does this cause for you?
! The error msg will be a confuse to log/alert monitor infrastructures and really hope it can be fixed.
As I wrote before: I have a 8-core laptop that can work ~ 4 hours on battery (enough for a mid-range flight). When I run Kubernetes cluster via hyperkube, it loads 15% of my CPU resources (more than 1 core!) in a tight loop draining my battery less than in hour.
Assigning (all seem related--secret volume tear down issues with hyperkube) to for investigation.
After setting up a hypekube and digging the logs, I think the following happened and caused the issue. Hyperkube automatically creates a few pods for dns, dashboard, proxy services etc. But there is something wrong when setting up and mounting the volumes. The following shows some logs from one such pod. (1) 21:20:03. ] SyncLoop (ADD, "api"): "kubernetes-dashboard-fnivxkube-system(-29d0-11e6-bafc-)" (2) 21:20:03. ] Mount command: nsenter [--mount=/rootfs/proc/1/ns/mnt -- /bin/mount -t tmpfs tmpfs /var/lib/kubelet/pods/-29d0-11e6-bafc-]  These two steps show that volume is mounted (3) 21:20:07. ] IsLikelyNotMountPoint findmnt output for path /var/lib/kubelet/pods/-29d0-11e6-bafc- /var/lib/kubelet/pods/-29d0-11e6-bafc-  Very strange thing here is when SychPod check the mount path, the last letter "7" of the path is missing. Now the step (4) show that the volume is mounted again (4) 21:20:07. ] Mount command: nsenter [--mount=/rootfs/proc/1/ns/mnt -- /bin/mount -t tmpfs tmpfs /var/lib/kubelet/pods/-29d0-11e6-bafc-] From this point, each time SyncPod check the volume, it mounts again the again which is proved by the findmnt output in the log. After a few hours running, there are over hundreds of mounts. I am working on finding the reason why the last letter of the returned mount point is missing. I manually tested the findmnt command inside of the container and it returned the correct result. There is also an issue for umount. The umount code will first move the mount dir to a temporary dir, but failed with "device or resource busy" error. I also tried to mount tmpfs in the container with command mount -t tmpfs tmpfs userdir Then tried to mv userdir also failed with the same error. Will continue working on solving this issue. Please let me know if you any comments
Excellent debugging Looking forward to seeing what the root cause is for both issues.
Some updates for this bug issue one that path name missing the last character: I tried to run the "findmnt" command directly on terminal, the output is fine. But when executing the command though exec.command.Output(), the target path name's last character always misses. I tried out different findmnt options, it does not work. Only if the mounting path name is relatively short (looks like under 90~100 characters), there is no missing character. Also no matter how long the path is, only the last character is missing. Our current code in function nsenter_mount.IsLikelyNotMountPoint() uses exact string match to detect whether the path is already mounted or not, which is not robust. We can change to substring match instead. umount issue. The tearDown also invokes IsLikelyNotMountPoint(). If it returns error message, GetMountMedium() function will regard the mount medium as default, but instead it should be memory. Then teaddown will invokes tearDownDefault and tries to rename directories and delete them, which in turn results in "Device or resource busy" error. Also it does not handle the case if path is mounted multiple times (this case should not happen in normal situation) I will try to modify the IsLikeLyNotMountPoint() to make it more robust. Please let me know if you have any comments or suggestions. Thanks!
PR and PR both modified the behavior of to fix issues and evidently introduced different issues. So watch out for regressions in any potential fix. CC Also, have you observed the missing character issue in a non-hyperkube environment, or is there something unique about running locally inside a docker container that causes this to happen?
I see isLikelyNotMountPoint() function is implmented in two places: and My understanding is nsenterMounter is only used when kubelet is running inside of container. nscentermount uses findmnt command result to check the mount point. In , it uses a different approach: It checks whether the directory has a different device as parent, then it is a mountpoint I also wrote a small go program to test the "findmnt" command in my local environment, it also truncate the last letter if the mounting path is relatively long. So this "findmnt" behavior is not related to hyperkube environment I think.
Thanks for the clarification
Thanks for debugging this. May I set the milestone, as I think we should fix this before release?
Sure. THanks! Jing , Lucas Käldström wrote:
Please note that findmnt command does not exist on docker host vm for virtualbox/windows setup (i.e. 'boot2docker'). I'm pretty sure this will be the case for Mac too. The errors that I get are: ] Failed findmnt command for path /var/lib/kubelet/pods/-31ba-11e6-bdbe- exit status 1 and these don't seem to get cleaned up
This problem is still there. Right now I'm observing it on my GKE instances (managed VM, Kubernetes 1.3.5): I noticed it after CPU load jumped from typical 15% to crazy 100% on both my nodes: !
Thank you for providing this information. After checking the log offline, the problem should be fixed in v1.3.6. Please upgrade to v1.3.6 and let us know if you have any problem. Thanks!

Discussion:
glog.V(5).Infof("findmnt: directory %s does not exist", file) return true, err } <del> args := []string{"--mount=/rootfs/proc/1/ns/mnt", "--", n.absHostPath("findmnt"), "-o", "target", "--noheadings", "--target", file} </del> <ins> // Add --first-only option: since we are testing for the absense of a mountpoint, it is sufficient to get only // the first of multiple possible mountpoints using --first-only. // Also add fstype output to make sure that the output of target file will give the full path // TODO: Need more refactoring for this function. Track the solution with issue #26996 args := []string{"--mount=/rootfs/proc/1/ns/mnt", "--", n.absHostPath("findmnt"), "-o", "target,fstype", "--noheadings", "--first-only", "--target", file} </ins> glog.V(5).Infof("findmnt command: %v %v", nsenterPath, args) exec := exec.New()

// It's safer to assume that it's not a mount point. return true, nil } <del> strOut := strings.TrimSuffix(string(out), "n") </del> <ins> mountTarget := strings.Split(string(out), " ")[0] mountTarget = strings.TrimSuffix(mountTarget, "n") glog.V(5).Infof("IsLikelyNotMountPoint findmnt output for path %s: %v:", file, mountTarget) </ins> <del> glog.V(5).Infof("IsLikelyNotMountPoint findmnt output for path %s: %v", file, strOut) if strOut == file { </del> <ins> if mountTarget == file { glog.V(5).Infof("IsLikelyNotMountPoint: %s is a mount point", file) </ins> return false, nil } <ins> glog.V(5).Infof("IsLikelyNotMountPoint: %s is not a mount point", file) </ins> return true, nil }