Change:
I'm running Kubernetes v1.2.3 using the single-node instructions here: with Docker v1.10. Everything worked as expected for several hours, but now pods are not being scheduled or removed. I've attempted to update a deployment several times over the last hour. Here is for that deployment: The ContainerCreating pod's last event from is: When I delete a pod, the scheduler does seem to start the new pod almost immediately, and the kubelet logs: The kubelet then begins logging a readiness check failure every 10s: but I can't see any evidence that it actually tries to run the container. Docker doesn't emit any logs. The Docker daemon is responsive, and the image is definitely present. When I run the image using it works fine. Anything I can check? Thanks!
At a glance, looks like some bug in state transitions at the kubelet.
is there any message in the kubelet log related to image pulling? Without further information, it seems like kubelet could be stuck at pulling the images. It'd be helpful to see if there are related information.
Can you run docker ps -a to check if docker failed to create / run containers? Could you please upload related Kubelet log somewhere so that we can access. Based on the initial description, it looks like the image is successfully pulled.
I was just making a wild guess because image pulling is usually where the pod workers get stuck and become unresponsive for a long time :) From the description, the "cluster" was working for a few hours, but the pods were newly created.
and , does not show any attempted containers of the relevant image. Here are the logs of the kubelet. I d the existing pod in the deployment, which was in the state; and a new pod was immediately scheduled. These logs cover those events. The Readiness probes continue at 10s intervals. I sent you the full logs of the kubelet in a Slack pm.
just an update, the kubelet is still in the same state: Some context that may be helpful is this happened after I had been experimenting with deploying an NFS server using the current Kubernetes example, , and . I changed the replication controller at to a deployment, and updated it several times. I was changing the code inside the container, so some of those attempts may have contained incorrect NFS configuration and/or failed to clean up completely before the container exited. I haven't seen this particular kubelet state before. I have seen slow pod state changes, , but have never seen them halt completely. This state is also different because the kubelet is not logging anything. Previously when I have seen slow pod state changes, it has logged repeatedly about orphaned volumes. Is it plausible that this issue has something to do with the state of the kernel after my experimentation with NFS?
/cc
If this is related to stuck image pulling, what you'll see in the docker daemon's log is something like this: If pulling is stuck, this is all that you'll see. If pulling isn't stuck, then you should see something like the following: And in the kubelet's log, this is the last entry you should expect to see for a pod that is stuck Pending / ContainerCreating if pulling is indeed stuck:
the image is definitely present on the node. I've verified by hand: and also the kubelet doesn't log about creating an image. The logs here: are all it says after a pod is scheduled. Note also that several pods have been in the 'Terminating' state for three days.
can you ssh to the node and curl the kubelet port to get a stack trace? e.g.
Stack trace attached.
goroutine 192 looks
What does it mean?
I think it's trying to mount the NFS directory. You said you had an incorrect configuration at one point?
There are several goroutines in this state
It's safe to assume I had an incorrect configuration at one point, yeah.
Wrong IP address for the NFS server, or ? , Anand Patil wrote:
To clarify, that deployment doesn't consume any NFS persistent volumes, and none of my versions did. It consumes a host volume and runs an NFS server. The pod spec is attached.
So it can't be hanging trying to mount an NFS volume, it's trying to create an NFS volume.
By "incorrect configuration", I meant that my version of this: could have been wrong.
Oh ok. Here's one of several hung pod workers: I don't know that this pod worker is necessarily associated with the nfs server pod. Regardless, it's trying to run And hanging. You probably need to get some help from when they have time.
OK, I appreciate your attention to this issue, I look forward to hearing from them.
do you have installed on the Docker host?
xref
Hi unfortunately the machine where this issue was observed has been shut down. I saw something similar since then, though, and have a hunch about what was going on. A running pod had an NFS volume source, then the NFS server was shut down. The result was that there were some folders in in which recursive walkers like and would hang as described here: . I can try to create a simple repro case, if that would be helpful.
Assigning (all seem related--secret volume tear down issues with hyperkube) to for investigation.

Discussion:
glog.V(5).Infof("findmnt: directory %s does not exist", file) return true, err } <del> args := []string{"--mount=/rootfs/proc/1/ns/mnt", "--", n.absHostPath("findmnt"), "-o", "target", "--noheadings", "--target", file} </del> <ins> // Add --first-only option: since we are testing for the absense of a mountpoint, it is sufficient to get only // the first of multiple possible mountpoints using --first-only. // Also add fstype output to make sure that the output of target file will give the full path // TODO: Need more refactoring for this function. Track the solution with issue #26996 args := []string{"--mount=/rootfs/proc/1/ns/mnt", "--", n.absHostPath("findmnt"), "-o", "target,fstype", "--noheadings", "--first-only", "--target", file} </ins> glog.V(5).Infof("findmnt command: %v %v", nsenterPath, args) exec := exec.New()

// It's safer to assume that it's not a mount point. return true, nil } <del> strOut := strings.TrimSuffix(string(out), "n") </del> <ins> mountTarget := strings.Split(string(out), " ")[0] mountTarget = strings.TrimSuffix(mountTarget, "n") glog.V(5).Infof("IsLikelyNotMountPoint findmnt output for path %s: %v:", file, mountTarget) </ins> <del> glog.V(5).Infof("IsLikelyNotMountPoint findmnt output for path %s: %v", file, strOut) if strOut == file { </del> <ins> if mountTarget == file { glog.V(5).Infof("IsLikelyNotMountPoint: %s is a mount point", file) </ins> return false, nil } <ins> glog.V(5).Infof("IsLikelyNotMountPoint: %s is not a mount point", file) </ins> return true, nil }