Change:
I've been struggling on a problem over the past week, that a few pods always fail to run. When I tried to manually run one of those, I got: I could succesfully run the image on another os with different docker version and different backend filesystem, so I did many tests, to conclude that the problem is neither with btrfs, nor with docker, nor with coreos. I could create another aws autoscaling group, reusing the same versions of those tools, but without kubernetes, and I could sucessfully run that image there as well. I then removed all components from my cloudformation template, and readded them progressively, to notice that pulling this image only fails when the kubelet is running. I then tried to run the kubelet with different options, but the bare minimum is enough for docker to fail on that image. Looking at the traceback: Does kubelet try to query when only and are available? I still don't understand what does kubelet has to do with the btrfs error raised by docker, but the evidence is that everytime I stop kubelet, docker has no problem with the image: This issue is a blocker. Since I cannot get the cluster-side logging to work yet (), I'm trying to have a temporary fluentd pod running for application-side logging at first. Any suggestion?
Could this and be related? If so, think this should be p0. Who feels best equipped to dig on this one? , Adrien Lemaire wrote:
Yes, I believe it's the same issue :)
Maybe not related, but CoreOS moved from btrfs to overlay, see kubernetes issues -- sorry I don't have a CoreOS issue.
is most likely person to be able to know more about this issue.
If it happens again, it could help if someone could grab for all kubelet and dockerd threads (). That would allow us to determine if anyone is currently blocked in the kernel in a devicemapper-related system call.
Which version of kubernetes / kubelet are you running?
Ok I could re-produce the issue on aufs from HEAD: when kubelet is running: After I stopped kubelet process, every runs fine. I am going to mark this as P0, and assign it to myself.
Ok, I think I know the root cause of the issue here. Actually it is indeed caused by Image GC collection () we introduced recently. Current support of gc of image depending on docker's "dangling=true", which gives us a wrong set unused images on a node. Instead it only returns a list of image layers without tag. When a image is pulling, all layers are untagged initially, then tagged later. In this case, some intermediate layers might be removed by kubelet GC, especially with a big image. We fixed the similar issues by adding lock during docker image pull triggered by kubernetes. But there is protection for docker image pull outside kubelet. To prove my hyperthesis, I add some debugging line to Kubelet, then run From below, one can see the same image layer is being removed. I did reopen to re-do Image GC, but didn't treat it as the highest issue in my queue. Now with these issues being reported, I think our current image GC does nothing except introducing issues, and we should remove it or completely disable it: 1) Issues reported like this. And it is very hard, or unnecessary complicated to fix this issue. 2) I think when we decided if a image should be GC, it should deal the entire image stacks, not one or two image layers. Even we might fix all race issues today, but we might break the logic later. Removing one or two intermediate layers is very hard to debug. 3) Since both docker and kubernetes pro all images having tag now. Any images with proper tag wouldn't be removed at all even they are not referred by any container for long time. cc/
is trying to disable garbage collection temporarily.
thank you !
I got this error on CoreOS, current stable (at this date) on DigitalOcean, for a single container that I had never built before -- rebuilding it after changing a single file making up the image, made the problem go away.

Discussion:
type ContainerCommandRunner interface { RunInContainer(containerID string, cmd []string) ([]byte, error) } <del> func GetUnusedImages(client DockerInterface) ([]string, error) { // IMPORTANT: this is _unsafe_ to do while there are active pulls // See https://github.com/docker/docker/issues/8926 for details images, err := client.ListImages(docker.ListImagesOptions{ Filters: map[string][]string{ "dangling": {"true"}, }, }) if err != nil { return nil, err } result := make([]string, len(images)) for ix := range images { result[ix] = images[ix].ID } return result, nil } </del>

if err := kl.GarbageCollectContainers(); err != nil { glog.Errorf("Garbage collect failed: %v", err) } <del> if err := kl.GarbageCollectImages(); err != nil { glog.Errorf("Garbage collect images failed: %v", err) } </del> }, time.Minute*1) } <del> func (kl *Kubelet) getUnusedImages() ([]string, error) { kl.pullLock.Lock() defer kl.pullLock.Unlock() return dockertools.GetUnusedImages(kl.dockerClient) } func (kl *Kubelet) GarbageCollectImages() error { images, err := kl.getUnusedImages() if err != nil { return err } for ix := range images { if err := kl.dockerClient.RemoveImage(images[ix]); err != nil { glog.Errorf("Failed to remove image: %q (%v)", images[ix], err) } } return nil } </del> // TODO: Also enforce a maximum total number of containers. func (kl *Kubelet) GarbageCollectContainers() error { if kl.maxContainerCount == 0 {

fakeDocker.Unlock() } <del> func TestGarbageCollectImages(t *testing.T) { kubelet, fakeDocker := newTestKubelet(t) fakeDocker.Images = []docker.APIImages{ { ID: "foo", }, { ID: "bar", }, } if err := kubelet.GarbageCollectImages(); err != nil { t.Errorf("unexpected error: %v", err) } if len(fakeDocker.RemovedImages) != 2 || !fakeDocker.RemovedImages.Has("foo") || !fakeDocker.RemovedImages.Has("bar") { t.Errorf("unexpected images removed: %v", fakeDocker.RemovedImages) } } </del> func TestParseResolvConf(t *testing.T) { testCases := []struct { data        string