Change:
W0312 15:24:56. 792 ] found a container with the "k8s" prefix, but too few fields (5): "k8scadvisor.8d424740cadvisor-" These are all brand new pods - they should not be invalid
I've been tracking this issue, this seems to be old, not new containers. Typically happens when a GCE cluster is started (static pods are brought up by the old Kubelet) when the new Kubelet comes up it kills the old pods but we can't GC them because we can't recognize them. The log is from GC. Assigning to myself and I'll fix today.
dup of ?
Thanks closed that in favor of this. , Dawn Chen wrote:
thanks for handling this. Let me know if there's anything I can help. We should GC them as long as the container name starts with k8s.
I saw it today on a brand new cluster. On Mar 12, 2015 12:58 PM, "Yu-Ju Hong" wrote:
that should be expected. The first Kubelet that comes up is the Kubelet we have built-in the Container VM image and it doesn't do the new names yet.
No, it keeps happening, even later. I manually updated kubelet, it keeps popping up On Mar 12, 2015 1:10 PM, "Victor Marmol" wrote:
taking a look
indicates what happened above: 1) When node first bootstrap, it has containers running with old format 2) kubelet is upgrade, and kill old containers 3) GC periodically queries the dead containers with k8s prefix, and found those containers with old format, and throw log message. How to fix? just GC them.
Or simply lower down the log level to VLOG(4) since it should be a short-term thing.
Just sent out with the fix. The issue was container GC. Old Kubelet created containers New Kubelet comes up and kills old containers GC runs, but since the containers don't have a UID we don't remove them (repeat) Verified on a new cluster.

Discussion:
if err != nil { return err } <ins> type unidentifiedContainer struct { // Docker ID. id string // Docker container name name string } unidentifiedContainers := make([]unidentifiedContainer, 0) </ins> uidToIDMap := map[string][]string{} for _, container := range containers { _, uid, name, _ := dockertools.ParseDockerName(container.Names[0]) <ins> if uid == "" && name == "" { unidentifiedContainers = append(unidentifiedContainers, unidentifiedContainer{ id:   container.ID, name: container.Names[0], }) continue } </ins> uidName := string(uid) + "." + name uidToIDMap[uidName] = append(uidToIDMap[uidName], container.ID) } <ins> // Remove all non-running unidentified containers. for _, container := range unidentifiedContainers { data, err := kl.dockerClient.InspectContainer(container.id) if err != nil { return err } if data.State.Running { continue } glog.Infof("Removing unidentified dead container %q with ID %q", container.name, container.id) err = kl.dockerClient.RemoveContainer(docker.RemoveContainerOptions{ID: container.id}) if err != nil { return err } } // Evict dead containers according to our policies. </ins> for _, list := range uidToIDMap { if len(list) <= kl.maxContainerCount { continue

}, expectedRemoved: []string{"1706", "1876"}, }, <ins> // Remove non-running unidentified Kubernetes containers. { containers: []docker.APIContainers{ { // Unidentified Kubernetes container. Names: []string{"/k8s_unidentified"}, ID:    "1876", }, { // Unidentified (non-running) Kubernetes container. Names: []string{"/k8s_unidentified"}, ID:    "2309", }, { // Regular Kubernetes container. Names: []string{"/k8s_POD_foo_new_.deadbeef_42"}, ID:    "3876", }, }, containerDetails: map[string]*docker.Container{ "1876": { State: docker.State{ Running: false, }, ID:      "1876", Created: time.Now(), }, "2309": { State: docker.State{ Running: true, }, ID:      "2309", Created: time.Now(), }, }, expectedRemoved: []string{"1876"}, }, </ins> } for _, test := range tests { kubelet, fakeDocker, _, _ := newTestKubelet(t)