Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I tried to use virtlet with Kubernetes release 1.6.6, but a pod with a VM in it gets stuck in "ContainerCreating" state. This problem does not happen with Kubernetes release 1.6.3. The log of the responsible kubelet shows a panic when trying to restart the container. I do not know why the container is being restarted, but that should not cause a panic in any case. Here is an excerpt from the kubelet's log: It is easy to see a code path that can cause this panic. The offending line in is this: and there is indeed a way for to be nil at this point. Earlier in that function we see the line and there is no intervening statement that checks whether is ; ends with . What you expected to happen: I expected the virtlet to run, like it does with Kubernetes release 1.6.3. How to reproduce it (as minimally and precisely as possible): That would take some work, I hope the diagnosis above makes this moot. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: "bare metal" (really VMs, but who cares?) OS (e.g. from /etc/os-release): Ubuntu 16.04.2 LTS Kernel (e.g. ): Linux n11rdu1ctrl 4.4.0-34-generic -Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: I installed Kubernetes and virtlet using some local Ansible Others: I am using a local fork of virtlet
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig node
Reiterating the mentions to trigger a notification: <detailsIn response to : </details
, did you do any update action against that Pods?
I upped the kubelet's to 4, and got an interesting clue when I tried the experiment again. Here is an excerpt from the more verbose log. Note the empty container name in the "Killing unwanted container" entry. That telling log entry comes from these lines in which is soon followed by .
I only created the pod, no updates.
Is there a running container with id something like when the panic occurs? A quick fix is to add checking whether containerSpec is nil as you mentioned. But it's weird that gets a container without name.
Did virtlet set correctly? Seems it is "", which caused the problem.
Create a PR () that checking containerSpec to avoid panic. And it seems there's no UT case for , I'll create related PRs for
The actual Docker containers are all gone now. But here is what I got from shortly before opening this issue: I will look into the question asked.

Discussion:
msg, handlerErr := m.runner.Run(kubeContainerID, pod, container, container.Lifecycle.PostStart) if handlerErr != nil { m.recordContainerEvent(pod, container, kubeContainerID.ID, v1.EventTypeWarning, events.FailedPostStartHook, msg) <del> m.killContainer(pod, kubeContainerID, container.Name, "FailedPostStartHook", nil) </del> <ins> if err := m.killContainer(pod, kubeContainerID, container.Name, "FailedPostStartHook", nil); err != nil { glog.Errorf("Failed to kill container %q(id=%q) in pod %q: %v, %v", container.Name, kubeContainerID.String(), format.Pod(pod), ErrPostStartHook, err) } </ins> return msg, ErrPostStartHook } }

func (m *kubeGenericRuntimeManager) killContainer(pod *v1.Pod, containerID kubecontainer.ContainerID, containerName string, reason string, gracePeriodOverride *int64) error { var containerSpec *v1.Container if pod != nil { <del> containerSpec = kubecontainer.GetContainerSpec(pod, containerName) </del> <ins> if containerSpec = kubecontainer.GetContainerSpec(pod, containerName); containerSpec == nil { return fmt.Errorf("failed to get containerSpec %q(id=%q) in pod %q when killing container for reason %q", containerName, containerID.String(), format.Pod(pod), reason) } </ins> } else { // Restore necessary information if one of the specs is nil. restoredPod, restoredContainer, err := m.restoreSpecsFromContainerLabels(containerID)

} pod, containerSpec = restoredPod, restoredContainer } // From this point , pod and container must be non-nil. gracePeriod := int64(minimumGracePeriodInSeconds) switch {

assert.Empty(t, containers) } <ins> // TestKillContainer tests killing the container in a Pod. func TestKillContainer(t *testing.T) { _, _, m, _ := createTestRuntimeManager() tests := []struct { caseName            string pod                 *v1.Pod containerID         kubecontainer.ContainerID containerName       string reason              string gracePeriodOverride int64 succeed             bool }{ { caseName: "Failed to find container in pods, expect to return error", pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{UID: "pod1_id", Name: "pod1", Namespace: "default"}, Spec:       v1.PodSpec{Containers: []v1.Container{{Name: "empty_container"}}}, }, containerID:         kubecontainer.ContainerID{Type: "docker", ID: "not_exist_container_id"}, containerName:       "not_exist_container", reason:              "unknown reason", gracePeriodOverride: 0, succeed:             false, }, } for _, test := range tests { err := m.killContainer(test.pod, test.containerID, test.containerName, test.reason, &test.gracePeriodOverride) if test.succeed != (err == nil) { t.Errorf("%s: expected %v, got %v (%v)", test.caseName, test.succeed, (err == nil), err) } } } </ins> // TestToKubeContainerStatus tests the converting the CRI container status to // the internal type (i.e., toKubeContainerStatus()) for containers in // different states.