Change:
What happened: A container which had exclusive CPUs assigned to it (and an entry in /var/lib/kubelet/cpumanagerstate specifying those CPUs) was killed. When it came back, it was using the default cpuset, and its entry in /var/lib/kubelet/cpumanagerstate was gone. What you expected to happen: It should have been given exclusive CPUs. How to reproduce it (as minimally and precisely as possible): Create a pod/container with "guaranteed" QoS. Determine the exclusive CPUs allocated to the container. From the host, kill pid1 within the container. ("kill -9" works.) The container will restart within the existing pod, but it will be allocated cpus from the default cpuset (which will now include the CPUs that had been exclusively allocated to the container earlier). Anything else we need to know?: I some instrumentation to kubelet. What seems to be happening is this: 1) The container is killed (or dies). Kubelet then immediately creates a new container and calls PolicyStatic.Allocate() to allocate exclusive CPUs for it. This updates the entry in /var/lib/kubelet/cpumanagerstate (and the in-memory cache) which is mapped by podUID and container name, such that it contains the cpuset for the new container. 2) The InternalContainerLifecycle.PostStopContainer() code runs, which calls i.cpuManager.RemoveContainer(containerID). This uses the containerID of the exited container to look up the podUID/container-name tuple, then deletes the cpuset entry from /var/lib/kubelet/cpumanagerstate (and the in-memory cache). But the entry that was deleted was actually the one for the new container. 3) Kubelet calls the CPU manager's AddContainer() function, which calls m.state.GetCPUSetOrDefault(string(p.UID), c.Name) to get the cpuset based on the podUID and container name of the new container. The cpuset entry was erroneously deleted, so this returns the default cpuset, and the container which should have gotten exclusive CPUs ends up running on the default cpuset. An solution suggested by is to delay calling cpuManager.RemoveContainer() until the whole pod is deleted. This would preserve the CPU allocations. An alternate solution also suggested by is to avoid calling cpuManager.RemoveContainer() from the container lifecycle code, but instead call removeStaleState() in the CpuManager.Allocate() function. Environment: Kubernetes version (use ): 1.18.1 Cloud provider or hardware configuration: x86, SMP OS (e.g: ): CentOS 7 Kernel (e.g. ): 3.10.0-1062.1.2.rt56.1025.el7
/sig sig-node
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
Considering the following change based on discussion over Slack with
Sorry that bother you via this close issue, but I was tracing back to this issue via the PR. As per your statement above, if I understand correctly, this problem only happened when the container inside Pod got killed. but an simple Pod restarted (e.g. delete/re-create via kubectl delete pod command) should not trigger the problem, right? Or any container(inside Pod) or Pod restart with guaranteed Qos will trigger this issue?

Discussion:
} func (m *manager) Allocate(p *v1.Pod, c *v1.Container) error { <ins> // Garbage collect any stranded resources before allocating CPUs. m.removeStaleState() </ins> m.Lock() defer m.Unlock()

} if cstatus.State.Terminated != nil { <del> // Since the container is terminated, we know it is safe to // remove it without any reconciliation. Removing the container // will also remove it from the `containerMap` so that this // container will be skipped next time around the loop. </del> <ins> // The container is terminated but we can't call m.RemoveContainer() // here because it could remove the allocated cpuset for the container // which may be in the process of being restarted.  That would result // in the container losing any exclusively-allocated CPUs that it // was allocated. </ins> _, _, err := m.containerMap.GetContainerRef(containerID) if err == nil { <del> klog.Warningf("[cpumanager] reconcileState: skipping container; already terminated (pod: %s, container id: %s)", pod.Name, containerID) err := m.RemoveContainer(containerID) if err != nil { klog.Errorf("[cpumanager] reconcileState: failed to remove container (pod: %s, container id: %s, error: %v)", pod.Name, containerID, err) failure = append(failure, reconciledContainer{pod.Name, container.Name, containerID}) } </del> <ins> klog.Warningf("[cpumanager] reconcileState: ignoring terminated container (pod: %s, container id: %s)", pod.Name, containerID) </ins> } continue }

state:             state, containerRuntime:  mockRuntimeService{}, containerMap:      containermap.NewContainerMap(), <del> activePods:        func() []*v1.Pod { return nil }, </del> podStatusProvider: mockPodStatusProvider{}, <ins> sourcesReady:      &sourcesReadyStub{}, activePods: func() []*v1.Pod { return []*v1.Pod{testCase.pod} }, </ins> } containers := append(

err: testCase.updateErr, }, containerMap:      containermap.NewContainerMap(), <del> activePods:        func() []*v1.Pod { return nil }, </del> podStatusProvider: mockPodStatusProvider{}, <ins> sourcesReady:      &sourcesReadyStub{}, </ins> } pod := makePod("fakePod", "fakeContainer", "2", "2") container := &pod.Spec.Containers[0] <ins> mgr.activePods = func() []*v1.Pod { return []*v1.Pod{pod} } </ins> err := mgr.Allocate(pod, container) if !reflect.DeepEqual(err, testCase.expAllocateErr) { t.Errorf("CPU Manager Allocate() error (%v). expected error: %v but got: %v",

} func (i *internalContainerLifecycleImpl) PreStopContainer(containerID string) error { <del> if i.cpuManager != nil { return i.cpuManager.RemoveContainer(containerID) } </del> return nil } func (i *internalContainerLifecycleImpl) PostStopContainer(containerID string) error { <del> if i.cpuManager != nil { err := i.cpuManager.RemoveContainer(containerID) if err != nil { return err } } </del> if utilfeature.DefaultFeatureGate.Enabled(kubefeatures.TopologyManager) { err := i.topologyManager.RemoveContainer(containerID) if err != nil {