Code change:
/kind bug /sig node When the is set to and a container that was granted exclusive CPUs terminates with a normal exit status, the CPU Manager never receives a call. Kuberuntime should be updated to invoke the container manager's hook on detecting normal termination for these containers. This method is already idempotent since it is currently invoked both from kuberuntime's and (the latter of which is invoked by GC.) How to reproduce it (as minimally and precisely as possible): Start singleton cluster with static CPU manager policy enabled. Start a guaranteed pod with a single container that consumes 1 CPU (to get an exclusive allocation) with restartPolicy never that simply exits . Try to start a similar container that won't fit on the node unless the first container's CPU is reaped. Observe the second container fails with insufficient CPUs, and this problem does not resolve even after an extended period of time. Environment: Kubernetes version (use ): v1.8 beta Cloud provider or hardware configuration**: all OS (e.g. from /etc/os-release): all cc
/assign
Yikes, that's a fun issue. I guess we need to listen for PLEG events?
There's a spot in the kuberuntime sync where we think we can make the call.
well, i knew i would miss something ;-) good catch.
All thanks to workload perf testing
By the way, where is of CPU manager called if is set to or ? invoked by GC?
Hi, Besides, I wonder do we have a mechanism to recover things in state of CPU manager when kubelet restarts? Here I have a trial, seems it doesn't work as expected, that is: a few logs before and after the action of addcontainer and removecontainer in policy, to see the content of state: a pod that declares an exclusive CPU, and saw logs: kubelet the pod, to compare the state with that , I think if recovered, the content of them should be the same. but saw logs: Seems that the content in state did not recovered, this way, the CPU manager would behavior far from what we expected if kubelet restart. Did I miss something important?
is called as a part of internal container lifecycle pre-stop and post-stop hooks (see and ). Regarding Kubelet restarts, it is a known issue and captured in
priority/important-soon
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Well, I understand business in , what I wonder is upper behavor, that is, the behavior of in containerRuntime, When a container with restartStrategy exits, It seems that the runtime try to restart new container directly, without calling things like . So I failed to see the place where remove related hooks called in this case, so here I have a trial: a pod whose container is in restartStrategy of and would exit after a perid exited after a peiod: try to restart it but failed: restart after quite a preiod: So I think the resource is not recycled synchronously, and the remove related hook is called in GC. Am I right? Does it works in the way we expected?
Yes, that's exactly correct. The linked PR with the fix for this issue also helps the situation you described to converge faster.

Related discussion:
text: // check the status of containers. for idx, container := range pod.Spec.Containers { containerStatus := podStatus.FindContainerStatusByName(container.Name) <ins> // Call internal container post-stop lifecycle hook for any non-running container so that any // allocated cpus are released immediately. If the container is restarted, cpus will be re-allocated // to it. if containerStatus != nil && containerStatus.State != kubecontainer.ContainerStateRunning { if err := m.internalLifecycle.PostStopContainer(containerStatus.ID.ID); err != nil { glog.Errorf("internal container post-stop lifecycle hook failed for container %v in pod %v with error %v", container.Name, pod.Name, err) } } </ins> // If container does not exist, or is not running, check whether we // need to restart it. if containerStatus == nil || containerStatus.State != kubecontainer.ContainerStateRunning {