Change:
/kind bug What happened: Sometimes when an deployment creates new pods they get scheduled on a node successfully but the containers fail to start with: What you expected to happen: The scheduler shouldn't put pods on nodes without enough resources. How to reproduce it (as minimally and precisely as possible): I don't know how it got into this situation. Deleting the pods and have the RS recreate them works around this issue. Anything else we need to know?: We're using the CPUManager with static policy which might contribute to this problem. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): coreos /sig scheduling
/sig node (for CPU manager)
do the Kubelet logs seem similar to what's reported in this PR description?
/assign
Hi! :) No, I don't see these 'still running' messages. This is what I've got in the kubelet log (slightly redacted):
Can you share more information on this? Deployment manifest, platform details, frequency of these errors?
Fortunately we have open source the platform deployment: - but in short it's: AWS (c5.4xlarge instances for the workers where this issue is happening) multi master w/ kubeadm Enabled CPUManager and set to static policy Running 5 ASG: Controller, GPU-enabled workers and 3x regular workers (one per AZ) Using cluster autoscaler and HPA We have this problem with some pods pretty much all the time. Since we have pretty varying workloads, the CA often moves pods around and scales up/down which works around the problem by deleting the affected pods. So sometimes all pods are healthy, sometime maybe 15% of the pods have this issue. The biggest issue for us currently is that it's blocking rolling upgrades: Since new pods take forever to start, it blocks our CI. There are several deployments affected. All use HPA and all have resource limits of 2 cpus. One has 5Gi, the other 2Gi memory limits but that shouldn't matter. Here is one deployment manifest (slightly edited for internals) Why does this "Internal PreStartContainer hook" even exists? Shouldn't the scheduler be what knows about available resources and decides about this? Is there some doc that could help me understand how this is suppose to work? I'd imaging if there is some limits the scheduler doesn't know about (like exclusively available cores or something), it would explain the issue but it would be unclear how this can work at all?
I've just realize that only one specific node (which is suppose to be the same as all other nodes) seems to be affected on which all pods that require full CPUs (triggering the static allocation) fail. Going to taint and drain the node now and will decomision it ~tomorrow, so let me know if you want me to pull something from the node first. Unfortunately I don't have long enough log rentention to see what kubelet might have logged before it got into this state.
My kubelet args:
Maybe relevant: Now that I'm trying to drain the node, I realize that there are ~100 argo () workflows objects "mapped" to the node. I don't know the details and how they are related to the node, I only know that I can't drain the node because of the workflows (that's another issue with argo or CRDs in general, see ). Maybe this is somehow blocking the cpus?
Ah, I misunderstood that error. There are ~100 completed pods that were created by argo. Could it by simply that somehow the finished pods don't free their cpu resources somehow?
We have fixed the completed pods related issue before:
would you mind digging into the cpu manager state checkpointed to disk? This will confirm or refute for us that we failed to release CPUs from a terminal container. The state file is located by default in and the content is simply utf-8 JSON. We should be able to see some container-to-cpuset mappings in there. If we can verify that some of the mappings are for terminal container IDs then that's where we should focus the search. My next question would be, were the PreStopContainer and PostStopContainer callbacks invoked, and if so, was there a failed write to the checkpoint file. PreStopContainer is not guaranteed to be called, in case the container process exits on its own. PostStopContainer is called as part of container GC. If there's some other problem interfering with the GC logic then that may have prevented cleanup of our CPU manager state too. Thanks for helping us to debug this remotely!
Ah damn, looks like the cluster autoscaler terminated that node already so unfortunately can't provide the cpumanagerstate.. Would be nice to have the relevant bits exposed as Prometheus metrics. I'll update this issue if this happens again.
"Good" news, problem just re-appeared - this time on two nodes. Here is one: cpumanagerstate: One of the containers doesn't exist at all and another one is exited: Here is the state field from for the exited container: It has "": "koko-intelligence--np2zh", set and according to k8s this pod is still running on that host: I found this in the kubelet log for this container:
Oh and it's indeed running, now in container which is also properly accounted for in the state file. Now regarding the non-existent container, I also see it in the kubelet log: Looks like this belongs to koko-gateway-track--6mljm, which doesn't exist anymore. But most likely because it just got gracefully removed eventually. Maybe some race condition preventing it to remove it and leading to these inconsistencies?
This is awesome info — thanks for updating. I won’t have time to look at this immediately, but perhaps can help trace it. I suspect some kind of race in reconcileState.
Thanks for the info I will take a look at this. /assign
/ Do you have a rough estimate for this? Fully understand if you have other things on your plate but then I'd probably revert the static policy since this becomes a bigger and bigger problem for us (it looks like it got worse over time, now even nodes are affected that were just brought up).
/ Is there some way I can help you debug/fix this? Seems unrelated to anything special in our setup from what I can tell. We had to disable this for all our deployments because it was causing severe issues.
/ Do you have any update/ETA on this?
Note that this is fixed in 1.12, but backports to 1.11 and 1.10 are not merged yet:
This still happens occasionally in 1.12 (with ) . provides cleanup for stalled entries.
Out of curiousity, does anyone still seeing this issue utilize containers in their pod Specs? If so, the curlprit is likely addressed by the following:
The issue is still existing in kubernetes v1.14.4, without init containers to pod spec. Some way below may reproduce it: container crash on a node frequently and then delete the pod. a new pod and schedule to this node. Is this already fixed on master branch?

Discussion:
err = m.updateContainerCPUSet(containerID, cpus) if err != nil { glog.Errorf("[cpumanager] AddContainer error: %v", err) <del> return err </del> <ins> m.Lock() err := m.policy.RemoveContainer(m.state, containerID) if err != nil { glog.Errorf("[cpumanager] AddContainer rollback state error: %v", err) } m.Unlock() </ins> } <del> } else { glog.V(5).Infof("[cpumanager] update container resources is skipped due to cpu set is empty") </del> <ins> return err </ins> } <ins> glog.V(5).Infof("[cpumanager] update container resources is skipped due to cpu set is empty") </ins> return nil }

if err != nil { glog.Errorf("[cpumanager] reconcileState: failed to add container (pod: %s, container: %s, container id: %s, error: %v)", pod.Name, container.Name, containerID, err) failure = append(failure, reconciledContainer{pod.Name, container.Name, containerID}) <ins> continue </ins> } } else { // if DeletionTimestamp is set, pod has already been removed from state

"testing" "time" <del> cadvisorapi "github.com/google/cadvisor/info/v1" </del> "io/ioutil" <ins> "os" cadvisorapi "github.com/google/cadvisor/info/v1" </ins> "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" runtimeapi "k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1alpha2" "k8s.io/kubernetes/pkg/kubelet/cm/cpumanager/state" <ins> "k8s.io/kubernetes/pkg/kubelet/cm/cpumanager/topology" </ins> "k8s.io/kubernetes/pkg/kubelet/cm/cpuset" <del> "os" </del> ) type mockState struct {

} func TestCPUManagerAdd(t *testing.T) { <ins> testPolicy := NewStaticPolicy( &topology.CPUTopology{ NumCPUs:    4, NumSockets: 1, NumCores:   4, CPUDetails: map[int]topology.CPUInfo{ 0: {CoreID: 0, SocketID: 0}, 1: {CoreID: 1, SocketID: 0}, 2: {CoreID: 2, SocketID: 0}, 3: {CoreID: 3, SocketID: 0}, }, }, 0) </ins> testCases := []struct { description string <del> regErr      error </del> updateErr   error <ins> policy      Policy expCPUSet   cpuset.CPUSet </ins> expErr      error }{ { description: "cpu manager add - no error", <del> regErr:      nil, </del> updateErr:   nil, <ins> policy:      testPolicy, expCPUSet:   cpuset.NewCPUSet(3, 4), </ins> expErr:      nil, }, { description: "cpu manager add - policy add container error", <del> regErr:      fmt.Errorf("fake reg error"), </del> updateErr:   nil, <del> expErr:      fmt.Errorf("fake reg error"), </del> <ins> policy: &mockPolicy{ err: fmt.Errorf("fake reg error"), }, expCPUSet: cpuset.NewCPUSet(1, 2, 3, 4), expErr:    fmt.Errorf("fake reg error"), </ins> }, { description: "cpu manager add - container update error", <del> regErr:      nil, </del> updateErr:   fmt.Errorf("fake update error"), <del> expErr:      nil, </del> <ins> policy:      testPolicy, expCPUSet:   cpuset.NewCPUSet(1, 2, 3, 4), expErr:      fmt.Errorf("fake update error"), </ins> }, } for _, testCase := range testCases { mgr := &manager{ <del> policy: &mockPolicy{ err: testCase.regErr, }, </del> <ins> policy: testCase.policy, </ins> state: &mockState{ assignments:   state.ContainerCPUAssignments{}, <del> defaultCPUSet: cpuset.NewCPUSet(), </del> <ins> defaultCPUSet: cpuset.NewCPUSet(1, 2, 3, 4), </ins> }, containerRuntime: mockRuntimeService{ err: testCase.updateErr,

podStatusProvider: mockPodStatusProvider{}, } <del> pod := makePod("1000", "1000") </del> <ins> pod := makePod("2", "2") </ins> container := &pod.Spec.Containers[0] err := mgr.AddContainer(pod, container, "fakeID") if !reflect.DeepEqual(err, testCase.expErr) { t.Errorf("CPU Manager AddContainer() error (%v). expected error: %v but got: %v", testCase.description, testCase.expErr, err) } <ins> if !testCase.expCPUSet.Equals(mgr.state.GetDefaultCPUSet()) { t.Errorf("CPU Manager AddContainer() error (%v). expected cpuset: %v but got: %v", testCase.description, testCase.expCPUSet, mgr.state.GetDefaultCPUSet()) } </ins> } }