Code change:
What happened: Kops runs from a pod manifest stored on the node, having the annotation. We've experienced the case where the cluster autoscaler turned on a new node and, once the node has been to the cluster, application pods have been admitted before . Once it was the turn of , all system resources were utilized, so the pod has not been regularly admitted. However, since it's a critical pod, the executed the , which has tried to kill () a pod whose container wasn't started yet (it was downloading the Docker image), which has caused to timeout and return the error . The admission of failed with the following error: The pod execution has not been retried, so we've got a node without running and its networking broken. What you expected to happen: The should be eventually admitted. Anything else we need to know?: A minimal log extraction for comprehension: From the "Failed to admit pod" log I have been able to trace back the core code involved: () Logs: "Failed to admit pod [POD] - [ERROR]" () Error message: "preemption: pod [POD] failed to evict [ERROR]" () -() Error message: "timeout waiting to kill pod" Environment: Kubernetes version: Cloud provider: AWS OS: ubuntu 16.04.6 LTS (Xenial Xerus) Kernel: Install tools: kops Network plugin and version: kubenet
/sig node
I think the following change may alleviate the situation you encountered:
Thanks for your reply. To my understanding, when the is called it will return the minimum set of pods required to free up resources, so even if only 1 pod eviction fails it's likely that at the end of the the will not have enough resources anyway to admit the critical pod. In your opinion / experience: be an option introducing retries in the in case the eviction of 1 pod fails? yes, do you have any suggestion of a similar design adopted somewhere else in the ?
Looking at getPodsToPreempt, I am not sure it returns the minimum set of Pods. For , retrying may not help much since the pod to be killed may not respond to the subsequent kill command.
Sorry to reiterate on this, but I double checked the code and - if I'm not missing anything - returns the minimum set of pods. Let's look at logic: pods by QOS class using that the total sum of evictable pods resources is enough to satify the requirements (resources to evict for the critical pod) the list of pods to evict, as the merge of three sublists: : the minimum set of pods to evict assuming we've already evicted all and all : the minimum set of pods to evict assuming we've already evicted all and the required : the minimum set of pods to evict to satify the requirements left Am I missing anything? A possible different approach to solve the issue is to refactor to return all input pods sorted by "should be evicted first". Then iterates over the full sorted list of potential pods to evict until the requirements are not satisfied (skipping to subtract pod resources from requirements if a pod fails on ) and breaking the loop as soon as the requirements are satisfied. What's your take on this? /cc who originally implemented this
w.r.t. approach outlined above, it makes sense.
Whether this evicts the "minimum set" depends on how you define the "minimum set". At the time this was implemented (k8s v1.5.0 IIRC), we didn't have PodPriority, so we used QoS to mirror the kubelet's eviction algorithm. It should probably prioritize low-priority pods for eviction today given the recent changes to the eviction algorithm to incorporate PodPriority. The current approach minimizes the eviction of guaranteed pods, and prioritizes evicting few larger pods over many smaller ones. Since eviction marks the evicted pod as failed, it should suffice to just admit the pod regardless of what is returned by the evictPod function. In future syncPod loops, the kubelet will retry the pod deletion steps that it was stuck on. It would eliminate this class of bugs entirely, which was the purpose of the feature in the first place.
See if the following aligns with your suggestion:
Something like that, yeah. I'm not sure the "Failed to admit pod" log warning is still required, but we can iron those things out during code review.
Doing like this it means we do always admit pods, while we should do it only for critical pods, correct?
Please see this The change is for critical pod(s).

Related discussion:
text: // this is a blocking call and should only return when the pod and its containers are killed. err := c.killPodFunc(pod, status, nil) if err != nil { <del> return fmt.Errorf("preemption: pod %s failed to evict %v", format.Pod(pod), err) </del> <ins> klog.Warningf("preemption: pod %s failed to evict %v", format.Pod(pod), err) // In future syncPod loops, the kubelet will retry the pod deletion steps that it was stuck on. continue </ins> } klog.Infof("preemption: pod %s evicted successfully", format.Pod(pod)) }