Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Eviction manager evicts the below static and critical pod when the node observed MemoryPressure. The k8s server version is v1.13.2, and the pod priority is enabled. As far as I can see, the eviction manager does not evict such pods. What you expected to happen: Static pods with are not deleted as well as those with a critical pod annotation. How to reproduce it (as minimally and precisely as possible): pod with system-node-critical priority using by kubelet option memory load to the target node until eviction manager observes memory pressure. critical and static pod is evicted. Anything else we need to know?: When I set critical-pod annotation (and enables ) insted of pod priority, the eviction manager DOES NOT evict the static and critical pod. Environment: Kubernetes version (use ): v1.13.2 Cloud provider or hardware configuration: baremetal
/sig node
from enable priority is , more than I can not understand why it has happened
Right. It seems the expected behavior and the actual behavior are different. will remove next k8s v1.14 (it's written in the document), but I'd like to wait for it until the critical priority can prevent eviction.
I see this as well. Start from the bottom and read up. It ranks ibm-master-proxy-static which is a static pod that has priority system-node-critical. It actually is always ranked first as well.
It can be replicated by causing disk pressure (using ) And then viewing the eviction manager logs. `journalctl -u kubelet | grep eviction There's also the piece that when the static pod is evicted kubelet never ends up starting it back up again.
Enabling ExperimentalCriticalPodAnnotation=true makes it not fail and detect properly
Setting on the static pod also works. It appears that a static pod doesn't get set based on . If this is expected then it should be documented. Although it seems like the priority class name should be honored for static pods.
I've confirmed that the problem occurs when a static pod has set to but does not set accordingly. doesn't see the priority as being set and thus thinks the pod isn't critical which ultimately causes the eviction. I would have expected to translate to .
The code fix is available in master branch right now, which means it will be auto included in the upcoming 1.14 release. For lower releases, you need to wait for a while before cherrypick PRs get approved and merged.
Cherrypick PRs have been created and waiting for approval. 1.13 - 1.12 - 1.11 -
Your patch does not work for me. The case in that you completely replaces static pods with the mirror pods. But static pods contains important annotations such as . For static pods is , for mirror - . As a result, eviction manager supposes that pod is not static.
I doubt it. A mirror pod should be annotated with : Otherwise the PR won't pass the e2e test.
When I get static pod through I see . But when I try to debug eviction manager, I see that pod has annotation
Thanks for the info. Then we are sort of in a dilemma - on one hand annotation is needed to check if it's a static pod, on the other hand we also need the corresponding mirror pod to get the priority value. I will provide a supplemental fix later.
can we please hold on those cherry-picks until the supplemental fix is also available?
sure.
Here is a detailed explanation on why previous PR passed the e2e test: As , the critical static pod failed on IsStaticPod() check (because of mirror pod is tagged as " api") And it's reasonable that it's being marked as "to be deleted": However, as , there is some weird log showing: It's because the pod we passed onto () is a mirror pod, and it's deleted (or probably a dummy delete) by container runtime without any error. To take an analogy, it's sort of like a "controllee" (mirror pod) gets deleted, but "controller" (static pod) is still alive, so it causes data inconsistency, and the above error msg popped up. And in user's perspective, the static pod is "not" evicted, then it "looks" like previous PR works.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Status of cherry pick PRs: 1.13 - - merged 1.12 - - merged 1.11 - - merged
/close As all PRs have been cherrypicked.
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
When will this be released for 1.11.x? We are getting this same issue with and believe it's related: The api log:
Maybe in v1.11.10 (but I didn't see it in v1.11.10-beta.0 yet)
Thanks, I assume this is pretty high priority if it's causing to be evicted?
Yes. Prior to get it in 1.11.x, you can work it around by manually setting the priority value (not only setting priority name) to the control plane pods (apiserver, controller-manager and scheduler).
I think the priority value is correct:
Did you get that from ? If so, that's not sufficient. You should work around by hardcoding it into your static pod yaml file.
I did get it from , will try hardcoding. Thanks!
Were you successful ? I'm running into this same problem (unfortunately on a 1.10 cluster, which doesn't have this patch) and received the error: Looks like it comes from
I see this as well but the actual disk is not even full.

Discussion:
// GetActivePods returns non-terminal pods func (kl *Kubelet) GetActivePods() []*v1.Pod { <del> allPods := kl.podManager.GetPods() </del> <ins> allPods, mirrorPods := kl.podManager.GetPodsAndMirrorPods() mirrorPodSet := make(map[string]*v1.Pod) for _, p := range mirrorPods { mirrorPodSet[kubecontainer.GetPodFullName(p)] = p } for i := range allPods { podFullName := kubecontainer.GetPodFullName(allPods[i]) // replace static pod with mirror pod as some info (e.g. spec.Priority) // is needed to make further decisions (e.g. eviction) if mirrorPod, ok := mirrorPodSet[podFullName]; ok { allPods[i] = mirrorPod } } </ins> activePods := kl.filterOutTerminatedPods(allPods) return activePods }

"runtime_conformance_test.go", "security_context_test.go", "summary_test.go", <ins> "system_node_critical_test.go", </ins> "volume_manager_test.go", ], embed = [":go_default_library"],

<ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package e2e_node import ( "fmt" "os" "time" "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" "k8s.io/apimachinery/pkg/util/uuid" kubeapi "k8s.io/kubernetes/pkg/apis/core" kubeletconfig "k8s.io/kubernetes/pkg/kubelet/apis/config" evictionapi "k8s.io/kubernetes/pkg/kubelet/eviction/api" "k8s.io/kubernetes/test/e2e/framework" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) var _ = framework.KubeDescribe("SystemNodeCriticalPod [Slow] [Serial] [Disruptive] [NodeFeature:SystemNodeCriticalPod]", func() { f := framework.NewDefaultFramework("system-node-critical-pod-test") // this test only manipulates pods in kube-system f.SkipNamespaceCreation = true Context("when create a system-node-critical pod", func() { tempSetCurrentKubeletConfig(f, func(initialConfig *kubeletconfig.KubeletConfiguration) { diskConsumed := resource.MustParse("200Mi") summary := eventuallyGetSummary() availableBytes := *(summary.Node.Fs.AvailableBytes) initialConfig.EvictionHard = map[string]string{string(evictionapi.SignalNodeFsAvailable): fmt.Sprintf("%d", availableBytes-uint64(diskConsumed.Value()))} initialConfig.EvictionMinimumReclaim = map[string]string{} }) // Place the remainder of the test within a context so that the kubelet config is set before and after the test. Context("", func() { var staticPodName, mirrorPodName, podPath string ns := kubeapi.NamespaceSystem BeforeEach(func() { By("create a static system-node-critical pod") staticPodName = "static-disk-hog-" + string(uuid.NewUUID()) mirrorPodName = staticPodName + "-" + framework.TestContext.NodeName podPath = framework.TestContext.KubeletConfig.StaticPodPath // define a static pod consuming disk gradually // the upper limit is 1024 (iterations) * 10485760 bytes (10MB) = 10GB err := createStaticSystemNodeCriticalPod( podPath, staticPodName, ns, busyboxImage, v1.RestartPolicyNever, 1024, "dd if=/dev/urandom of=file${i} bs=10485760 count=1 2>/dev/null; sleep .1;", ) Expect(err).ShouldNot(HaveOccurred()) By("wait for the mirror pod to be running") Eventually(func() error { return checkMirrorPodRunning(f.ClientSet, mirrorPodName, ns) }, time.Minute, time.Second*2).Should(BeNil()) }) It("should not be evicted upon DiskPressure", func() { By("wait for the node to have DiskPressure condition") Eventually(func() error { if hasNodeCondition(f, v1.NodeDiskPressure) { return nil } msg := fmt.Sprintf("NodeCondition: %s not encountered yet", v1.NodeDiskPressure) framework.Logf(msg) return fmt.Errorf(msg) }, time.Minute*2, time.Second*4).Should(BeNil()) By("check if it's running all the time") Consistently(func() error { err := checkMirrorPodRunning(f.ClientSet, mirrorPodName, ns) if err == nil { framework.Logf("mirror pod %q is running", mirrorPodName) } else { framework.Logf(err.Error()) } return err }, time.Minute*8, time.Second*4).ShouldNot(HaveOccurred()) }) AfterEach(func() { By("delete the static pod") err := deleteStaticPod(podPath, staticPodName, ns) Expect(err).ShouldNot(HaveOccurred()) By("wait for the mirror pod to disappear") Eventually(func() error { return checkMirrorPodDisappear(f.ClientSet, mirrorPodName, ns) }, time.Minute, time.Second*2).Should(BeNil()) }) }) }) }) func createStaticSystemNodeCriticalPod(dir, name, namespace, image string, restart v1.RestartPolicy, iterations int, command string) error { template := ` apiVersion: v1 kind: Pod metadata: name: %s namespace: %s spec: priorityClassName: system-node-critical containers: - name: %s image: %s restartPolicy: %s command: ["sh", "-c", "i=0; while [ $i -lt %d ]; do %s i=$(($i+1)); done; while true; do sleep 5; done"] ` file := staticPodPath(dir, name, namespace) podYaml := fmt.Sprintf(template, name, namespace, name, image, string(restart), iterations, command) f, err := os.OpenFile(file, os.O_RDWR|os.O_TRUNC|os.O_CREATE, 0666) if err != nil { return err } defer f.Close() _, err = f.WriteString(podYaml) return err } </ins>