Change:
What happened: A Pod was scheduled for deletion because a taint w/ effect was observed in the Node the Pod was assigned to, and after the taint is removed from the Node, the Pod scheduled deletion isn't canceled. After scheduling pod deletion, the only cancels pod deletion if all taints are removed from the node, including any taints that the user relies on for their use-cases (example below). What you expected to happen: After the taint that triggered scheduling pod deletion is removed, cancels the scheduled pod deletion. How to reproduce it (as minimally and precisely as possible): A node is tainted w/ , so any workloads already assigned to it which don't tolerate this taint are evicted. A pod is created w/ tolerations: w/ unspecified and w/ . is assigned to . At some point in time after the previous steps took place, is tainted as . gets an update event for and observes two taints. It proceeds to calculate the minimum time (in seconds) that tolerates for the two taints which at this point is 300 seconds, and marks for deletion in ~300 seconds - this happens in-memory, in a timed-queue. At some point in time after the previous steps took place, and before the 300 seconds trigger happen, taint is removed from . gets an update event for and observes only one taint, . It proceeds to calculate the minimum time (in seconds) that tolerates for the this taint which is infinity, and returns, not canceling the previous deletion. It completely ignores the fact that the taint that triggered deletion is no longer observed. Anything else we need to know?: Worked w/ to figure this out after all pods in a node got evicted without apparent reason. Environment: Kubernetes version (use ): 1.17.3 (but looking at the code seems to affect all of 1.17.x and 1.18.x and master. Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): 5.x Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A cc (owners of this code)
There seems to be at least .
You can sort of get a feel for what's going on here by looking at these two code snippets: This is what happens in when the node does not have any NoExecute taints: If you have any NoExecute taints on the node processPodOnNode gets called and this routine is really, really eager to evict pods. So : We are currently working around this issue by changing over our NoExecute taints to NoSchedule.
/sig scheduling node
/cc
/remove-sig scheduling scheduler isn't involved in this case.
Which SIG owns the controller-manager or is the node lifecycle manager (and its NoExecuteTaintManager) owned by sig-node alone?
I think node lifecycle is jointly owned by sig-node and sig-cloud-provider
API machinery owns the mechanics of the controller manager (controller loop setup and management), but the individual controllers are SIG-specific
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Looks like we lost some momentum here and I'd like to avoid this falling into the abyss of bugs that never get fixed. When this bug is triggered the outcome is pretty terrible and once it does happen it is very difficult to dig into and understand what actually happened. Did we ever get this routed to the right owner?
adding sig-node leads for triage/routing. this seems like a clearly reproducible bug with significant correctness issues. /assign
thanks for the detailed report and reproducer. to help accelerate this, do you have a form of your reproducer we could add in the e2e suite (even if its failing initially)?
custom taints are considered disruptive, so it couldn't run in the main suite
It's been a while since I wrote tests for the e2e suite but if you are available over slack in case guidance is needed, I'm willing to put the effort.
Do all suites need to pass for a fix to be merged? I'd assume it is the case, otherwise how would a regression be prevented after the fix lands?
i was more inquiring if there was a test that could automate the reproduction, so even just a linked commit may be helpful for someone that wants to pick this up. i am happy to help if you want to reach out over slack later this week.
looks like a unit test could reproduce the issue, opened a WIP fix in
our code is not publicly accessible, I'm sorry. PR linked by seems to cover both the fix and the test that proves it.
/remove-lifecycle frozen
frozen keeps the bot from auto-closing
Ah, sorry then.

Discussion:
"//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/fields:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/labels:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/types:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", "//staging/src/k8s.io/client-go/kubernetes/fake:go_default_library", "//staging/src/k8s.io/client-go/testing:go_default_library",

minTolerationTime := getMinTolerationTime(usedTolerations) // getMinTolerationTime returns negative value to denote infinite toleration. if minTolerationTime < 0 { <del> klog.V(4).Infof("New tolerations for %v tolerate forever. Scheduled deletion won't be cancelled if already scheduled.", podNamespacedName.String()) </del> <ins> klog.V(4).Infof("Current tolerations for %v tolerate forever, cancelling any scheduled deletion.", podNamespacedName.String()) tc.cancelWorkWithEvent(podNamespacedName) </ins> return }

v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/labels" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/client-go/kubernetes/fake" "k8s.io/kubernetes/pkg/controller/testutil"

} type nodeHolder struct { <ins> lock sync.Mutex </ins> node *v1.Node } <ins> func (n *nodeHolder) setNode(node *v1.Node) { n.lock.Lock() defer n.lock.Unlock() n.node = node } </ins> func (n *nodeHolder) getNode(name string) (*v1.Node, error) { <ins> n.lock.Lock() defer n.lock.Unlock() </ins> return n.node, nil }

for _, item := range testCases { stopCh := make(chan struct{}) fakeClientset := fake.NewSimpleClientset(&v1.PodList{Items: item.pods}) <del> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{item.node}).getNode, getPodsAssignedToNode(fakeClientset)) </del> <ins> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{node: item.node}).getNode, getPodsAssignedToNode(fakeClientset)) </ins> controller.recorder = testutil.NewFakeRecorder() go controller.Run(stopCh) controller.NodeUpdated(nil, item.node)

for _, item := range testCases { stopCh := make(chan struct{}) fakeClientset := fake.NewSimpleClientset(&v1.PodList{Items: item.pods}) <del> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{item.newNode}).getNode, getPodsAssignedToNode(fakeClientset)) </del> <ins> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{node: item.newNode}).getNode, getPodsAssignedToNode(fakeClientset)) </ins> controller.recorder = testutil.NewFakeRecorder() go controller.Run(stopCh) controller.NodeUpdated(item.oldNode, item.newNode)

} } <ins> func TestUpdateNodeWithMultipleTaints(t *testing.T) { taint1 := createNoExecuteTaint(1) taint2 := createNoExecuteTaint(2) minute := int64(60) pod := testutil.NewPod("pod1", "node1") pod.Spec.Tolerations = []v1.Toleration{ {Key: taint1.Key, Operator: v1.TolerationOpExists, Effect: v1.TaintEffectNoExecute}, {Key: taint2.Key, Operator: v1.TolerationOpExists, Effect: v1.TaintEffectNoExecute, TolerationSeconds: &minute}, } podNamespacedName := types.NamespacedName{Namespace: pod.Namespace, Name: pod.Name} untaintedNode := testutil.NewNode("node1") doubleTaintedNode := testutil.NewNode("node1") doubleTaintedNode.Spec.Taints = []v1.Taint{taint1, taint2} singleTaintedNode := testutil.NewNode("node1") singleTaintedNode.Spec.Taints = []v1.Taint{taint1} stopCh := make(chan struct{}) fakeClientset := fake.NewSimpleClientset(pod) holder := &nodeHolder{node: untaintedNode} controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (holder).getNode, getPodsAssignedToNode(fakeClientset)) controller.recorder = testutil.NewFakeRecorder() go controller.Run(stopCh) // no taint holder.setNode(untaintedNode) controller.handleNodeUpdate(nodeUpdateItem{"node1"}) // verify pod is not queued for deletion if controller.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String()) != nil { t.Fatalf("pod queued for deletion with no taints") } // no taint -> infinitely tolerated taint holder.setNode(singleTaintedNode) controller.handleNodeUpdate(nodeUpdateItem{"node1"}) // verify pod is not queued for deletion if controller.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String()) != nil { t.Fatalf("pod queued for deletion with permanently tolerated taint") } // infinitely tolerated taint -> temporarily tolerated taint holder.setNode(doubleTaintedNode) controller.handleNodeUpdate(nodeUpdateItem{"node1"}) // verify pod is queued for deletion if controller.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String()) == nil { t.Fatalf("pod not queued for deletion after addition of temporarily tolerated taint") } // temporarily tolerated taint -> infinitely tolerated taint holder.setNode(singleTaintedNode) controller.handleNodeUpdate(nodeUpdateItem{"node1"}) // verify pod is not queued for deletion if controller.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String()) != nil { t.Fatalf("pod queued for deletion after removal of temporarily tolerated taint") } // verify pod is not deleted for _, action := range fakeClientset.Actions() { if action.GetVerb() == "delete" && action.GetResource().Resource == "pods" { t.Error("Unexpected deletion") } } close(stopCh) } </ins> func TestUpdateNodeWithMultiplePods(t *testing.T) { testCases := []struct { description         string

stopCh := make(chan struct{}) fakeClientset := fake.NewSimpleClientset(&v1.PodList{Items: item.pods}) sort.Sort(item.expectedDeleteTimes) <del> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{item.newNode}).getNode, getPodsAssignedToNode(fakeClientset)) </del> <ins> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{node: item.newNode}).getNode, getPodsAssignedToNode(fakeClientset)) </ins> controller.recorder = testutil.NewFakeRecorder() go controller.Run(stopCh) controller.NodeUpdated(item.oldNode, item.newNode)

stopCh := make(chan struct{}) fakeClientset := fake.NewSimpleClientset(&v1.PodList{Items: item.pods}) holder := &podHolder{} <del> controller := NewNoExecuteTaintManager(fakeClientset, holder.getPod, (&nodeHolder{item.newNode}).getNode, getPodsAssignedToNode(fakeClientset)) </del> <ins> controller := NewNoExecuteTaintManager(fakeClientset, holder.getPod, (&nodeHolder{node: item.newNode}).getNode, getPodsAssignedToNode(fakeClientset)) </ins> controller.recorder = testutil.NewFakeRecorder() go controller.Run(stopCh)