Code change:
} } <ins> func TestUpdateNodeWithMultipleTaints(t *testing.T) { taint1 := createNoExecuteTaint(1) taint2 := createNoExecuteTaint(2) minute := int64(60) pod := testutil.NewPod("pod1", "node1") pod.Spec.Tolerations = []v1.Toleration{ {Key: taint1.Key, Operator: v1.TolerationOpExists, Effect: v1.TaintEffectNoExecute}, {Key: taint2.Key, Operator: v1.TolerationOpExists, Effect: v1.TaintEffectNoExecute, TolerationSeconds: &minute}, } podNamespacedName := types.NamespacedName{Namespace: pod.Namespace, Name: pod.Name} untaintedNode := testutil.NewNode("node1") doubleTaintedNode := testutil.NewNode("node1") doubleTaintedNode.Spec.Taints = []v1.Taint{taint1, taint2} singleTaintedNode := testutil.NewNode("node1") singleTaintedNode.Spec.Taints = []v1.Taint{taint1} stopCh := make(chan struct{}) fakeClientset := fake.NewSimpleClientset(pod) holder := &nodeHolder{node: untaintedNode} controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (holder).getNode, getPodsAssignedToNode(fakeClientset)) controller.recorder = testutil.NewFakeRecorder() go controller.Run(stopCh) // no taint holder.setNode(untaintedNode) controller.handleNodeUpdate(nodeUpdateItem{"node1"}) // verify pod is not queued for deletion if controller.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String()) != nil { t.Fatalf("pod queued for deletion with no taints") } // no taint -> infinitely tolerated taint holder.setNode(singleTaintedNode) controller.handleNodeUpdate(nodeUpdateItem{"node1"}) // verify pod is not queued for deletion if controller.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String()) != nil { t.Fatalf("pod queued for deletion with permanently tolerated taint") } // infinitely tolerated taint -> temporarily tolerated taint holder.setNode(doubleTaintedNode) controller.handleNodeUpdate(nodeUpdateItem{"node1"}) // verify pod is queued for deletion if controller.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String()) == nil { t.Fatalf("pod not queued for deletion after addition of temporarily tolerated taint") } // temporarily tolerated taint -> infinitely tolerated taint holder.setNode(singleTaintedNode) controller.handleNodeUpdate(nodeUpdateItem{"node1"}) // verify pod is not queued for deletion if controller.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String()) != nil { t.Fatalf("pod queued for deletion after removal of temporarily tolerated taint") } // verify pod is not deleted for _, action := range fakeClientset.Actions() { if action.GetVerb() == "delete" && action.GetResource().Resource == "pods" { t.Error("Unexpected deletion") } } close(stopCh) } </ins> func TestUpdateNodeWithMultiplePods(t *testing.T) { testCases := []struct { description         string

Related discussion:
text: You can sort of get a feel for what's going on here by looking at these two code snippets: This is what happens in when the node does not have any NoExecute taints: If you have any NoExecute taints on the node processPodOnNode gets called and this routine is really, really eager to evict pods. So : We are currently working around this issue by changing over our NoExecute taints to NoSchedule.
/sig scheduling node
/cc
/remove-sig scheduling scheduler isn't involved in this case.
Which SIG owns the controller-manager or is the node lifecycle manager (and its NoExecuteTaintManager) owned by sig-node alone?
I think node lifecycle is jointly owned by sig-node and sig-cloud-provider
API machinery owns the mechanics of the controller manager (controller loop setup and management), but the individual controllers are SIG-specific
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Looks like we lost some momentum here and I'd like to avoid this falling into the abyss of bugs that never get fixed. When this bug is triggered the outcome is pretty terrible and once it does happen it is very difficult to dig into and understand what actually happened. Did we ever get this routed to the right owner?
adding sig-node leads for triage/routing. this seems like a clearly reproducible bug with significant correctness issues. /assign
thanks for the detailed report and reproducer. to help accelerate this, do you have a form of your reproducer we could add in the e2e suite (even if its failing initially)?
custom taints are considered disruptive, so it couldn't run in the main suite
It's been a while since I wrote tests for the e2e suite but if you are available over slack in case guidance is needed, I'm willing to put the effort.
Do all suites need to pass for a fix to be merged? I'd assume it is the case, otherwise how would a regression be prevented after the fix lands?