Code change:
} } <ins> func TestUpdateNodeWithMultipleTaints(t *testing.T) { taint1 := createNoExecuteTaint(1) taint2 := createNoExecuteTaint(2) minute := int64(60) pod := testutil.NewPod("pod1", "node1") pod.Spec.Tolerations = []v1.Toleration{ {Key: taint1.Key, Operator: v1.TolerationOpExists, Effect: v1.TaintEffectNoExecute}, {Key: taint2.Key, Operator: v1.TolerationOpExists, Effect: v1.TaintEffectNoExecute, TolerationSeconds: &minute}, } podNamespacedName := types.NamespacedName{Namespace: pod.Namespace, Name: pod.Name} untaintedNode := testutil.NewNode("node1") doubleTaintedNode := testutil.NewNode("node1") doubleTaintedNode.Spec.Taints = []v1.Taint{taint1, taint2} singleTaintedNode := testutil.NewNode("node1") singleTaintedNode.Spec.Taints = []v1.Taint{taint1} stopCh := make(chan struct{}) fakeClientset := fake.NewSimpleClientset(pod) holder := &nodeHolder{node: untaintedNode} controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (holder).getNode, getPodsAssignedToNode(fakeClientset)) controller.recorder = testutil.NewFakeRecorder() go controller.Run(stopCh) // no taint holder.setNode(untaintedNode) controller.handleNodeUpdate(nodeUpdateItem{"node1"}) // verify pod is not queued for deletion if controller.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String()) != nil { t.Fatalf("pod queued for deletion with no taints") } // no taint -> infinitely tolerated taint holder.setNode(singleTaintedNode) controller.handleNodeUpdate(nodeUpdateItem{"node1"}) // verify pod is not queued for deletion if controller.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String()) != nil { t.Fatalf("pod queued for deletion with permanently tolerated taint") } // infinitely tolerated taint -> temporarily tolerated taint holder.setNode(doubleTaintedNode) controller.handleNodeUpdate(nodeUpdateItem{"node1"}) // verify pod is queued for deletion if controller.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String()) == nil { t.Fatalf("pod not queued for deletion after addition of temporarily tolerated taint") } // temporarily tolerated taint -> infinitely tolerated taint holder.setNode(singleTaintedNode) controller.handleNodeUpdate(nodeUpdateItem{"node1"}) // verify pod is not queued for deletion if controller.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String()) != nil { t.Fatalf("pod queued for deletion after removal of temporarily tolerated taint") } // verify pod is not deleted for _, action := range fakeClientset.Actions() { if action.GetVerb() == "delete" && action.GetResource().Resource == "pods" { t.Error("Unexpected deletion") } } close(stopCh) } </ins> func TestUpdateNodeWithMultiplePods(t *testing.T) { testCases := []struct { description         string

Related discussion:
text: What happened: A Pod was scheduled for deletion because a taint w/ effect was observed in the Node the Pod was assigned to, and after the taint is removed from the Node, the Pod scheduled deletion isn't canceled. After scheduling pod deletion, the only cancels pod deletion if all taints are removed from the node, including any taints that the user relies on for their use-cases (example below). What you expected to happen: After the taint that triggered scheduling pod deletion is removed, cancels the scheduled pod deletion. How to reproduce it (as minimally and precisely as possible): A node is tainted w/ , so any workloads already assigned to it which don't tolerate this taint are evicted. A pod is created w/ tolerations: w/ unspecified and w/ . is assigned to . At some point in time after the previous steps took place, is tainted as . gets an update event for and observes two taints. It proceeds to calculate the minimum time (in seconds) that tolerates for the two taints which at this point is 300 seconds, and marks for deletion in ~300 seconds - this happens in-memory, in a timed-queue. At some point in time after the previous steps took place, and before the 300 seconds trigger happen, taint is removed from . gets an update event for and observes only one taint, . It proceeds to calculate the minimum time (in seconds) that tolerates for the this taint which is infinity, and returns, not canceling the previous deletion. It completely ignores the fact that the taint that triggered deletion is no longer observed. Anything else we need to know?: Worked w/ to figure this out after all pods in a node got evicted without apparent reason. Environment: Kubernetes version (use ): 1.17.3 (but looking at the code seems to affect all of 1.17.x and 1.18.x and master. Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): 5.x Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A cc (owners of this code)
There seems to be at least .