Change:
What happened: Occasionally we get a Daemonset stuck with less Current replicas than Desired, causing a Daemonset sync tends to fix it, e.g (add a new node). This happens to all our Daemonsets. I'm using fluentd as an example. There are no Daemonsets pods in pending nor are there any errors when describing the Daemonset What you expected to happen: The Daemonset Pod to be scheduled How to reproduce it (as minimally and precisely as possible): Reproducing this state is very tricky, I've had the most success by terminating nodes in our ASG causing a new one to come up. However I think I've identified why this is happening. The issue appears to be related to daemonset Pod deletion, a Pod gets deleted causing to trigger, this lowers the expectations by as expected. The delete pod event then triggers a sync of the daemonset, in which it finds which involves fetching pods from the informer cache This is somehow returning the deleted pod. You can see in the logs here thinks that it needs to delete the pod that was deleted. It is also setting the expectation back to after the delete pod event previously lowered it by 1. At this point you're stuck as the function can't get past this line As the expectation can not be satisfied as syncNodes set it back to and there's nothing it to get it back to . So I believe this is what's causing us to get less replicas than expected, the function is stuck due to the expectation so it can't perform a full sync to schedule the new pods. There's also not much to trigger a new sync once the expectation has expired until a new node comes up or one goes down. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g: ):Kernel (e.g. ):
/sig apps
I investigated this as much as I could without getting some external input, it feels like it's some kind of race condition in the informer cache. If anyone has ideas of if places to dig deeper that would be handy.
Reiterating the mentions to trigger a notification: <detailsIn response to : I investigated this as much as I could without getting some external input, it feels like it's some kind of race condition in the informer cache. If anyone has ideas of if places to dig deeper that would be handy. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This problem is definitely related to node termination, it's trying to delete the already deleted pod from the node that's been terminated. The problem seems to go away if is set, I think although this kind of race condition may have always been possible, it's being frequently caused via Which is finding the non-existent pod in the pod cache and seeing that it is assigned to a non-running node, resulting in a request to delete the non-existent pod. cc
Reiterating the mentions to trigger a notification: <detailsIn response to : This problem is definitely related to node termination, it's trying to delete the already deleted pod from the node that's been terminated. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign could you please take a look at this? It seems that it is related to the code you in .
Thanks for reporting the bug. Bug was introduced in which deleting pods without nodes. There is race condition between removing pod from cache and listing pods for DaemonSet. Bug occurs during following scenario: is deleted. for node is deleted by PodGCController. handler is executed by informer () set is to queue by delete handler () is processed and pods are taken from cache before deleted pod is removed. for deleted pod is not found so we add deleted pod to podsToDelete list (). are set to del:1() succeeds as pod is not found(). are never met so we need to wait till expectations expire. As fix I will send a change to so only pods that can't be deleted by PodGCController are deleted by DaemonSet controller.

Discussion:
failedPodsObserved += failedPodsObservedOnNode } <del> // Remove pods assigned to not existing nodes when daemonset pods are scheduled by default scheduler. </del> <ins> // Remove unscheduled pods assigned to not existing nodes when daemonset pods are scheduled by scheduler. </ins> // If node doesn't exist then pods are never scheduled and can't be deleted by PodGCController. if utilfeature.DefaultFeatureGate.Enabled(features.ScheduleDaemonSetPods) { <del> podsToDelete = append(podsToDelete, getPodsWithoutNode(nodeList, nodeToDaemonPods)...) </del> <ins> podsToDelete = append(podsToDelete, getUnscheduledPodsWithoutNode(nodeList, nodeToDaemonPods)...) </ins> } // Label new pods using the hash label value of the current history when creating them

return fmt.Sprintf("%s/%d/%s", ds.UID, ds.Status.ObservedGeneration, nodeName) } <del> // getPodsWithoutNode returns list of pods assigned to not existing nodes. func getPodsWithoutNode(runningNodesList []*v1.Node, nodeToDaemonPods map[string][]*v1.Pod) []string { </del> <ins> // getUnscheduledPodsWithoutNode returns list of unscheduled pods assigned to not existing nodes. // Returned pods can't be deleted by PodGCController so they should be deleted by DaemonSetController. func getUnscheduledPodsWithoutNode(runningNodesList []*v1.Node, nodeToDaemonPods map[string][]*v1.Pod) []string { </ins> var results []string isNodeRunning := make(map[string]bool) for _, node := range runningNodesList {

for n, pods := range nodeToDaemonPods { if !isNodeRunning[n] { for _, pod := range pods { <del> results = append(results, pod.Name) </del> <ins> if len(pod.Spec.NodeName) == 0 { results = append(results, pod.Name) } </ins> } } }

} } <del> func TestDeletePodForNotExistingNode(t *testing.T) { </del> <ins> func TestDeleteUnscheduledPodForNotExistingNode(t *testing.T) { </ins> for _, f := range []bool{true, false} { defer utilfeaturetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ScheduleDaemonSetPods, f)() for _, strategy := range updateStrategies() {

addNodes(manager.nodeStore, 0, 1, nil) addPods(manager.podStore, "node-0", simpleDaemonSetLabel, ds, 1) addPods(manager.podStore, "node-1", simpleDaemonSetLabel, ds, 1) <ins> podScheduledUsingAffinity := newPod("pod1-node-3", "", simpleDaemonSetLabel, ds) podScheduledUsingAffinity.Spec.Affinity = &v1.Affinity{ NodeAffinity: &v1.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &v1.NodeSelector{ NodeSelectorTerms: []v1.NodeSelectorTerm{ { MatchFields: []v1.NodeSelectorRequirement{ { Key:      schedulerapi.NodeFieldSelectorKeyNodeName, Operator: v1.NodeSelectorOpIn, Values:   []string{"node-2"}, }, }, }, }, }, }, } manager.podStore.Add(podScheduledUsingAffinity) </ins> if f { syncAndValidateDaemonSets(t, manager, ds, podControl, 0, 1, 0) } else {