Change:
What happened: DaemonSet doesn't delete pods that: weren't scheduled yet, node was deleted so pod can't be scheduled. What you expected to happen: Pod should be deleted when node is removed. How to reproduce it (as minimally and precisely as possible): I encountered this problem when doing tests with cluster with 1000 nodes and pods running. During tests I was deleting and adding nodes using GCE API without draining them. I reproduced problem on much smaller cluster but it requires stopping K8s scheduler. Scenario: cluster with 2 nodes. till fluentd-gcp pods are running on nodes. K8s scheduler. fluentd-gcp pods. till daemonset controller creates pending pods. one node. scheduler. of the fluentd-gcp pods will be pending and scheduler won't be able to schedule. Anything else we need to know?: Environment: GKE Kubernetes version (use ): <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig apps
/assign janetkuo
/assign
We have PodGC to clean up orphaned pods every 20s, is that enough?
I'm removing pods that have only node affinity set and don't have pod.Spec.NodeName set. These pods were never touched by scheduler before node was removed and after node was removed can't be scheduled as node is doesn't exist anymore. Such pods are pending forever and PodGC can't remove them as pod.Spec.NodeName is empty.
Yes, that's right :) let me take few time to review the PR, thanks.
Should I backport fix to 1.13 and 1.12?
+1

Discussion:
failedPodsObserved += failedPodsObservedOnNode } <ins> // Remove pods assigned to not existing nodes. podsToDelete = append(podsToDelete, getPodsWithoutNode(nodeList, nodeToDaemonPods)...) </ins> // Label new pods using the hash label value of the current history when creating them if err = dsc.syncNodes(ds, podsToDelete, nodesNeedingDaemonPods, hash); err != nil { return err

func failedPodsBackoffKey(ds *apps.DaemonSet, nodeName string) string { return fmt.Sprintf("%s/%d/%s", ds.UID, ds.Status.ObservedGeneration, nodeName) } <ins> // getPodsWithoutNode returns list of pods assigned to not existing nodes. func getPodsWithoutNode( runningNodesList []*v1.Node, nodeToDaemonPods map[string][]*v1.Pod) []string { var results []string isNodeRunning := make(map[string]bool) for _, node := range runningNodesList { isNodeRunning[node.Name] = true } for n, pods := range nodeToDaemonPods { if !isNodeRunning[n] { for _, pod := range pods { results = append(results, pod.Name) } } } return results } </ins>

} } <ins> func TestDeletePodForNotExistingNode(t *testing.T) { for _, f := range []bool{true, false} { defer utilfeaturetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ScheduleDaemonSetPods, f)() for _, strategy := range updateStrategies() { ds := newDaemonSet("foo") ds.Spec.UpdateStrategy = *strategy manager, podControl, _, err := newTestController(ds) if err != nil { t.Fatalf("error creating DaemonSets controller: %v", err) } manager.dsStore.Add(ds) addNodes(manager.nodeStore, 0, 1, nil) addPods(manager.podStore, "node-0", simpleDaemonSetLabel, ds, 1) addPods(manager.podStore, "node-1", simpleDaemonSetLabel, ds, 1) syncAndValidateDaemonSets(t, manager, ds, podControl, 0, 1, 0) } } } </ins> func TestGetNodesToDaemonPods(t *testing.T) { for _, f := range []bool{true, false} { defer utilfeaturetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ScheduleDaemonSetPods, f)()