Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: I have a daemonsets with 250 replicas and maxUnavailable is 20%. When I try to update its image, the ds will scale down 50 old replicas and wait for SatisfiedExpectations, after that it will scale up 50 new replicas. But if a pod delete failed (node is notready, and pod stuck at terminating status), ds controller will never reach the expectation and just stuck here. What you expected to happen: When a pod stuck at terminating, ds controller should start new pods like deployment controller, instead of waiting for the delete batch finish and ready for next poll. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.1-8+", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-25T13:04:04Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig apps /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
DaemonSet controller waits for old pods to terminate before creating new ones for best-effort at-most-one pod per node. This is a deliberate design decision. In most cases, DaemonSets won't tolerate two pods scheduled on the same node at the same time; for example, it's common to communicate with DaemonSet pods via . I'm going to close this ticket as this works as designed. Please feel free to reopen the ticket if you think there are still issues to be resolved.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
why does the DaemonSet controller wait for all pods 50 to be done, before moving to the NEXT 50 ? I completely understand that we want only at most 1 pod per node. But if a pod deletion fails on the node, move on to the next pod as long as the total unavailable pods are less than maxUnavailable. This will allow us to make progress in the face of bad nodes or failed pod deletions due to failing tear down of bad volumes or other reasons.
/assign
thoughts on this one ?
Any progress on this issue? We also encountered this problem. There are thousands of nodes in our prod cluster and nodes often go down, which cause daemonset rolling update stucked, this problem really bother us.
PTAL
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
// podsShouldBeOnNode figures out the DaemonSet pods to be created and deleted on the given node: //   - nodesNeedingDaemonPods: the pods need to start on the node //   - podsToDelete: the Pods need to be deleted on the node <del> //   - failedPodsObserved: the number of failed pods on node </del> //   - err: unexpected error func (dsc *DaemonSetsController) podsShouldBeOnNode( node *v1.Node, nodeToDaemonPods map[string][]*v1.Pod, ds *apps.DaemonSet, <del> ) (nodesNeedingDaemonPods, podsToDelete []string, failedPodsObserved int, err error) { </del> <ins> ) (nodesNeedingDaemonPods, podsToDelete []string, err error) { </ins> wantToRun, shouldSchedule, shouldContinueRunning, err := dsc.nodeShouldRunDaemonPod(node, ds) if err != nil {

continue } if pod.Status.Phase == v1.PodFailed { <del> failedPodsObserved++ </del> // This is a critical place where DS is often fighting with kubelet that rejects pods. // We need to avoid hot looping and backoff. backoffKey := failedPodsBackoffKey(ds, node.Name)

} } <del> return nodesNeedingDaemonPods, podsToDelete, failedPodsObserved, nil </del> <ins> return nodesNeedingDaemonPods, podsToDelete, nil </ins> } // manage manages the scheduling and running of Pods of ds on nodes.

// For each node, if the node is running the daemon pod but isn't supposed to, kill the daemon // pod. If the node is supposed to run the daemon pod, but isn't, create the daemon pod on the node. var nodesNeedingDaemonPods, podsToDelete []string <del> var failedPodsObserved int </del> for _, node := range nodeList { <del> nodesNeedingDaemonPodsOnNode, podsToDeleteOnNode, failedPodsObservedOnNode, err := dsc.podsShouldBeOnNode( </del> <ins> nodesNeedingDaemonPodsOnNode, podsToDeleteOnNode, err := dsc.podsShouldBeOnNode( </ins> node, nodeToDaemonPods, ds) if err != nil {

nodesNeedingDaemonPods = append(nodesNeedingDaemonPods, nodesNeedingDaemonPodsOnNode...) podsToDelete = append(podsToDelete, podsToDeleteOnNode...) <del> failedPodsObserved += failedPodsObservedOnNode </del> } // Remove unscheduled pods assigned to not existing nodes when daemonset pods are scheduled by scheduler.

return err } <del> // Throw an error when the daemon pods fail, to use ratelimiter to prevent kill-recreate hot loop if failedPodsObserved > 0 { return fmt.Errorf("deleted %d failed pods of DaemonSet %s/%s", failedPodsObserved, ds.Namespace, ds.Name) } </del> return nil }