Change:
Starting with a cluster of three nodes, with a healthy deamonset pod running on each, with . When performing three updates to the daemonset, each with some reason that prevents the new pods from becoming ready, I ended up with zero ready pods. I would expect that when I apply an update, and the new pods fail to become ready, when I apply a second update the "new" failing pods are replaced before considering any of the oldest healthy pods. This is similar behavior to how Deployments work. On a three node (or any count really) cluster, create this daemonset, which is just an nginx pod with a startup probe hitting port 80. Update the startup probe port to something other than port 80, which should cause the pod to never become ready, and apply again. You can see the new unhealthy pod, and the generation of each pod Repeat this process for as many nodes/pods as you have (2 more in this example), changing anything that would cause the pods to be recreated (in this case I just cycled the startup probe port a few times). After the 2nd update, you can see that it has replaced both the 2nd generation pod (the one that was failing), and one of the first generation pods. Im now down to 1 out of 3, when I had specified . One more apply / update, and all generation 1 pods are gone, and all pods are unhealthy. No response <details</details<detailsReproduced this on bare metal, and GKE/GCP </detailsNo response No response No response No response
/sig apps
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/triage accepted /priority important-longterm

Discussion:
allowedReplacementPods = make([]string, 0, len(nodeToDaemonPods)) } allowedReplacementPods = append(allowedReplacementPods, oldPod.Name) <ins> numUnavailable++ </ins> case numUnavailable >= maxUnavailable: // no point considering any other candidates continue

clearExpectations(t, manager, ds, podControl) } <ins> func TestDaemonSetUpdatesSomeOldPodsNotReady(t *testing.T) { _, ctx := ktesting.NewTestContext(t) ds := newDaemonSet("foo") manager, podControl, _, err := newTestController(ctx, ds) if err != nil { t.Fatalf("error creating DaemonSets controller: %v", err) } maxUnavailable := 2 addNodes(manager.nodeStore, 0, 5, nil) err = manager.dsStore.Add(ds) if err != nil { t.Fatal(err) } expectSyncDaemonSets(t, manager, ds, podControl, 5, 0, 0) markPodsReady(podControl.podStore) ds.Spec.Template.Spec.Containers[0].Image = "foo2/bar2" ds.Spec.UpdateStrategy.Type = apps.RollingUpdateDaemonSetStrategyType intStr := intstr.FromInt(maxUnavailable) ds.Spec.UpdateStrategy.RollingUpdate = &apps.RollingUpdateDaemonSet{MaxUnavailable: &intStr} err = manager.dsStore.Update(ds) if err != nil { t.Fatal(err) } // All old pods are available, should update 2 clearExpectations(t, manager, ds, podControl) expectSyncDaemonSets(t, manager, ds, podControl, 0, maxUnavailable, 0) clearExpectations(t, manager, ds, podControl) expectSyncDaemonSets(t, manager, ds, podControl, maxUnavailable, 0, 0) clearExpectations(t, manager, ds, podControl) expectSyncDaemonSets(t, manager, ds, podControl, 0, 0, 0) clearExpectations(t, manager, ds, podControl) // Perform another update, verify we delete and create 2 pods, three available should remain untouched ds.Spec.Template.Spec.Containers[0].Image = "foo2/bar3" err = manager.dsStore.Update(ds) if err != nil { t.Fatal(err) } clearExpectations(t, manager, ds, podControl) expectSyncDaemonSets(t, manager, ds, podControl, 0, maxUnavailable, 0) clearExpectations(t, manager, ds, podControl) expectSyncDaemonSets(t, manager, ds, podControl, maxUnavailable, 0, 0) clearExpectations(t, manager, ds, podControl) expectSyncDaemonSets(t, manager, ds, podControl, 0, 0, 0) clearExpectations(t, manager, ds, podControl) readyCount := 0 expectedReadyCount := 5 - maxUnavailable for _, obj := range podControl.podStore.List() { pod := obj.(*v1.Pod) n, condition := podutil.GetPodCondition(&pod.Status, v1.PodReady) if n != -1 && condition.Status == v1.ConditionTrue { readyCount++ } } if readyCount != expectedReadyCount { t.Fatalf("Expected %d old ready pods, but found %d", expectedReadyCount, readyCount) } } </ins> func TestDaemonSetUpdatesAllOldPodsNotReady(t *testing.T) { _, ctx := ktesting.NewTestContext(t) ds := newDaemonSet("foo")