Code change:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The deployment controller for a deployment with a single replica failed to create a new ReplicaSet when the old ReplicaSet included an Evicted pod. Typically, the eviction was caused by "node low on resource: nodefs". When we manually deleted the evicted pod, the new RS and accompanying pod were created immediately. We have seen this happen at least a dozen times. Here are the audit logs for a recent case: ! What you expected to happen: The deployment controller ignores the evicted pod and creates the updated replicaset/pod. How to reproduce it (as minimally and precisely as possible): To reproduce, you would need a way to trigger a kubelet-driven eviction on a node that is currently running a single-replica Deployment with . Wait for the pod to be evicted and the new pod to be created, confirm that the evicted pod still exists, and then make a trivial change to the deployment spec. The running pod will be deleted, and no new replicaset/pod will be created. This issue does not seem to occur for manual evictions (via ); pods I evict that way disappear from the API server immediately. Anything else we need to know?: Possibly related to and/or Environment: Kubernetes version (use ): v1.8.5-gke.0 Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Container-Optimized OS
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks for the detailed bug report! From Here's what happened: Deployment was updated by the user, so it scaled all old ReplicaSets to 0 old ReplicaSet was scaled to 0 (ReplicaSet's == 0), but still owned the evicted pod, which wouldn't be killed automatically because of the above comment (ReplicaSet's 0) Deployment found that there were still old Pods, and kept waiting for them to be gone ~(wait for old ReplicaSet's to be 0)~ (Edit: Deployment also checks those old Pods by listing them to make sure there are none, not just by checking ReplicaSet's status) before starting the new rollout. We can either modify Deployment controller (to ignore evicted old pods) or ReplicaSet controller (to kill evicted pods ~when it gets scaled down~), if this is not the intended behavior. WDYT?
I am actually debugging likely the same production issue with Recreate strategy in two clusters right now. (We run on patched 1.9.1.) I have traced it down to broken RS as well. in 1 cluster RS leaves around pod failed matching the selector in the other one those are evicted pods. all replicas are scaled down to 0 and report 0 in status but pods with valid controllerRefs to them are still present. Same is 0 for me but pod is still there I think the correct behaviour is for replicaSet scaled to 0 to delete all of its pod, not to leave them laying around
xref - this should fix it

Related discussion:
text: oldRSs []*extensions.ReplicaSet podMap map[types.UID]*v1.PodList <del> expected bool </del> <ins> hasOldPodsRunning bool </ins> }{ { <del> name:     "no old RSs", expected: false, </del> <ins> name:              "no old RSs", hasOldPodsRunning: false, </ins> }, { <del> name:     "old RSs with running pods", oldRSs:   []*extensions.ReplicaSet{rsWithUID("some-uid"), rsWithUID("other-uid")}, podMap:   podMapWithUIDs([]string{"some-uid", "other-uid"}), expected: true, </del> <ins> name:              "old RSs with running pods", oldRSs:            []*extensions.ReplicaSet{rsWithUID("some-uid"), rsWithUID("other-uid")}, podMap:            podMapWithUIDs([]string{"some-uid", "other-uid"}), hasOldPodsRunning: true, </ins> }, { <del> name:     "old RSs without pods but with non-zero status replicas", oldRSs:   []*extensions.ReplicaSet{newRSWithStatus("rs-blabla", 0, 1, nil)}, expected: true, </del> <ins> name:              "old RSs without pods but with non-zero status replicas", oldRSs:            []*extensions.ReplicaSet{newRSWithStatus("rs-1", 0, 1, nil)}, hasOldPodsRunning: true, </ins> }, { <del> name:     "old RSs without pods or non-zero status replicas", oldRSs:   []*extensions.ReplicaSet{newRSWithStatus("rs-blabla", 0, 0, nil)}, expected: false, </del> <ins> name:              "old RSs without pods or non-zero status replicas", oldRSs:            []*extensions.ReplicaSet{newRSWithStatus("rs-1", 0, 0, nil)}, hasOldPodsRunning: false, }, { name:   "old RSs with zero status replicas but pods in terminal state are present", oldRSs: []*extensions.ReplicaSet{newRSWithStatus("rs-1", 0, 0, nil)}, podMap: map[types.UID]*v1.PodList{ "uid-1": { Items: []v1.Pod{ { Status: v1.PodStatus{ Phase: v1.PodFailed, }, }, { Status: v1.PodStatus{ Phase: v1.PodSucceeded, }, }, }, }, }, hasOldPodsRunning: false, }, { name:   "old RSs with zero status replicas but pod in unknown phase present", oldRSs: []*extensions.ReplicaSet{newRSWithStatus("rs-1", 0, 0, nil)}, podMap: map[types.UID]*v1.PodList{ "uid-1": { Items: []v1.Pod{ { Status: v1.PodStatus{ Phase: v1.PodUnknown, }, }, }, }, }, hasOldPodsRunning: true, }, { name:   "old RSs with zero status replicas with pending pod present", oldRSs: []*extensions.ReplicaSet{newRSWithStatus("rs-1", 0, 0, nil)}, podMap: map[types.UID]*v1.PodList{ "uid-1": { Items: []v1.Pod{ { Status: v1.PodStatus{ Phase: v1.PodPending, }, }, }, }, }, hasOldPodsRunning: true, }, { name:   "old RSs with zero status replicas with running pod present", oldRSs: []*extensions.ReplicaSet{newRSWithStatus("rs-1", 0, 0, nil)}, podMap: map[types.UID]*v1.PodList{ "uid-1": { Items: []v1.Pod{ { Status: v1.PodStatus{ Phase: v1.PodRunning, }, }, }, }, }, hasOldPodsRunning: true, }, { name:   "old RSs with zero status replicas but pods in terminal state and pending are present", oldRSs: []*extensions.ReplicaSet{newRSWithStatus("rs-1", 0, 0, nil)}, podMap: map[types.UID]*v1.PodList{ "uid-1": { Items: []v1.Pod{ { Status: v1.PodStatus{ Phase: v1.PodFailed, }, }, { Status: v1.PodStatus{ Phase: v1.PodSucceeded, }, }, }, }, "uid-2": { Items: []v1.Pod{}, }, "uid-3": { Items: []v1.Pod{ { Status: v1.PodStatus{ Phase: v1.PodPending, }, }, }, }, }, hasOldPodsRunning: true, </ins> }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { <del> if expected, got := test.expected, oldPodsRunning(test.newRS, test.oldRSs, test.podMap); expected != got { </del> <ins> if expected, got := test.hasOldPodsRunning, oldPodsRunning(test.newRS, test.oldRSs, test.podMap); expected != got { </ins> t.Errorf("%s: expected %t, got %t", test.name, expected, got) } })