Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: a statefulset which will never be running: down it the pods seems like only running pods will be deleted. And CrashLoopBackOff pods are not included into "CURRENT" pod. What you expected to happen: I guess the failss-0 should be deleted after replicas set to 0, but it doesn't. Is such behavior designed by purpose? How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Change to , and it can be scaled down... Now the crashloopbackoff pods got deleted... Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): debian Kernel (e.g. ): Install tools: Others:
Update: Cannot scale down to 0, if the pods keep . How to reproduce it any statefulset whose PVC cannot be bound. the replicas to 0
/sig apps because statefulset
I'll have a look at this issue
I'm afraid it requires bigger code/design change because controller first updates its pods/replicas and when every replica is and then it starts to terminating redundant pods (scaling down). What do you think about changing the order? I mean scale down first and then update every StatefulSet's pod. For more details take a look here:
I think when set the replicas to 0, all pending or crashed pods should be included in the array and then removed.
I've observed something similar as well - I think it may be a Kubelet issue though (or there are two issues together), because what I see is: a stateful set size 1 with a pod that crash loops after 5-10s the pod directly pod is in , pod is deleted quickly pod is in , pod takes tens of seconds to be deleted My suspicion is that there is a node level issue as well where the pod worker on the kubelet is interacting with crash loop backoff incorrectly. was seeing this in 1.9
I just tried this with a bare pod and could not reproduce on master. create pod that immediately fails wait for cashloop backoff to reach 5m attempt to delete the pod In my case, the pod deleted immediately, even though it had just started the 5m backoff interval. So this is either fixed already in master (1.10) there is some element of the recreate that is not captured when creating bare pods
Re-reading through the issue, seems like there is some element of Stateful sets at play here that does not effect bare pods or ReplicationControllers.
I was able to recreate with a StatefulSet. He is the kubelet + controller-manager log for the time between the pod delete request and the StatefulSet controller creating the new one (with a 10s yet unexplained delay) There are a number of that seem to be holding up the deletion of the pod. However, the the REMOVE comes in from the API server 10s after the first DELETE forces the pod the cleanup and the StatefulSet to start a new pod.
Nevermind the part about not being able to create it with a bare pod. I can. I forgot I set a termination grace period of 0 on the pod and the issue is related to the grace period. What happens is that the kubelet is not cleaning up the failed container and thus the pod is not cleaned up and not being completely deleted. If there are any containers that still exist in the pod, then the delete can't be finalized. Thus, the delete waits until the grace period is over and then the pod is removed from the API server causing the kubelet to clean up along another path. Still running down where this should be cleaned up.
/sig node
Hi, I am also encountering this issue using StatefulSet on 1.8. Is there any known workaround we can use in the short term?

Discussion:
return false } if len(runtimeStatus.ContainerStatuses) > 0 { <del> glog.V(3).Infof("Pod %q is terminated, but some containers have not been cleaned up: %+v", format.Pod(pod), runtimeStatus.ContainerStatuses) </del> <ins> var statusStr string for _, status := range runtimeStatus.ContainerStatuses { statusStr += fmt.Sprintf("%+v ", *status) } glog.V(3).Infof("Pod %q is terminated, but some containers have not been cleaned up: %s", format.Pod(pod), statusStr) </ins> return false } if kl.podVolumesExist(pod.UID) && !kl.keepTerminatedPodVolumes {

containersToKeep := p.containersToKeep if removeAll { containersToKeep = 0 <ins> filterContainerID = "" </ins> } for _, candidate := range getContainersToDeleteInPod(filterContainerID, podStatus, containersToKeep) {