Change:
What happened: What you expected to happen: The statefulset should have scaled down to 0. Instead, even after 10 minutes: How to reproduce it (as minimally and precisely as possible): Run the e2e test on a gce cluster running the latest and greatest kubernetes. cc
if the petset has replicas=0 it's not going to do anything
When I a running petset, and set its scale to 0, that seems to work and delete the pets, and we should have a functioning scaler. Am I misunderstanding something? I'm using the same teardown process for the e2e test I linked in the issue as the other tests (), but it times out sometimes after network partition.
yeah is scale in the petset controller noticing deletion timestamp? because the test code you showed doesn't care
As I mentioned yesterday, it isn't even issuing a delete request even after 10 minutes, because the pods continue to be "running". I'll check the controller manager logs today. I suspect it might be the backoff in the rate limited work-queue.
Hmm, the AGE fields on your pets don't make sense if it isn't even issuing a delete?
In the output I pasted above, at the 7 minute mark, it created the new replacement pet-0, after the network partition was healed. Then tried to scale down the StatefulSet and waited 10 minutes for the pods to disappear.
I see a bunch of different failures modes on the bug: 3 08:58:44.475: Failed waiting for pods to enter running: timed out waiting for the condition Failed to scale statefulset to 0 in 10m0s. Remaining pods: [] 3 08:11:30.728: Unexpected error: <nilBasically 3 translates to We should clarify. 1 and 2 both have events like guessing the node controlelr shouldn't be setting timestamp?
Looking into the case 2: The controller manager log goes: So I guess even deletions are being blocked by the health/readiness.
Both issues appear to be due to the testsuite and the way we use the "initialized" annotation in tests. I'll send a PR to fix these shortly. Thanks

Discussion:
Expect(err).NotTo(HaveOccurred()) pst := statefulSetTester{c: c} <del> pst.saturate(ps) </del> nn := framework.TestContext.CloudConfig.NumNodes nodeNames, err := framework.CheckNodesReady(f.ClientSet, framework.NodeReadyInitialTimeout, nn)

petMounts := []api.VolumeMount{{Name: "datadir", MountPath: "/data/"}} podMounts := []api.VolumeMount{{Name: "home", MountPath: "/home"}} ps := newStatefulSet(psName, ns, headlessSvcName, 3, petMounts, podMounts, labels) <ins> setInitializedAnnotation(ps, "false") </ins> _, err := c.Apps().StatefulSets(ns).Create(ps) Expect(err).NotTo(HaveOccurred())

petMounts := []api.VolumeMount{{Name: "datadir", MountPath: "/data/"}} podMounts := []api.VolumeMount{{Name: "home", MountPath: "/home"}} ps := newStatefulSet(psName, ns, headlessSvcName, 2, petMounts, podMounts, labels) <ins> setInitializedAnnotation(ps, "false") </ins> _, err := c.Apps().StatefulSets(ns).Create(ps) Expect(err).NotTo(HaveOccurred())

Replicas: replicas, Template: api.PodTemplateSpec{ ObjectMeta: api.ObjectMeta{ <del> Labels: labels, Annotations: map[string]string{ "pod.alpha.kubernetes.io/initialized": "false", }, </del> <ins> Labels:      labels, Annotations: map[string]string{}, </ins> }, Spec: api.PodSpec{ Containers: []api.Container{

}, } } <ins> func setInitializedAnnotation(ss *apps.StatefulSet, value string) { ss.Spec.Template.ObjectMeta.Annotations["pod.alpha.kubernetes.io/initialized"] = value } </ins>