Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Statefulsets are sometimes continually rolling pods and I don't know why statefulset created: storageclass referenced: kubectl describe statefulset statefulset-test: As you can see from the above describe, the pod in the statefulset has been restarted a lot of times. Observing the created pod, 1) The pod comes up ready 2) immediately transitions to terminating 3) Is terminated 4) A new pod is created 5) Go back to (1) (repeats forever) Despite no updates to the statefulset spec ocurring. Making a change to the statefulset template (for example a trivial change adding a nonsense annotation) stops the rolling and fixes the issue When this statefulset was created, there was a mutating webhook enabled. After the observed pod rolling. I disabled this by making its configuration invalid and making it non required, and the rolling persisted. The status of the statefulset: : You can see that there is a discrepency between currentRevision and updateRevision. currentRevision: statefulset-test- updateRevision: statefulset-test- A short while later the status is changed to contain currentRevision: statefulset-test- updateRevision: statefulset-test- This change happened at Sun Apr 1 19:48:48 2018 Presumably after the rollout of statefulset-test- is complete. Here are all the controllerrevisions from above: kubectl get controllerrevision statefulset-test- -o yaml kubectl get controllerrevision statefulset-test- -o yaml: kubectl get controllerrevision statefulset-test- -o yaml: The differences between the above controllerrevisions are insignificant (the hash, the creation timestamp, the name, the resource version, the uid, the self link) Interestingly all these controllerrevisions were created very close together in time shortly after I created the statefulset The current controllerrevisions are After watching the state change for a while, it actually seems like the statefulset is just cycling through the updateRevision being one of the above 10 controllerrevisions Leaders during this time From kube-controller-manager logs during this time, here is one excerpt: The error sync in there is suspicous, and as you can see the controllerrevisions are changing in the logs I have the full logs from a 5 minute window on either side of the above if necessary. The reason I'm inclined to think this is a kubernetes bug is that any update to the statefulset template fixes the problem. How to reproduce it (as minimally and precisely as possible): Not really sure. Worth just trying to create the above statefulset (requires creating the I tried deleting and recreating the above statefulset and it didn't happen, so something flaky is going on Anything else we need to know?: kubernetes 1.9.3 (client and server) cloud provider: aws
/sig apps
I realize this will probably need some further debugging, happy to get some direction
Consistent repro: Create a statefulset that creates a controllerrevision whose hash is between max int32 and max uint32 (this shouldn't be that hard to do, possibly a 50/50 chance of this). Manually create a controllerrevision identical to the first (same revision), incrementing both the name and hash by one. See that the statefulset controller will continually switch which controller revision should be the target of the statefulset.
Due to the issue fixed here: , under some circumstances controllerrevisions whose revision is the same were being considered equal. This ends up causing this bug to manifest as follows: ListRevisions finds many controllerrevisions with the same revision (unclear how this state happens, but it can happen) The sort here randomly sorts them Due to the linked bug , FindEqualRevisions incorrectly finds that all these controller revisions are equal to the proposed udpate. The updateRevision selects the last revision found here Therefore, updateRevision, is randomly selected and causes the bug. This fixes the immediate issue, but the sort should probably be made stable (i.e not just look at the revision number, perhaps just consider the hash for tie breakers)
As an update, fixing the equality check will not be sufficient because it is possible for multiple controller revisions to be created that are actually equal, including the same hash, that have the same revision number. I believe the fix will need to be equal controllerrevisions with the same revision number from being created sort the controllerrevisions so that the controllerrevision that is the target of the statefulset is stable
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
ping I use the K8S which version is 1.11.4 and encounter the same problem. I want to reproduce the problem and find your way: Manually create a controllerrevision identical to the first (same revision), incrementing both the name and hash by one. Could you tell me how to mannually create a same controllerrevision identical to the first? Thanks in anvance.

Discussion:
// SortControllerRevisions sorts revisions by their Revision. func SortControllerRevisions(revisions []*apps.ControllerRevision) { <del> sort.Sort(byRevision(revisions)) </del> <ins> sort.Stable(byRevision(revisions)) </ins> } // EqualRevision returns true if lhs and rhs are either both nil, or both point to non-nil ControllerRevisions that

return len(br) } <ins> // Less breaks ties first by creation timestamp, then by name </ins> func (br byRevision) Less(i, j int) bool { <ins> if br[i].Revision == br[j].Revision { if br[j].CreationTimestamp.Equal(&br[i].CreationTimestamp) { return br[i].Name < br[j].Name } return br[j].CreationTimestamp.After(br[i].CreationTimestamp.Time) } </ins> return br[i].Revision < br[j].Revision }

"k8s.io/apimachinery/pkg/util/strategicpatch" core "k8s.io/client-go/testing" <ins> "time" </ins> ) func TestRealHistory_ListControllerRevisions(t *testing.T) {

want      []string } testFn := func(test *testcase, t *testing.T) { <del> SortControllerRevisions(test.revisions) for i := range test.revisions { if test.revisions[i].Name != test.want[i] { t.Errorf("%s: want %s at %d got %s", test.name, test.want[i], i, test.revisions[i].Name) </del> <ins> t.Run(test.name, func(t *testing.T) { SortControllerRevisions(test.revisions) for i := range test.revisions { if test.revisions[i].Name != test.want[i] { t.Errorf("%s: want %s at %d got %s", test.name, test.want[i], i, test.revisions[i].Name) } </ins> } <del> } </del> <ins> }) </ins> } ss1 := newStatefulSet(3, "ss1", types.UID("ss1"), map[string]string{"foo": "bar"}) ss1.Status.CollisionCount = new(int32)

if err != nil { t.Fatal(err) } ss1Rev1.Namespace = ss1.Namespace ss1Rev2, err := NewControllerRevision(ss1, parentKind, ss1.Spec.Template.Labels, rawTemplate(&ss1.Spec.Template), 2, ss1.Status.CollisionCount) if err != nil { t.Fatal(err) } ss1Rev2.Namespace = ss1.Namespace ss1Rev3, err := NewControllerRevision(ss1, parentKind, ss1.Spec.Template.Labels, rawTemplate(&ss1.Spec.Template), 3, ss1.Status.CollisionCount) if err != nil { t.Fatal(err) } <del> ss1Rev3.Namespace = ss1.Namespace </del> <ins> ss1Rev3Time2, err := NewControllerRevision(ss1, parentKind, ss1.Spec.Template.Labels, rawTemplate(&ss1.Spec.Template), 3, ss1.Status.CollisionCount) if err != nil { t.Fatal(err) } ss1Rev3Time2.Namespace = ss1.Namespace ss1Rev3Time2.CreationTimestamp = metav1.Time{Time: ss1Rev3.CreationTimestamp.Add(time.Second)} ss1Rev3Time2Name2, err := NewControllerRevision(ss1, parentKind, ss1.Spec.Template.Labels, rawTemplate(&ss1.Spec.Template), 3, ss1.Status.CollisionCount) if err != nil { t.Fatal(err) } ss1Rev3Time2Name2.Namespace = ss1.Namespace ss1Rev3Time2Name2.CreationTimestamp = metav1.Time{Time: ss1Rev3.CreationTimestamp.Add(time.Second)} </ins> tests := []testcase{ {

want:      []string{ss1Rev1.Name, ss1Rev2.Name, ss1Rev3.Name}, }, { <ins> name:      "with ties", revisions: []*apps.ControllerRevision{ss1Rev3, ss1Rev3Time2, ss1Rev2, ss1Rev1}, want:      []string{ss1Rev1.Name, ss1Rev2.Name, ss1Rev3.Name, ss1Rev3Time2.Name, ss1Rev3Time2Name2.Name}, }, { </ins> name:      "empty", revisions: nil, want:      nil,