Change:
The kube-controller-manage may crash when processing StatefulSet with spec.podManagementPolicy is Parallel. With logs: we expect the kube-controller-manage will work well. The following example can reproduce this problem: After digging into the source code, we found that when using the pod management policy, the statefulset controller will use to start parallel processing. But the which is called during parallel processing may directly update the claims' labels without a lock. This finally causes a concurrent map-accessing problem. <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
Thanks for the report. Do we know when this issue was introduced? cc
Maybe it was imported by this pr
We can reproduce this issue in unit testing.
The reported versions is and it seems that PR got backported further /priority important-soon
/assign
The original report looks like could have introduced it, yes. I wonder if we intentionally modify template without creating a copy (and as it seems, assign the exact same labels every time we use this template for any pod): Regardless, that was probably safe before the pod creations were parallelized. I don't think the proposed unit test catches the bug (more on that below), but we can probably reproduce it by timing pod state transitions so that many pods are simultaneously pending and missing a PVC. I ran this test on (before the cherry-pick of parallel statefulset scaling), and it also failed. This is because the test itself introduces a concurrent map write. It probably doesn't replicate the issue, and I don't think it can be merged, as it'll fail even if the original PR is reverted. For reproduction we'd need to catch a concurrent write without adding one in test code, just as it happened based on the original report.
Turns out we never tested this path at all, since the test PVC didn't have any labels of its own. After adding labels, existing tests are sufficient to detect race condition. PR showing that improved test detects the issue: PR with fix and improved test:
Sweet! thanks !

Discussion:
} { t.Run(tc.desc, func(t *testing.T) { set := burst(newStatefulSet(0)) <ins> set.Spec.VolumeClaimTemplates[0].ObjectMeta.Labels = map[string]string{"test": "test"} </ins> parallelScale(t, set, tc.replicas, tc.desiredReplicas, assertBurstInvariants) }) }

templates := set.Spec.VolumeClaimTemplates claims := make(map[string]v1.PersistentVolumeClaim, len(templates)) for i := range templates { <del> claim := templates[i] claim.Name = getPersistentVolumeClaimName(set, &claim, ordinal) </del> <ins> claim := templates[i].DeepCopy() claim.Name = getPersistentVolumeClaimName(set, claim, ordinal) </ins> claim.Namespace = set.Namespace if claim.Labels != nil { for key, value := range set.Spec.Selector.MatchLabels {

} else { claim.Labels = set.Spec.Selector.MatchLabels } <del> claims[templates[i].Name] = claim </del> <ins> claims[templates[i].Name] = *claim </ins> } return claims }