Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Almost every update to a DaemonSet can lead to 2~3 error logs in kube-controller-manager even when the DaemonSet manages very few Pods, for example, there were 3 error logs when updating a daemonset in a small cluster of 2 Nodes. Other controllers like deployment don't have this issue when they manage much more Pods. After looking at DaemonSet controller code, it seems the retry logic in "storeDaemonSetStatus" was not implemented correctly. It never really retried and always made an useless GET call. DaemonSet controller should produce less error logs in normal case. Create or update a DaemonSet to trigger Pod recreation. Kubernetes version (use ): v1.20.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps /assign created to fix the retry logic.
Same problem with me. </brCould you explain why other controllers(like deployment controller) is not easy to get conflict when updating status? I think there are many in every controller.
the number of workers doesn't matter here as the workqueue guarantees that they will not process the same resource at the same time. The conflict comes from multiple successive processing of a resource because of the update delay of the resource's cache in . For example, when a DaemonSet manages 2 Pods and they are getting ready. first Pod is ready and triggers updating Daemonset, whose resourceVersion is in Lister. updates DaemonSet's ready count to 1 with resourceVersion set to and it succeeds. The resourceVersion in server becomes and the Informer/Lister hasn't received the update. second Pod is ready and triggers updating Daemonset, whose resourceVersion is still in Lister. updates DaemonSet's ready count to 2 with resourceVersion set to and it fails with the "the object has been modified" error. To avoid this error, retry is the most straightforward way but it is implemented differently in controllers. In deployment controller, it actually has the same conflict but it only logs the error with so normally you don't see it, and it requeues the resource and re-processes it upon conflict. DaemonSet controller and Replicaset controller retry directly when calling fails. However, the former made a mistake when checking the loop's boundary condition and it never retried. is fixing it.
Very clear explanation. Thanks.

Discussion:
toUpdate := ds.DeepCopy() var updateErr, getErr error <del> for i := 0; i < StatusUpdateRetries; i++ { </del> <ins> for i := 0; ; i++ { </ins> if updateObservedGen { toUpdate.Status.ObservedGeneration = ds.Generation }

return nil } <ins> // Stop retrying if we exceed statusUpdateRetries - the DaemonSet will be requeued with a rate limit. if i >= StatusUpdateRetries { break } </ins> // Update the set with the latest resource version for the next poll if toUpdate, getErr = dsClient.Get(context.TODO(), ds.Name, metav1.GetOptions{}); getErr != nil { // If the GET fails we can't trust status.Replicas anymore. This error

t.Errorf("unexpected deletesnexpected: %vn  actual: %v", expected.List(), actual.List()) } } <ins> func TestStoreDaemonSetStatus(t *testing.T) { getError := fmt.Errorf("fake get error") updateError := fmt.Errorf("fake update error") tests := []struct { name                 string updateErrorNum       int getErrorNum          int expectedUpdateCalled int expectedGetCalled    int expectedError        error }{ { name:                 "succeed immediately", updateErrorNum:       0, getErrorNum:          0, expectedUpdateCalled: 1, expectedGetCalled:    0, expectedError:        nil, }, { name:                 "succeed after one update failure", updateErrorNum:       1, getErrorNum:          0, expectedUpdateCalled: 2, expectedGetCalled:    1, expectedError:        nil, }, { name:                 "fail after two update failures", updateErrorNum:       2, getErrorNum:          0, expectedUpdateCalled: 2, expectedGetCalled:    1, expectedError:        updateError, }, { name:                 "fail after one update failure and one get failure", updateErrorNum:       1, getErrorNum:          1, expectedUpdateCalled: 1, expectedGetCalled:    1, expectedError:        getError, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { ds := newDaemonSet("foo") fakeClient := &fake.Clientset{} getCalled := 0 fakeClient.AddReactor("get", "daemonsets", func(action core.Action) (bool, runtime.Object, error) { getCalled += 1 if getCalled <= tt.getErrorNum { return true, nil, getError } return true, ds, nil }) updateCalled := 0 fakeClient.AddReactor("update", "daemonsets", func(action core.Action) (bool, runtime.Object, error) { updateCalled += 1 if updateCalled <= tt.updateErrorNum { return true, nil, updateError } return true, ds, nil }) if err := storeDaemonSetStatus(fakeClient.AppsV1().DaemonSets("default"), ds, 2, 2, 2, 2, 2, 2, 2, true); err != tt.expectedError { t.Errorf("storeDaemonSetStatus() got %v, expected %v", err, tt.expectedError) } if getCalled != tt.expectedGetCalled { t.Errorf("Get() was called %v times, expected %v times", getCalled, tt.expectedGetCalled) } if updateCalled != tt.expectedUpdateCalled { t.Errorf("UpdateStatus() was called %v times, expected %v times", updateCalled, tt.expectedUpdateCalled) } }) } } </ins>