Code change:
From an e2e test failure: The command should retry resource version mismatches like scale and rollingupdate. cc
/cc fyi.
From perspective , Isn't the failure intentional (i.e. optimistic mutate) ? or... maybe I'm (probably) missing something. I would think its the user's job to retry... ?
The desired behavior that we settled on previously is to detect version mismatches at the client and error. User can force replacement using --force. See , and .
What is the mismatch here? If it is just resourceVersion, kubectl should retry. It should only bail on a meaningful conflict. We're working on eliminating flaky tests (and flaky parts of the system). Also, that log message isn't very useful.
per discussion on , we can't determine the correct behavior when the resource version changes without more information, so best to catch version mismatch at the client and bail, and let the user replace the object with --force if they want. We can also make the error message less verbose.
I don't see any mention of resourceVersion in . It shouldn't be serialized in the annotation, because the user shouldn't specify it in their config. The resourceVersion-based precondition is intended to guard against concurrent read-modify-write operations. kubectl apply should be able retry the entire diff&patch in the case of this type of conflict. That said, it's not obvious to me from the log message what the actual problem was in this case.
I'm sorry, my mistake. You're right that resourceVersion is not in the annotation, and that we should retry on resourceVersion failure. I was confusing this with the api version change issue discussed in . This is actually a much simpler issue, in the sense that it's an optimistic lock conflict. If we encounter a lock conflict, then we should fetch a fresh copy of the object, and try again to apply the new configuration supplied by the user. If we don't get a merge conflict when we resubmit, then all's well. If we get a merge conflict because the object has changed out from under the user, then we report it in those terms, and the user can decide what to do (e.g., replace using --force). Does this fit your expectation of how it should be handled?
This is causing test flake:
yes, your understanding is correct. Do you have time to fix this? We're trying to get all our tests fixed. Additionally this sounds like a legit system bug, not just a flaky test problem.
Maybe. Swamped right now, but can fit it in given enough notice. When is the fix needed?
asap. We'll find someone else.
This should just be a generic GET/Update retry loop, eg:
Go for it. Jack Greenfield | Google , Prashanth B wrote:
Chao, do you have cycles to take a look?
I thought Patch is retried in the server. I'll investigate.
Patch is retried at server-side five times before an error is returned: , so I expect this issue to occur very rarely. In fact, it has passed last ~400 kubernetes-gce-e2e tests. If we want to make this test less flaky, I'd recommend increase the number of the retries at the server-side. What do you guys think?
It'll fail if an actual conflicting change was made, though. Do we know what the nature of the conflict was? Maybe first we need to add some log messages to figure out what the conflict was next time it occurs. , Chao Xu wrote:
This happened last week:
you are right. I thought an actual conflicting change would result in a different error message but it's not. I'll add the logs.
In I a log entry if there is a meaningful conflict. That said, I don't think we have a meaningful conflict in this particular e2e test. After taking a closer look at the implementation of , I think the submitted is precise, so we shouldn't retry at the client side if server reports a meaningful conflict. [edit] And if the conflict is not meaningful, we should just rely on the server-side patch handler to retry.
does the milestone on this need to be changed to v1.2?
How often is the test flaking?
Last time I checked, I'll check it again.
Last time I checked the results was two days ago, in the past 48 hours, the test has consecutively passed another 124 builds ( to ).
I only see 3 reports of failures since October, so I'm going to say, no, it doesn't need to be fixed in 1.2.
any progress on this now? I returned back to the kubectl apply issues, and hope I can fix all apply related issues before 1.3.0 release.
I'll try to check the flake rate again. IMO if the flake rate is non-negligible, we should increase the retry limits of Patch, or more aggressively, just remove the limit.
Infinite retry still scares me, mostly because we may have failure modes that could race for a very long time. , Chao Xu wrote:
I don't think anyone has worked on this. Though the original failing test/logs weren't recorded, it appears to be a failure of "should apply a new configuration to an existing RC": There is a retry loop in the Patch handler in apiserver, but if resourceVersion is specified, the retry loop will always fail in the case of a conflict. (We should detect that and bail out, but I don't think we do.) In the case of such a conflict, apply should diff again, create another patch, and try again. I agree with that it shouldn't retry forever. We should use the same retry limits we use for other commands.
we have an infinite loop in GuaranteedUpdate (), maybe we should apply a retry limit there as well?
Agree. Just note that the test flake doesn't specify resourceVersion.
Since this is causing flakey test failures, it is something we should focus on. Is this something I should pick up?
FYI: has a PR out for this under review
Making this a P3 (no release block) since I looks like we haven't seen many instances since Feb 2016

Related discussion:
text: return nil } <ins> type patcher struct { encoder runtime.Encoder decoder runtime.Decoder mapping *meta.RESTMapping helper  *resource.Helper backOff clockwork.Clock } func NewPatcher(encoder runtime.Encoder, decoder runtime.Decoder, mapping *meta.RESTMapping, helper *resource.Helper) *patcher { return &patcher{ encoder: encoder, decoder: decoder, mapping: mapping, helper:  helper, backOff: clockwork.NewRealClock(), } } func (p *patcher) patchSimple(obj runtime.Object, modified []byte, source, namespace, name string) ([]byte, error) { // Serialize the current configuration of the object from the server. current, err := runtime.Encode(p.encoder, obj) if err != nil { return nil, cmdutil.AddSourceToErr(fmt.Sprintf("serializing current configuration from:n%vnfor:", obj), source, err) } // Retrieve the original configuration of the object from the annotation. original, err := kubectl.GetOriginalConfiguration(p.mapping, obj) if err != nil { return nil, cmdutil.AddSourceToErr(fmt.Sprintf("retrieving original configuration from:n%vnfor:", obj), source, err) } // Create the versioned struct from the original from the server for // strategic patch. // TODO: Move all structs in apply to use raw data. Can be done once // builder has a RawResult method which delivers raw data instead of // internal objects. versionedObject, _, err := p.decoder.Decode(current, nil, nil) if err != nil { return nil, cmdutil.AddSourceToErr(fmt.Sprintf("converting encoded server-side object back to versioned struct:n%vnfor:", obj), source, err) } // Compute a three way strategic merge patch to send to server. patch, err := strategicpatch.CreateThreeWayMergePatch(original, modified, current, versionedObject, true) if err != nil { format := "creating patch with:noriginal:n%snmodified:n%sncurrent:n%snfor:" return nil, cmdutil.AddSourceToErr(fmt.Sprintf(format, original, modified, current), source, err) } _, err = p.helper.Patch(namespace, name, api.StrategicMergePatchType, patch) return patch, err } func (p *patcher) patch(current runtime.Object, modified []byte, source, namespace, name string) ([]byte, error) { var getErr error patchBytes, err := p.patchSimple(current, modified, source, namespace, name) for i := 1; i <= maxPatchRetry && errors.IsConflict(err); i++ { if i > triesBeforeBackOff { p.backOff.Sleep(backOffPeriod) } current, getErr = p.helper.Get(namespace, name, false) if getErr != nil { return nil, getErr } patchBytes, err = p.patchSimple(current, modified, source, namespace, name) } return patchBytes, err } </ins>