Change:
Which jobs are failing: pull-kubernetes-integration Which test(s) are failing: https://k8s- Not showing up in triage, oddly, but seen in individual failures: not sure if this specific to custom resources, but that's where I'm seeing the flakes /sig api-machinery /priority important-soon /area custom-resources
adding to the milestone until root-caused
/cc
I suspect there is some cache/race that keeps the old CR information around during the update from the CR that fails UID check during an update etcd log: few read (other other version defined by CRD. the port difference indicates a different client) the timestamp, the failure happened after the write a successful CR from another test in that file (TestValidateOnlyStatus, TestGeneration, etc) (different etcd key prefix from the same etcd log) few read (update) few read (the second version. I'm surprised that the port didn't change) few read
leaving in the milestone for visibility as we root cause whether this is a test-only issue or not
I spent a while trying to recreate this and could not (again). I made one of the tests hotloop with QPS 100 doing get/update of the status subresource, then ran that in parallel with the other integration tests in the package. No failures found after running for ~10 minutes. Updated description with testgrid link to locate recent flakes (https://k8s-), found a couple in the last week.
iiuc, the error comes from apiserver the UID against the . The requested obj is up-to-date since it was from etcd, but I didn't see mechanism that guarantees the watch cache to be up-to-date after a "delete-create" or "delete-create-get" sequence (the cacher implementation for Delete, Create and Get(rv="") go through etcd directly). Is this failure an expected behavior, and should we just retry on precondition failure?
the triage page seems to be stale, filed
talked with offline, we get it reproduced by adding some delay to the watch cache: as suggested preconditions should retry on stale watch data instead of surfacing such error, this is not a test issue. I will send a fix

Discussion:
} } <ins> func TestPreconditionalUpdateWithCachedObject(t *testing.T) { podName := "foo" pod := &example.Pod{ ObjectMeta: metav1.ObjectMeta{Name: podName}, Spec:       example.PodSpec{NodeName: "machine"}, } ctx := genericapirequest.WithNamespace(genericapirequest.NewContext(), "test") destroyFunc, registry := newTestGenericStoreRegistry(t, scheme, false) defer destroyFunc() // cached object has old UID oldPod, err := registry.Create(ctx, pod, rest.ValidateAllObjectFunc, &metav1.CreateOptions{}) if err != nil { t.Fatal(err) } registry.Storage.Storage = &staleGuaranteedUpdateStorage{Interface: registry.Storage.Storage, cachedObj: oldPod} // delete and re-create the same object with new UID _, _, err = registry.Delete(ctx, podName, rest.ValidateAllObjectFunc, nil) if err != nil { t.Fatal(err) } obj, err := registry.Create(ctx, pod, rest.ValidateAllObjectFunc, &metav1.CreateOptions{}) if err != nil { t.Fatal(err) } newPod, ok := obj.(*example.Pod) if !ok { t.Fatalf("unexpected object: %#v", obj) } // update the object should not fail precondition newPod.Spec.NodeName = "machine2" res, _, err := registry.Update(ctx, podName, rest.DefaultUpdatedObjectInfo(newPod), rest.ValidateAllObjectFunc, rest.ValidateAllObjectUpdateFunc, false, &metav1.UpdateOptions{}) if err != nil { t.Fatal(err) } // the update should have succeeded r, ok := res.(*example.Pod) if !ok { t.Fatalf("unexpected update result: %#v", res) } if r.Spec.NodeName != "machine2" { t.Fatalf("unexpected, update didn't take effect: %#v", r) } } </ins> // TestRetryDeleteValidation checks if the deleteValidation is called again if // the GuaranteedUpdate in the Delete handler conflicts with a simultaneous // Update.

transformContext := authenticatedDataString(key) for { if err := preconditions.Check(key, origState.obj); err != nil { <del> return err </del> <ins> // If our data is already up to date, return the error if !mustCheckData { return err } // It's possible we were working with stale data // Actually fetch origState, err = getCurrentState() if err != nil { return err } mustCheckData = false // Retry continue </ins> } ret, ttl, err := s.updateState(origState, tryUpdate)