Change:
ctx context.Context, key string, destination runtime.Object, ignoreNotFound bool, preconditions *storage.Preconditions, tryUpdate storage.UpdateFunc, dryRun bool, cachedExistingObject runtime.Object) error { if dryRun { <del> err := s.Storage.Get(ctx, key, storage.GetOptions{IgnoreNotFound: ignoreNotFound}, destination) </del> <ins> var current runtime.Object v, err := conversion.EnforcePtr(destination) if err != nil { return fmt.Errorf("unable to convert output object to pointer: %v", err) } if u, ok := v.Addr().Interface().(runtime.Unstructured); ok { current = u.NewEmptyInstance() } else { current = reflect.New(v.Type()).Interface().(runtime.Object) } err = s.Storage.Get(ctx, key, storage.GetOptions{IgnoreNotFound: ignoreNotFound}, current) </ins> if err != nil { return err } <del> err = preconditions.Check(key, destination) </del> <ins> err = preconditions.Check(key, current) </ins> if err != nil { return err } <del> rev, err := s.Versioner().ObjectResourceVersion(destination) </del> <ins> rev, err := s.Versioner().ObjectResourceVersion(current) </ins> if err != nil { return err } <del> updated, _, err := tryUpdate(destination, storage.ResponseMeta{ResourceVersion: rev}) </del> <ins> updated, _, err := tryUpdate(current, storage.ResponseMeta{ResourceVersion: rev}) </ins> if err != nil { return err }

Discussion:
Apparently, the bug happens in 1.26.3 but not in 1.27.1 so maybe it's been fixed somewhere. Here are the steps to reproduce: Note that the metrics list is atomic, so the list should be entirely replaced every-time, not merged/appended. /wg api-expression /triage accepted /priority important-soon
I tried to add a printf of the json'd merged object right before leaving the ssa code, and the field isn't duplicated. So something else must be trying to merge somehow!?
I also checked that it's not kubectl doing anything crazy, the response from the server includes the duplicated field.
Checked both before and after admission, object seems to be properly merged.
Ah, something not mentioned in the bug, applying without dry-run doesn't trigger the same behavior.
storage version changed to v2 in 1.27 () HPA conversion is really weird (taking v2 fields and putting them in annotations in v1, then shipping annotations back into appended items when converting back to v2)... I'd suspect something in the storage → conversion path
Right, the exact conversions that we do are different in dry-run vs non-dry-run because we have to storage version. I'm still investigating the conversion path to see if anything seems shady and I've made progress on that.
OK, this is the trace that leads to the internal HorizontalPodAutoscaler object to have the metrics field duplicated is this one: So definitely in the path to dry-run. The following test reproduces the error, as it fails with two identical metrics. Problem is probably that 1. the "out" object isn't cleared before for dry-run. 2. conversion doesn't detect the duplication/remove previous entry from out. I'll check to see how the non-dry-run version clears the out object.
Did anyone find a workaround for this bug? Currently trying to compare equality between two objects after a dry-run to populate default fields.