Code change:
We use a to nil out the : However on a re-sync, this results in a data race that was observed when running unit tests: The read had occurred previously during a callback. On re-sync, the retrieves every object from the cache, invokes the transform function, and re-queues them. However, the object instances are the same as those stored in the cache so mutating them outside the cache without lock protection is unsafe. It is documented that the "TransformFunc sees the object before any other actor, and it is now safe to mutate the object in place instead of making a copy" however this is not the case on a re-sync. The should make a copy of the objects retrieved from the cache store on a re-sync or perhaps elide the on a re-sync since the objects would already have been transformed. As a workaround, I changed our to only nil the if not already nil but this is a latent bug that others will likely run into.
/assign That means that and introduced possible panic regressions into kube-controller-manager and kube-scheduler /priority critical-urgent /kind regression
/transfer kubernetes
/sig api-machinery
kube-controller-manager and kube-scheduler do the same in 1.29+ and likely have the same issue
I think delete has the same problem, and replace may have the same problem: Passing the same object to the transformer after it is already in the cache is unsafe and we should not do that at all.
/assign /cc
Looking into this. I may take time to come up with a proper fix. But would like to evaluate the risk for the current situation. Not entirely sure how delete and replace would be triggered in normal settings, but resync is fortunately low frequency as the default setting for resync period is between 12h and 24h for (--min-resync-period duration Default: 12h0m0s, see ) . As for the resync function is Just think out aloud, I read a thread discussion somewhere that resync was a mechanism in the past to prevent data inconsistency due to bugs but client-go has been mature these days, resync is no-longer needed in theory but for a safe guard. So maybe disabling resync is a reasonable fix if delete/replace can be fixed.
You're mixing resync with relist - I believe relist is no longer needed. Resync is actually used. FWIW - I opened an initial version of the fix:
I think it's resync. The thread I mentioned is
/triage accepted

Related discussion:
text: // queueActionLocked appends to the delta list for the object. // Caller must lock first. func (f *DeltaFIFO) queueActionLocked(actionType DeltaType, obj interface{}) error { <ins> return f.queueActionInternalLocked(actionType, actionType, obj) } // queueActionInternalLocked appends to the delta list for the object. // The actionType is emitted and must honor emitDeltaTypeReplaced. // The internalActionType is only used within this function and must // ignore emitDeltaTypeReplaced. // Caller must lock first. func (f *DeltaFIFO) queueActionInternalLocked(actionType, internalActionType DeltaType, obj interface{}) error { </ins> id, err := f.KeyOf(obj) if err != nil { return KeyError{obj, err} } // Every object comes through this code path once, so this is a good <del> // place to call the transform func.  If obj is a // DeletedFinalStateUnknown tombstone, then the containted inner object // will already have gone through the transformer, but we document that // this can happen. In cases involving Replace(), such an object can // come through multiple times. </del> <ins> // place to call the transform func. // // If obj is a DeletedFinalStateUnknown tombstone or the action is a Sync, // then the object have already gone through the transformer. // // If the objects already present in the cache are passed to Replace(), // the transformer must be idempotent to avoid re-mutating them, // or coordinate with all readers from the cache to avoid data races. // Default informers do not pass existing objects to Replace. </ins> if f.transformer != nil { <del> var err error obj, err = f.transformer(obj) if err != nil { return err </del> <ins> _, isTombstone := obj.(DeletedFinalStateUnknown) if !isTombstone && internalActionType != Sync { var err error obj, err = f.transformer(obj) if err != nil { return err } </ins> } }