Code change:
We use a to nil out the : However on a re-sync, this results in a data race that was observed when running unit tests: The read had occurred previously during a callback. On re-sync, the retrieves every object from the cache, invokes the transform function, and re-queues them. However, the object instances are the same as those stored in the cache so mutating them outside the cache without lock protection is unsafe. It is documented that the "TransformFunc sees the object before any other actor, and it is now safe to mutate the object in place instead of making a copy" however this is not the case on a re-sync. The should make a copy of the objects retrieved from the cache store on a re-sync or perhaps elide the on a re-sync since the objects would already have been transformed. As a workaround, I changed our to only nil the if not already nil but this is a latent bug that others will likely run into.
/assign That means that and introduced possible panic regressions into kube-controller-manager and kube-scheduler /priority critical-urgent /kind regression
/transfer kubernetes
/sig api-machinery
kube-controller-manager and kube-scheduler do the same in 1.29+ and likely have the same issue
I think delete has the same problem, and replace may have the same problem: Passing the same object to the transformer after it is already in the cache is unsafe and we should not do that at all.
/assign /cc
Looking into this. I may take time to come up with a proper fix. But would like to evaluate the risk for the current situation. Not entirely sure how delete and replace would be triggered in normal settings, but resync is fortunately low frequency as the default setting for resync period is between 12h and 24h for (--min-resync-period duration Default: 12h0m0s, see ) . As for the resync function is Just think out aloud, I read a thread discussion somewhere that resync was a mechanism in the past to prevent data inconsistency due to bugs but client-go has been mature these days, resync is no-longer needed in theory but for a safe guard. So maybe disabling resync is a reasonable fix if delete/replace can be fixed.
You're mixing resync with relist - I believe relist is no longer needed. Resync is actually used. FWIW - I opened an initial version of the fix:
I think it's resync. The thread I mentioned is
/triage accepted

Related discussion:
text: } // TransformFunc allows for transforming an object before it will be processed. <del> // TransformFunc (similarly to ResourceEventHandler functions) should be able // to correctly handle the tombstone of type cache.DeletedFinalStateUnknown. // // New in v1.27: In such cases, the contained object will already have gone // through the transform object separately (when it was added / updated prior // to the delete), so the TransformFunc can likely safely ignore such objects // (i.e., just return the input object). </del> // // The most common usage pattern is to clean-up some parts of the object to // reduce component memory usage if a given component doesn't care about them. // <del> // New in v1.27: unless the object is a DeletedFinalStateUnknown, TransformFunc // sees the object before any other actor, and it is now safe to mutate the // object in place instead of making a copy. </del> <ins> // New in v1.27: TransformFunc sees the object before any other actor, and it // is now safe to mutate the object in place instead of making a copy. // // It's recommended for the TransformFunc to be idempotent. // It MUST be idempotent if objects already present in the cache are passed to // the Replace() to avoid re-mutating them. Default informers do not pass // existing objects to Replace though. </ins> // // Note that TransformFunc is called while inserting objects into the // notification queue and is therefore extremely performance sensitive; please