Code change:
We use a to nil out the : However on a re-sync, this results in a data race that was observed when running unit tests: The read had occurred previously during a callback. On re-sync, the retrieves every object from the cache, invokes the transform function, and re-queues them. However, the object instances are the same as those stored in the cache so mutating them outside the cache without lock protection is unsafe. It is documented that the "TransformFunc sees the object before any other actor, and it is now safe to mutate the object in place instead of making a copy" however this is not the case on a re-sync. The should make a copy of the objects retrieved from the cache store on a re-sync or perhaps elide the on a re-sync since the objects would already have been transformed. As a workaround, I changed our to only nil the if not already nil but this is a latent bug that others will likely run into.
/assign That means that and introduced possible panic regressions into kube-controller-manager and kube-scheduler /priority critical-urgent /kind regression
/transfer kubernetes
/sig api-machinery
kube-controller-manager and kube-scheduler do the same in 1.29+ and likely have the same issue
I think delete has the same problem, and replace may have the same problem: Passing the same object to the transformer after it is already in the cache is unsafe and we should not do that at all.
/assign /cc
Looking into this. I may take time to come up with a proper fix. But would like to evaluate the risk for the current situation. Not entirely sure how delete and replace would be triggered in normal settings, but resync is fortunately low frequency as the default setting for resync period is between 12h and 24h for (--min-resync-period duration Default: 12h0m0s, see ) . As for the resync function is Just think out aloud, I read a thread discussion somewhere that resync was a mechanism in the past to prevent data inconsistency due to bugs but client-go has been mature these days, resync is no-longer needed in theory but for a safe guard. So maybe disabling resync is a reasonable fix if delete/replace can be fixed.
You're mixing resync with relist - I believe relist is no longer needed. Resync is actually used. FWIW - I opened an initial version of the fix:
I think it's resync. The thread I mentioned is
/triage accepted

Related discussion:
text: close(stopCh) } <ins> func TestTransformingInformerRace(t *testing.T) { // source simulates an apiserver object endpoint. source := fcache.NewFakeControllerSource() label := "to-be-transformed" makePod := func(name string) *v1.Pod { return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      name, Namespace: "namespace", Labels:    map[string]string{label: "true"}, }, Spec: v1.PodSpec{ Hostname: "hostname", }, } } badTransform := atomic.Bool{} podTransformer := func(obj interface{}) (interface{}, error) { pod, ok := obj.(*v1.Pod) if !ok { return nil, fmt.Errorf("unexpected object type: %T", obj) } if pod.ObjectMeta.Labels[label] != "true" { badTransform.Store(true) return nil, fmt.Errorf("object already transformed: %#v", obj) } pod.ObjectMeta.Labels[label] = "false" return pod, nil } numObjs := 5 for i := 0; i < numObjs; i++ { source.Add(makePod(fmt.Sprintf("pod-%d", i))) } type event struct{} events := make(chan event, numObjs) recordEvent := func(eventType watch.EventType, previous, current interface{}) { events <- event{} } checkEvents := func(count int) { for i := 0; i < count; i++ { <-events } } store, controller := NewTransformingInformer( source, &v1.Pod{}, 5*time.Millisecond, ResourceEventHandlerDetailedFuncs{ AddFunc:    func(obj interface{}, isInInitialList bool) { recordEvent(watch.Added, nil, obj) }, UpdateFunc: func(oldObj, newObj interface{}) { recordEvent(watch.Modified, oldObj, newObj) }, DeleteFunc: func(obj interface{}) { recordEvent(watch.Deleted, obj, nil) }, }, podTransformer, ) stopCh := make(chan struct{}) go controller.Run(stopCh) checkEvents(numObjs) // Periodically fetch objects to ensure no access races. wg := sync.WaitGroup{} errors := make(chan error, numObjs) for i := 0; i < numObjs; i++ { wg.Add(1) go func(index int) { defer wg.Done() key := fmt.Sprintf("namespace/pod-%d", index) for { select { case <-stopCh: return default: } obj, ok, err := store.GetByKey(key) if !ok || err != nil { errors <- fmt.Errorf("couldn't get the object for %v", key) return } pod := obj.(*v1.Pod) if pod.ObjectMeta.Labels[label] != "false" { errors <- fmt.Errorf("unexpected object: %#v", pod) return } } }(i) } // Let resyncs to happen for some time. time.Sleep(time.Second) close(stopCh) wg.Wait() close(errors) for err := range errors { t.Error(err) } if badTransform.Load() { t.Errorf("unexpected transformation happened") } } </ins> func TestDeletionHandlingObjectToName(t *testing.T) { cm := &v1.ConfigMap{ ObjectMeta: metav1.ObjectMeta{