Change:
continue } delete(f.items, id) <ins> // Only log traces if the queue depth is greater than 10 and it takes more than // 100 milliseconds to process one item from the queue. // Queue depth never goes high because processing an item is locking the queue, // and new items can't be added until processing finish. // https://github.com/kubernetes/kubernetes/issues/103789 if depth > 10 { trace := utiltrace.New("DeltaFIFO Pop Process", utiltrace.Field{Key: "ID", Value: id}, utiltrace.Field{Key: "Depth", Value: depth}, utiltrace.Field{Key: "Reason", Value: "slow event handlers blocking the queue"}) defer trace.LogIfLong(100 * time.Millisecond) } </ins> err := process(item) if e, ok := err.(ErrRequeue); ok { f.addIfNotPresent(id, item)

Discussion:
I am using client-go/tools/cache package NewInformer, find the StreamWatcher and seeing extremely high memory allocations. Attached is a memory profile: From this we have 2.3 GB of memory allocated by the StreamWatcher. To get to this state I had a script to create pod and delete pod over and over again, and every 10 seconds I run it again. I left this running 30 minutes and saw this memory usage. watch code as follows, cache.NewInformer return controller and storewriter is chan queue, receiving event and distribute to queue
/sig api-machinery
Thanks for taking a look
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
i find that infomer handler be stuck will lead to streamWatcher memory high allocationsdon't hang informer handler and you will resolve that question.
/assign /triage accepted
can you check with a cpu profile if some handler is blocking?
cpu profile is normalas followscode callback handler func (w *Watcher) AddEvent(obj interface{}) { data := w.genSyncData(obj, action.SyncDataActionAdd) if data == nil { return } w.writer.Sync(data) } writer is a buffer chan with length and write chan timeout for one secondprobably is will be stuck when produce lots of eventsfinallyI store events to fifo queue when it produce lots of eventsand consume fifo queue event to writer chanthe issue is solved
So, the problem is that your handlers are slower processing the objects from the delta fifo queue, than the rate at which the events are obtained from the apiserver, this is because of this objects are processed by the handlers locking the deltaFifo queue, hence the streamwatcher accumulates the object in memory until the handlers process the object and Pop() release the lock. The code comments inform about this problem // Add() and Get(), won't be blocked for too long. the only solution I see is logging some errors warning about this (processing withouth locking doesn't seem an option, right?)

I've tried to use queue depth, but that is not an option, since the contention happens before events get into the queue and it nevers goes high. I've tried adding trace for slow handlers operations, however, those maybe harmless if the incoming rate is still lower than the draining rate, unless we decide that we want to trace process handlers that take more than X seconds Is adding another intermediate queue between the streamwatecher and the deltaFifo an option? that will free the streamwatchers ...
Yeah. I don't see it as a high priority, but that could be helpful. Following the standard hander to workqueue pattern, it isn't a practical problem. /priority backlog
i don't hang informer hander by fifo queue, i find that memory will come down when lots of create and delete podsbut i running code onlinethe streamwatcher still exist high memory usage as follows pprof Please take a look at it
i am also facing same issue any guidance for the watcher memory increase in high pod churn