Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We recently had a production issue where one of our large cluster (~3k nodes) has it's API server OOMKilled. This caused all kubelets' connection to master broken, and caused a global synchronization of kubelet's reconnecting to master. Due to the burst of API call (18K QPS, 7x than normal), API server went into even worse condition, and caused more replicas to get OOMKilled. Current reflector impl has fixed 1sec backoff before calling next , which shall be the root cause of such instability. What you expected to happen: reflectors to backoff with jitter How to reproduce it (as minimally and precisely as possible): a cluster with a few thousands of nodes api server and reboot it Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign /sig api-machinery scalability
/cc

Discussion:
store Store // listerWatcher is used to perform lists and watches. listerWatcher ListerWatcher <del> // period controls timing between an unsuccessful watch ending and // the beginning of the next list. period time.Duration // The period at which ShouldResync is invoked </del> <ins> // backoff manages backoff of ListWatch backoffManager wait.BackoffManager </ins> resyncPeriod time.Duration // ShouldResync is invoked periodically and whenever it returns `true` the Store's Resync operation is invoked ShouldResync func() bool

// NewNamedReflector same as NewReflector, but with a specified name for logging func NewNamedReflector(name string, lw ListerWatcher, expectedType interface{}, store Store, resyncPeriod time.Duration) *Reflector { <ins> realClock := &clock.RealClock{} </ins> r := &Reflector{ name:          name, listerWatcher: lw, store:         store, <del> period:        time.Second, resyncPeriod:  resyncPeriod, clock:         &clock.RealClock{}, </del> <ins> // We used to make the call every 1sec (1 QPS), the goal here is to achieve ~98% traffic reduction when // API server is not healthy. With these parameters, backoff will stop at [30,60) sec interval which is // 0.22 QPS. If we don't backoff for 2min, assume API server is healthy and we reset the backoff. backoffManager: wait.NewExponentialBackoffManager(800*time.Millisecond, 30*time.Second, 2*time.Minute, 2.0, 1.0, realClock), resyncPeriod:   resyncPeriod, clock:          realClock, </ins> } r.setExpectedType(expectedType) return r

// objects and subsequent deltas. // Run will exit when stopCh is closed. func (r *Reflector) Run(stopCh <-chan struct{}) { <del> klog.V(3).Infof("Starting reflector %v (%s) from %s", r.expectedTypeName, r.resyncPeriod, r.name) wait.Until(func() { </del> <ins> klog.V(2).Infof("Starting reflector %s (%s) from %s", r.expectedTypeName, r.resyncPeriod, r.name) wait.BackoffUntil(func() { </ins> if err := r.ListAndWatch(stopCh); err != nil { utilruntime.HandleError(err) } <del> }, r.period, stopCh) </del> <ins> }, r.backoffManager, true, stopCh) klog.V(2).Infof("Stopping reflector %s (%s) from %s", r.expectedTypeName, r.resyncPeriod, r.name) </ins> } var (