Change:
Currently, reflector uses two backoff managers: One is used for initial list call and other for watch call. Imagine following scenaraion when k8s apiserver is under very heavy load: the beginning, reflector spams k8s apiserver (with exponential backoff) with calls until one of them succeeds (due to APF throtling and receiving ) tries to run request and uses separate backoffManager - so again spams k8s apiserver with requests Watch does not succeed during Etcd compaction window (2.5 - 5 minutes usually) at some point it will receive and go back to point 1, spamming k8s apiserver again with List calls (because first backoffManager for List got reseted after 2 minutes of retries for Watch requests) I propose we combine these two backoff managers into one. If we combine these two backoffManagers into one, it will work like this: At the beginning, reflector spams k8s apiserver (still with exponential backoff) with calls until one of them succeeds Tries to make first watch call instantly (as there is no backoff on this path) If it fails, it will makre retries every (30, 60)s If watch request fails all the times during Etcd compaction window it will go back to List call, but already being throttled with (30, 60)s window. /sig scalability
/cc
/assign
I think the proposed solution makes sense to me. Probably before we start, we should to understand why we ended up with two back off managers by looking into PRs touching them. Maybe the code authors had some specific case in mind.
for thoughts too I agree it makes sense, but I don't know the historical context here.
/sig api-machinery
/triage accepted

Discussion:
listerWatcher ListerWatcher // backoff manages backoff of ListWatch backoffManager wait.BackoffManager <del> // initConnBackoffManager manages backoff the initial connection with the Watch call of ListAndWatch. initConnBackoffManager wait.BackoffManager resyncPeriod           time.Duration </del> <ins> resyncPeriod   time.Duration </ins> // clock allows tests to manipulate time clock clock.Clock // paginatedResult defines whether pagination should be forced for list calls.

// We used to make the call every 1sec (1 QPS), the goal here is to achieve ~98% traffic reduction when // API server is not healthy. With these parameters, backoff will stop at [30,60) sec interval which is // 0.22 QPS. If we don't backoff for 2min, assume API server is healthy and we reset the backoff. <del> backoffManager:         wait.NewExponentialBackoffManager(800*time.Millisecond, 30*time.Second, 2*time.Minute, 2.0, 1.0, reflectorClock), initConnBackoffManager: wait.NewExponentialBackoffManager(800*time.Millisecond, 30*time.Second, 2*time.Minute, 2.0, 1.0, reflectorClock), clock:                  reflectorClock, watchErrorHandler:      WatchErrorHandler(DefaultWatchErrorHandler), expectedType:           reflect.TypeOf(expectedType), </del> <ins> backoffManager:    wait.NewExponentialBackoffManager(800*time.Millisecond, 30*time.Second, 2*time.Minute, 2.0, 1.0, reflectorClock), clock:             reflectorClock, watchErrorHandler: WatchErrorHandler(DefaultWatchErrorHandler), expectedType:      reflect.TypeOf(expectedType), </ins> } if r.name == "" {

select { case <-stopCh: return nil <del> case <-r.initConnBackoffManager.Backoff().C(): </del> <ins> case <-r.backoffManager.Backoff().C(): </ins> continue } }

select { case <-stopCh: return nil <del> case <-r.initConnBackoffManager.Backoff().C(): </del> <ins> case <-r.backoffManager.Backoff().C(): </ins> continue } case apierrors.IsInternalError(err) && retry.ShouldRetry():

isErrorRetriableWithSideEffectsFn := func(err error) bool { if canRetry := isWatchErrorRetriable(err); canRetry { klog.V(2).Infof("%s: watch-list of %v returned %v - backing off", r.name, r.typeDescription, err) <del> <-r.initConnBackoffManager.Backoff().C() </del> <ins> <-r.backoffManager.Backoff().C() </ins> return true } if isExpiredError(err) || isTooLargeResourceVersionError(err) {

}, } r := &Reflector{ <del> name:                   "test-reflector", listerWatcher:          lw, store:                  NewFIFO(MetaNamespaceKeyFunc), initConnBackoffManager: bm, clock:                  fakeClock, watchErrorHandler:      WatchErrorHandler(DefaultWatchErrorHandler), </del> <ins> name:              "test-reflector", listerWatcher:     lw, store:             NewFIFO(MetaNamespaceKeyFunc), backoffManager:    bm, clock:             fakeClock, watchErrorHandler: WatchErrorHandler(DefaultWatchErrorHandler), </ins> } start := fakeClock.Now() err := r.ListAndWatch(stopCh)

} r := &Reflector{ <del> name:                   "test-reflector", listerWatcher:          lw, store:                  NewFIFO(MetaNamespaceKeyFunc), initConnBackoffManager: bm, clock:                  clock, watchErrorHandler:      WatchErrorHandler(DefaultWatchErrorHandler), </del> <ins> name:              "test-reflector", listerWatcher:     lw, store:             NewFIFO(MetaNamespaceKeyFunc), backoffManager:    bm, clock:             clock, watchErrorHandler: WatchErrorHandler(DefaultWatchErrorHandler), </ins> } stopCh := make(chan struct{})

} r := &Reflector{ <del> name:                   "test-reflector", listerWatcher:          lw, store:                  NewFIFO(MetaNamespaceKeyFunc), initConnBackoffManager: bm, clock:                  fakeClock, watchErrorHandler:      WatchErrorHandler(DefaultWatchErrorHandler), </del> <ins> name:              "test-reflector", listerWatcher:     lw, store:             NewFIFO(MetaNamespaceKeyFunc), backoffManager:    bm, clock:             fakeClock, watchErrorHandler: WatchErrorHandler(DefaultWatchErrorHandler), </ins> } r.MaxInternalErrorRetryDuration = tc.maxInternalDuration