Change:
} } <ins> func TestRetryInternalError(t *testing.T) { testCases := []struct { name                string maxInternalDuration time.Duration rewindTime          int wantRetries         int }{ { name:                "retries off", maxInternalDuration: time.Duration(0), wantRetries:         0, }, { name:                "retries on, all calls fail", maxInternalDuration: time.Second * 30, wantRetries:         31, }, { name:                "retries on, one call successful", maxInternalDuration: time.Second * 30, rewindTime:          10, wantRetries:         40, }, } for _, tc := range testCases { err := apierrors.NewInternalError(fmt.Errorf("etcdserver: no leader")) fakeClock := testingclock.NewFakeClock(time.Now()) bm := &fakeBackoff{clock: fakeClock} counter := 0 lw := &testLW{ ListFunc: func(options metav1.ListOptions) (runtime.Object, error) { return &v1.PodList{ListMeta: metav1.ListMeta{ResourceVersion: "1"}}, nil }, WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) { counter = counter + 1 t.Logf("Counter: %v", counter) if counter == tc.rewindTime { t.Logf("Rewinding") fakeClock.Step(time.Minute) } fakeClock.Step(time.Second) w := watch.NewFakeWithChanSize(1, false) status := err.Status() w.Error(&status) return w, nil }, } r := &Reflector{ name:                   "test-reflector", listerWatcher:          lw, store:                  NewFIFO(MetaNamespaceKeyFunc), initConnBackoffManager: bm, clock:                  fakeClock, watchErrorHandler:      WatchErrorHandler(DefaultWatchErrorHandler), } r.MaxInternalErrorRetryDuration = tc.maxInternalDuration stopCh := make(chan struct{}) r.ListAndWatch(stopCh) close(stopCh) if counter-1 != tc.wantRetries { t.Errorf("%v unexpected number of retries: %d", tc, counter-1) } } } </ins> func TestReflectorResync(t *testing.T) { iteration := 0 stopCh := make(chan struct{})

Discussion:
When etcd is unable to elect leader for 3 election cycles, etcd returns "no leader" error through watches. This can be seen in api-server logs as . Also, when apiserver receives this error, it terminates all watchers: . In large clusters, this can destabilize master, as all these watchers need to be recreated almost at the same time, overwhelming api-server quite often. I propose adding new api-server flag that allows to specify timeout, during which we are retrying to reestablish apiserver <-etcd watches, without terminating all api-server watchers. If we are unable to reestablish watches within this specified timeout, we start terminating all watchers. Losing etcd leader for ~2-3s does not destabilize api-server. Trigger prolonged reelection of etcd leader. No response Observed in all kubernetes versions 1.19-1.24 N/A <details</details<details</details<details</details<details</details
/sig scalability /sig api-machinery /priority important-soon
/wg reliability
/assign
/triage accepted
indeed, this is a source of instability. Is this coming from ?
I believe it's coming from this change. As I mentioned, the idea to mitigate it is to add some parameter that will allow us to ignore this error coming from etcd for specific duration without terminating all watchers, for example 10s be default.
With this comment thread in mind: It would probably be cleaner to allow for simply setting the "timeout" as part of WithRequireLeader option (instead of using the constant of 3). But that would require etcd changes, including the API and we want to workaround that in much smaller timeframe. But a comment about how to make that cleaner with some additional context from here would help.