Change:
<ins> /* Copyright 2022 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package cache import ( "fmt" "testing" "time" apierrors "k8s.io/apimachinery/pkg/api/errors" testingclock "k8s.io/utils/clock/testing" ) type retryScenarioStep struct { clockStep time.Duration err       error wantRetry bool } func TestRetryWithDeadline(t *testing.T) { internalError := apierrors.NewInternalError(fmt.Errorf("etcdserver: no leader")) otherError := fmt.Errorf("some other error") testCases := []struct { name        string duration    time.Duration reset       time.Duration isRetryable func(error) bool scenario    []retryScenarioStep }{ { name:        "Never retry when duration is zero", duration:    time.Duration(0), reset:       time.Second * 30, isRetryable: func(err error) bool { return false }, scenario: []retryScenarioStep{ { clockStep: time.Second * 1, err:       nil, wantRetry: false, }, { clockStep: time.Second * 0, err:       internalError, wantRetry: false, }, { clockStep: time.Second * 1, err:       otherError, wantRetry: false, }, }, }, { name:        "Retry when internal error happens only within duration", duration:    time.Second * 1, reset:       time.Second * 30, isRetryable: apierrors.IsInternalError, scenario: []retryScenarioStep{ { clockStep: time.Second * 1, err:       internalError, wantRetry: true, }, { clockStep: time.Second * 1, err:       internalError, wantRetry: true, }, { clockStep: time.Second * 1, err:       internalError, wantRetry: false, }, }, }, { name:        "Don't retry when other error happens", duration:    time.Second * 1, reset:       time.Second * 30, isRetryable: apierrors.IsInternalError, scenario: []retryScenarioStep{ { clockStep: time.Second * 1, err:       otherError, wantRetry: false, }, }, }, { name:        "Ignore other errors for retries", duration:    time.Second * 1, reset:       time.Second * 30, isRetryable: apierrors.IsInternalError, scenario: []retryScenarioStep{ { clockStep: time.Second * 1, err:       internalError, wantRetry: true, }, { clockStep: time.Second * 0, err:       otherError, wantRetry: true, }, { clockStep: time.Second * 1, err:       internalError, wantRetry: true, }, }, }, } for _, tc := range testCases { fakeClock := testingclock.NewFakeClock(time.Now()) retry := NewRetryWithDeadline(tc.duration, tc.reset, tc.isRetryable, fakeClock) for i, step := range tc.scenario { fakeClock.Step(step.clockStep) retry.After(step.err) result := retry.ShouldRetry() if result != step.wantRetry { t.Errorf("%v unexpected retry, step %d, result %v want %v", tc, i, result, step.wantRetry) break } } } } </ins>

Discussion:
When etcd is unable to elect leader for 3 election cycles, etcd returns "no leader" error through watches. This can be seen in api-server logs as . Also, when apiserver receives this error, it terminates all watchers: . In large clusters, this can destabilize master, as all these watchers need to be recreated almost at the same time, overwhelming api-server quite often. I propose adding new api-server flag that allows to specify timeout, during which we are retrying to reestablish apiserver <-etcd watches, without terminating all api-server watchers. If we are unable to reestablish watches within this specified timeout, we start terminating all watchers. Losing etcd leader for ~2-3s does not destabilize api-server. Trigger prolonged reelection of etcd leader. No response Observed in all kubernetes versions 1.19-1.24 N/A <details</details<details</details<details</details<details</details
/sig scalability /sig api-machinery /priority important-soon
/wg reliability
/assign
/triage accepted
indeed, this is a source of instability. Is this coming from ?
I believe it's coming from this change. As I mentioned, the idea to mitigate it is to add some parameter that will allow us to ignore this error coming from etcd for specific duration without terminating all watchers, for example 10s be default.
With this comment thread in mind: It would probably be cleaner to allow for simply setting the "timeout" as part of WithRequireLeader option (instead of using the constant of 3). But that would require etcd changes, including the API and we want to workaround that in much smaller timeframe. But a comment about how to make that cleaner with some additional context from here would help.