Change:
/sig scalability Debugging below done by Failed test attempt: It failed with a number of "WaitingForPods: objects timed out" errors (46 failed objects in total). Sample failing pods: 580 nodes were not ready at 18:37: nodelifecyclecontroller said that gce-scale-cluster-minion-group-2-66d8's status hasn't been updated for 41 seconds. apiserver's logs show that kubelet was doing 'PUT ' every 10 seconds. Something weird happened to watch by 'shared-informers': One watch started at 18:36:52 and finished 9ms later. The next one started at 18:40:34. So it seems that there was a gap in lease watche between 18:36:52 and 18:40:34. Between 18:36:52 and 18:40:34 there is a number of leases LIST requests in the following pattern: a single /leases?limit=500 LIST request that finishes with 200 a single /leases/continue=XXX request that finishes with 400 The kube-controller-manager is retrying every 1 second up to 18:40:34.
/assign
Looking further at this it seems that number of LIST leases by kube-controller-manager increased significantly ~week ago in large scale tests (both gce and kubemark). GCE: ! Kubemark: ! This was caused by which as a side effect enabled pagination of LIST calls when starting/restarting watchers. In some cases we're getting 400 errors from api-server for this calls which results in the symptoms described above, i.e nodes becoming not-ready and pods stuck in running but not ready state for some time. GCE: ! Kubemark: !
There are actually two issues here: that we switched from non-paginated LIST calls to paginated the 10x increase in # calls is expected, but what is worrying is that we already had ~50 LISTs in every test. Such LIST happens only when watcher restarts which shouldn't happen in a healthy cluster This will be address in need to figure out why we're sometimes getting 400 response (BAD REQUEST).
/milestone v1.17
Logs when then 400 errors occurred api-server kube-controller-manager
So what happens here is: in reflector ends with an error different than so the doesn't get reset to next list is issued with set to page is fetched successfully, the next page is requested with the got from the first page and second page fails with due to error the doesn't get cleared anywhere so we go back to 2 and the 400 error keeps repeating until we get 410 (resourceVersionTooOld, on the next etcd compaction) and the is set to takes long enough for the NodeController to mark many nodes as not ready as it doesn't get any updates from watch about node leases This was indeed introduced by but the actual bug is in the which sets both and on the subsequent pages while it should set only . I will send a PR to fix it today.
FYI
Great finding.
Does this mean we have no tests that actually check if a reflector works with paging?
I see, it's the rare situation where "Watch in reflector ends with an error different than 410" that is untested. Can we set up a unit test that exercises this scenario by wrapping etcd3 storage to allow injecting an error in a watch attempt (similar to how we wrapped storage to test stale behavior in )?
I don't think it's the case - see . It clearly seems to be that pager wasn't properly tested. The problem seems to be that we're testing it with artifical stuff, whereas in real life etcd would return "bad request" errors.
Right, I'd like to see if we can trigger the error condition in a way that the restart/paging requests go to a real storage backend.
/milestone v1.18 (v1.17 code freeze)
- this is a regression introduced in 1.17, I actually believe we should fix that in 1.17.
/milestone v1.17 as noted above, this is a release blocking bug

Discussion:
for i := 0; i < 8; i++ { pods[i] = v1.Pod{ObjectMeta: metav1.ObjectMeta{Name: fmt.Sprintf("pod-%d", i), ResourceVersion: fmt.Sprintf("%d", i)}} } <del> switch options.ResourceVersion { case "0": </del> <ins> rvContinueLimit := func(rv, c string, l int64) metav1.ListOptions { return metav1.ListOptions{ResourceVersion: rv, Continue: c, Limit: l} } switch rvContinueLimit(options.ResourceVersion, options.Continue, options.Limit) { // initial limited list case rvContinueLimit("0", "", 4): </ins> return &v1.PodList{ListMeta: metav1.ListMeta{ResourceVersion: "10"}, Items: pods[0:4]}, nil <del> case "10": switch options.Limit { case 4: switch options.Continue { case "": return &v1.PodList{ListMeta: metav1.ListMeta{Continue: "C1", ResourceVersion: "11"}, Items: pods[0:4]}, nil case "C1": return nil, apierrs.NewResourceExpired("The resourceVersion for the provided watch is too old.") default: t.Fatalf("Unrecognized Continue: %s", options.Continue) } case 0: return &v1.PodList{ListMeta: metav1.ListMeta{ResourceVersion: "11"}, Items: pods[0:8]}, nil default: t.Fatalf("Unrecognized Limit: %d", options.Limit) } </del> <ins> // first page of the rv=10 list case rvContinueLimit("10", "", 4): return &v1.PodList{ListMeta: metav1.ListMeta{Continue: "C1", ResourceVersion: "11"}, Items: pods[0:4]}, nil // second page of the above list case rvContinueLimit("", "C1", 4): return nil, apierrs.NewResourceExpired("The resourceVersion for the provided watch is too old.") // rv=10 unlimited list case rvContinueLimit("10", "", 0): return &v1.PodList{ListMeta: metav1.ListMeta{ResourceVersion: "11"}, Items: pods[0:8]}, nil </ins> default: <del> t.Fatalf("Unrecognized ResourceVersion: %s", options.ResourceVersion) </del> <ins> err := fmt.Errorf("unexpected list options: %#v", options) t.Error(err) return nil, err </ins> } return nil, nil },

r.WatchListPageSize = 4 // Initial list should use RV=0 <del> r.ListAndWatch(stopCh) </del> <ins> if err := r.ListAndWatch(stopCh); err != nil { t.Fatal(err) } </ins> results := s.List() if len(results) != 4 { t.Errorf("Expected 4 results, got %d", len(results)) } <del> // relist should use lastSyncResourceVersions (RV=10) and since second page of RV=10 is expired, it should full list with RV=10 </del> <ins> // relist should use lastSyncResourceVersions (RV=10) and since second page of that expired, it should full list with RV=10 </ins> stopCh = make(chan struct{}) <del> r.ListAndWatch(stopCh) </del> <ins> if err := r.ListAndWatch(stopCh); err != nil { t.Fatal(err) } </ins> results = s.List() if len(results) != 8 { t.Errorf("Expected 8 results, got %d", len(results)) } <del> expectedRVs := []string{"0", "10", "10", "10"} </del> <ins> expectedRVs := []string{"0", "10", "", "10"} </ins> if !reflect.DeepEqual(listCallRVs, expectedRVs) { <del> t.Errorf("Expected series of list calls with resource versiosn of %v but got: %v", expectedRVs, listCallRVs) </del> <ins> t.Errorf("Expected series of list calls with resource versiosn of %#v but got: %#v", expectedRVs, listCallRVs) </ins> } }

if options.Limit == 0 { options.Limit = p.PageSize } <ins> requestedResourceVersion := options.ResourceVersion </ins> var list *metainternalversion.List for { select {

if !errors.IsResourceExpired(err) || !p.FullListIfExpired || options.Continue == "" { return nil, err } <del> // the list expired while we were processing, fall back to a full list </del> <ins> // the list expired while we were processing, fall back to a full list at // the requested ResourceVersion. </ins> options.Limit = 0 options.Continue = "" <ins> options.ResourceVersion = requestedResourceVersion </ins> return p.PageFn(ctx, options) } m, err := meta.ListAccessor(obj)

// set the next loop up options.Continue = m.GetContinue() <ins> // Clear the ResourceVersion on the subsequent List calls to avoid the // `specifying resource version is not allowed when using continue` error. // See https://github.com/kubernetes/kubernetes/issues/85221#issuecomment-553748143. options.ResourceVersion = "" </ins> } }

p.t.Errorf("invariant violated, expected limit %d and continue %s, got %#v", p.expectPage, expectedContinue, options) return nil, fmt.Errorf("invariant violated") } <ins> if options.Continue != "" && options.ResourceVersion != "" { p.t.Errorf("invariant violated, specifying resource version (%s) is not allowed when using continue (%s).", options.ResourceVersion, options.Continue) return nil, fmt.Errorf("invariant violated") } </ins> var list metainternalversion.List total := options.Limit if total == 0 {

args: args{}, want: list(21, "rv:20"), }, <ins> { name:   "two pages with resourceVersion", fields: fields{PageSize: 10, PageFn: (&testPager{t: t, expectPage: 10, remaining: 11, rv: "rv:20"}).PagedList}, args:   args{options: metav1.ListOptions{ResourceVersion: "rv:10"}}, want:   list(11, "rv:20"), }, </ins> } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) {