Code change:
In a cluster with large enough numbers of objects, it seems to be impossible to clean them up with . Because the deletecollection is failing while listing the objects from etcd. The failure happens after exactly 34 seconds, due to the 34-second max timeout introduced in PR (cc for insight on that PR). The deletecollection should be able to run for longer, and succeed. I recognise that there are still risks of the API server using too much RAM, if deleting a very large collection due to issue . But, until that issue is fixed, it seems like allowing to run for longer might be a good idea. As far as I can tell, the 34 second timeout does not apply to normal LIST calls, so on that basis I feel it should not apply to the giant unchunked LIST call that happens inside . a few hundred thousand objects of some particular type. (Argo workflowtaskresults happen to be the type I'm looking at while I write this) deletecollection. (Optionally passing some kind of fieldSelector, but I don't think that's essential) that a 504 failure code is logged in kube-audit logs, and that a log entry similar to this is logged by the API Server (In practice, the total time may be a little over ms). far as I can tell, the delete collection really has failed at this point. I notice that PR says that, in general, "The go-routine can keep running, while the timeout logic will return a timeout to the client" but I don't think that's what's happening here. I think that, because the 34 sec timeout is used in the context of the etcd call, the etcd list call really does fail, and so the code never proceeds to actually deleting anything. No response 22 AKS <details</details<details</details<details</details<details</details
/sig scalability
/cc
the deadline has been in effect before PR was introduced If we remove this deadline from here, then the deletecollection request will have at most to complete (default). The list call that is failing in your case, is a direct call to the storage, right? on the contrary, we don't enforce the timeout for LIST thinking out loud here: I don't know if it makes sense to remove the from should we allow timeout for DELETECOLLECTION to exceed ? thoughts?
The list call that is failing in your case, is a direct call to the storage, right? It reads from etcd, not from API Server Cache. Is that what you were asking Looking at the bigger picture, I think your question about 60s timeout is a good one. Especially in the context of this: In particular, changing the timeouts won't solve all of the problems with DELETECOLLECTION. My assumption, when logging this issue, is that it might be a quick way to solve just some of them.
I think we should remove this - it's effectively a regression that we introduced with the PR. That's a good question, and I'm not entirely sure. Maybe we should, but I would do that separately. In the meantime, I would much more prefer to switch the DeleteCollection implementation to internally use pagination, as it would allow us to 1) not block on full list (and as a result make some progress) 2) would be some preparation towards potentially exposing a Limit option in the API... /kind regression
/sig api-machinery
/triage accepted
I suggest the following: - can you please take care of reverting the 34s timeout for DeleteCollection calls? this is done, let's close this issue and continue operating on
sounds good, thanks! /assign

Related discussion:
text: }) } } <ins> func TestDeleteCollectionWithNoContextDeadlineEnforced(t *testing.T) { var invokedGot, hasDeadlineGot int32 fakeDeleterFn := func(ctx context.Context, _ rest.ValidateObjectFunc, _ *metav1.DeleteOptions, _ *metainternalversion.ListOptions) (runtime.Object, error) { // we expect CollectionDeleter to be executed once atomic.AddInt32(&invokedGot, 1) // we don't expect any context deadline to be set if _, hasDeadline := ctx.Deadline(); hasDeadline { atomic.AddInt32(&hasDeadlineGot, 1) } return nil, nil } // do the minimum setup to ensure that it gets as far as CollectionDeleter scope := &RequestScope{ Namer: &mockNamer{}, Serializer: &fakeSerializer{ serializer: runtime.NewCodec(runtime.NoopEncoder{}, runtime.NoopDecoder{}), }, } handler := DeleteCollection(fakeCollectionDeleterFunc(fakeDeleterFn), false, scope, nil) request, err := http.NewRequest("GET", "/test", nil) if err != nil { t.Fatalf("unexpected error: %v", err) } // the request context should not have any deadline by default if _, hasDeadline := request.Context().Deadline(); hasDeadline { t.Fatalf("expected request context to not have any deadline") } recorder := httptest.NewRecorder() handler.ServeHTTP(recorder, request) if atomic.LoadInt32(&invokedGot) != 1 { t.Errorf("expected collection deleter to be invoked") } if atomic.LoadInt32(&hasDeadlineGot) > 0 { t.Errorf("expected context to not have any deadline") } } type fakeCollectionDeleterFunc func(ctx context.Context, deleteValidation rest.ValidateObjectFunc, options *metav1.DeleteOptions, listOptions *metainternalversion.ListOptions) (runtime.Object, error) func (f fakeCollectionDeleterFunc) DeleteCollection(ctx context.Context, deleteValidation rest.ValidateObjectFunc, options *metav1.DeleteOptions, listOptions *metainternalversion.ListOptions) (runtime.Object, error) { return f(ctx, deleteValidation, options, listOptions) } type fakeSerializer struct { serializer runtime.Serializer } func (n *fakeSerializer) SupportedMediaTypes() []runtime.SerializerInfo { return []runtime.SerializerInfo{ { MediaType:        "application/json", MediaTypeType:    "application", MediaTypeSubType: "json", }, } } func (n *fakeSerializer) EncoderForVersion(serializer runtime.Encoder, gv runtime.GroupVersioner) runtime.Encoder { return n.serializer } func (n *fakeSerializer) DecoderToVersion(serializer runtime.Decoder, gv runtime.GroupVersioner) runtime.Decoder { return n.serializer } </ins>