Change:
I am doing v1.28.3 kube-apiserver testing in the test environment, and I often encounter kube-apiserver panic, especially when kube-apiserver call webhook fails. The following are the captured panic logs: and: kube-apiserver does not panic even if calling webhook times out or fails Deploy a webhook, remove the service corresponding to the webhook, and then create pods in batches to trigger webhook calls No response <details</details<detailsprivate cloud </details<details</details<details</details<details</details<details</details
/sig api-machinery
somewhat related to
The scenarios that trigger panic are similar(the apiserver fails to call webhook), but the log shows that the location of panic is different.
/assign For context of  Thank you. /triage accepted
again, full stack logs:
I'll try to test the mr:
Let me try to explain the panic - both goroutines are trying to write to the of the first goroutine: This request has timed out, and consequently the filter is trying to send a response to the caller. This is the client facing goroutine from the timeout filter. Note that the call to is protected by a mutex, that means that if the other goroutine is using the delegated object via the adapter, then this panic should not happen. second goroutine: let's check the other goroutine, it is also trying to write to the of the object: Now uses an unprotected version of the ResponseWriter object: The filter precedes the filter in the execution order, so the in is not using the thread safe implementation. This is the order today: one quick resolution is to move the after the timeout filter: , but there could be code that precedes that needs to write warning? Also note that the panic mentioned in is not as severe, it does not cause the apiserver to crash. For a more proper solution, i think we would want any thoughts? /cc
I tend to remove the timeout filter, the sooner the better, and I think the Apiserver panic is the P0-level bug, directly affecting the availability of the cluster I have cherry-picked and tested it in my cluster, and I still need to continue to observe for 1-2 days
I opened a minimal WIP fix for the warning writer in since the timeout filter removal is a much bigger change to reason through
thanks a lot, I will try to test and cherry-pick the above minimal WIP fix in
sorry, I completely overlooked , I only saw and that looked too big to take safely right now I prefer to
Can we backport the fix to 1.28?
Why 1.28 specifically? I'm not aware of anything that changed in 1.28 that made this more or less prevalent. If this meets backport criteria, going to all active branches could make sense (1.27+)

Discussion:
handler = genericfilters.WithCORS(handler, c.CorsAllowedOriginList, nil, nil, nil, "true") <ins> // WithWarningRecorder must be wrapped by the timeout handler // to make the addition of warning headers threadsafe handler = genericapifilters.WithWarningRecorder(handler) </ins> // WithTimeoutForNonLongRunningRequests will call the rest of the request handling in a go-routine with the // context with deadline. The go-routine can keep running, while the timeout logic will return a timeout to the client. handler = genericfilters.WithTimeoutForNonLongRunningRequests(handler, c.LongRunningFunc)

if c.SecureServing != nil && !c.SecureServing.DisableHTTP2 && c.GoawayChance > 0 { handler = genericfilters.WithProbabilisticGoaway(handler, c.GoawayChance) } <del> handler = genericapifilters.WithWarningRecorder(handler) </del> handler = genericapifilters.WithCacheControl(handler) handler = genericfilters.WithHSTS(handler, c.HSTSDirectives) if c.ShutdownSendRetryAfter {

openapinamer "k8s.io/apiserver/pkg/endpoints/openapi" "k8s.io/apiserver/pkg/registry/rest" genericfilters "k8s.io/apiserver/pkg/server/filters" <ins> "k8s.io/apiserver/pkg/warning" </ins> "k8s.io/client-go/informers" "k8s.io/client-go/kubernetes/fake" restclient "k8s.io/client-go/rest"

t.Errorf("Timed out waiting for response.") } } <ins> func TestWarningWithRequestTimeout(t *testing.T) { type result struct { err        interface{} stackTrace string } clientDoneCh, resultCh := make(chan struct{}), make(chan result, 1) testHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // this will catch recoverable panic like 'Header called after Handler finished'. // go runtime crashes the program if it detects a program-ending // panic like 'concurrent map iteration and map write', so this // panic can not be caught. defer func() { result := result{} result.err = recover() if result.err != nil { // Same as stdlib http server code. Manually allocate stack // trace buffer size to prevent excessively large logs const size = 64 << 10 buf := make([]byte, size) buf = buf[:goruntime.Stack(buf, false)] result.stackTrace = string(buf) } resultCh <- result }() // add warnings while we're waiting for the request to timeout to catch read/write races loop: for { select { case <-r.Context().Done(): break loop default: warning.AddWarning(r.Context(), "a", "1") } } // the request has just timed out, write to catch read/write races warning.AddWarning(r.Context(), "agent", "text") // give time for the timeout response to be written, then try to // write a response header to catch the "Header after Handler finished" panic <-clientDoneCh warning.AddWarning(r.Context(), "agent", "text") }) handler := newGenericAPIServerHandlerChain(t, "/test", testHandler) server := httptest.NewUnstartedServer(handler) server.EnableHTTP2 = true server.StartTLS() defer server.Close() request, err := http.NewRequest("GET", server.URL+"/test?timeout=100ms", nil) if err != nil { t.Fatalf("unexpected error: %v", err) } client := server.Client() response, err := client.Do(request) close(clientDoneCh) if err != nil { t.Errorf("expected server to return an HTTP response: %v", err) } if want := http.StatusGatewayTimeout; response == nil || response.StatusCode != want { t.Errorf("expected server to return %d, but got: %v", want, response) } var resultGot result select { case resultGot = <-resultCh: case <-time.After(wait.ForeverTestTimeout): t.Errorf("the handler never returned a result") } if resultGot.err != nil { t.Errorf("Expected no panic, but got: %v", resultGot.err) t.Errorf("Stack Trace: %s", resultGot.stackTrace) } } // builds a handler chain with the given user handler as used by GenericAPIServer. func newGenericAPIServerHandlerChain(t *testing.T, path string, handler http.Handler) http.Handler { config, _ := setUp(t) s, err := config.Complete(nil).New("test", NewEmptyDelegate()) if err != nil { t.Fatalf("Error in bringing up the server: %v", err) } s.Handler.NonGoRestfulMux.Handle(path, handler) return s.Handler } </ins>