Change:
Apiserver crashed with stacktrace: Not to crash I was able to reproduce it by "slowing down" deleting headers in Authentication filter (): And making requests (around 5-10 requests to trigger this race condition): Crash was caused by race condition - one goroutine was modifying headers of requests (for example authentication filter). The other goroutine was trying to get UserAgent from request, which is also taken from request's headers map. This causes unrecoverable panic. <detailsAll versions, tested on 1.21 </details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery /priority important-soon
/triage accepted
Reopening - since we need to cherrypick it to previous releasses.
- friendly ping on opening cherrypicks
I've opened two cherry-picks:
One more for 1.20:
All merged - thanks!

Discussion:
"net/http" "runtime" "strings" <ins> "sync" </ins> "time" "k8s.io/apiserver/pkg/endpoints/metrics"

// the http.ResponseWriter. We can recover panics from go-restful, and // the logging value is questionable. type respLogger struct { <del> hijacked           bool statusRecorded     bool status             int statusStack        string </del> <ins> hijacked       bool statusRecorded bool status         int statusStack    string // mutex is used when accessing addedInfo and addedKeyValuePairs. // They can be modified by other goroutine when logging happens (in case of request timeout) mutex              sync.Mutex </ins> addedInfo          strings.Builder addedKeyValuePairs []interface{} startTime          time.Time captureErrorOutput bool <del> req *http.Request w   http.ResponseWriter </del> <ins> req       *http.Request userAgent string w         http.ResponseWriter </ins> logStacktracePred StacktracePred }

logger := &respLogger{ startTime:         startTime, req:               req, <ins> userAgent:         req.UserAgent(), </ins> w:                 w, logStacktracePred: DefaultStacktracePred, }

// Addf adds additional data to be logged with this request. func (rl *respLogger) Addf(format string, data ...interface{}) { <ins> rl.mutex.Lock() defer rl.mutex.Unlock() </ins> rl.addedInfo.WriteString("n") rl.addedInfo.WriteString(fmt.Sprintf(format, data...)) }

} func (rl *respLogger) AddKeyValue(key string, value interface{}) { <ins> rl.mutex.Lock() defer rl.mutex.Unlock() </ins> rl.addedKeyValuePairs = append(rl.addedKeyValuePairs, key, value) }

"verb", verb, "URI", rl.req.RequestURI, "latency", latency, <del> "userAgent", rl.req.UserAgent(), </del> <ins> // We can't get UserAgent from rl.req.UserAgent() here as it accesses headers map, // which can be modified in another goroutine when apiserver request times out. // For example authentication filter modifies request's headers, // This can cause apiserver to crash with unrecoverable fatal error. // More info about concurrent read and write for maps: https://golang.org/doc/go1.6#runtime "userAgent", rl.userAgent, </ins> "audit-ID", auditID, "srcIP", rl.req.RemoteAddr, } <ins> // Lock for accessing addedKeyValuePairs and addedInfo rl.mutex.Lock() defer rl.mutex.Unlock() </ins> keysAndValues = append(keysAndValues, rl.addedKeyValuePairs...) if rl.hijacked {