Change:
width := widthEstimator.EstimateWidth(r) digest := utilflowcontrol.RequestDigest{RequestInfo: requestInfo, User: user, Width: width} <del> fcIfc.Handle(ctx, digest, note, func(inQueue bool) { if inQueue { noteWaitingDelta(1) } else { noteWaitingDelta(-1) </del> <ins> if isWatchRequest { // This channel blocks calling handler.ServeHTTP() until closed, and is closed inside execute(). // If APF rejects the request, it is never closed. shouldStartWatchCh := make(chan struct{}) watchInitializationSignal := newInitializationSignal() // This wraps the request passed to handler.ServeHTTP(), // setting a context that plumbs watchInitializationSignal to storage var watchReq *http.Request // This is set inside execute(), prior to closing shouldStartWatchCh. // If the request is rejected by APF it is left nil. var forgetWatch utilflowcontrol.ForgetWatchFunc defer func() { // Protect from the situation when request will not reach storage layer // and the initialization signal will not be send. if watchInitializationSignal != nil { watchInitializationSignal.Signal() } // Forget the watcher if it was registered. // // // This is race-free because by this point, one of the following occurred: // case <-shouldStartWatchCh: execute() completed the assignment to forgetWatch // case <-resultCh: Handle() completed, and Handle() does not return //   while execute() is running if forgetWatch != nil { forgetWatch() } }() execute := func() { noteExecutingDelta(1) defer noteExecutingDelta(-1) served = true setResponseHeaders(classification, w) forgetWatch = fcIfc.RegisterWatch(requestInfo) // Notify the main thread that we're ready to start the watch. close(shouldStartWatchCh) // Wait until the request is finished from the APF point of view // (which is when its initialization is done). watchInitializationSignal.Wait() </ins> } <del> }, execute) </del> <ins> // Ensure that an item can be put to resultCh asynchronously. resultCh := make(chan interface{}, 1) // Call Handle in a separate goroutine. // The reason for it is that from APF point of view, the request processing // finishes as soon as watch is initialized (which is generally orders of // magnitude faster then the watch request itself). This means that Handle() // call finishes much faster and for performance reasons we want to reduce // the number of running goroutines - so we run the shorter thing in a // dedicated goroutine and the actual watch handler in the main one. go func() { defer func() { err := recover() // do not wrap the sentinel ErrAbortHandler panic value if err != nil && err != http.ErrAbortHandler { // Same as stdlib http server code. Manually allocate stack // trace buffer size to prevent excessively large logs const size = 64 << 10 buf := make([]byte, size) buf = buf[:runtime.Stack(buf, false)] err = fmt.Sprintf("%vn%s", err, buf) } // Ensure that the result is put into resultCh independently of the panic. resultCh <- err }() // We create handleCtx with explicit cancelation function. // The reason for it is that Handle() underneath may start additional goroutine // that is blocked on context cancellation. However, from APF point of view, // we don't want to wait until the whole watch request is processed (which is // when it context is actually cancelled) - we want to unblock the goroutine as // soon as the request is processed from the APF point of view. // // Note that we explicitly do NOT call the actuall handler using that context // to avoid cancelling request too early. handleCtx, handleCtxCancel := context.WithCancel(ctx) defer handleCtxCancel() // Note that Handle will return irrespective of whether the request // executes or is rejected. In the latter case, the function will return // without calling the passed `execute` function. fcIfc.Handle(handleCtx, digest, note, queueNote, execute) }() select { case <-shouldStartWatchCh: watchCtx := utilflowcontrol.WithInitializationSignal(ctx, watchInitializationSignal) watchReq = r.WithContext(watchCtx) handler.ServeHTTP(w, watchReq) // Protect from the situation when request will not reach storage layer // and the initialization signal will not be send. // It has to happen before waiting on the resultCh below. watchInitializationSignal.Signal() // TODO: Consider finishing the request as soon as Handle call panics. if err := <-resultCh; err != nil { panic(err) } case err := <-resultCh: if err != nil { panic(err) } } } else { execute := func() { noteExecutingDelta(1) defer noteExecutingDelta(-1) served = true setResponseHeaders(classification, w) handler.ServeHTTP(w, r) } fcIfc.Handle(ctx, digest, note, queueNote, execute) } </ins> if !served { setResponseHeaders(classification, w)

Discussion:
As visible in this graph: ! pod-startup time in 5k-node tests increased around beginning of June (only 99% though). It seems to be connected to "run-to-watch" phase. We should investigate and see what we can do with this. /sig scalability /kind bug
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : As visible in this graph: ! /kind bugs Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted Once PR that merged around that time is P&F support for watch-initialitation: As a first step, we may want to confirm/reject if it's this PR and see if we aren't somehow misconfigured here.
I've briefly checked test-infra/perf-tests to see if we had any relevant changes around that time, but I don't see any. Most likely it's related to k/k changes.
/assign
Good news is that we can also see it regression in kubemark (although not as bad as in regular 5k tests): http://perf- But this can help us with finding root cause faster/cheaper. ! I've picked two runs, one before this change, one after: 26th May: Timeline: 12th June: Timeline: First run vs second run I can see significantly higher spike on waiting time in P&F, but according to timeline it seems like it was during huge service and not pod startup latency: 26th May: ! 12th June: ! I can see that number of apiserver goroutines significantly increased (from ~500k to ~900k). I would expect to have 1 more goroutine per watch, we have around 200k watches). ! vs ! Worst 100 pods with slowest latency runtowatch: I've picked the slowest pod "test-2xppkn-33/latency-deployment-60--p7nwz" to analyse.

This pod was bound to node gce-scale-cluster-minion-group-4-txww: Kubelet log for this pod: CL2 logs for this pod: In apiserver logs we can see: So it looks like updating pod status was delayed by P&F by ~500 ms for this pod. Timeline for this pod: 18:37:46. scheduled 18:37:48. container started 18:37:49. kubelet gets pod (1.3s latency with 610 ms delay from P&F) 18:37:50. patch status of pod (545ms latency with 500ms delay from P&F) 18:37:51. observed by CL2 as running To compare with previous run we have this timeline for pod "test-rcdjgd-26/latency-deployment-69--6s95x 2.724031741s": 18:57:08. scheduled on node gce-scale-cluster-minion-group-3-l21s 18:57:10. container started 18:57:10. kubelet gets pod (364 ms latency with 345 ms delay from P&F) 18:57:11. patched status od pod (360 ms latency with 254 ms delay from P&F) So to compare, for these two pods major difference was "get pod" + "patch status" in kubelet: 1.8s vs 700ms. 1s of 1.8s of latency was due to P&F and 0.6s of 0.7s was due to P&F.
In both runs I don't see significant waiting time (99th percentile) on any of Priority levels during pod startup latency phase: 26th May: ! 12th June: !
1) Do we have data about 100% waiting time in P&F? With 99% it my not be enough to to see that data. 2) If really waiting time is small, then the question is what exactly. (a) it might be related to the number of goroutines (b) it might be locking (c) it might be something else... I have just opened the WIP PR: that should help with first two (at least to some extent). Will you be able to test if that helps?

Max waiting time in P&F for pod startup latency phase: 26th May: ! 12th June: ! Buckets are ..., 0.2, 0.5, ... so I don't think it's significant. will test it.
This is definitely non-negligible regression that we should address in 1.22 timeframe. I'm actually have a promissing results for it. /milestone v1.22 /priority critical-urgent
OK - so we proved that addresses the problem: it reduces back to previous value the number of goroutines ! it brings back pod startup time back to less than 3.4s on 99th percentile it brings back API call latencies I'm going to work with reviewers on merging this to fix the regression. - FYI