Change:
On a cluster with many APIs provided by aggregated apiservers, with controllers managing those APIs, the number of occupied seats effectively has a floor of 1 per watch. If the combined max-inflight limit is reduced low enough, priority levels tend to become saturated by these long-running watch requests and other requests are starved. Watch requests consume a seat only during initialization. a PriorityLevelConfiguration and a corresponding FlowSchema to match watches against a resource served by an aggregated apiserver. some watch requests. the metric climb on the priority level per watch until all seats are exhausted. Kubernetes version (use ): (OpenShift 4.9) /sig api-machinery
cc
D'oh! I can see two obvious possible ways to cope: one apiserver not manage watch requests the are delegated to another apiserver; or a marker into the notification stream so that the non-origin apiservers can detect when initialization is done.
I would even say that for regular (non-long-running) requests, it's not obvious that they should be managed by P&F even. [I agree that for watch initialization it's particularly bad though.] - do we have a simple way to detect if the request will be served from our kube-apiserver or from some other aggregated apiserver?
There is not currently a way to know that ahead of time in the filters today. It require a list/watch, but it ought not be excessively expensive to make a 99% correct prediction, but the actual decision is made by the aggregated apiserver later in the request.
/triage accepted
: is there a way that we can plumb that knowledge forward to the filters?

Discussion:
} } <ins> // RequestDelegated informs the priority and fairness dispatcher that // a given request has been delegated to an aggregated API // server. No-op when priority and fairness is disabled. func RequestDelegated(ctx context.Context) { // The watch initialization signal doesn't traverse request // boundaries, so we generously fire it as soon as we know // that the request won't be serviced locally. Safe to call // for non-watch requests. WatchInitialized(ctx) } </ins> // InitializationSignal is an interface that allows sending and handling // initialization signals. type InitializationSignal interface {

genericfeatures "k8s.io/apiserver/pkg/features" "k8s.io/apiserver/pkg/server/egressselector" utilfeature "k8s.io/apiserver/pkg/util/feature" <ins> utilflowcontrol "k8s.io/apiserver/pkg/util/flowcontrol" </ins> "k8s.io/apiserver/pkg/util/x509metrics" restclient "k8s.io/client-go/rest" "k8s.io/client-go/transport"

handler := proxy.NewUpgradeAwareHandler(location, proxyRoundTripper, true, upgrade, &responder{w: w}) handler.InterceptRedirects = utilfeature.DefaultFeatureGate.Enabled(genericfeatures.StreamingProxyRedirects) handler.RequireSameHostRedirects = utilfeature.DefaultFeatureGate.Enabled(genericfeatures.ValidateProxyRedirects) <ins> utilflowcontrol.RequestDelegated(req.Context()) </ins> handler.ServeHTTP(w, newReq) }

"k8s.io/apiserver/pkg/authentication/user" genericapirequest "k8s.io/apiserver/pkg/endpoints/request" "k8s.io/apiserver/pkg/server/egressselector" <ins> utilflowcontrol "k8s.io/apiserver/pkg/util/flowcontrol" </ins> "k8s.io/component-base/metrics" "k8s.io/component-base/metrics/legacyregistry" apiregistration "k8s.io/kube-aggregator/pkg/apis/apiregistration/v1"

} } <ins> type fcInitSignal struct { nSignals int32 } func (s *fcInitSignal) SignalCount() int { return int(atomic.SwapInt32(&s.nSignals, 0)) } func (s *fcInitSignal) Signal() { atomic.AddInt32(&s.nSignals, 1) } func (s *fcInitSignal) Wait() { } type hookedListener struct { l        net.Listener onAccept func() } func (wl *hookedListener) Accept() (net.Conn, error) { wl.onAccept() return wl.l.Accept() } func (wl *hookedListener) Close() error { return wl.l.Close() } func (wl *hookedListener) Addr() net.Addr { return wl.l.Addr() } func TestFlowControlSignal(t *testing.T) { for _, tc := range []struct { Name           string Local          bool Available      bool Request        http.Request SignalExpected bool }{ { Name:           "local", Local:          true, SignalExpected: false, }, { Name:           "unavailable", Local:          false, Available:      false, SignalExpected: false, }, { Name:           "request performed", Local:          false, Available:      true, SignalExpected: true, }, { Name:      "upgrade request performed", Local:     false, Available: true, Request: http.Request{ Header: http.Header{"Connection": []string{"Upgrade"}}, }, SignalExpected: true, }, } { t.Run(tc.Name, func(t *testing.T) { okh := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusOK) }) var sig fcInitSignal var signalCountOnAccept int32 backend := httptest.NewUnstartedServer(okh) backend.Listener = &hookedListener{ l: backend.Listener, onAccept: func() { atomic.StoreInt32(&signalCountOnAccept, int32(sig.SignalCount())) }, } backend.Start() defer backend.Close() p := proxyHandler{ localDelegate:   okh, serviceResolver: &mockedRouter{destinationHost: backend.Listener.Addr().String()}, } server := httptest.NewServer(contextHandler( http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { p.ServeHTTP(w, r.WithContext(utilflowcontrol.WithInitializationSignal(r.Context(), &sig))) }), &user.DefaultInfo{ Name:   "username", Groups: []string{"one", "two"}, }, )) defer server.Close() p.handlingInfo.Store(proxyHandlingInfo{ local:             tc.Local, serviceAvailable:  tc.Available, proxyRoundTripper: backend.Client().Transport, }) surl, err := url.Parse(server.URL) if err != nil { t.Fatalf("unexpected error: %v", err) } req := tc.Request req.URL = surl _, err = server.Client().Do(&req) if err != nil { t.Fatalf("unexpected error: %v", err) } if fired := (atomic.LoadInt32(&signalCountOnAccept) > 0); tc.SignalExpected && !fired { t.Errorf("flow control signal expected but not fired") } else if fired && !tc.SignalExpected { t.Errorf("flow control signal fired but not expected") } }) } } </ins> func getCertAndKeyPaths(t *testing.T) (string, string, string) { dir, err := ioutil.TempDir(os.TempDir(), "k8s-test-handler-proxy-cert") if err != nil {