Change:
func (fakeRL) Stop()             {} func (f fakeRL) CanAccept() bool { return bool(f) } <ins> func expectHTTP(url string, code int, t *testing.T) { r, err := http.Get(url) if err != nil { t.Errorf("unexpected error: %v", err) } if r.StatusCode != code { t.Errorf("unexpected response: %v", r.StatusCode) } } func TestMaxInFlight(t *testing.T) { const Iterations = 3 block := sync.WaitGroup{} block.Add(1) sem := make(chan bool, Iterations) re := regexp.MustCompile("[.*/watch][^/proxy.*]") server := httptest.NewServer(MaxInFlightLimit(sem, re, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if strings.Contains(r.URL.Path, "dontwait") { return } block.Wait() }))) defer server.Close() // These should hang, but not affect accounting. for i := 0; i < Iterations; i++ { // These should hang waiting on block... go func() { expectHTTP(server.URL+"/foo/bar/watch", http.StatusOK, t) }() } for i := 0; i < Iterations; i++ { // These should hang waiting on block... go func() { expectHTTP(server.URL+"/proxy/foo/bar", http.StatusOK, t) }() } expectHTTP(server.URL+"/dontwait", http.StatusOK, t) for i := 0; i < Iterations; i++ { // These should hang waiting on block... go func() { expectHTTP(server.URL, http.StatusOK, t) }() } // There's really no more elegant way to do this.  I could use a WaitGroup, but even then // it'd still be racy. time.Sleep(1 * time.Second) expectHTTP(server.URL+"/dontwait/watch", http.StatusOK, t) // Do this multiple times to show that it rate limit rejected requests don't block. for i := 0; i < 2; i++ { expectHTTP(server.URL, errors.StatusTooManyRequests, t) } block.Done() // Show that we recover from being blocked up. expectHTTP(server.URL, http.StatusOK, t) } </ins> func TestRateLimit(t *testing.T) { for _, allow := range []bool{true, false} { rl := fakeRL(allow)

Discussion:
Related with issue , the API server should have a cap on the number of outstanding operations/requests it's handling, and return an error otherwise. A simple semaphore is an idea for this.
+1 !
The good news/bad news on this is that there is a rate limiter in the API server (), but the defaults for it seem too high: 10 qps average, 200 burst (). We could start by lowering the burst level to something smaller, like 30. It may be interesting to also differentiate between GET and PUT/POST requests.
I think that we need limits on the number of outstanding requests, not the number of QPS.
-An max outstanding would be useful. But it may force us to have separate buckets for different types of requests, i.e. a small number of long running requests causing API server to reject short requests. -Heard that you might be looking into adding support for flow control (or something like it?). We may need it soon to help avoid bugs like runaway node status updates (see issue ).
I had not - the things we were looking at were the bastion changes to apiserver which will break logs and proxying into their own rest resources so they can be granularly controlled. Being able to timebox and classify requests differently would be good. ----- Original Message -----
- Ah, np, I probably misheard.
What are our plans for handling this client side? Do we want clients to handle "you're being throttled" error themselves, or do we plan to handle it in the library? I'm asking because currently kubelet does not have a 'short' retry loop, hence it tries to report NodeStatus every X (2) seconds, whether it succeed or not. Neither we have a sidechannel for heartbeat messages. This would mean, that if API server will be under heavy load, and with not-negligible probability NodeStatus reporting will be throttled, we may incorrectly mark a Node as unreachable. Currently it's not a huge problem, because X << Y, where Y is the time we wait before we decide a Node is unreachable. But because of the effort somewhere close to it'll probably get worse.
The client library should have options for retry on client.Config which include the ability to disable the behavior. Automatic retry is probably an "opt-out" behavior.

----- Original Message -----
says there are people working on rate limiting client connections to API server.
I have in the past adjusted the rate limiter to support the creation of large clusters. I also the logic. If you adjust the rate limiter settings, please make sure it is still possible to create large clusters.
I'm closing this as fixed, given the original subject. If we want to expand client-side support for retry, please open a dedicated issue. (Also note that QPS based throttling was to the client recently too)