Change:
Related with issue , the API server should have a cap on the number of outstanding operations/requests it's handling, and return an error otherwise. A simple semaphore is an idea for this.
+1 !
The good news/bad news on this is that there is a rate limiter in the API server (), but the defaults for it seem too high: 10 qps average, 200 burst (). We could start by lowering the burst level to something smaller, like 30. It may be interesting to also differentiate between GET and PUT/POST requests.
I think that we need limits on the number of outstanding requests, not the number of QPS.
-An max outstanding would be useful. But it may force us to have separate buckets for different types of requests, i.e. a small number of long running requests causing API server to reject short requests. -Heard that you might be looking into adding support for flow control (or something like it?). We may need it soon to help avoid bugs like runaway node status updates (see issue ).
I had not - the things we were looking at were the bastion changes to apiserver which will break logs and proxying into their own rest resources so they can be granularly controlled. Being able to timebox and classify requests differently would be good. ----- Original Message -----
- Ah, np, I probably misheard.
What are our plans for handling this client side? Do we want clients to handle "you're being throttled" error themselves, or do we plan to handle it in the library? I'm asking because currently kubelet does not have a 'short' retry loop, hence it tries to report NodeStatus every X (2) seconds, whether it succeed or not. Neither we have a sidechannel for heartbeat messages. This would mean, that if API server will be under heavy load, and with not-negligible probability NodeStatus reporting will be throttled, we may incorrectly mark a Node as unreachable. Currently it's not a huge problem, because X << Y, where Y is the time we wait before we decide a Node is unreachable. But because of the effort somewhere close to it'll probably get worse.
The client library should have options for retry on client.Config which include the ability to disable the behavior. Automatic retry is probably an "opt-out" behavior. ----- Original Message -----
says there are people working on rate limiting client connections to API server.
I have in the past adjusted the rate limiter to support the creation of large clusters. I also the logic. If you adjust the rate limiter settings, please make sure it is still possible to create large clusters.
I'm closing this as fixed, given the original subject. If we want to expand client-side support for retry, please open a dedicated issue. (Also note that QPS based throttling was to the client recently too)

Discussion:
"net" "net/http" "os" <ins> "regexp" </ins> "strconv" "strings" "time"

KubeletConfig              client.KubeletConfig ClusterName                string EnableProfiling            bool <ins> MaxRequestsInFlight        int LongRunningRequestRE       string </ins> } // NewAPIServer creates a new APIServer object with default parameters

fs.StringVar(&s.ClusterName, "cluster_name", s.ClusterName, "The instance prefix for the cluster") fs.BoolVar(&s.EnableProfiling, "profiling", false, "Enable profiling via web interface host:port/debug/pprof/") fs.StringVar(&s.ExternalHost, "external_hostname", "", "The hostname to use when generating externalized URLs for this master (e.g. Swagger API Docs.)") <ins> fs.IntVar(&s.MaxRequestsInFlight, "max_requests_inflight", 20, "The maximum number of requests in flight at a given time.  When the server exceeds this, it rejects requests.  Zero for no limit.") fs.StringVar(&s.LongRunningRequestRE, "long_running_request_regexp", "[.*/watch$][^/proxy.*]", "A regular expression matching long running requests which should be excluded from maximum inflight request handling.") </ins> } // TODO: Longer term we should read this from some config store, rather than a flag.

// See the flag commentary to understand our assumptions when opening the read-only and read-write ports. <ins> var sem chan bool if s.MaxRequestsInFlight > 0 { sem = make(chan bool, s.MaxRequestsInFlight) } longRunningRE := regexp.MustCompile(s.LongRunningRequestRE) </ins> if roLocation != "" { // Default settings allow 1 read-only request per second, allow up to 20 in a burst before enforcing. rl := util.NewTokenBucketRateLimiter(s.APIRate, s.APIBurst) readOnlyServer := &http.Server{ Addr:           roLocation, <del> Handler:        apiserver.RecoverPanics(apiserver.ReadOnly(apiserver.RateLimit(rl, m.InsecureHandler))), </del> <ins> Handler:        apiserver.MaxInFlightLimit(sem, longRunningRE, apiserver.RecoverPanics(apiserver.ReadOnly(apiserver.RateLimit(rl, m.InsecureHandler)))), </ins> ReadTimeout:    5 * time.Minute, WriteTimeout:   5 * time.Minute, MaxHeaderBytes: 1 << 20,

if secureLocation != "" { secureServer := &http.Server{ Addr:           secureLocation, <del> Handler:        apiserver.RecoverPanics(m.Handler), </del> <ins> Handler:        apiserver.MaxInFlightLimit(sem, longRunningRE, apiserver.RecoverPanics(m.Handler)), </ins> ReadTimeout:    5 * time.Minute, WriteTimeout:   5 * time.Minute, MaxHeaderBytes: 1 << 20,

"watch":    true, } <ins> // Constant for the retry-after interval on rate limiting. // TODO: maybe make this dynamic? or user-adjustable? const RetryAfter = "1" </ins> // IsReadOnlyReq() is true for any (or at least many) request which has no observable // side effects on state of apiserver (though there may be internal side effects like // caching and logging).

}) } <ins> // MaxInFlight limits the number of in-flight requests to buffer size of the passed in channel. func MaxInFlightLimit(c chan bool, longRunningRequestRE *regexp.Regexp, handler http.Handler) http.Handler { if c == nil { return handler } return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if longRunningRequestRE.MatchString(r.URL.Path) { // Skip tracking long running events. handler.ServeHTTP(w, r) return } select { case c <- true: defer func() { <-c }() handler.ServeHTTP(w, r) default: tooManyRequests(w) } }) } </ins> // RateLimit uses rl to rate limit accepting requests to 'handler'. func RateLimit(rl util.RateLimiter, handler http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {

handler.ServeHTTP(w, req) return } <del> // Return a 429 status indicating "Too Many Requests" w.Header().Set("Retry-After", "1") w.WriteHeader(errors.StatusTooManyRequests) fmt.Fprintf(w, "Rate limit is 10 QPS or a burst of 200") </del> <ins> tooManyRequests(w) </ins> }) } <ins> func tooManyRequests(w http.ResponseWriter) { // Return a 429 status indicating "Too Many Requests" w.Header().Set("Retry-After", RetryAfter) w.WriteHeader(errors.StatusTooManyRequests) fmt.Fprintf(w, "Too many requests, please try again later.") } </ins> // RecoverPanics wraps an http Handler to recover and log panics. func RecoverPanics(handler http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {

"net/http" "net/http/httptest" "reflect" <ins> "regexp" </ins> "strings" <ins> "sync" </ins> "testing" <ins> "time" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/api" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/api/errors" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/api/latest" "github.com/GoogleCloudPlatform/kubernetes/pkg/util" )

func (fakeRL) Stop()             {} func (f fakeRL) CanAccept() bool { return bool(f) } <ins> func expectHTTP(url string, code int, t *testing.T) { r, err := http.Get(url) if err != nil { t.Errorf("unexpected error: %v", err) } if r.StatusCode != code { t.Errorf("unexpected response: %v", r.StatusCode) } } func TestMaxInFlight(t *testing.T) { const Iterations = 3 block := sync.WaitGroup{} block.Add(1) sem := make(chan bool, Iterations) re := regexp.MustCompile("[.*/watch][^/proxy.*]") server := httptest.NewServer(MaxInFlightLimit(sem, re, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if strings.Contains(r.URL.Path, "dontwait") { return } block.Wait() }))) defer server.Close() // These should hang, but not affect accounting. for i := 0; i < Iterations; i++ { // These should hang waiting on block... go func() { expectHTTP(server.URL+"/foo/bar/watch", http.StatusOK, t) }() } for i := 0; i < Iterations; i++ { // These should hang waiting on block... go func() { expectHTTP(server.URL+"/proxy/foo/bar", http.StatusOK, t) }() } expectHTTP(server.URL+"/dontwait", http.StatusOK, t) for i := 0; i < Iterations; i++ { // These should hang waiting on block... go func() { expectHTTP(server.URL, http.StatusOK, t) }() } // There's really no more elegant way to do this.  I could use a WaitGroup, but even then // it'd still be racy. time.Sleep(1 * time.Second) expectHTTP(server.URL+"/dontwait/watch", http.StatusOK, t) // Do this multiple times to show that it rate limit rejected requests don't block. for i := 0; i < 2; i++ { expectHTTP(server.URL, errors.StatusTooManyRequests, t) } block.Done() // Show that we recover from being blocked up. expectHTTP(server.URL, http.StatusOK, t) } </ins> func TestRateLimit(t *testing.T) { for _, allow := range []bool{true, false} { rl := fakeRL(allow)