Change:
On a large cluster, the routecontroller basically makes O(n) requests all at once, and most of them error out with "Rate Limit Exceeded". We then wait for operations to complete on the ones that did stick, and damn the rest, they'll be caught next reconcile (which is a while). In fact, in some cases it appears we're spamming hard enough to get rate limited by the upper-level API DOS protections and seeing something akin to , e.g.: We can do better than this!
c.f. , which is the other leg of this problem
I took a look into code and it seems that we are doing full reconciliation every 10s. If we need to create 2000 routes at the beginning it seems to frequently. I think that we need to: first compute how many routes we have to create based on that try to spread them over some time to reduce possibility of causing 'rate limit exceeded' I will try to put some small PR together as a starting point.
I'm reopening this one, since this doesn't seem to be solved. So basically something strange is happening here. With PR we have 10/s limit on API calls send to GCE. And we are still rate-limitted. However, according to documentation we are allowed for 20/s: So something either doesn't work or is weird.
What is super interesting from the logs of recent run: It starts to creating routes: After 2 minutes from that, we are hitting "Rate limit Exceeded" for the first time: And only after 9 minutes we create the first route ever. The second one is created after 15 minutes, and from that point we create a number of them So it seems that nothing useful is happening for the first 2 minutes, and then between 3m30 and 9m30
OK - so I think that what is happening is that all controllers are sharing GCE interface (and thuse they have common throttling). So if one controller is generating a lot of API calls, then other controllers may be throttled. has a hypothesis that it may be caused by nodecontroller.
One thing my PR didn't address (because it was late): Now we have some obnoxious differences in logging when you go to make a request, because the versus could actually be absorbing a fair amount of ratelimiter. It could before because of the operation rate limiter and operation polling anyways, but now could be worse.
- yes I'm aware of that. And probably this is exactly the case here.
I got fooled by that - I guess we should fix it somehow.
One approach to fixing it is a similar approach to the operation polling code, but in the : warn if we ratelimited for more than N seconds (and dump the request or something? not clear how to clarify what we were sending).
I found something else - Kubelet is also building GCE interface: We have 2000 kubelets, so if all of them send multiple requests to GCE, we may hit limits.
Any idea in what circumstances Kubelet will contact cloud provider?
Hmm - it seems it contacts GCE exactly once at the beginning. So it doesn't explain much.
I think it's only here: I've rarely seen GCE ratelimit readonly calls, in practice, and kubelets come up pretty well staggered.
OK - so with logs that I , it is pretty clear that we are very heavily throttled on GCE api calls. In particular, when I was running 1000-node cluster, I see lines where we were throttle for 1m30. I'm pretty sure it's significantly higher in 2000-node cluster. I'm going to investigate a bit more where all those reuqests come from.
Actually - it seems that we have a pretty big problem here. Basically, Kubelet seems to send a lot of requests to GCE. So to clarify: every time Kubelet is updating NodeStatus (so every 10s by default) it is sending an API call to GCE to get it's not addresses: that means, that if we have 2000 nodes, kubelets generate 200 QPS to GCE (which is significantly more than we can afford) We should think how to solve this issue, but I think the options are: increasing QPS quota or calling GCE only once per X statusUpdates.
The second issue is that we are throttled at the controller level too, and this one I'm tryin to understand right now.
- that said, any rate-limitting on our side, will not solve the kubelet-related problem
Hmm - actually looking into implementation it seems that NodeAddresses are only contacting metadata server, so I'm not longer that convinced...
Also, I think that important thing is that a single call to GCE API can translate to multiple api calls to GCE. In particular - CreateRoute translates to: get instance insert route a bunch of getOperation (until this is finished)
OK - so I think that what is happening here is that since CreateRoute translates to: get instance insert route a bunch of getOperations That means that if we call say 2000 CreateRoute() at the same time, all of them will try to issue: getInstance at the beginning. So we will quickly accumulate 2000 GET requests in the queue. Once they are processed, we generate the "POST route" then for any processed. So It's kind of expected what is happening here. So getting back to the Zach`s PR - I think that throttling POST and GET and OPERATION calls separate is kind of good idea in general.
But I still don't really understand why do we get "RateLimitExceeded". Is that because of Kubelets?
I think that one problem we have at the RouteController level is that if CreateRoute fails, we simply don't react on it. And we will wait at least 5 minutes before even retrying it. We should add some retries at the route controller level too. I will send out some short PR for it.
But I still don't understand why do we get those "RateLimitExceeded" error
OK - so as an update. In GCE, we have a separate rate-limit for number of in-flight CreateRoute call per project. is supposed to address this issue.
OK - so this one is actually fixed. I mean that it is still long, but currently it's purely blocked on GCE.
As a comment - we have an internal bug for it already.

Discussion:
// GCECloud is an implementation of Interface, LoadBalancer and Instances for Google Compute Engine. type GCECloud struct { <del> service                  *compute.Service containerService         *container.Service projectID                string region                   string localZone                string   // The zone in which we are running managedZones             []string // List of zones we are spanning (for Ubernetes-Lite, primarily when running on master) networkURL               string nodeTags                 []string // List of tags to use on firewall rules for load balancers useMetadataServer        bool operationPollRateLimiter flowcontrol.RateLimiter </del> <ins> service           *compute.Service containerService  *container.Service projectID         string region            string localZone         string   // The zone in which we are running managedZones      []string // List of zones we are spanning (for Ubernetes-Lite, primarily when running on master) networkURL        string nodeTags          []string // List of tags to use on firewall rules for load balancers useMetadataServer bool </ins> } type Config struct {

return g.service } <ins> type rateLimitedRoundTripper struct { rt      http.RoundTripper limiter flowcontrol.RateLimiter } func (rl *rateLimitedRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) { rl.limiter.Accept() return rl.rt.RoundTrip(req) } </ins> func getProjectAndZone() (string, string, error) { result, err := metadata.Get("instance/zone") if err != nil {

} client := oauth2.NewClient(oauth2.NoContext, tokenSource) <ins> // Override the transport to make it rate-limited. client.Transport = &rateLimitedRoundTripper{ rt:      client.Transport, limiter: flowcontrol.NewTokenBucketRateLimiter(10, 100), // 10 qps, 100 bucket size. } </ins> svc, err := compute.New(client) if err != nil { return nil, err

glog.Infof("managing multiple zones: %v", managedZones) } <del> operationPollRateLimiter := flowcontrol.NewTokenBucketRateLimiter(10, 100) // 10 qps, 100 bucket size. </del> return &GCECloud{ <del> service:                  svc, containerService:         containerSvc, projectID:                projectID, region:                   region, localZone:                zone, managedZones:             managedZones, networkURL:               networkURL, nodeTags:                 nodeTags, useMetadataServer:        useMetadataServer, operationPollRateLimiter: operationPollRateLimiter, </del> <ins> service:           svc, containerService:  containerSvc, projectID:         projectID, region:            region, localZone:         zone, managedZones:      managedZones, networkURL:        networkURL, nodeTags:          nodeTags, useMetadataServer: useMetadataServer, </ins> }, nil }

opName := op.Name return wait.Poll(operationPollInterval, operationPollTimeoutDuration, func() (bool, error) { start := time.Now() <del> gce.operationPollRateLimiter.Accept() </del> duration := time.Now().Sub(start) if duration > 5*time.Second { glog.Infof("pollOperation: waited %v for %v", duration, opName)

package gce import ( <ins> "net/http" "net/http/httptest" </ins> "reflect" "testing" compute "google.golang.org/api/compute/v1" <ins> "k8s.io/kubernetes/pkg/util/flowcontrol" </ins> "k8s.io/kubernetes/pkg/util/rand" <ins> utiltesting "k8s.io/kubernetes/pkg/util/testing" </ins> ) func TestGetRegion(t *testing.T) {

// } } } <ins> func TestRateLimitedRoundTripper(t *testing.T) { handler := utiltesting.FakeHandler{StatusCode: 200} server := httptest.NewServer(&handler) defer server.Close() method := "GET" path := "/foo/bar" req, err := http.NewRequest(method, server.URL+path, nil) if err != nil { t.Errorf("unexpected error: %v", err) } // TODO(zmerlynn): Validate the rate limiter is actually getting called. client := http.Client{ Transport: &rateLimitedRoundTripper{ rt:      http.DefaultTransport, limiter: flowcontrol.NewFakeAlwaysRateLimiter(), }, } _, err = client.Do(req) if err != nil { t.Errorf("unexpected error: %v", err) } handler.ValidateRequest(t, path, method, nil) } </ins>