Code change:
Hello, I am having issues with a deployment on AWS with version 1.0.1. I had a collision in security group when I launch a service and there already was a security group from an earlier deployment of the "same" cluster. I deleted the security group and then kubernetes managed to create the load balancer so thats good, but since then I am having these messages in a loop: E0729 14:48:07. 8 ] Could not delete route k8s-v1.0.1-10.244.2.0/24 10.244.2.0/24: error deleting AWS route (10.244.2.0/24): RequestLimitExceeded: Request limit exceeded. status code: 503, request id: [] E0729 14:48:07. 8 ] Could not create route -35f5-11e5-8163- 10.244.2.0/24: error configuring source-dest-check on instance i-: RequestLimitExceeded: Request limit exceeded. status code: 503, request id: [] E0729 14:48:07. 8 ] Could not create route -35f5-11e5-8163- 10.244.3.0/24: error creating AWS route (10.244.3.0/24): RequestLimitExceeded: Request limit exceeded. status code: 503, request id: [] E0729 14:48:17. 8 ] Could not create route -35f5-11e5-8163- 10.244.1.0/24: error configuring source-dest-check on instance i-: RequestLimitExceeded: Request limit exceeded. status code: 503, request id: [] How can I stop them at least to get under the limit request and try again ? Thanks
I'm seeing the same thing, and AWS let me know they didn't like it: It seems like maybe when we describe the route tables Kubernetes is not seeing them as being consistent with what it expects internally. Any ideas on what the root cause might be here? I haven't looked at this piece closely yet, but will try to find and fix the issue.
Some research - I believe this has to do with the name that Kubernetes has assigned to my instances internally. tells me that my node looks like this: Whereas describe-route-tables returns the amazon instance identifier: Based on the above, I'm guessing we should just use node.Spec.ExternalID instead of node.Name when doing the reconciliation. I'll start on a PR and testing this.
I am not doing the calls, Kubernetes is for me :). It tried to do a load balancer and failed, after that I helped when i saw the logs saying that there already was some security group, deleted it and then it managed to do the load balancer. So i guess the routes making comes from that, My use case is not something out of the ordinary, I launched a reverse proxy and it tried to create an external load balancer on aws. I could try to create the routes manually in the aws console, just tell me what are the routes that kubernetes is trying to create. And you received a message from AWS on my account or is it one of yours ? Adam Guldemann
Yep, sorry, my flurry of details might have been confusing. What I was trying to say was: Kubernetes is doing the same thing on my AWS account and they actually sent me that message to let me know. The load balancer problem you saw is actually another issue that I believe has been merged and should be in the next release, see . No need to create the routes manually, they have probably been created by Kubernetes. The problem, at least as I understand it, is that Kubernetes tries to reconcile the routes it sees on AWS against what it knows internally. That reconciliation fails because of a mismatch between how AWS and Kubernetes identify nodes. I'm working on a fix for that and will have a PR pending some testing on my cluster. I'll keep you posted about the progress on that here.
thank you for taking this one on - feels pretty serious if AWS are sending warnings out, so I appreciate it. Please tag me when the PR is up (and/or put up a WIP PR and I can test too!) LMK if there's anything I can do to assist.
I attempted a fix for this on . I'm testing now and am considering adding some logging at a higher log level for future debugging.
We should probably add a rate limiter into the AWS client, like we did for GCE:
agree. I can take a shot at that once I get my fix for route reconciliation done.
I am having issues with redis / rethinkdb cluster on my AWS cluster, I am wondering if that could from the fact that it can't create routes or maybe deleting and recreating ? Here is a ping request from one container to another for like 5 minutes: .... 64 bytes from 10.244.1.10: icmpseq=962 ttl=62 time=0.693 ms 64 bytes from 10.244.1.10: icmpseq=963 ttl=62 time=0.670 ms 64 bytes from 10.244.1.10: icmpseq=964 ttl=62 time=0.636 ms 64 bytes from 10.244.1.10: icmpseq=965 ttl=62 time=0.732 ms 64 bytes from 10.244.1.10: icmpseq=966 ttl=62 time=0.590 ms 64 bytes from 10.244.1.10: icmpseq=967 ttl=62 time=0.533 ms 64 bytes from 10.244.1.10: icmpseq=968 ttl=62 time=0.544 ms 64 bytes from 10.244.1.10: icmpseq=969 ttl=62 time=0.605 ms 64 bytes from 10.244.1.10: icmpseq=970 ttl=62 time=0.636 ms 64 bytes from 10.244.1.10: icmpseq=971 ttl=62 time=0.603 ms 64 bytes from 10.244.1.10: icmpseq=972 ttl=62 time=0.548 ms 64 bytes from 10.244.1.10: icmpseq=973 ttl=62 time=0.632 ms 64 bytes from 10.244.1.10: icmpseq=974 ttl=62 time=0.596 ms 64 bytes from 10.244.1.10: icmpseq=975 ttl=62 time=0.436 ms 64 bytes from 10.244.1.10: icmpseq=976 ttl=62 time=0.516 ms 64 bytes from 10.244.1.10: icmpseq=977 ttl=62 time=0.660 ms 64 bytes from 10.244.1.10: icmpseq=978 ttl=62 time=0.532 ms 64 bytes from 10.244.1.10: icmpseq=979 ttl=62 time=0.544 ms 64 bytes from 10.244.1.10: icmpseq=980 ttl=62 time=0.588 ms 64 bytes from 10.244.1.10: icmpseq=981 ttl=62 time=0.553 ms 64 bytes from 10.244.1.10: icmpseq=982 ttl=62 time=0.604 ms 64 bytes from 10.244.1.10: icmpseq=983 ttl=62 time=0.380 ms 64 bytes from 10.244.1.10: icmpseq=984 ttl=62 time=0.531 ms 64 bytes from 10.244.1.10: icmpseq=985 ttl=62 time=0.569 ms 64 bytes from 10.244.1.10: icmp_seq=986 ttl=62 time=0.606 ms ^C--- 10.244.1.10 ping statistics --- 987 packets transmitted, 802 packets received, 18% packet loss round-trip min/avg/max/stddev = 0.323/0.547/4.269/0.165 ms I see 18% packet loss, any ideas where that could come from ?
If I ping google from the same container 0% packet loss to compare: 64 bytes from 8.8.8.8: icmpseq=193 ttl=54 time=0.988 ms 64 bytes from 8.8.8.8: icmpseq=194 ttl=54 time=1.014 ms 64 bytes from 8.8.8.8: icmpseq=195 ttl=54 time=0.952 ms 64 bytes from 8.8.8.8: icmpseq=196 ttl=54 time=1.069 ms 64 bytes from 8.8.8.8: icmpseq=197 ttl=54 time=1.038 ms 64 bytes from 8.8.8.8: icmpseq=198 ttl=54 time=1.094 ms 64 bytes from 8.8.8.8: icmpseq=199 ttl=54 time=1.083 ms 64 bytes from 8.8.8.8: icmpseq=200 ttl=54 time=1.005 ms 64 bytes from 8.8.8.8: icmpseq=201 ttl=54 time=1.042 ms 64 bytes from 8.8.8.8: icmpseq=202 ttl=54 time=1.014 ms ^C--- 8.8.8.8 ping statistics --- 203 packets transmitted, 203 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.887/1.027/1.166/0.055 ms
Closed, but obviously please reopen if the problem continues And I think we're tracking the ping problems separately in ; I have my fingers crossed that we will fix them with this path also.
we're def seeing this right now on AWS. We spun up about 40 nodes using the default k8s aws installation, and are at times crippled by this... Even to the point where pods fail to start because k8s is unable to mount their EBS volumes due to rate limits. Any ideas re workarounds? Anything we can do interim? FWIW, we've spoken to AWS premium support, and they are pretty adamant about not increasing their API quota. They claim that they should be sufficient, and that they won't even raise those for their biggest customers...
I'm curious if you had any luck with this? Unfortunately 1 year later we are experiencing the same problems with EBS volumes. Makes me wonder if the Persistent Volume API makes a ton of calls for EBS volumes. As a side note, we are also having problems with terraform and even packer when running in the same account as our kubernetes cluster, which leads me to believe it is indeed the kubernetes cluster

Related discussion:
text: continue } <ins> instance, err := s.getInstanceById(instanceID) if err != nil { return nil, err } instanceName := orEmpty(instance.PrivateDNSName) </ins> routeName := clusterName + "-" + destinationCIDR <del> routes = append(routes, &cloudprovider.Route{routeName, instanceID, destinationCIDR}) </del> <ins> routes = append(routes, &cloudprovider.Route{routeName, instanceName, destinationCIDR}) </ins> } return routes, nil