Change:
After setting up a few services, I noticed the following output in : I assume the service tried to configure an ELB in the past (not sure when or how or what details it tried to use) and has kept trying and now can't even use the AWS API. How can I either: 1) tell it to please stop so that I can use my AWS account again 2) fix it so that it does configure the ELB /cc
Sorry about the problem. Looks like we need a sleep / backoff policy here! To get you up and running again, if you redeploy the nginx service without Type=LoadBalancer, we will (should) stop trying to create a load balancer, and you should get out of the AWS API penalty-box. Sorry about this, and thanks for the report. Looks like this is pretty important to fix. Do you happen to know why you can't create an ELB? My guess is that you have hit your quota?
I'll remove the service, wait for the penalty-box to expire, then build it again so that we can see what the real error was.
Adding the LoadBalancer service back in, I'm seeing:
I took a few different steps and seem to have gotten it both working in different ways: First I deleted the service because it was erroring, but I noticed the count was still going up. So I deleted the security group it said was a duplicate and then it finished setting the ELB up! That meant I needed to manually clean up the ELB resources and security groups because k8 thought it was already deleted. After cleaning up all the resources in AWS again, I the service again and it seemed to work fine! I'm not sure how/why it initially confused itself with the duplicate SG, but it doesn't necessarily seem to happen every time. The code could likely just, in the event that it sees the SG already exists, continue on with the setup. I could send you a PR to handle that if you'd like!
I have a PR that I think covers this: . I thought it was a one-off error, but it sounds like it wasn't. Thanks so much for pausing & retrying also - that was super-helpful. Even if we think covers this, let's leave this issue open to track the need for a backoff to avoid overloading the AWS API. (And presumably this aggressive retrying happens on other clouds also, though maybe they have overload a sleep built in).
Certainly looks like should fix it. I'm still wondering why I'd see it failing literally for hours in a loop though. If it was a problem with AWS being eventually consistent, then after an hour I'd think that first check, looking to see if the securityGroup existed, would start to return the group! I don't know the k8 code well yet, though so perhaps it was looping in a different place that wasn't running that code to see if the security group already existed.
you're right; my explanation wasn't entirely correct... So security group names are unique per VPC. We also have a per-cluster name, and we filter all our AWS API interactions to only select resources tagged with this name. If we had a security group name collision across two k8s clusters in the same VPC, that would be bad. Thankfully the security group name we're creating includes a UUID, so this "won't happen". However, the problem is that we created the security group, and then failed to tag it. But we were filtering based on our cluster. So our "describe security groups" operation would never see the SG, because it wasn't appropriately tagged. We have a few workarounds to make this much less likely, and I have opened the "big picture" issue . And I'm going to think about whether we should in fact be filtering when we DescribeSecurityGroups - I think the answer is no in the case where the name has a UUID in it.
Ah! That totally makes sense. The race between resource creation and resource tagging can created "invisible" resources if the process of tagging fails. Maybe an anti-entropy function that can look at all the resources and normalize them. In the case of invisible resources, you can just delete them and hope that recreating them won't fail the tagging process.
I see this as well when frequently deploying/testing. Unfortunately, it happens far too often (running v1.0.0).
+1
I've been having a similar issue to this, Kubernetes can't create a load balancer as it says the security group already exists. The workaround I've found is to manually delete the security group and wait for Kubernetes to successfully create it again. I've generally had the problem when quickly deleting a service with a and then creating it again (I see now I should use ). I wonder if it is something to do with AWS reassigning the same load balancer back to you if you haven't finished deleting it coupled with the tagging issue mentioned earlier.
+1
cc
I had an email discussion with on this and he said that what's remaining shouldn't be considered a blocker for 1.1, so I'll move it to 1.2-candidate.
Can you summarize what's remaining to do for this issue? Should we just close it in favor of ?
I see you the v1.2 milestone -- will you be able to look into this in the next few days?

Discussion:
// apply tags if volumeOptions.Tags != nil { <del> tags := []*ec2.Tag{} for k, v := range *volumeOptions.Tags { tag := &ec2.Tag{} tag.Key = aws.String(k) tag.Value = aws.String(v) tags = append(tags, tag) } tagRequest := &ec2.CreateTagsInput{} tagRequest.Resources = []*string{&awsID} tagRequest.Tags = tags if _, err := s.createTags(tagRequest); err != nil { </del> <ins> if err := s.createTags(awsID, *volumeOptions.Tags); err != nil { </ins> // delete the volume and hope it succeeds _, delerr := s.DeleteDisk(volumeName) if delerr != nil {

return true, nil } <ins> // Ensure that a resource has the correct tags // If it has no tags, we assume that this was a problem caused by an error in between creation and tagging, // and we add the tags.  If it has a different cluster's tags, that is an error. func (s *AWSCloud) ensureClusterTags(resourceID string, tags []*ec2.Tag) error { actualTags := make(map[string]string) for _, tag := range tags { actualTags[aws.StringValue(tag.Key)] = aws.StringValue(tag.Value) } addTags := make(map[string]string) for k, expected := range s.filterTags { actual := actualTags[k] if actual == expected { continue } if actual == "" { glog.Warningf("Resource %q was missing expected cluster tag %q.  Will add (with value %q)", resourceID, k, expected) addTags[k] = expected } else { return fmt.Errorf("resource %q has tag belonging to another cluster: %q=%q (expected %q)", resourceID, k, actual, expected) } } if err := s.createTags(resourceID, addTags); err != nil { return fmt.Errorf("error adding missing tags to resource %q: %v", resourceID, err) } return nil } </ins> // Makes sure the security group exists // Returns the security group id or error func (s *AWSCloud) ensureSecurityGroup(name string, description string, vpcID string) (string, error) {

newEc2Filter("group-name", name), newEc2Filter("vpc-id", vpcID), } <del> request.Filters = s.addFilters(filters) </del> <ins> // Note that we do _not_ add our tag filters; group-name + vpc-id is the EC2 primary key. // However, we do check that it matches our tags. // If it doesn't have any tags, we tag it; this is how we recover if we failed to tag before. // If it has a different cluster's tags, that is an error. // This shouldn't happen because name is expected to be globally unique (UUID derived) request.Filters = filters </ins> securityGroups, err := s.ec2.DescribeSecurityGroups(request) if err != nil {

if len(securityGroups) > 1 { glog.Warning("Found multiple security groups with name:", name) } <del> return orEmpty(securityGroups[0].GroupId), nil </del> <ins> err := s.ensureClusterTags(aws.StringValue(securityGroups[0].GroupId), securityGroups[0].Tags) if err != nil { return "", err } return aws.StringValue(securityGroups[0].GroupId), nil </ins> } createRequest := &ec2.CreateSecurityGroupInput{}

return "", fmt.Errorf("created security group, but id was not returned: %s", name) } <del> tags := []*ec2.Tag{} for k, v := range s.filterTags { tag := &ec2.Tag{} tag.Key = aws.String(k) tag.Value = aws.String(v) tags = append(tags, tag) } if len(tags) > 0 { tagRequest := &ec2.CreateTagsInput{} tagRequest.Resources = []*string{&groupID} tagRequest.Tags = tags if _, err := s.createTags(tagRequest); err != nil { // Not clear how to recover fully from this; we're OK because we don't match on tags, but that is a little odd return "", fmt.Errorf("error tagging security group: %v", err) } </del> <ins> err := s.createTags(groupID, s.filterTags) if err != nil { // If we retry, ensureClusterTags will recover from this - it // will add the missing tags.  We could delete the security // group here, but that doesn't feel like the right thing, as // the caller is likely to retry the create return "", fmt.Errorf("error tagging security group: %v", err) </ins> } return groupID, nil }

// createTags calls EC2 CreateTags, but adds retry-on-failure logic // We retry mainly because if we create an object, we cannot tag it until it is "fully created" (eventual consistency) // The error code varies though (depending on what we are tagging), so we simply retry on all errors <del> func (s *AWSCloud) createTags(request *ec2.CreateTagsInput) (*ec2.CreateTagsOutput, error) { </del> <ins> func (s *AWSCloud) createTags(resourceID string, tags map[string]string) error { if tags == nil || len(tags) == 0 { return nil } var awsTags []*ec2.Tag for k, v := range tags { tag := &ec2.Tag{ Key:   aws.String(k), Value: aws.String(v), } awsTags = append(awsTags, tag) } request := &ec2.CreateTagsInput{} request.Resources = []*string{&resourceID} request.Tags = awsTags </ins> // TODO: We really should do exponential backoff here attempt := 0 maxAttempts := 60 for { <del> response, err := s.ec2.CreateTags(request) </del> <ins> _, err := s.ec2.CreateTags(request) </ins> if err == nil { <del> return response, err </del> <ins> return nil </ins> } // We could check that the error is retryable, but the error code changes based on what we are tagging

attempt++ if attempt > maxAttempts { glog.Warningf("Failed to create tags (too many attempts): %v", err) <del> return response, err </del> <ins> return err </ins> } glog.V(2).Infof("Failed to create tags; will retry.  Error was %v", err) time.Sleep(1 * time.Second)