Code change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: We created a LoadBalancer service with type 'nlb' with two listeners forwarding traffic on two different ports. The service was created correctly along with the target groups and was functionning correctly. Around 20 hours later, no traffic was flowing from clients to any k8s nodes. While troubleshooting we discovered that the k8s node instance security group was lacking the input rules allowing traffic to the node ports for this LB. We deleted the service and recreated it. The service started working normally again, but we found out that the two target groups were missing their k8s tags. The logs contain the following error: What you expected to happen: We expected the NLB target groups to have k8s tags, so that k8s could remember it created them. How to reproduce it (as minimally and precisely as possible): Create a k8s service with NLB: Add a simple deployment that is matched by this service. Anything else we need to know?: Looking at AWS cloudtrail we could see the following error event: So the API request was called with two Resource ARNs, which is confirmed by the code: Since we have two target groups, there are two resources to tag. Except that the AWS API while allowing to send more than one resource ARNs doesn't allow it returning a validation error. This could be solved by doing one per target groups. Note that this can't happen on a Classic LB because there's no target groups. I believe that those missing tags on the target groups are the root cause of the deletion of the security group rules allowing traffic to reach the nodeports, with this (hypothetical and unproved) scenario: 8s creates the target groups, the NLB, the listeners, the sg input rules 8s fails to tag the target groups, but doesn't rollback anything a subsequent convergence loops, k8s notices that there are two k8s-managed Security Group input rules, but those don't match any Target Groups or match Target Groups that are not managed by k8s (because they don't have the owner tags) 8s deletes those rules Unfortunately we don't see the rule deletion in the controller manager logs, but it appears in Cloudtrail: Let me know if you have any questions, Environment: Kubernetes version: 1.10.7 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CoreOS 1967.3.0 Kernel (e.g. ): Linux 4.14.88-coreos Install tools: home grown Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig aws /assign
Hi, the root cause of your problem should already be solved by It's cherry-picked into old versions and available as v1.11.7, v1.12.5 and v1.13.3(hasn't released yet)
, Thanks, you're most certainly right about the root cause of the issue being . It indeed looks exactly like what we've experienced. Still there's the problem of target groups not being tagged.

Related discussion:
text: if err != nil { return nil, fmt.Errorf("Error creating listener: %q", err) } <del> addTagsInput.ResourceArns = append(addTagsInput.ResourceArns, targetGroupArn) </del> <ins> resourceArns = append(resourceArns, targetGroupArn) </ins> } // Add tags to targets <ins> targetGroupTags := make([]*elbv2.Tag, 0, len(tags)) </ins> for k, v := range tags { <del> addTagsInput.Tags = append(addTagsInput.Tags, &elbv2.Tag{ </del> <ins> targetGroupTags = append(targetGroupTags, &elbv2.Tag{ </ins> Key: aws.String(k), Value: aws.String(v), }) } <del> if len(addTagsInput.ResourceArns) > 0 && len(addTagsInput.Tags) > 0 { _, err = c.elbv2.AddTags(addTagsInput) if err != nil { return nil, fmt.Errorf("Error adding tags after creating Load Balancer: %q", err) </del> <ins> if len(resourceArns) > 0 && len(targetGroupTags) > 0 { // elbv2.AddTags doesn't allow to tag multiple resources at once for _, arn := range resourceArns { _, err = c.elbv2.AddTags(&elbv2.AddTagsInput{ ResourceArns: []*string{arn}, Tags:         targetGroupTags, }) if err != nil { return nil, fmt.Errorf("Error adding tags after creating Load Balancer: %q", err) } </ins> } } } else {