Change:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After a rolling update of the cluster, NLB kept failing health checks because security group did not have rules for new EC2 instance. What you expected to happen: Security group should have been updated with both TCP rules for NLBs to work correctly. How to reproduce it (as minimally and precisely as possible): Create a service object with type LoadBalancer and adding the annotation for NLBs. FWIW, I had 6 services and each had their own NLBs. It’s possible that the bug happens when there are multiple NLBs causing multiple AWS updates. Perform a rolling update of the instance group. This causes the instances to be terminated and new ones to be brought up. AFAICT, this should result in two changes to the security group per EC2 instance. When the instance is terminated, the rules for that nodePort should be deleted. When the new node is created, the rule should be again. However, after the rolling update, the rule doesn’t get . As a result, the NLB’s health checks fail. Anything else we need to know?: As mentioned earlier, this happened on a cluster where there were multiple NLBs. This doesn’t seem to be a deterministic failure. The log has the following error message:I manually edited the Service object to see if it would simply retrigger an update on the cloud resources. It actually did. The rules got to the security group. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: kops 1.8.1 Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
did you find any workarounds? we've run into this issue, where the security group rules got deleted. Specifically, we found that the security group missed the following: 1) TCP rule to allow traffic from NLB to Node Port 2) TCP rule to allow traffic from NLB to Healthcheck Port We these manually back in to fix, but were surprised these were deleted in the first place.
This stackoverflow also is relevant:
One workaround was to simply refresh the Kubernetes service object. We /updated a label in the Kubernetes service object and found that it would make the corresponding changes in AWS.
thanks have you been able to root cause why the rule doesn't get ? We're trying to re-produce the failure ourselves, too.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Any update on this bug fix release please?
Sorry to bother you with commenting on this issue. But I cannot find out in what version of Kubernetes this is expected to be released / was released. Any clarification highly appreciated.

Discussion:
response := []*ec2.SecurityGroup{} clientRule := fmt.Sprintf("%s=%s", NLBClientRuleDescription, lbName) healthRule := fmt.Sprintf("%s=%s", NLBHealthCheckRuleDescription, lbName) <ins> alreadyAdded := sets.NewString() </ins> for i := range securityGroups { for j := range securityGroups[i].IpPermissions { for k := range securityGroups[i].IpPermissions[j].IpRanges { description := aws.StringValue(securityGroups[i].IpPermissions[j].IpRanges[k].Description) if description == clientRule || description == healthRule { <del> response = append(response, securityGroups[i]) </del> <ins> sgIDString := aws.StringValue(securityGroups[i].GroupId) if !alreadyAdded.Has(sgIDString) { response = append(response, securityGroups[i]) alreadyAdded.Insert(sgIDString) } </ins> } } }

// if clientTraffic is false, then only update HealthCheck rules func (c *Cloud) updateInstanceSecurityGroupsForNLBTraffic(actualGroups []*ec2.SecurityGroup, desiredSgIds []string, ports []int64, lbName string, clientCidrs []string, clientTraffic bool) error { <ins> klog.V(8).Infof("updateInstanceSecurityGroupsForNLBTraffic: actualGroups=%v, desiredSgIds=%v, ports=%v, clientTraffic=%v", actualGroups, desiredSgIds, ports, clientTraffic) </ins> // Map containing the groups we want to make changes on; the ports to make // changes on; and whether to add or remove it. true to add, false to remove portChanges := map[string]map[int64]bool{}

if add { if clientTraffic { klog.V(2).Infof("Adding rule for client MTU discovery from the network load balancer (%s) to instances (%s)", clientCidrs, instanceSecurityGroupID) <del> klog.V(2).Infof("Adding rule for client traffic from the network load balancer (%s) to instances (%s)", clientCidrs, instanceSecurityGroupID) </del> <ins> klog.V(2).Infof("Adding rule for client traffic from the network load balancer (%s) to instances (%s), port (%v)", clientCidrs, instanceSecurityGroupID, port) </ins> } else { <del> klog.V(2).Infof("Adding rule for health check traffic from the network load balancer (%s) to instances (%s)", clientCidrs, instanceSecurityGroupID) </del> <ins> klog.V(2).Infof("Adding rule for health check traffic from the network load balancer (%s) to instances (%s), port (%v)", clientCidrs, instanceSecurityGroupID, port) </ins> } } else { if clientTraffic { klog.V(2).Infof("Removing rule for client MTU discovery from the network load balancer (%s) to instances (%s)", clientCidrs, instanceSecurityGroupID) <del> klog.V(2).Infof("Removing rule for client traffic from the network load balancer (%s) to instance (%s)", clientCidrs, instanceSecurityGroupID) </del> <ins> klog.V(2).Infof("Removing rule for client traffic from the network load balancer (%s) to instance (%s), port (%v)", clientCidrs, instanceSecurityGroupID, port) </ins> } <del> klog.V(2).Infof("Removing rule for health check traffic from the network load balancer (%s) to instance (%s)", clientCidrs, instanceSecurityGroupID) </del> <ins> klog.V(2).Infof("Removing rule for health check traffic from the network load balancer (%s) to instance (%s), port (%v)", clientCidrs, instanceSecurityGroupID, port) </ins> } if clientTraffic {

package aws import ( <ins> "fmt" </ins> "testing" "github.com/aws/aws-sdk-go/aws" <ins> "github.com/aws/aws-sdk-go/service/ec2" </ins> ) func TestElbProtocolsAreEqual(t *testing.T) {

} } } <ins> func TestSecurityGroupFiltering(t *testing.T) { grid := []struct { in          []*ec2.SecurityGroup name        string expected    int description string }{ { in: []*ec2.SecurityGroup{ { IpPermissions: []*ec2.IpPermission{ { IpRanges: []*ec2.IpRange{ { Description: aws.String("an unmanaged"), }, }, }, }, }, }, name:        "unmanaged", expected:    0, description: "An environment without managed LBs should have %d, but found %d SecurityGroups", }, { in: []*ec2.SecurityGroup{ { IpPermissions: []*ec2.IpPermission{ { IpRanges: []*ec2.IpRange{ { Description: aws.String("an unmanaged"), }, { Description: aws.String(fmt.Sprintf("%s=%s", NLBClientRuleDescription, "managedlb")), }, { Description: aws.String(fmt.Sprintf("%s=%s", NLBHealthCheckRuleDescription, "managedlb")), }, }, }, }, }, }, name:        "managedlb", expected:    1, description: "Found %d, but should have %d Security Groups", }, } for _, g := range grid { actual := len(filterForIPRangeDescription(g.in, g.name)) if actual != g.expected { t.Errorf(g.description, actual, g.expected) } } } </ins>