Change:
What happened: Create SG outside of K8s - a SG rule has a range of ports which are opened via K8s Add this into Ensure load balancer switches from the range back to individual ports, the CIDR will be 0.0.0.0/0 if loadBalancerSourceRanges was removed too What you expected to happen: K8S doesn't modify the SG How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.13.7 Cloud provider or hardware configuration: EKS OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig aws
Note - I'm not sure if this is related to the aws cloud config, which I can't change as I'm using EKS Add aws cloud config: [global] disableSecurityGroupIngress = true
/sig cloud-provider
We're also having this issue which makes kinda unusable. shouldn't ensure permissions if an SG is a custom one here . could you please take a look?
This is also an issue for us, using these service annotations should have been straightforward. A custom security group id used in the should ensure it's attached to the ELB/LB, and remain untouched. Instead, this customer security group gets purged, leaving the port mapping untouched, but all sources become 0.0.0.0/0.
unfortunately I can't, I don't have access at AWS resources to work on that. To be honest, it was pointed out a number of times that the load balancer code will need a refactor and better tests, I would suggest pinging AWS directly or bringing it up during the meetings.
Suppose you have an existing EKS cluster, gateways, the whole nine yards... Also, suppose you have a custom security group with entries already defined, for example, an SG that holds the rules of your corporate office network ranges, or even customer-specific IPs. You expect, by using this SG id (sg-) in conjunction with this service annotation that: All other security groups kubernetes created during init would be "wiped away", and your custom SG would be the load-balancer's only security group (assuming you only had one). All entries in your custom security group are intact, and would be retained by the control plane. Gotcha, looks like it doesn't. It's reverted back to 0.0.0.0/0 as the source for all rules. Temporary solution: Run the override script (in our case, assign the serviceAnnotations to the gateway). Still use a custom security group, but leave it empty. When all and done, add your custom entries to the SG. As of 08/22, this cheap workaround, works. Still needs investigating.
Any subsequent updates to the cluster, reverts all configurations back to 0.0.0.0/0. EKS was updated from eks.2 to eks.3. Which somehow triggered the changes with istio.
Hi, It looks I have the same issue. Company policy dictates that all SG's have a specific name. Before: we just had service deployment which result in an auto-generated ELB with an auto-generated SG with the 'wrong' name, but everything was working fine. I'm now in the process of getting the correct name. So I created a custom SG "A" which has ingress rules SG "A1" and "A2". I add this with Now the elb is still auto-generated (as expected), but it only has 1 of the SG's from "A", being "A2" and a autogenerated one. I the a file with contents in the location. But I cannot get it to work. We use as , which probably creates this extra rule. I cannot find ( or am probably not able to read correctly) the correct config. Should the aws-load-balancer-internal be used or not? Do I need another directive? Maybe someone can help me out with some pointers? Thanks in advance.
It's possible to work around this by setting to something bogus (i.e. ) and putting your desired SG in the annotation instead (k8s won't modify this one).
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale

Discussion:
// Returns true if and only if changes were made // The security group must already exist func (c *Cloud) setSecurityGroupIngress(securityGroupID string, permissions IPPermissionSet) (bool, error) { <del> // We do not want to make changes to the Global defined SG if securityGroupID == c.cfg.Global.ElbSecurityGroup { return false, nil } </del> group, err := c.findSecurityGroup(securityGroupID) if err != nil { klog.Warningf("Error retrieving security group %q", err)

// Extra groups can be specified via annotation, as can extra tags for any // new groups. The annotation "ServiceAnnotationLoadBalancerSecurityGroups" allows for // setting the security groups specified. <del> func (c *Cloud) buildELBSecurityGroupList(serviceName types.NamespacedName, loadBalancerName string, annotations map[string]string) ([]string, error) { </del> <ins> func (c *Cloud) buildELBSecurityGroupList(serviceName types.NamespacedName, loadBalancerName string, annotations map[string]string) ([]string, bool, error) { </ins> var err error var securityGroupID string <ins> // We do not want to make changes to a Global defined SG var setupSg = false </ins> sgList := getSGListFromAnnotation(annotations[ServiceAnnotationLoadBalancerSecurityGroups]) <del> // The below code changes makes sure that when we have Security Groups  specified with the ServiceAnnotationLoadBalancerSecurityGroups // annotation we don't create a new default Security Groups </del> // If no Security Groups have been specified with the ServiceAnnotationLoadBalancerSecurityGroups annotation, we add the default one. if len(sgList) == 0 { if c.cfg.Global.ElbSecurityGroup != "" { <del> securityGroupID = c.cfg.Global.ElbSecurityGroup </del> <ins> sgList = append(sgList, c.cfg.Global.ElbSecurityGroup) </ins> } else { // Create a security group for the load balancer sgName := "k8s-elb-" + loadBalancerName

securityGroupID, err = c.ensureSecurityGroup(sgName, sgDescription, getLoadBalancerAdditionalTags(annotations)) if err != nil { klog.Errorf("Error creating load balancer security group: %q", err) <del> return nil, err </del> <ins> return nil, setupSg, err </ins> } <ins> sgList = append(sgList, securityGroupID) setupSg = true </ins> } <del> sgList = append(sgList, securityGroupID) </del> } extraSGList := getSGListFromAnnotation(annotations[ServiceAnnotationLoadBalancerExtraSecurityGroups]) sgList = append(sgList, extraSGList...) <del> return sgList, nil </del> <ins> return sgList, setupSg, nil </ins> } // buildListener creates a new listener from the given port, adding an SSL certificate

loadBalancerName := c.GetLoadBalancerName(ctx, clusterName, apiService) serviceName := types.NamespacedName{Namespace: apiService.Namespace, Name: apiService.Name} <del> securityGroupIDs, err := c.buildELBSecurityGroupList(serviceName, loadBalancerName, annotations) </del> <ins> securityGroupIDs, setupSg, err := c.buildELBSecurityGroupList(serviceName, loadBalancerName, annotations) </ins> if err != nil { return nil, err }

return nil, fmt.Errorf("[BUG] ELB can't have empty list of Security Groups to be assigned, this is a Kubernetes bug, please report") } <del> { </del> <ins> if setupSg { </ins> ec2SourceRanges := []*ec2.IpRange{} for _, sourceRange := range sourceRanges.StringSlice() { ec2SourceRanges = append(ec2SourceRanges, &ec2.IpRange{CidrIp: aws.String(sourceRange)})

t.Run(test.name, func(t *testing.T) { serviceName := types.NamespacedName{Namespace: "default", Name: "myservice"} <del> sgList, err := c.buildELBSecurityGroupList(serviceName, "aid", test.annotations) </del> <ins> sgList, setupSg, err := c.buildELBSecurityGroupList(serviceName, "aid", test.annotations) </ins> assert.NoError(t, err, "buildELBSecurityGroupList failed") extraSGs := sgList[1:] assert.True(t, sets.NewString(test.expectedSGs...).Equal(sets.NewString(extraSGs...)), "Security Groups expected=%q , returned=%q", test.expectedSGs, extraSGs) <ins> assert.True(t, setupSg, "Security Groups Setup Permissions Flag expected=%t , returned=%t", true, setupSg) </ins> }) } }

t.Run(test.name, func(t *testing.T) { serviceName := types.NamespacedName{Namespace: "default", Name: "myservice"} <del> sgList, err := c.buildELBSecurityGroupList(serviceName, "aid", test.annotations) </del> <ins> sgList, setupSg, err := c.buildELBSecurityGroupList(serviceName, "aid", test.annotations) </ins> assert.NoError(t, err, "buildELBSecurityGroupList failed") assert.True(t, sets.NewString(test.expectedSGs...).Equal(sets.NewString(sgList...)), "Security Groups expected=%q , returned=%q", test.expectedSGs, sgList) <ins> assert.False(t, setupSg, "Security Groups Setup Permissions Flag expected=%t , returned=%t", false, setupSg) </ins> }) } }