Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: We run Kuberentes on AWS. We are trying out service type. We use Terraform heavily for managing infrastructure on AWS. As a result we have several shared security groups with "perfect" rules. One of them is supposed to be attached to ELBs so our own IPs, partners, etc are whitelisted. Right now, it seems there isn't a way for Kubernetes to use the security group as is. There are several alternatives: Pass my existing security group as annotation. However, Kubernetes still tries to create one first, or use a global security group. If I don't provide any IP range, Kubernetes will whitelist . This seems insecure. Provide my own IP ranges as either annotation or . Then Kubernetes will create a security group for this ELB with the given IPs. This means I need to duplicate my IPs in all microservices. Provide my own IP ranges as either annotation or , and use a global security group. Then Kubernetes will modify the global SG with the given IPs. This seems problematic when lots of microservices share one SG but some want different IPs whitelisted. I want to entertain the idea below: Allow users to sepcify another security group to or object, which Kubernetes will take it and attach it to ELBs as is. If such security group is provided, skip other IP whitelisting steps. What you expected to happen: I can give Kubernetes existing security group(s). Kuberentes just attaches the security group(s) to the managed ELBs without modifying the rules. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: It may already be possible to achieve this, and I'm not aware of. Or there are other concerns so that Kubernetes didn't choose to implement this feature. Environment: Kubernetes version (use ): v1.7.1 Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig aws
PR may help with this. . .
This problem could be really critical if one has the Kubernetes dashboard installed. If for some reasons the service recreates AWS ELB, the related security group will be recreated as well with the "0.0.0.0/0" input/output masks and then botnets could find the ELB and . JFYI, in our case in the one of clusters with K8s 1.5, the deployment and replication controller were installed from the dashboard using this vector of attack. Then special pods were created and root volume mounted inside. The system crontab was modified by processes in these pods and the crypto miner appeared in all K8s nodes on the system level. After this another pods were installed and the miner was executed inside the Kubernetes pods in parallel with the system level.
just had some downtime on a production cluster because of this. i really need it too ;)
+1
As for me, it isn't a feature, it's a kind of bug.
Some comments on which may be relevant - please comment if an alternative annotation that did replace the default security-groups (as opposed to adding to them) would/wouldn't meet your use-case:
For me, I don't add any annotation, I just want to change its security group rule. After it will create a aws elb, and a specificed secruity group with 0.0.0.0, I change 0.0.0.0 to However, the security group rule will change back to 0.0.0.0 after some minutes. Is this by design? Is there any way for me keep my changes, thanks.
I guess this is part of how the controller works. If you look at , the PR fixes this problem by allowing you to specify the behaviour you want with an annotation. For the way Kubernetes works, the ELB is owned by Kubernetes and you should never be forced to modify the resource manually, thing that ultimately will not work.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale This is still relevant cause is not merged yet.
This is a good change. The situation pre-1.13.0 is somewhat surprising because explicitly setting to an empty array like so: has no effect, and the Security Group Ingress is still .

Discussion:
const ServiceAnnotationLoadBalancerCrossZoneLoadBalancingEnabled = "service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled" // ServiceAnnotationLoadBalancerExtraSecurityGroups is the annotation used <del> // one the service to specify additional security groups to be added to ELB created </del> <ins> // on the service to specify additional security groups to be added to ELB created </ins> const ServiceAnnotationLoadBalancerExtraSecurityGroups = "service.beta.kubernetes.io/aws-load-balancer-extra-security-groups" <ins> // ServiceAnnotationLoadBalancerSecurityGroups is the annotation used // on the service to specify the security groups to be added to ELB created. Differently from the annotation // "service.beta.kubernetes.io/aws-load-balancer-extra-security-groups", this replaces all other security groups previously assigned to the ELB. const ServiceAnnotationLoadBalancerSecurityGroups = "service.beta.kubernetes.io/aws-load-balancer-security-groups" </ins> // ServiceAnnotationLoadBalancerCertificate is the annotation used on the // service to request a secure listener. Value is a valid certificate ARN. // For more, see http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/elb-listener-config.html

// attached to ELB created by a service. List always consist of at least // 1 member which is an SG created for this service or a SG from the Global config. // Extra groups can be specified via annotation, as can extra tags for any <del> // new groups. </del> <ins> // new groups. The annotation "ServiceAnnotationLoadBalancerSecurityGroups" allows for // setting the security groups specified. </ins> func (c *Cloud) buildELBSecurityGroupList(serviceName types.NamespacedName, loadBalancerName string, annotations map[string]string) ([]string, error) { var err error var securityGroupID string

return nil, err } } <del> sgList := []string{securityGroupID} </del> <ins> sgList := []string{} for _, extraSG := range strings.Split(annotations[ServiceAnnotationLoadBalancerSecurityGroups], ",") { extraSG = strings.TrimSpace(extraSG) if len(extraSG) > 0 { sgList = append(sgList, extraSG) } } // If no Security Groups have been specified with the ServiceAnnotationLoadBalancerSecurityGroups annotation, we add the default one. if len(sgList) == 0 { sgList = append(sgList, securityGroupID) } </ins> for _, extraSG := range strings.Split(annotations[ServiceAnnotationLoadBalancerExtraSecurityGroups], ",") { extraSG = strings.TrimSpace(extraSG)

} } <ins> func TestLBSecurityGroupsAnnotation(t *testing.T) { awsServices := newMockedFakeAWSServices(TestClusterId) c, _ := newAWSCloud(CloudConfig{}, awsServices) sg1 := map[string]string{ServiceAnnotationLoadBalancerSecurityGroups: "sg-000001"} sg2 := map[string]string{ServiceAnnotationLoadBalancerSecurityGroups: "sg-000002"} sg3 := map[string]string{ServiceAnnotationLoadBalancerSecurityGroups: "sg-000001, sg-000002"} tests := []struct { name string annotations map[string]string expectedSGs []string }{ {"SG specified", sg1, []string{sg1[ServiceAnnotationLoadBalancerSecurityGroups]}}, {"Multiple SGs specified", sg3, []string{sg1[ServiceAnnotationLoadBalancerSecurityGroups], sg2[ServiceAnnotationLoadBalancerSecurityGroups]}}, } awsServices.ec2.(*MockedFakeEC2).expectDescribeSecurityGroups(TestClusterId, "k8s-elb-aid", "cluster.test") for _, test := range tests { t.Run(test.name, func(t *testing.T) { serviceName := types.NamespacedName{Namespace: "default", Name: "myservice"} sgList, err := c.buildELBSecurityGroupList(serviceName, "aid", test.annotations) assert.NoError(t, err, "buildELBSecurityGroupList failed") assert.True(t, sets.NewString(test.expectedSGs...).Equal(sets.NewString(sgList...)), "Security Groups expected=%q , returned=%q", test.expectedSGs, sgList) }) } } </ins> // Test that we can add a load balancer tag func TestAddLoadBalancerTags(t *testing.T) { loadBalancerName := "test-elb"