Code change:
What happened: We were running a service with LoadBalancer type with AWS cloud provider and after deleting this service, the cleanup process on AWS removed a wrong ingress input rule which provoked that all traffic between nodes was unable to reach out the destination. The service we were running had a configuration similar to: After applying this manfiest on the cluster, the cloud provider created the associated load balancer with two security groups: sg- (due to extra security group annotation) sg- (a new SG created by the AWS cloud provider with the rules defined on the ) Cloud provider also created a new input rule on the nodes security group which allows all traffic from all ports from (which is the sg created for the newly created load balancer in our example). One particularity of our setup is that the security group we as was the nodes security group. The input rules of this security group after the creation of this service was similar to: Everything worked fine with this setup until we deleted this service and the cloud provider started the deletion process of the LoadBalancer on AWS. We've been debugging the deletion code and we found that the first thing it does is cleaning up the security group. To do it, it first removes the input rule from the nodes security group which allows traffic from the load balancer security group (sg-, on this example) and then, as it's not referenced, it removes the security group. The expected output of this process, would be that nodes security groups would be: But, in our case, it ended up as: which dropped all the traffic between nodes (as it didn't allow any traffic between them). The cleanup process didn't properly remove the input rule from the node security group (the one which allows all traffic from source sg-) and it also did not remove the sg- security group. In our case, as we were using the nodes security group on the extra security groups, this bug provoked a network partition between nodes, but we think that if is used, the cleanup process may be broken and have unexpected side-effects like those we observed. We've been debugging and we found that the process which removes the input rule from the nodes security groups starts here: The first thing this function does is this: : When extra security groups are used, will contain the security group created for the LoadBalancer and all the extra security groups. We also found that AWS returns this SecurityGroups list in distinct orders after changing the LoadBalancer configuration. That code is always assuming that latest security group on the list is the , which may not be true when extra security groups are used. On the cases where the load balancer security group is not returned on the latest position, the cleanup process does not remove the security group and the input rule from the nodes security group is not removed as expected. We've seen that master branch still contains this code and we think this may be hapenning on newer versions: ), so we think that newer versions may be affected by this bug (we didn't try it). In our case, we saw several times this log during the removal process: Additionally, the code tried to remove the security group (but being unable to do so), after wrongly revoking the rule we have talked about before. What you expected to happen: The cleanup process should properly detect the load balancer security group, remove the input rule from nodes security group and remove the load balancer security group, regardless of the order in which AWS returns the security group list on . How to reproduce it (as minimally and precisely as possible): We did not find any consistent process to create the scenario, as it happens randomly (we are not sure if due to AWS or any kind of race condition on AWS cloud provider). Setup: Create a Service with running controller manager with AWS cloud provider with tha annotation containing one or more security groups (as the one the section) Wait until load balancer is created Run Check if does not have the newly created security group for the balancer in the latest position. Repeat until the balancer does not have the balancer security group on the latest position Trigger the bug: Result:The log will appear multiple times on controller-manager logs Load balancer associated security group is not removed Nodes security group input rule allowing all traffic from load balancer security group is not removed If the latest security group on the list was the nodes security group, the input rule allowing all traffic on all ports from all the other nodes is removed and no traffic is allowed between nodes on the cluster. Environment: Kubernetes version (use ): version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.10", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-05-01T04:05:01Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: aws OS (e.g: ):Kernel (e.g. ): Linux ip-10-2-13-67 4.9.0-7-amd64 SMP Debian 4.9.110-3+deb9u2 (2018-08-13) x8664 GNU/Linux Install tools: kops
/sig cloud-provider
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/area provider/aws /lifecycle frozen
Working on a repro for this issue (on 1.11 first, then I'll try on master) and I've not quite observed the same behavior yet. Depending on the order of the security groups on the load balancer, I only see the ingress rule for the new LB security group to the node security group if the new security group comes last. Otherwise, if the NODE security group comes last, the ingress rule is not , and I see in the logs. Even without a full repro yet, I find it troubling that behavior is dependent on the order of security groups attached to the load balancer. One question I have related to your setup, can you describe the reason why you're adding the node security group as an ? I'd like to understand your use case a little better.
Hi Nicholas. I have a working patch for this specific issue: This bug was the subject of a talk for a local meetup I did recently. I also published a personal blog post talking about it: There you can see more details about how it can be reproduced easily. If you think the patch is valid, I'll try to add a few test cases for it and then propose the change upstream. Thanks!
It has been a while since we got this error and I'm not sure if we saw this error. What I experienced is that the order is changed by Kubernetes or by some call to AWS. It took me not more than 3 attempts to get this behavior. There's something I can do to help you reproduce the error? Do you need more details of our setup? This was a misconfiguration on our side. When we noticed that this setup was not correct we removed this security group from , which removed the mentioned security group rule and took down the Kubernetes cluster (no communication between nodes was allowed).

Related discussion:
text: assert.Equal(t, testCase.region, result) } } <ins> func TestCloud_sortELBSecurityGroupList(t *testing.T) { type args struct { securityGroupIDs []string annotations      map[string]string } tests := []struct { name                 string args                 args wantSecurityGroupIDs []string }{ { name: "with no annotation", args: args{ securityGroupIDs: []string{"sg-1"}, annotations:      map[string]string{}, }, wantSecurityGroupIDs: []string{"sg-1"}, }, { name: "with service.beta.kubernetes.io/aws-load-balancer-security-groups", args: args{ securityGroupIDs: []string{"sg-2", "sg-1", "sg-3"}, annotations: map[string]string{ "service.beta.kubernetes.io/aws-load-balancer-security-groups": "sg-3,sg-2,sg-1", }, }, wantSecurityGroupIDs: []string{"sg-3", "sg-2", "sg-1"}, }, { name: "with service.beta.kubernetes.io/aws-load-balancer-extra-security-groups", args: args{ securityGroupIDs: []string{"sg-2", "sg-1", "sg-3", "sg-4"}, annotations: map[string]string{ "service.beta.kubernetes.io/aws-load-balancer-extra-security-groups": "sg-3,sg-2,sg-1", }, }, wantSecurityGroupIDs: []string{"sg-4", "sg-3", "sg-2", "sg-1"}, }, { name: "with both annotation", args: args{ securityGroupIDs: []string{"sg-2", "sg-1", "sg-3", "sg-4", "sg-5", "sg-6"}, annotations: map[string]string{ "service.beta.kubernetes.io/aws-load-balancer-security-groups":       "sg-3,sg-2,sg-1", "service.beta.kubernetes.io/aws-load-balancer-extra-security-groups": "sg-6,sg-5", }, }, wantSecurityGroupIDs: []string{"sg-3", "sg-2", "sg-1", "sg-4", "sg-6", "sg-5"}, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { c := &Cloud{} c.sortELBSecurityGroupList(tt.args.securityGroupIDs, tt.args.annotations) assert.Equal(t, tt.wantSecurityGroupIDs, tt.args.securityGroupIDs) }) } } </ins>