Change:
Is this a BUG REPORT or FEATURE REQUEST?: /kind feature /sig aws Anything else we need to know?: AWS NLBs support static IP addresses by associating an EIP per AZ. An already allocated EIP can be supplied with the . A user can then supply EIP allocation IDs in the service definition of kubernetes. By that the service will be available via those static IPs.
eip's for white listing are really important in the enterprise, we need this feature now, as a result we have to manually create nlb's, would really like to see this
This is also important for hosting providers, who might have huge numbers of apex domain DNS records pointed at EIPs. Associating an already-allocated EIP with a new Kubernetes-generated NLB would be huge!
This feature would save us weeks of work reimplementing the cloud controller just so we can preserve a special ELB that has static IP addresses... I really wish it was available
I'd like to contribute time to working on this. Is anyone else actively working on adding support?
That's awesome news. Thanks for working on this. Do you need assistance in testing this in an AWS environment? I might help out with that ...
Sure, the PR has been approved for testing so the binaries are available. I'm moving the code into our soak environment to get it some more exposure this week.
I've been trying with: But I only go a NLB without the EIPs deployed. Didn't find any errors in the logs. Do I have to enable a feature flag for it or kind of?
The reviewer found a typo where I may not be collecting the correct annotation. I'm waiting for the new build to complete and then rolling to our soak test. I'm passing through any AWS error and also error if the number of EIP and Subnets don't match. Let me get it online with the changes from code review and I'll dig in deeper. Also, are you using the binaries directly from the build system or just Kubernetes 1.12?
I checked with the fixes you provided yesterday. Works :) Created a service which spawns over 3AZs and supplied 3 EIP allocations. NLBs gets created correctly.
Can you also comment on the PR? I'm sure having multiple successful tests would be helpful to the reviewer.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Hey, thanks for doing this, I was just looking for a way to associate one eip to a nlb, one question though why do you need one eip for each AZ? now the nlb creates a 1 public IP that balance to 3 AZ using traefik in my case when I deploy the service, thanks
Due to the fact that EIPs are represented by network hardware in AWS's datacenter you'd like to have redundant instances. If you run with only one EIP and the AZ of this EIP goes down, you won't be able to reach the (still up and running) traefik deployment in the other two AZs.
Hi, I've seen the PR () has been waiting for an update for three months. do you need a hand? do you want me to pick it up? I am really interested on this to be merged :smile:
Can you help write the tests, ? I'm completely lost there to be honest.
Hi I've found out that test-coverage on the aws provider is low and there is no tests on the so I can't find tests on NLB. I am going to open a new issue and work on improving coverage for that and hopefully I'll find out how to add tests for the new annotation. As you can see below current coverage for and is low and can't find tests for NLB:
Thanks for the help. I thought I was just missing being able to locate them.
+1 need this ASAP We are hosted on EKS so only official version supported. Any idea when this will be merged?
.... As a workaround, I'm just manually creating a second NLB (with EIPs) and changing the listeners to forward the traffic to the automatically created target groups (from the original svc-linked NLB).
Waiting for this as well on EKS. Thanks!
would be great to have that in place. thanks! I can confirm that the manual workaround by works fine, but it is hard to automate that flow.
+1 Waiting for this as well on EKS.
+1
I have created a node project that runs as a cronjob and monitors a SOURCE target group and clones the targets to a DESTINATION target group. So when you create a LoadBalancer of type 'nlb' in kubernetes, all you need to do is manually create a new 'nlb' in AWS Console, assign your EIP to your NLB for a specific 'subnet'(availability zone), then create a target group with no targets; use the ARN from this target group as the destination and use the ARN of the kubernetes target group as the SOURCE. More information here: Public docker image:
I have hard time getting this working with NLB using ingress controller : Even subnets are not taking effect here , its not passing my configurations in the API that creates the NLB kind: Service apiVersion: v1 metadata: name: ingress-nginx namespace: ingress-nginx labels: ingress-nginx ingress-nginx annotations: nlb "eipalloc-,eipalloc-,eipalloc-" "subnet-,subnet-" spec: type: LoadBalancer selector: ingress-nginx ingress-nginx ports: name: http port: 80 targetPort: http name: https port: 443 targetPort: https =======================================================``
I am also getting the same error can anyone help me with this ??
what version of k8s you are using ? this is not supported before 1.15 as far as I know so my issue was that I tried to implement on EKS
Hi i am using the latest version of k8 1.15 .5
I am not using EKS with my solution but the problem is the same as you mentioned
New to EKS. Could you elaborate how the connectivity from NLB that we have created manually (as you mentioned ) to the ingress-nginx pod?
Has it worked for you in k8s 1.15 EKS?
How to get Allocation IDs?
my elastic eip ids cannot be found at service creation, so it just sits in pending forever. if I resort to listing the subnets on their own annotation, (not using subnet discovery) same thing. no matter what I do it seems to not find me elastic ip addresses. Are you sure nlb type is supported to work with Elastic IPs? From the AWS docs it seem they are pretty heavily trying to discourage use of the nlb type (Cloud Provider Load Balancer) in favor of pushing the new AWS Load balancer.
See available annotations: https://kubernetes- If you have done that: what do your aws-lb-controller logs tell you? I guess there’s an access problem with your iam role. Is your aws-lb-controller iam autorized to list and modify eip’s? Are the eip annotations exactly correct, can they be found by the controller? The controller’s log stream should have some kind of an error there. I have successfully assigned eips using service annotations to provision an aws nlb, but pay attention, you need to install the aws-lb-controller yourself. The kubernetes built-in aws library does not support eips.
Here is the annotation needed for assigning a static Elastic IP to your NLB using the AWS Load Balancer (Ingress) Controller https://kubernetes- New Controller link https://kubernetes-

Discussion:
"io" "net" "path" <ins> "sort" </ins> "strconv" "strings" "sync"

"github.com/aws/aws-sdk-go/service/kms" "github.com/aws/aws-sdk-go/service/sts" "gopkg.in/gcfg.v1" <ins> "k8s.io/api/core/v1" </ins> "k8s.io/klog" <del> "k8s.io/api/core/v1" </del> "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types"

// service to specify, in seconds, the interval between health checks. const ServiceAnnotationLoadBalancerHCInterval = "service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval" <ins> // ServiceAnnotationLoadBalancerEIPAllocations is the annotation used on the // service to specify a comma separated list of EIP allocations to use as // static IP addresses for the NLB. Only supported on elbv2 (NLB) const ServiceAnnotationLoadBalancerEIPAllocations = "service.beta.kubernetes.io/aws-load-balancer-eip-allocations" </ins> // Event key when a volume is stuck on attaching state when being attached to a volume const volumeAttachmentStuck = "VolumeAttachmentStuck"

continue } <ins> var azNames []string for key := range subnetsByAZ { azNames = append(azNames, key) } sort.Strings(azNames) </ins> var subnetIDs []string <del> for _, subnet := range subnetsByAZ { subnetIDs = append(subnetIDs, aws.StringValue(subnet.SubnetId)) </del> <ins> for _, key := range azNames { subnetIDs = append(subnetIDs, aws.StringValue(subnetsByAZ[key].SubnetId)) </ins> } return subnetIDs, nil

createRequest.Scheme = aws.String("internal") } <ins> var allocationIDs []string if eipList, present := annotations[ServiceAnnotationLoadBalancerEIPAllocations]; present { allocationIDs = strings.Split(eipList, ",") if len(allocationIDs) != len(subnetIDs) { return nil, fmt.Errorf("Error creating load balancer: Must have same number of EIP AllocationIDs (%d) and SubnetIDs (%d)", len(allocationIDs), len(subnetIDs)) } } </ins> // We are supposed to specify one subnet per AZ. // TODO: What happens if we have more than one subnet per AZ? <del> createRequest.SubnetMappings = createSubnetMappings(subnetIDs) </del> <ins> createRequest.SubnetMappings = createSubnetMappings(subnetIDs, allocationIDs) </ins> for k, v := range tags { createRequest.Tags = append(createRequest.Tags, &elbv2.Tag{

return true } <del> func createSubnetMappings(subnetIDs []string) []*elbv2.SubnetMapping { </del> <ins> func createSubnetMappings(subnetIDs []string, allocationIDs []string) []*elbv2.SubnetMapping { </ins> response := []*elbv2.SubnetMapping{} <del> for _, id := range subnetIDs { // Ignore AllocationId for now response = append(response, &elbv2.SubnetMapping{SubnetId: aws.String(id)}) </del> <ins> for index, id := range subnetIDs { sm := &elbv2.SubnetMapping{SubnetId: aws.String(id)} if len(allocationIDs) > 0 { sm.AllocationId = aws.String(allocationIDs[index]) } response = append(response, sm) </ins> } return response