Change:
What happened: What you expected to happen: kubernetes creating an AWS NLB for UDP How to reproduce it (as minimally and precisely as possible): Create service with annotation and port with Anything else we need to know?: AWS recently announced that UDP is available within their NLB Environment: Kubernetes version (use ): 1.11.9 Cloud provider or hardware configuration: AWS
/sig aws
/kind feature
/remove-kind bug
/milestone v1.16
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : /milestone v1.16 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.16
I am taking up this Feature Development. How can this be assigned to me?
/assign
+1
+1 I am experiencing the same issue. NLB also supports UDP_TCP type, any service using "nlb" should be able to mix TCP and UDP port configs.
Any progress? I can collaborate on this, if you need a hand.
+1
+1 On Thu, Jul 18, 2019 at 12:58 Adam Murphy <:
+1
I just ran into this. Any updates?
Attempting to fix it with
Thanks, The Changes looks good.
This should be closed I think.
This feature only for 1.16?
Could feature back ported into 1.15 1.14 1.13
Could someone review please?
/sig cloud-provider
Any update on this?
As we are in the code freeze and this issue is still not fixed, I'll move it to the 1.17 milestone. I'm also going to add the PR that fixes this issue to the milestone. If there are any concerns, please ping me or ping SIG-Release on Slack. /milestone v1.17
Do you have a timeline when this will be released?
This is a breaking issue in multiple versions but the is not prioritized. Does anyone have a workaround at the moment?
For AWS you can associate elastic IP with pod. Need make aws cli call. If you need static private IP then you can create DaemonSet attach NIC with static IP as second NIC to EC2 instance and node some label (like ). Later you start pod with network, get node label, bound to required host IP. If you need static public and private IPs, then you can customize node-group init script and attach NIC with permanent private IP and Elastic IP at node startup. Later simple start pod with network
Bug triage for release 1.17 here. Since this issue has been tagged for the milestone, we would like to check it's status. Code freeze is on Nov. 14. Will this issue be resolved before then? If not, we should move it out of the milestone.
Hi bug triage team for 1.17, will this issue be resolved before code freeze on Nov. 14 (Next Thursday)?
/milestone v1.18 (v1.17 code freeze)
looking for a first issue to work, can I take this
gave some good feedback to the original PR. Does this need a reset with a fresh PR? I also have a need for this. I don't have a strong urge to write, or reuse, an operator for NLBs in AWS for UDP. This will eventually get done in k8s anyways, so I'd rather put the effort there.
BTW, you can always setup an NLB manually to forward UDP to a nodePort service on your k8s instances. The issue with that is that you have to pick a NodePort service for UDP before hand, and manually handle the NLB and the target groups. One cluster upgrade and your target group behind the NLB disappears since all of the instances get replaced. It's not ideal but in a pinch...
I don't mind updating a the PR, seems like a good first issue for me.
/assign
+1
+1
+1
+1
+1
Hello ! Bug Triage team here for the 1.18 release. This is a friendly reminder that code freeze is scheduled for 5 March, which is a few weeks from now. Is this issue still intended for milestone 1.18? Thanks in advance!
C/O Code freeze for release 1.18 is starting in 5 March (less than 1 week from now) and we want to ensure that each issue has a chance to be finished on time. Is this still on track for this deadline?
/remove-kind bug
Is there any work remaining to be done on this? Willing to contribute if so, would really love to see this make it into 1.18!
Thanks for the reply This has been a long running issue, and given that there has not been any recent activity, would it make more sense to once again move the milestone? We are 2 days away from code freeze (5 March).
I wonder what "activity" you are expecting.
We are now in code freeze for 1.18, so I am moving this to the next milestone. /milestone v1.19
Hello, as of now, I see is 65% completed. May we please have an update on estimated release date of UDP support for AWS NLB?
People have been trying to be constructive here. But the team doesn't even try to tell us what is pending. Sad.
Would also love to see this feature for EKS - and am willing to help out. Even if we could get it in 1.19 it would still take a long time to become available in EKS (which is currently at 1.15). So the sooner we can start work on this - the better :-)
+1 Also hope we could get in 1.19 and happy to help out!
any update on this pretty please ?
whats the update here?
From what I can see, the main issues that remain unresolved since the first PR last year: is still pending if you need a service that supports both TCP and UDP, e.g., DNS. DNS is also a more-complicated example because it supports TCP and UDP on the same port number, and that'll need more refactoring here, as AWS treats that as a single multi-protocol target. Edit: To be clear, not really a blocker for UDP support, but it's been raised in the discussion earlier. AWS NLB can only do TCP-based health checks (including HTTP and HTTPS), so your service needs to have a health-check TCP port listening. Current PRs silently assume it's on the same port number as the UDP service you're advertising. If you're using , then it'll use the health check from  instead anyway. So this is probably only an issue for LoadBalancer Services with set. See "Things to know" on the .

Discussion:
sslPorts := getPortSets(annotations[ServiceAnnotationLoadBalancerSSLPorts]) for _, port := range apiService.Spec.Ports { <del> if port.Protocol != v1.ProtocolTCP { return nil, fmt.Errorf("Only TCP LoadBalancer is supported for AWS ELB") </del> <ins> if err := checkProtocol(port, annotations); err != nil { return nil, err </ins> } if port.NodePort == 0 { klog.Errorf("Ignoring port without NodePort defined: %v", port) continue

} certificateARN := annotations[ServiceAnnotationLoadBalancerCertificate] <del> if certificateARN != "" && (sslPorts == nil || sslPorts.numbers.Has(int64(port.Port)) || sslPorts.names.Has(port.Name)) { </del> <ins> if port.Protocol != v1.ProtocolUDP && certificateARN != "" && (sslPorts == nil || sslPorts.numbers.Has(int64(port.Port)) || sslPorts.names.Has(port.Name)) { </ins> portMapping.FrontendProtocol = elbv2.ProtocolEnumTls portMapping.SSLCertificateARN = certificateARN portMapping.SSLPolicy = annotations[ServiceAnnotationLoadBalancerSSLNegotiationPolicy]

} v2Mappings = append(v2Mappings, portMapping) <ins> } else { listener, err := buildListener(port, annotations, sslPorts) if err != nil { return nil, err } listeners = append(listeners, listener) </ins> } <del> listener, err := buildListener(port, annotations, sslPorts) if err != nil { return nil, err } listeners = append(listeners, listener) </del> } if apiService.Spec.LoadBalancerIP != "" {

return KubernetesInstanceID(node.Spec.ProviderID).MapToAWSInstanceID() } <ins> func checkProtocol(port v1.ServicePort, annotations map[string]string) error { // nlb supports tcp, udp if isNLB(annotations) && (port.Protocol == v1.ProtocolTCP || port.Protocol == v1.ProtocolUDP) { return nil } // elb only supports tcp if !isNLB(annotations) && port.Protocol == v1.ProtocolTCP { return nil } return fmt.Errorf("Protocol %s not supported by LoadBalancer", port.Protocol) } </ins> func setNodeDisk( nodeDiskMap map[types.NodeName]map[KubernetesVolumeID]bool, volumeID KubernetesVolumeID,

} // actual maps FrontendPort to an elbv2.Listener <del> actual := map[int64]*elbv2.Listener{} </del> <ins> actual := map[int64]map[string]*elbv2.Listener{} </ins> for _, listener := range listenerDescriptions.Listeners { <del> actual[*listener.Port] = listener </del> <ins> if actual[*listener.Port] == nil { actual[*listener.Port] = map[string]*elbv2.Listener{} } actual[*listener.Port][*listener.Protocol] = listener </ins> } actualTargetGroups, err := c.elbv2.DescribeTargetGroups(

// Handle additions/modifications for _, mapping := range mappings { frontendPort := mapping.FrontendPort <ins> frontendProtocol := mapping.FrontendProtocol </ins> nodePort := mapping.TrafficPort // modifications <del> if listener, ok := actual[frontendPort]; ok { </del> <ins> if listener, ok := actual[frontendPort][frontendProtocol]; ok { </ins> listenerNeedsModification := false if aws.StringValue(listener.Protocol) != mapping.FrontendProtocol {

dirty = true } <del> frontEndPorts := map[int64]bool{} </del> <ins> frontEndPorts := map[int64]map[string]bool{} </ins> for i := range mappings { <del> frontEndPorts[mappings[i].FrontendPort] = true </del> <ins> if frontEndPorts[mappings[i].FrontendPort] == nil { frontEndPorts[mappings[i].FrontendPort] = map[string]bool{} } frontEndPorts[mappings[i].FrontendPort][mappings[i].FrontendProtocol] = true </ins> } // handle deletions <del> for port, listener := range actual { if _, ok := frontEndPorts[port]; !ok { err := c.deleteListenerV2(listener) if err != nil { return nil, err </del> <ins> for port := range actual { for protocol := range actual[port] { if _, ok := frontEndPorts[port][protocol]; !ok { err := c.deleteListenerV2(actual[port][protocol]) if err != nil { return nil, err } dirty = true </ins> } <del> dirty = true </del> } } } if err := c.reconcileLBAttributes(aws.StringValue(loadBalancer.LoadBalancerArn), annotations); err != nil { return nil, err }

{ clientPorts := sets.Int64{} <ins> clientProtocol := "tcp" </ins> healthCheckPorts := sets.Int64{} for _, port := range portMappings { clientPorts.Insert(port.TrafficPort) healthCheckPorts.Insert(port.HealthCheckPort) <ins> if port.TrafficProtocol == string(v1.ProtocolUDP) { clientProtocol = "udp" } </ins> } clientRuleAnnotation := fmt.Sprintf("%s=%s", NLBClientRuleDescription, lbName) healthRuleAnnotation := fmt.Sprintf("%s=%s", NLBHealthCheckRuleDescription, lbName)

if err := c.updateInstanceSecurityGroupForNLBTraffic(sgID, sgPerms, healthRuleAnnotation, "tcp", healthCheckPorts, vpcCIDRs); err != nil { return err } <del> if err := c.updateInstanceSecurityGroupForNLBTraffic(sgID, sgPerms, clientRuleAnnotation, "tcp", clientPorts, clientCIDRs); err != nil { </del> <ins> if err := c.updateInstanceSecurityGroupForNLBTraffic(sgID, sgPerms, clientRuleAnnotation, clientProtocol, clientPorts, clientCIDRs); err != nil { </ins> return err } } else { if err := c.updateInstanceSecurityGroupForNLBTraffic(sgID, sgPerms, healthRuleAnnotation, "tcp", nil, nil); err != nil { return err } <del> if err := c.updateInstanceSecurityGroupForNLBTraffic(sgID, sgPerms, clientRuleAnnotation, "tcp", nil, nil); err != nil { </del> <ins> if err := c.updateInstanceSecurityGroupForNLBTraffic(sgID, sgPerms, clientRuleAnnotation, clientProtocol, nil, nil); err != nil { </ins> return err } }

c.EnsureLoadBalancer(context.TODO(), TestClusterName, &v1.Service{ObjectMeta: metav1.ObjectMeta{Name: "myservice", UID: "id"}}, []*v1.Node{}) } <ins> func TestCheckProtocol(t *testing.T) { tests := []struct { name        string annotations map[string]string port        v1.ServicePort wantErr     error }{ { name:        "TCP with ELB", annotations: make(map[string]string), port:        v1.ServicePort{Protocol: v1.ProtocolTCP, Port: int32(8080)}, wantErr:     nil, }, { name:        "TCP with NLB", annotations: map[string]string{ServiceAnnotationLoadBalancerType: "nlb"}, port:        v1.ServicePort{Protocol: v1.ProtocolTCP, Port: int32(8080)}, wantErr:     nil, }, { name:        "UDP with ELB", annotations: make(map[string]string), port:        v1.ServicePort{Protocol: v1.ProtocolUDP, Port: int32(8080)}, wantErr:     fmt.Errorf("Protocol UDP not supported by load balancer"), }, { name:        "UDP with NLB", annotations: map[string]string{ServiceAnnotationLoadBalancerType: "nlb"}, port:        v1.ServicePort{Protocol: v1.ProtocolUDP, Port: int32(8080)}, wantErr:     nil, }, } for _, test := range tests { tt := test t.Run(tt.name, func(t *testing.T) { t.Parallel() err := checkProtocol(tt.port, tt.annotations) if tt.wantErr != nil && err == nil { t.Errorf("Expected error: want=%s got =%s", tt.wantErr, err) } if tt.wantErr == nil && err != nil { t.Errorf("Unexpected error: want=%s got =%s", tt.wantErr, err) } }) } } </ins> func TestBuildListener(t *testing.T) { tests := []struct { name string