Change:
What would you like to be : Support for additional load balancer health probe configurations (specifically HTTP and HTTPS) for Azure Load Balancers. Why is this needed: The generated Azure Load Balancers for type services only support TCP health probes. Those guarantee that the backend pool consists of nodes listening on the target port but do not validate whether or not anything is responding. We've also noticed elevated volumes SSL handshake errors generated by the TCP health checks targeting 443 when using more verbose log level configurations in ingress-nginx. It would be valuable to configure the type of health probe used by the AKS-created Azure Load Balancer.
/sig azure
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Is there any update on this?
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
what do you think about this feature requests?
I had a look at a possible implementation. I did not create a PR yet, because the relevant files are in and the said there should only be bug fixes, no new features. In , there is some logic which would lead to the load balancer created with protocol HTTP (which is what we want) The check is from here: and the "RequestsOnlyLocalTraffic" is about the : However, I do not understand the relationship here, why setting should result in HTTP health check. What I think we could do is add a new annotation like and when this is set to , the health check will use the HTTP protocol in . We could also add an additional annotation to set the property of the health check probe, this could be used so that people can set the path to or similar.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
We have a similar requirement for having an http health probe with a custom path. According to comment, we could get an http probe by setting but our service expects health probes on a different path, not . Looks like proposed solution would work for us too. Are there any workarounds to achieve this currently?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/assign

Discussion:
// TODO(feiskyer): disable-tcp-reset annotations has been depracated since v1.18, it would removed on v1.20. ServiceAnnotationLoadBalancerDisableTCPReset = "service.beta.kubernetes.io/azure-load-balancer-disable-tcp-reset" <ins> // ServiceAnnotationLoadBalancerHealthProbeProtocol determines the network protocol that the load balancer health probe use. // If not set, the local service would use the HTTP and the cluster service would use the TCP by default. ServiceAnnotationLoadBalancerHealthProbeProtocol = "service.beta.kubernetes.io/azure-load-balancer-health-probe-protocol" // ServiceAnnotationLoadBalancerHealthProbeRequestPath determines the request path of the load balancer health probe. // This is only useful for the HTTP and HTTPS, and would be ignored when using TCP. If not set, // `/healthz` would be configured by default. ServiceAnnotationLoadBalancerHealthProbeRequestPath = "service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path" </ins> // serviceTagKey is the service key applied for public IP tags. serviceTagKey = "service" // clusterNameKey is the cluster name key applied for public IP tags.

return nil } <ins> func parseHealthProbeProtocolAndPath(service *v1.Service) (string, string) { var protocol, path string if v, ok := service.Annotations[ServiceAnnotationLoadBalancerHealthProbeProtocol]; ok { protocol = v } else { return protocol, path } // ignore the request path if using TCP if strings.EqualFold(protocol, string(network.ProbeProtocolHTTP)) || strings.EqualFold(protocol, string(network.ProbeProtocolHTTPS)) { if v, ok := service.Annotations[ServiceAnnotationLoadBalancerHealthProbeRequestPath]; ok { path = v } } return protocol, path } </ins> func (az *Cloud) reconcileLoadBalancerRule( service *v1.Service, wantLb bool,

return expectedProbes, expectedRules, err } <ins> probeProtocol, requestPath := parseHealthProbeProtocolAndPath(service) </ins> if servicehelpers.NeedsHealthCheck(service) { podPresencePath, podPresencePort := servicehelpers.GetServiceHealthCheckPathPort(service) <ins> if probeProtocol == "" { probeProtocol = string(network.ProbeProtocolHTTP) } if requestPath == "" { requestPath = podPresencePath } </ins> expectedProbes = append(expectedProbes, network.Probe{ Name: &lbRuleName, ProbePropertiesFormat: &network.ProbePropertiesFormat{ <del> RequestPath:       to.StringPtr(podPresencePath), Protocol:          network.ProbeProtocolHTTP, </del> <ins> RequestPath:       to.StringPtr(requestPath), Protocol:          network.ProbeProtocol(probeProtocol), </ins> Port:              to.Int32Ptr(podPresencePort), IntervalInSeconds: to.Int32Ptr(5), NumberOfProbes:    to.Int32Ptr(2),

}) } else if protocol != v1.ProtocolUDP && protocol != v1.ProtocolSCTP { // we only add the expected probe if we're doing TCP <ins> if probeProtocol == "" { probeProtocol = string(*probeProto) } var actualPath *string if !strings.EqualFold(probeProtocol, string(network.ProbeProtocolTCP)) { if requestPath != "" { actualPath = to.StringPtr(requestPath) } else { actualPath = to.StringPtr("/healthz") } } </ins> expectedProbes = append(expectedProbes, network.Probe{ Name: &lbRuleName, ProbePropertiesFormat: &network.ProbePropertiesFormat{ <del> Protocol:          *probeProto, </del> <ins> Protocol:          network.ProbeProtocol(probeProtocol), RequestPath:       actualPath, </ins> Port:              to.Int32Ptr(port.NodePort), IntervalInSeconds: to.Int32Ptr(5), NumberOfProbes:    to.Int32Ptr(2),

service         v1.Service loadBalancerSku string wantLb          bool <ins> probeProtocol   string probePath       string </ins> expectedProbes  []network.Probe expectedRules   []network.LoadBalancingRule expectedErr     error

service:         getTestService("test1", v1.ProtocolTCP, map[string]string{"service.beta.kubernetes.io/azure-load-balancer-disable-tcp-reset": "true"}, false, 80), loadBalancerSku: "basic", wantLb:          true, <del> expectedProbes: []network.Probe{ { Name: to.StringPtr("atest1-TCP-80"), ProbePropertiesFormat: &network.ProbePropertiesFormat{ Protocol:          network.ProbeProtocol("Tcp"), Port:              to.Int32Ptr(10080), IntervalInSeconds: to.Int32Ptr(5), NumberOfProbes:    to.Int32Ptr(2), }, }, }, expectedRules: []network.LoadBalancingRule{ { Name: to.StringPtr("atest1-TCP-80"), LoadBalancingRulePropertiesFormat: &network.LoadBalancingRulePropertiesFormat{ Protocol: network.TransportProtocol("Tcp"), FrontendIPConfiguration: &network.SubResource{ ID: to.StringPtr("frontendIPConfigID"), }, BackendAddressPool: &network.SubResource{ ID: to.StringPtr("backendPoolID"), }, LoadDistribution:     "Default", FrontendPort:         to.Int32Ptr(80), BackendPort:          to.Int32Ptr(80), EnableFloatingIP:     to.BoolPtr(true), DisableOutboundSnat:  to.BoolPtr(false), IdleTimeoutInMinutes: to.Int32Ptr(0), Probe: &network.SubResource{ ID: to.StringPtr("/subscriptions/subscription/resourceGroups/rg/providers/" + "Microsoft.Network/loadBalancers/lbname/probes/atest1-TCP-80"), }, EnableTCPReset: nil, }, }, }, </del> <ins> expectedProbes:  getDefaultTestProbes("Tcp", ""), expectedRules:   getDefaultTestRules(false), </ins> }, { desc:            "reconcileLoadBalancerRule shall return corresponding probe and lbRule (slb without tcp reset)", service:         getTestService("test1", v1.ProtocolTCP, map[string]string{"service.beta.kubernetes.io/azure-load-balancer-disable-tcp-reset": "True"}, false, 80), loadBalancerSku: "standard", wantLb:          true, <del> expectedProbes: []network.Probe{ { Name: to.StringPtr("atest1-TCP-80"), ProbePropertiesFormat: &network.ProbePropertiesFormat{ Protocol:          network.ProbeProtocol("Tcp"), Port:              to.Int32Ptr(10080), IntervalInSeconds: to.Int32Ptr(5), NumberOfProbes:    to.Int32Ptr(2), }, }, }, expectedRules: []network.LoadBalancingRule{ { Name: to.StringPtr("atest1-TCP-80"), LoadBalancingRulePropertiesFormat: &network.LoadBalancingRulePropertiesFormat{ Protocol: network.TransportProtocol("Tcp"), FrontendIPConfiguration: &network.SubResource{ ID: to.StringPtr("frontendIPConfigID"), }, BackendAddressPool: &network.SubResource{ ID: to.StringPtr("backendPoolID"), }, LoadDistribution:     "Default", FrontendPort:         to.Int32Ptr(80), BackendPort:          to.Int32Ptr(80), EnableFloatingIP:     to.BoolPtr(true), DisableOutboundSnat:  to.BoolPtr(false), IdleTimeoutInMinutes: to.Int32Ptr(0), Probe: &network.SubResource{ ID: to.StringPtr("/subscriptions/subscription/resourceGroups/rg/providers/" + "Microsoft.Network/loadBalancers/lbname/probes/atest1-TCP-80"), }, EnableTCPReset: to.BoolPtr(true), }, }, }, </del> <ins> expectedProbes:  getDefaultTestProbes("Tcp", ""), expectedRules:   getDefaultTestRules(true), </ins> }, { desc:            "reconcileLoadBalancerRule shall return corresponding probe and lbRule(slb with tcp reset)", service:         getTestService("test1", v1.ProtocolTCP, nil, false, 80), loadBalancerSku: "standard", wantLb:          true, <del> expectedProbes: []network.Probe{ { Name: to.StringPtr("atest1-TCP-80"), ProbePropertiesFormat: &network.ProbePropertiesFormat{ Protocol:          network.ProbeProtocol("Tcp"), Port:              to.Int32Ptr(10080), IntervalInSeconds: to.Int32Ptr(5), NumberOfProbes:    to.Int32Ptr(2), }, }, }, expectedRules: []network.LoadBalancingRule{ { Name: to.StringPtr("atest1-TCP-80"), LoadBalancingRulePropertiesFormat: &network.LoadBalancingRulePropertiesFormat{ Protocol: network.TransportProtocol("Tcp"), FrontendIPConfiguration: &network.SubResource{ ID: to.StringPtr("frontendIPConfigID"), }, BackendAddressPool: &network.SubResource{ ID: to.StringPtr("backendPoolID"), }, LoadDistribution:     "Default", FrontendPort:         to.Int32Ptr(80), BackendPort:          to.Int32Ptr(80), EnableFloatingIP:     to.BoolPtr(true), DisableOutboundSnat:  to.BoolPtr(false), IdleTimeoutInMinutes: to.Int32Ptr(0), Probe: &network.SubResource{ ID: to.StringPtr("/subscriptions/subscription/resourceGroups/rg/providers/" + "Microsoft.Network/loadBalancers/lbname/probes/atest1-TCP-80"), }, EnableTCPReset: to.BoolPtr(true), }, }, }, </del> <ins> expectedProbes:  getDefaultTestProbes("Tcp", ""), expectedRules:   getDefaultTestRules(true), }, { desc:            "reconcileLoadBalancerRule shall respect the probe protocol and path configuration in the config file", service:         getTestService("test1", v1.ProtocolTCP, nil, false, 80), loadBalancerSku: "standard", wantLb:          true, probeProtocol:   "http", probePath:       "/healthy", expectedProbes:  getDefaultTestProbes("http", "/healthy"), expectedRules:   getDefaultTestRules(true), </ins> }, } for i, test := range testCases { az := GetTestCloud(ctrl) az.Config.LoadBalancerSku = test.loadBalancerSku <ins> service := test.service if test.probeProtocol != "" { service.Annotations[ServiceAnnotationLoadBalancerHealthProbeProtocol] = test.probeProtocol } if test.probePath != "" { service.Annotations[ServiceAnnotationLoadBalancerHealthProbeRequestPath] = test.probePath } </ins> probe, lbrule, err := az.reconcileLoadBalancerRule(&test.service, test.wantLb, "frontendIPConfigID", "backendPoolID", "lbname", to.Int32Ptr(0))

} } <ins> func getDefaultTestProbes(protocol, path string) []network.Probe { expectedProbes := []network.Probe{ { Name: to.StringPtr("atest1-TCP-80"), ProbePropertiesFormat: &network.ProbePropertiesFormat{ Protocol:          network.ProbeProtocol(protocol), Port:              to.Int32Ptr(10080), IntervalInSeconds: to.Int32Ptr(5), NumberOfProbes:    to.Int32Ptr(2), }, }, } if path != "" { expectedProbes[0].RequestPath = to.StringPtr(path) } return expectedProbes } func getDefaultTestRules(enableTCPReset bool) []network.LoadBalancingRule { expectedRules := []network.LoadBalancingRule{ { Name: to.StringPtr("atest1-TCP-80"), LoadBalancingRulePropertiesFormat: &network.LoadBalancingRulePropertiesFormat{ Protocol: network.TransportProtocol("Tcp"), FrontendIPConfiguration: &network.SubResource{ ID: to.StringPtr("frontendIPConfigID"), }, BackendAddressPool: &network.SubResource{ ID: to.StringPtr("backendPoolID"), }, LoadDistribution:     "Default", FrontendPort:         to.Int32Ptr(80), BackendPort:          to.Int32Ptr(80), EnableFloatingIP:     to.BoolPtr(true), DisableOutboundSnat:  to.BoolPtr(false), IdleTimeoutInMinutes: to.Int32Ptr(0), Probe: &network.SubResource{ ID: to.StringPtr("/subscriptions/subscription/resourceGroups/rg/providers/" + "Microsoft.Network/loadBalancers/lbname/probes/atest1-TCP-80"), }, }, }, } if enableTCPReset { expectedRules[0].EnableTCPReset = to.BoolPtr(true) } return expectedRules } </ins> func getTestLoadBalancer(name, rgName, clusterName, identifier *string, service v1.Service, lbSku string) network.LoadBalancer { lb := network.LoadBalancer{ Name: name,