Repository: kubernetes-sigs/cloud-provider-azure
Pull request #644 (2021-05-31 00:40:38)

cherry-pick of #645 to release-1.0
<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/release.md#issue-kind-label
3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/testing.md
4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
5. Follow the instructions for writing a release note: https://git.k8s.io/community/contributors/guide/release-notes.md
6. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

**What type of PR is this?**
> Uncomment only one ` /kind <>` line, hit enter to put that in a new line, and remove leading whitespaces from that line:
>
> /kind api-change
> /kind bug
> /kind cleanup
> /kind design
> /kind documentation
> /kind failing-test
> /kind feature
> /kind flake
/kind feature

**What this PR does / why we need it**:

This PR adds support for SCTP services. Because the public SLB only supports TCP and UDP protocols, the SCTP services would only be supported in internal SLB. Documents have also been added.

**Which issue(s) this PR fixes**:
<!-- 
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes #

**Special notes for your reviewer**:


**Release note**:
```
feat: add support for SCTP services
```


k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/644#" title="Author self-approved">feiskyer</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/master/OWNERS)~~ [feiskyer]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
nilo19: /lgtm
k8s-ci-robot: @feiskyer: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cloud-provider-azure-e2e | 077e1bad8931556d54408e0d49abc0eac4f63856 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/644/pull-cloud-provider-azure-e2e/1399163881313538048) | `/test pull-cloud-provider-azure-e2e`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=644). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Afeiskyer). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
nilo19: /retest
k8s-ci-robot: @feiskyer: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cloud-provider-azure-e2e | 077e1bad8931556d54408e0d49abc0eac4f63856 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/644/pull-cloud-provider-azure-e2e/1399267779407777792) | `/test pull-cloud-provider-azure-e2e`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=644). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Afeiskyer). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
feiskyer: /retest
k8s-ci-robot: @feiskyer: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm | 077e1bad8931556d54408e0d49abc0eac4f63856 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/644/pull-cloud-provider-azure-e2e-ccm/1399563460982870016) | `/test pull-cloud-provider-azure-e2e-ccm`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=644). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Afeiskyer). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
feiskyer: /retest
k8s-ci-robot: @feiskyer: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-basic-lb | 077e1bad8931556d54408e0d49abc0eac4f63856 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/644/pull-cloud-provider-azure-e2e-ccm-vmss-basic-lb/1399916348057849856) | `/test pull-cloud-provider-azure-e2e-ccm-vmss-basic-lb`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=644). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Afeiskyer). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
feiskyer: CI failed because the SLB has already been deleted.

/test pull-cloud-provider-azure-e2e-ccm-vmss-basic-lb


k8s-ci-robot: @feiskyer: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>

Patch:
diff --git a/pkg/provider/azure_loadbalancer.go b/pkg/provider/azure_loadbalancer.go
index 02252ad42..9773f4600 100644
--- a/pkg/provider/azure_loadbalancer.go
+++ b/pkg/provider/azure_loadbalancer.go
@@ -2023,6 +2023,10 @@ func (az *Cloud) getExpectedLBRules(
 	var expectedRules []network.LoadBalancingRule
 	highAvailabilityPortsEnabled := false
 	for _, port := range ports {
+		if !requiresInternalLoadBalancer(service) && port.Protocol == v1.ProtocolSCTP {
+			return nil, nil, fmt.Errorf("SCTP is only supported on internal LoadBalancer")
+		}
+
 		if highAvailabilityPortsEnabled {
 			// Since the port is always 0 when enabling HA, only one rule should be configured.
 			break
@@ -2111,7 +2115,7 @@ func (az *Cloud) getExpectedLBRules(
 
 		if requiresInternalLoadBalancer(service) &&
 			strings.EqualFold(az.LoadBalancerSku, consts.LoadBalancerSkuStandard) &&
-			strings.EqualFold(service.Annotations[consts.ServiceAnnotationLoadBalancerEnableHighAvailabilityPorts], consts.TrueAnnotationValue) {
+			(strings.EqualFold(service.Annotations[consts.ServiceAnnotationLoadBalancerEnableHighAvailabilityPorts], consts.TrueAnnotationValue) || port.Protocol == v1.ProtocolSCTP) {
 			expectedRule.FrontendPort = to.Int32Ptr(0)
 			expectedRule.BackendPort = to.Int32Ptr(0)
 			expectedRule.Protocol = network.TransportProtocolAll
diff --git a/pkg/provider/azure_standard.go b/pkg/provider/azure_standard.go
index d30137bcd..a68ceb40f 100644
--- a/pkg/provider/azure_standard.go
+++ b/pkg/provider/azure_standard.go
@@ -193,8 +193,12 @@ func getProtocolsFromKubernetesProtocol(protocol v1.Protocol) (*network.Transpor
 		transportProto = network.TransportProtocolUDP
 		securityProto = network.SecurityRuleProtocolUDP
 		return &transportProto, &securityProto, nil, nil
+	case v1.ProtocolSCTP:
+		transportProto = network.TransportProtocolAll
+		securityProto = network.SecurityRuleProtocolAsterisk
+		return &transportProto, &securityProto, nil, nil
 	default:
-		return &transportProto, &securityProto, &probeProto, fmt.Errorf("only TCP and UDP are supported for Azure LoadBalancers")
+		return &transportProto, &securityProto, &probeProto, fmt.Errorf("only TCP, UDP and SCTP are supported for Azure LoadBalancers")
 	}
 
 }
@@ -368,7 +372,7 @@ func (az *Cloud) serviceOwnsFrontendIP(fip network.FrontendIPConfiguration, serv
 			return false, isPrimaryService, nil
 		}
 
-		return false, isPrimaryService, fmt.Errorf("serviceOwnsFrontendIP: wrong parameters")
+		return false, isPrimaryService, nil
 	}
 
 	// for internal secondary service the private IP address on the frontend IP config should be checked
diff --git a/site/content/en/topics/loadbalancer.md b/site/content/en/topics/loadbalancer.md
index 26957f5a6..dc931e068 100644
--- a/site/content/en/topics/loadbalancer.md
+++ b/site/content/en/topics/loadbalancer.md
@@ -85,16 +85,41 @@ Here is the recommended way to define the [outbound rules](https://docs.microsof
 * Create a separate pool definition for outbound, and ensure all virtual machines or VMSS virtual machines are in this pool. Azure cloud provider will manage the load balancer rules with another pool, so that provisioning tools and the Azure cloud provider won't affect each other.
 * Define inbound with load balancing rules and inbound NAT rules as needed, and set `disableOutboundSNAT` to true on the load balancing rule(s).  Don't rely on the side effect from these rules for outbound connectivity. It makes it messier than it needs to be and limits your options.  Use inbound NAT rules to create port forwarding mappings for SSH access to the VM's rather than burning public IPs per instance.
 
-## Exclude nodes from the load balancer (v1.20.0)
+## Exclude nodes from the load balancer
 
-The kubernetes controller manager supports excluding nodes from the load balancer backend pools by enabling the feature gate `ServiceNodeExclusion`, which is in beta state since v1.19. This PR let users to exclude nodes from the LB by labeling `node.kubernetes.io/exclude-from-external-load-balancers=true` on the nodes. There are several things that need to be mentioned.
+> Excluding nodes from Azure LoadBalancer is supported since v1.20.0.
 
-1. To use the feature, the feature gate `ServiceNodeExclusion` should be on.
+The kubernetes controller manager supports excluding nodes from the load balancer backend pools by enabling the feature gate `ServiceNodeExclusion`. To exclude nodes from Azure LoadBalancer, label `node.kubernetes.io/exclude-from-external-load-balancers=true` should be added to the nodes.
+
+1. To use the feature, the feature gate `ServiceNodeExclusion` should be on (enabled by default since its beta on v1.19).
 
 2. The labeled nodes would be excluded from the LB in the next LB reconcile loop, which needs one or more LB typed services to trigger. Basically, users could trigger the update by creating a service. If there are one or more LB typed services existing, no extra operations are needed.
 
 3. To re-include the nodes, just remove the label and the update would be operated in the next LB reconcile loop.
 
+
+## Using SCTP
+
+SCTP protocol services are only supported on internal standard LoadBalancer, hence annotation `service.beta.kubernetes.io/azure-load-balancer-internal: "true"` should be added to SCTP protocol services. See below for an example:
+
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: sctpservice
+  annotations:
+    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
+spec:
+  type: LoadBalancer
+  selector:
+    app: sctpserver
+  ports:
+    - name: sctpserver
+      protocol: SCTP
+      port: 30102
+      targetPort: 30102
+```
+
 ## Load balancer limits
 
 The limits of the load balancer related resources are listed below:


Test patch:
diff --git a/pkg/provider/azure_loadbalancer_test.go b/pkg/provider/azure_loadbalancer_test.go
index ab63f7690..f1da84e3d 100644
--- a/pkg/provider/azure_loadbalancer_test.go
+++ b/pkg/provider/azure_loadbalancer_test.go
@@ -1769,7 +1769,17 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 			loadBalancerSku: "standard",
 			wantLb:          true,
 			expectedProbes:  getDefaultTestProbes("Tcp", ""),
-			expectedRules:   getHATestRules(true),
+			expectedRules:   getHATestRules(true, true, v1.ProtocolTCP),
+		},
+		{
+			desc: "reconcileLoadBalancerRule shall return corresponding probe and lbRule (slb with SCTP)",
+			service: getTestService("test1", v1.ProtocolSCTP, map[string]string{
+				"service.beta.kubernetes.io/azure-load-balancer-enable-high-availability-ports": "true",
+				"service.beta.kubernetes.io/azure-load-balancer-internal":                       "true",
+			}, false, 80),
+			loadBalancerSku: "standard",
+			wantLb:          true,
+			expectedRules:   getHATestRules(true, false, v1.ProtocolSCTP),
 		},
 		{
 			desc: "reconcileLoadBalancerRule shall return corresponding probe and lbRule (slb with HA enabled multi-ports services)",
@@ -1780,7 +1790,7 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 			loadBalancerSku: "standard",
 			wantLb:          true,
 			expectedProbes:  getDefaultTestProbes("Tcp", ""),
-			expectedRules:   getHATestRules(true),
+			expectedRules:   getHATestRules(true, true, v1.ProtocolTCP),
 		},
 	}
 	for i, test := range testCases {
@@ -1855,10 +1865,10 @@ func getDefaultTestRules(enableTCPReset bool) []network.LoadBalancingRule {
 	return expectedRules
 }
 
-func getHATestRules(enableTCPReset bool) []network.LoadBalancingRule {
+func getHATestRules(enableTCPReset, hasProbe bool, protocol v1.Protocol) []network.LoadBalancingRule {
 	expectedRules := []network.LoadBalancingRule{
 		{
-			Name: to.StringPtr("atest1-TCP-80"),
+			Name: to.StringPtr(fmt.Sprintf("atest1-%s-80", string(protocol))),
 			LoadBalancingRulePropertiesFormat: &network.LoadBalancingRulePropertiesFormat{
 				Protocol: network.TransportProtocol("All"),
 				FrontendIPConfiguration: &network.SubResource{
@@ -1867,22 +1877,26 @@ func getHATestRules(enableTCPReset bool) []network.LoadBalancingRule {
 				BackendAddressPool: &network.SubResource{
 					ID: to.StringPtr("backendPoolID"),
 				},
-				LoadDistribution:     "Default",
-				FrontendPort:         to.Int32Ptr(0),
-				BackendPort:          to.Int32Ptr(0),
-				EnableFloatingIP:     to.BoolPtr(true),
-				DisableOutboundSnat:  to.BoolPtr(false),
-				IdleTimeoutInMinutes: to.Int32Ptr(0),
-				Probe: &network.SubResource{
-					ID: to.StringPtr("/subscriptions/subscription/resourceGroups/rg/providers/" +
-						"Microsoft.Network/loadBalancers/lbname/probes/atest1-TCP-80"),
-				},
+				LoadDistribution:    "Default",
+				FrontendPort:        to.Int32Ptr(0),
+				BackendPort:         to.Int32Ptr(0),
+				EnableFloatingIP:    to.BoolPtr(true),
+				DisableOutboundSnat: to.BoolPtr(false),
 			},
 		},
 	}
+	if protocol == v1.ProtocolTCP {
+		expectedRules[0].IdleTimeoutInMinutes = to.Int32Ptr(0)
+	}
 	if enableTCPReset {
 		expectedRules[0].EnableTCPReset = to.BoolPtr(true)
 	}
+	if hasProbe {
+		expectedRules[0].Probe = &network.SubResource{
+			ID: to.StringPtr(fmt.Sprintf("/subscriptions/subscription/resourceGroups/rg/providers/"+
+				"Microsoft.Network/loadBalancers/lbname/probes/atest1-%s-80", string(protocol))),
+		}
+	}
 	return expectedRules
 }
 
diff --git a/pkg/provider/azure_standard_test.go b/pkg/provider/azure_standard_test.go
index b0eff89d8..6b0bbbe71 100644
--- a/pkg/provider/azure_standard_test.go
+++ b/pkg/provider/azure_standard_test.go
@@ -603,10 +603,17 @@ func TestGetProtocolsFromKubernetesProtocol(t *testing.T) {
 			expectedSecurityGroupProto: network.SecurityRuleProtocolUDP,
 			nilProbeProto:              true,
 		},
+		{
+			Name:                       "getProtocolsFromKubernetesProtocol should get SCTP protocol",
+			protocol:                   v1.ProtocolSCTP,
+			expectedTransportProto:     network.TransportProtocolAll,
+			expectedSecurityGroupProto: network.SecurityRuleProtocolAsterisk,
+			nilProbeProto:              true,
+		},
 		{
 			Name:           "getProtocolsFromKubernetesProtocol should report error",
-			protocol:       v1.ProtocolSCTP,
-			expectedErrMsg: fmt.Errorf("only TCP and UDP are supported for Azure LoadBalancers"),
+			protocol:       v1.Protocol("ICMP"),
+			expectedErrMsg: fmt.Errorf("only TCP, UDP and SCTP are supported for Azure LoadBalancers"),
 		},
 	}