Repository: kubernetes-sigs/cloud-provider-azure
Pull request #3659 (2023-03-27 05:57:05)

[release-1.23] Deprecate numberOfProbes and adopt ProbeThreshold to address a probe issue in nrp
This is an automated cherry-pick of #3623

```release-note
Deprecate numberOfProbes and adopt ProbeThreshold in network api to address a probe issue in nrp
```

k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/3659#" title="Author self-approved">feiskyer</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/release-1.23/OWNERS)~~ [feiskyer]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
feiskyer: /kind bug
coveralls: 
[![Coverage Status](https://coveralls.io/builds/58247787/badge)](https://coveralls.io/builds/58247787)

Coverage: 80.547% (+0.01%) from 80.537% when pulling **9710fd828fa5191afb84494efd04e639ab41a24c on feiskyer:fix-3623** into **c6c469c856e608581372f7a06c2797f62e96a46e on kubernetes-sigs:release-1.23**.

MartinForReal: 
/approve


k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/3659#" title="Author self-approved">feiskyer</a>*, *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/3659#issuecomment-1484588810" title="Approved">MartinForReal</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/release-1.23/OWNERS)~~ [MartinForReal,feiskyer]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
MartinForReal: /lgtm
k8s-ci-robot: @feiskyer: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-23 | 9710fd828fa5191afb84494efd04e639ab41a24c | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/3659/pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-23/1640231454023618560) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-23`
pull-cloud-provider-azure-e2e-ccm-capz-1-23 | 9710fd828fa5191afb84494efd04e639ab41a24c | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/3659/pull-cloud-provider-azure-e2e-ccm-capz-1-23/1640231454136864768) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz-1-23`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=3659). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Afeiskyer). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
MartinForReal: /retest
k8s-ci-robot: @feiskyer: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-23 | 9710fd828fa5191afb84494efd04e639ab41a24c | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/3659/pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-23/1640231454023618560) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-23`
pull-cloud-provider-azure-e2e-ccm-capz-1-23 | 9710fd828fa5191afb84494efd04e639ab41a24c | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/3659/pull-cloud-provider-azure-e2e-ccm-capz-1-23/1640281678607814656) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz-1-23`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=3659). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Afeiskyer). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @feiskyer: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-capz-1-23 | 9710fd828fa5191afb84494efd04e639ab41a24c | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/3659/pull-cloud-provider-azure-e2e-ccm-capz-1-23/1640281678607814656) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz-1-23`
pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-23 | 9710fd828fa5191afb84494efd04e639ab41a24c | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/3659/pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-23/1640281678561677312) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-23`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=3659). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Afeiskyer). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
MartinForReal: /retest
k8s-ci-robot: @feiskyer: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-capz-1-23 | 9710fd828fa5191afb84494efd04e639ab41a24c | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/3659/pull-cloud-provider-azure-e2e-ccm-capz-1-23/1640281678607814656) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz-1-23`
pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-23 | 9710fd828fa5191afb84494efd04e639ab41a24c | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/3659/pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-23/1640527783367020544) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-23`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=3659). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Afeiskyer). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @feiskyer: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-23 | 9710fd828fa5191afb84494efd04e639ab41a24c | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/3659/pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-23/1640527783367020544) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-23`
pull-cloud-provider-azure-e2e-ccm-capz-1-23 | 9710fd828fa5191afb84494efd04e639ab41a24c | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/3659/pull-cloud-provider-azure-e2e-ccm-capz-1-23/1640527783417352192) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz-1-23`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=3659). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Afeiskyer). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: New changes are detected. LGTM label has been removed.
k8s-ci-robot: @feiskyer: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-capz-1-23 | 9710fd828fa5191afb84494efd04e639ab41a24c | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/3659/pull-cloud-provider-azure-e2e-ccm-capz-1-23/1640527783417352192) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz-1-23`
pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-23 | 7580db184e8b1439a5c7b11bf684e1f802ceae1b | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/3659/pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-23/1640581949741338624) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-23`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=3659). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Afeiskyer). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @feiskyer: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-23 | 7580db184e8b1439a5c7b11bf684e1f802ceae1b | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/3659/pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-23/1640581949741338624) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-23`
pull-cloud-provider-azure-e2e-ccm-capz-1-23 | 7580db184e8b1439a5c7b11bf684e1f802ceae1b | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/3659/pull-cloud-provider-azure-e2e-ccm-capz-1-23/1640581949846196224) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz-1-23`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=3659). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Afeiskyer). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->

Patch:
diff --git a/pkg/provider/azure_loadbalancer.go b/pkg/provider/azure_loadbalancer.go
index 28466a46c..23894ab0f 100644
--- a/pkg/provider/azure_loadbalancer.go
+++ b/pkg/provider/azure_loadbalancer.go
@@ -2278,7 +2278,7 @@ func (az *Cloud) buildHealthProbeRulesForPort(annotations map[string]string, por
 		return nil, fmt.Errorf("total probe should be less than 120, please adjust interval and number of probe accordingly")
 	}
 	properties.IntervalInSeconds = probeInterval
-	properties.NumberOfProbes = numberOfProbes
+	properties.ProbeThreshold = numberOfProbes
 	properties.Port = &port.NodePort
 	probe := &network.Probe{
 		Name:                  &lbrule,
@@ -2315,7 +2315,7 @@ func (az *Cloud) getExpectedLBRules(
 				Protocol:          network.ProbeProtocolHTTP,
 				Port:              pointer.Int32(podPresencePort),
 				IntervalInSeconds: pointer.Int32(consts.HealthProbeDefaultProbeInterval),
-				NumberOfProbes:    pointer.Int32(consts.HealthProbeDefaultNumOfProbe),
+				ProbeThreshold:    pointer.Int32(consts.HealthProbeDefaultNumOfProbe),
 			},
 		}
 		expectedProbes = append(expectedProbes, *nodeEndpointHealthprobe)
@@ -3262,7 +3262,7 @@ func findProbe(probes []network.Probe, probe network.Probe) bool {
 			strings.EqualFold(string(existingProbe.Protocol), string(probe.Protocol)) &&
 			strings.EqualFold(pointer.StringDeref(existingProbe.RequestPath, ""), pointer.StringDeref(probe.RequestPath, "")) &&
 			pointer.Int32Deref(existingProbe.IntervalInSeconds, 0) == pointer.Int32Deref(probe.IntervalInSeconds, 0) &&
-			pointer.Int32Deref(existingProbe.NumberOfProbes, 0) == pointer.Int32Deref(probe.NumberOfProbes, 0) {
+			pointer.Int32Deref(existingProbe.ProbeThreshold, 0) == pointer.Int32Deref(probe.ProbeThreshold, 0) {
 			return true
 		}
 	}


Test patch:
diff --git a/pkg/provider/azure_loadbalancer_test.go b/pkg/provider/azure_loadbalancer_test.go
index 839730e19..c5ee49f55 100644
--- a/pkg/provider/azure_loadbalancer_test.go
+++ b/pkg/provider/azure_loadbalancer_test.go
@@ -2554,7 +2554,7 @@ func getTestProbe(protocol, path string, interval, port, numOfProbe *int32) netw
 			Protocol:          network.ProbeProtocol(protocol),
 			Port:              port,
 			IntervalInSeconds: interval,
-			NumberOfProbes:    numOfProbe,
+			ProbeThreshold:    numOfProbe,
 		},
 	}
 	if (strings.EqualFold(protocol, "Http") || strings.EqualFold(protocol, "Https")) && len(strings.TrimSpace(path)) > 0 {
@@ -2699,7 +2699,7 @@ func getTestLoadBalancer(name, rgName, clusterName, identifier *string, service
 						Port:              pointer.Int32(10080),
 						Protocol:          network.ProbeProtocolTCP,
 						IntervalInSeconds: pointer.Int32(5),
-						NumberOfProbes:    pointer.Int32(2),
+						ProbeThreshold:    pointer.Int32(2),
 					},
 				},
 			},
@@ -2972,7 +2972,7 @@ func TestReconcileLoadBalancer(t *testing.T) {
 				RequestPath:       pointer.String("/healthz"),
 				Protocol:          network.ProbeProtocolHTTP,
 				IntervalInSeconds: pointer.Int32(5),
-				NumberOfProbes:    pointer.Int32(2),
+				ProbeThreshold:    pointer.Int32(2),
 			},
 		},
 	}
@@ -3000,7 +3000,7 @@ func TestReconcileLoadBalancer(t *testing.T) {
 				Port:              pointer.Int32(10080),
 				Protocol:          network.ProbeProtocolTCP,
 				IntervalInSeconds: pointer.Int32(5),
-				NumberOfProbes:    pointer.Int32(2),
+				ProbeThreshold:    pointer.Int32(2),
 			},
 		},
 	}
diff --git a/tests/e2e/network/service_annotations.go b/tests/e2e/network/service_annotations.go
index d6e60ff43..129f9a7e7 100644
--- a/tests/e2e/network/service_annotations.go
+++ b/tests/e2e/network/service_annotations.go
@@ -563,8 +563,8 @@ var _ = Describe("Service with annotation", Label(utils.TestSuiteLabelServiceAnn
 		By("Validating health probe configs")
 		var numberOfProbes *int32
 		for _, probe := range targetProbes {
-			if probe.NumberOfProbes != nil {
-				numberOfProbes = probe.NumberOfProbes
+			if probe.ProbeThreshold != nil {
+				numberOfProbes = probe.ProbeThreshold
 			}
 		}
 		Expect(*numberOfProbes).To(Equal(int32(3)))
@@ -607,8 +607,25 @@ var _ = Describe("Service with annotation", Label(utils.TestSuiteLabelServiceAnn
 		Expect(err).NotTo(HaveOccurred())
 		// get lb from azure client
 		By("Validating health probe configs")
+		var probeThreshold *int32
+		var intervalInSeconds *int32
+		for _, probe := range targetProbes {
+			if probe.ProbeThreshold != nil {
+				probeThreshold = probe.ProbeThreshold
+			}
+			if probe.IntervalInSeconds != nil {
+				intervalInSeconds = probe.IntervalInSeconds
+			}
+		}
+		utils.Logf("Validating health probe config probeThreshold")
+		Expect(*probeThreshold).To(Equal(int32(2)))
+		utils.Logf("Validating health probe config intervalInSeconds")
+		Expect(*intervalInSeconds).To(Equal(int32(5)))
+		utils.Logf("Validating health probe config protocol")
 		Expect((len(targetProbes))).To(Equal(1))
-		Expect(targetProbes[0].Protocol).To(Equal(network.ProbeProtocolHTTP))
+		for _, targetProbe := range targetProbes {
+			Expect(targetProbe.Protocol).To(Equal(network.ProbeProtocolHTTP))
+		}
 	})
 
 	// Check if the following annotations are correctly set with Service LB IP