Repository: kubernetes-sigs/cloud-provider-azure
Pull request #2823 (2022-11-25 02:44:24)

[release-1.25]Add support for specifying probe protocol / probe port via annotation per service port
<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide/first-contribution.md#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?
/kind feature

<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature
/kind design

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->

#### What this PR does / why we need it:
This adds service.beta.kubernetes.io/port_<num>_health-probe_protocol and service.beta.kubernetes.io/port_<num>_health-probe_port annotations for LoadBalancer Services. These allow overriding the health check port or protocol for a given port. This is useful for ports that cannot respond affirmatively to the health checks and expose health information elsewhere.


#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes #

#### Special notes for your reviewer:

#### Does this PR introduce a user-facing change?
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://git.k8s.io/community/contributors/guide/release-notes.md
-->
```release-note
Added: support for new annotations **service.beta.kubernetes.io/port_<num>_health-probe_protocol** and **service.beta.kubernetes.io/port_<num>_health-probe_port** to allow explicitly setting the health probe protocol individually for each service port. Useful for services like Istio which have health check seperate from the main service port.
```

#### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

<!--
This section can be blank if this pull request does not require a release note.

When adding links which point to resources within git repositories, like
KEPs or supporting documentation, please reference a specific commit and avoid
linking directly to the master branch. This ensures that links reference a
specific point in time, rather than a document that may change over time.

See here for guidance on getting permanent links to files: https://help.github.com/en/articles/getting-permanent-links-to-files

Please use the following format for linking documentation:
- [KEP]: <link>
- [Usage]: <link>
- [Other doc]: <link>
-->
```docs

```


k8s-ci-robot: @MartinForReal: Adding label `do-not-merge/contains-merge-commits` because PR contains merge commits, which are not allowed in this repository.
Use `git rebase` to reapply your commits on top of the target branch. Detailed instructions for doing so can be found [here](https://git.k8s.io/community/contributors/guide/github-workflow.md#4-keep-your-branch-in-sync).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
k8s-ci-robot: @MartinForReal: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/2823#" title="Author self-approved">MartinForReal</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/master/OWNERS)~~ [MartinForReal]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
netlify[bot]: ### <span aria-hidden="true">👷</span> Deploy Preview for *kubernetes-sigs-cloud-provide-azure* processing.


|  Name | Link |
|---------------------------------|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | c6345be44a8bbf9e40595076504bb28f4548e430 |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/kubernetes-sigs-cloud-provide-azure/deploys/63802c0b8e59fe00084e88e0 |
k8s-ci-robot: @MartinForReal: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-check | c6345be44a8bbf9e40595076504bb28f4548e430 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2823/pull-cloud-provider-azure-check/1595971650912784384) | true | `/test pull-cloud-provider-azure-check`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | c6345be44a8bbf9e40595076504bb28f4548e430 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2823/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1595971650996670464) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=2823). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3AMartinForReal). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @MartinForReal: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-check | c6345be44a8bbf9e40595076504bb28f4548e430 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2823/pull-cloud-provider-azure-check/1595971650912784384) | true | `/test pull-cloud-provider-azure-check`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=2823). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3AMartinForReal). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @MartinForReal: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-check | c6345be44a8bbf9e40595076504bb28f4548e430 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2823/pull-cloud-provider-azure-check/1595971650912784384) | true | `/test pull-cloud-provider-azure-check`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | c6345be44a8bbf9e40595076504bb28f4548e430 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2823/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1595971650996670464) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-capz | c6345be44a8bbf9e40595076504bb28f4548e430 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2823/pull-cloud-provider-azure-e2e-ccm-capz/1595971651076362240) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=2823). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3AMartinForReal). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @MartinForReal: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-check | c6345be44a8bbf9e40595076504bb28f4548e430 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2823/pull-cloud-provider-azure-check/1595971650912784384) | true | `/test pull-cloud-provider-azure-check`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | c6345be44a8bbf9e40595076504bb28f4548e430 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2823/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1595971650996670464) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-capz | c6345be44a8bbf9e40595076504bb28f4548e430 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2823/pull-cloud-provider-azure-e2e-ccm-capz/1595971651076362240) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | c6345be44a8bbf9e40595076504bb28f4548e430 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2823/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1595971651122499584) | false | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=2823). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3AMartinForReal). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @MartinForReal: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-check | c6345be44a8bbf9e40595076504bb28f4548e430 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2823/pull-cloud-provider-azure-check/1595971650912784384) | true | `/test pull-cloud-provider-azure-check`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | c6345be44a8bbf9e40595076504bb28f4548e430 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2823/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1595971650996670464) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-capz | c6345be44a8bbf9e40595076504bb28f4548e430 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2823/pull-cloud-provider-azure-e2e-ccm-capz/1595971651076362240) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | c6345be44a8bbf9e40595076504bb28f4548e430 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2823/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1595971651122499584) | false | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-unit | c6345be44a8bbf9e40595076504bb28f4548e430 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2823/pull-cloud-provider-azure-unit/1595971651164442624) | true | `/test pull-cloud-provider-azure-unit`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=2823). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3AMartinForReal). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @MartinForReal: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-check | c6345be44a8bbf9e40595076504bb28f4548e430 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2823/pull-cloud-provider-azure-check/1595971650912784384) | true | `/test pull-cloud-provider-azure-check`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | c6345be44a8bbf9e40595076504bb28f4548e430 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2823/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1595971650996670464) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-capz | c6345be44a8bbf9e40595076504bb28f4548e430 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2823/pull-cloud-provider-azure-e2e-ccm-capz/1595971651076362240) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | c6345be44a8bbf9e40595076504bb28f4548e430 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2823/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1595971651122499584) | false | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-unit | c6345be44a8bbf9e40595076504bb28f4548e430 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2823/pull-cloud-provider-azure-unit/1595971651164442624) | true | `/test pull-cloud-provider-azure-unit`
pull-cloud-provider-azure-e2e-ccm-vmss-capz | c6345be44a8bbf9e40595076504bb28f4548e430 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2823/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1595971650950533120) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=2823). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3AMartinForReal). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/2823#" title="Author self-approved">MartinForReal</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/release-1.25/OWNERS)~~ [MartinForReal]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
MartinForReal: /hold

Patch:
diff --git a/pkg/consts/consts.go b/pkg/consts/consts.go
index 1aa91a26e..2fb05f10c 100644
--- a/pkg/consts/consts.go
+++ b/pkg/consts/consts.go
@@ -291,7 +291,7 @@ const (
 
 	// ServiceAnnotationLoadBalancerHealthProbeRequestPath determines the request path of the load balancer health probe.
 	// This is only useful for the HTTP and HTTPS, and would be ignored when using TCP. If not set,
-	// `/healthz` would be configured by default.
+	// `/` would be configured by default.
 	ServiceAnnotationLoadBalancerHealthProbeRequestPath = "service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path"
 
 	// ServiceAnnotationAzurePIPTags determines what tags should be applied to the public IP of the service. The cluster name
@@ -419,9 +419,27 @@ const RateLimited = "rate limited"
 // CreatedByTag tag key for CSI drivers
 const CreatedByTag = "k8s-azure-created-by"
 
+// port specific
+const (
+	PortAnnotationPrefixPattern            = "service.beta.kubernetes.io/port_%d_%s"
+	PortAnnotationNoLBRule      PortParams = "no_lb_rule"
+	// NoHealthProbeRule determines whether the port is only used for health probe. no lb probe rule will be created.
+	PortAnnotationNoHealthProbeRule PortParams = "no_probe_rule"
+)
+
+type PortParams string
+
 // health probe
 const (
-	HealthProbeAnnotationPrefixPattern = "service.beta.kubernetes.io/port_%d_health-probe_"
+	HealthProbeAnnotationPrefixPattern = "health-probe_%s"
+
+	// HealthProbeParamsProtocol determines the protocol for the health probe params.
+	// It always takes priority over spec.appProtocol or any other specified protocol
+	HealthProbeParamsProtocol HealthProbeParams = "protocol"
+
+	// HealthProbeParamsPort determines the probe port for the health probe params.
+	// It always takes priority over the NodePort of the spec.ports in a Service
+	HealthProbeParamsPort HealthProbeParams = "port"
 
 	// HealthProbeParamsProbeInterval determines the probe interval of the load balancer health probe.
 	// The minimum probe interval is 5 seconds and the default value is 5. The total duration of all intervals cannot exceed 120 seconds.
diff --git a/pkg/consts/helpers.go b/pkg/consts/helpers.go
index 13f49dd44..208dfc25f 100644
--- a/pkg/consts/helpers.go
+++ b/pkg/consts/helpers.go
@@ -50,6 +50,16 @@ func GetHealthProbeConfigOfPortFromK8sSvcAnnotation(annotations map[string]strin
 	return GetAttributeValueInSvcAnnotation(annotations, BuildHealthProbeAnnotationKeyForPort(port, key), validators...)
 }
 
+// IsHealthProbeRuleOnK8sServicePortDisabled return if port is for health probe only
+func IsHealthProbeRuleOnK8sServicePortDisabled(annotations map[string]string, port int32) (bool, error) {
+	return expectAttributeInSvcAnnotationBeEqualTo(annotations, BuildAnnotationKeyForPort(port, PortAnnotationNoHealthProbeRule), TrueAnnotationValue), nil
+}
+
+// IsHealthProbeRuleOnK8sServicePortDisabled return if port is for health probe only
+func IsLBRuleOnK8sServicePortDisabled(annotations map[string]string, port int32) (bool, error) {
+	return expectAttributeInSvcAnnotationBeEqualTo(annotations, BuildAnnotationKeyForPort(port, PortAnnotationNoLBRule), TrueAnnotationValue), nil
+}
+
 // Getint32ValueFromK8sSvcAnnotation get health probe configuration for port
 func Getint32ValueFromK8sSvcAnnotation(annotations map[string]string, key string, validators ...Int32BusinessValidator) (*int32, error) {
 	val, err := GetAttributeValueInSvcAnnotation(annotations, key)
@@ -59,9 +69,14 @@ func Getint32ValueFromK8sSvcAnnotation(annotations map[string]string, key string
 	return nil, err
 }
 
+// BuildHealthProbeAnnotationKeyForPort get health probe configuration key for port
+func BuildAnnotationKeyForPort(port int32, key PortParams) string {
+	return fmt.Sprintf(PortAnnotationPrefixPattern, port, string(key))
+}
+
 // BuildHealthProbeAnnotationKeyForPort get health probe configuration key for port
 func BuildHealthProbeAnnotationKeyForPort(port int32, key HealthProbeParams) string {
-	return fmt.Sprintf(HealthProbeAnnotationPrefixPattern, port) + string(key)
+	return BuildAnnotationKeyForPort(port, PortParams(fmt.Sprintf(HealthProbeAnnotationPrefixPattern, key)))
 }
 
 // GetInt32HealthProbeConfigOfPortFromK8sSvcAnnotation get health probe configuration for port
diff --git a/pkg/provider/azure_loadbalancer.go b/pkg/provider/azure_loadbalancer.go
index 2b88b8b63..d67412cce 100644
--- a/pkg/provider/azure_loadbalancer.go
+++ b/pkg/provider/azure_loadbalancer.go
@@ -2062,7 +2062,7 @@ func lbRuleConflictsWithPort(rule network.LoadBalancingRule, frontendIPConfigID
 // buildHealthProbeRulesForPort
 // for following sku: basic loadbalancer vs standard load balancer
 // for following protocols: TCP HTTP HTTPS(SLB only)
-func (az *Cloud) buildHealthProbeRulesForPort(annotations map[string]string, port v1.ServicePort, lbrule string) (*network.Probe, error) {
+func (az *Cloud) buildHealthProbeRulesForPort(serviceManifest *v1.Service, port v1.ServicePort, lbrule string) (*network.Probe, error) {
 	if port.Protocol == v1.ProtocolUDP || port.Protocol == v1.ProtocolSCTP {
 		return nil, nil
 	}
@@ -2070,19 +2070,45 @@ func (az *Cloud) buildHealthProbeRulesForPort(annotations map[string]string, por
 
 	properties := &network.ProbePropertiesFormat{}
 	var err error
-	if port.AppProtocol == nil {
-		if port.AppProtocol, err = consts.GetAttributeValueInSvcAnnotation(annotations, consts.ServiceAnnotationLoadBalancerHealthProbeProtocol); err != nil {
+
+	// order - Specific Override
+	// port_ annotation
+	// global annotation
+
+	// Select Protocol
+	//
+	var protocol *string
+
+	// 1. Look up port-specific override
+	protocol, err = consts.GetHealthProbeConfigOfPortFromK8sSvcAnnotation(serviceManifest.Annotations, port.Port, consts.HealthProbeParamsProtocol)
+	if err != nil {
+		return nil, fmt.Errorf("failed to parse annotation %s: %w", consts.BuildHealthProbeAnnotationKeyForPort(port.Port, consts.HealthProbeParamsProtocol), err)
+	}
+
+	// 2. If not specified, look up from AppProtocol
+	// Note - this order is to remain compatible with previous versions
+	if protocol == nil {
+		protocol = port.AppProtocol
+	}
+
+	// 3. If protocol is still nil, check the global annotation
+	if protocol == nil {
+		protocol, err = consts.GetAttributeValueInSvcAnnotation(serviceManifest.Annotations, consts.ServiceAnnotationLoadBalancerHealthProbeProtocol)
+		if err != nil {
 			return nil, fmt.Errorf("failed to parse annotation %s: %w", consts.ServiceAnnotationLoadBalancerHealthProbeProtocol, err)
 		}
-		if port.AppProtocol == nil {
-			port.AppProtocol = to.StringPtr(string(network.ProtocolTCP))
-		}
 	}
-	protocol := strings.TrimSpace(*port.AppProtocol)
+
+	// 4. Finally, if protocol is still nil, default to TCP
+	if protocol == nil {
+		protocol = to.StringPtr(string(network.ProtocolTCP))
+	}
+
+	*protocol = strings.TrimSpace(*protocol)
 	switch {
-	case strings.EqualFold(protocol, string(network.ProtocolTCP)):
+	case strings.EqualFold(*protocol, string(network.ProtocolTCP)):
 		properties.Protocol = network.ProbeProtocolTCP
-	case strings.EqualFold(protocol, string(network.ProtocolHTTPS)):
+	case strings.EqualFold(*protocol, string(network.ProtocolHTTPS)):
 		//HTTPS probe is only supported in standard loadbalancer
 		//For backward compatibility,when unsupported protocol is used, fall back to tcp protocol in basic lb mode instead
 		if !az.useStandardLoadBalancer() {
@@ -2090,21 +2116,81 @@ func (az *Cloud) buildHealthProbeRulesForPort(annotations map[string]string, por
 		} else {
 			properties.Protocol = network.ProbeProtocolHTTPS
 		}
-	case strings.EqualFold(protocol, string(network.ProtocolHTTP)):
+	case strings.EqualFold(*protocol, string(network.ProtocolHTTP)):
 		properties.Protocol = network.ProbeProtocolHTTP
 	default:
 		//For backward compatibility,when unsupported protocol is used, fall back to tcp protocol in basic lb mode instead
 		properties.Protocol = network.ProbeProtocolTCP
 	}
 
+	// Lookup or Override Health Probe Port
+	properties.Port = &port.NodePort
+
+	probePort, err := consts.GetHealthProbeConfigOfPortFromK8sSvcAnnotation(serviceManifest.Annotations, port.Port, consts.HealthProbeParamsPort, func(s *string) error {
+		if s == nil {
+			return nil
+		}
+		//nolint:gosec
+		port, err := strconv.Atoi(*s)
+		if err != nil {
+			//not a integer
+			for _, item := range serviceManifest.Spec.Ports {
+				if strings.EqualFold(item.Name, *s) {
+					//found the port
+					return nil
+				}
+			}
+			return fmt.Errorf("port %s not found in service", *s)
+		}
+		if port < 0 || port > 65535 {
+			return fmt.Errorf("port %d is out of range", port)
+		}
+		for _, item := range serviceManifest.Spec.Ports {
+			//nolint:gosec
+			if item.Port == int32(port) {
+				//found the port
+				return nil
+			}
+		}
+		return fmt.Errorf("port %s not found in service", *s)
+	})
+	if err != nil {
+		return nil, fmt.Errorf("failed to parse annotation %s: %w", consts.BuildHealthProbeAnnotationKeyForPort(port.Port, consts.HealthProbeParamsPort), err)
+	}
+
+	if probePort != nil {
+		//nolint:gosec
+		port, err := strconv.Atoi(*probePort)
+		if err != nil {
+			//not a integer
+			for _, item := range serviceManifest.Spec.Ports {
+				if strings.EqualFold(item.Name, *probePort) {
+					//found the port
+					properties.Port = to.Int32Ptr(item.NodePort)
+				}
+			}
+		} else {
+			if port >= 0 || port <= 65535 {
+				for _, item := range serviceManifest.Spec.Ports {
+					//nolint:gosec
+					if item.Port == int32(port) {
+						//found the port
+						properties.Port = to.Int32Ptr(item.NodePort)
+					}
+				}
+			}
+		}
+	}
+
+	// Select request path
 	if strings.EqualFold(string(properties.Protocol), string(network.ProtocolHTTPS)) || strings.EqualFold(string(properties.Protocol), string(network.ProtocolHTTP)) {
 		// get request path ,only used with http/https probe
-		path, err := consts.GetHealthProbeConfigOfPortFromK8sSvcAnnotation(annotations, port.Port, consts.HealthProbeParamsRequestPath)
+		path, err := consts.GetHealthProbeConfigOfPortFromK8sSvcAnnotation(serviceManifest.Annotations, port.Port, consts.HealthProbeParamsRequestPath)
 		if err != nil {
 			return nil, fmt.Errorf("failed to parse annotation %s: %w", consts.BuildHealthProbeAnnotationKeyForPort(port.Port, consts.HealthProbeParamsRequestPath), err)
 		}
 		if path == nil {
-			if path, err = consts.GetAttributeValueInSvcAnnotation(annotations, consts.ServiceAnnotationLoadBalancerHealthProbeRequestPath); err != nil {
+			if path, err = consts.GetAttributeValueInSvcAnnotation(serviceManifest.Annotations, consts.ServiceAnnotationLoadBalancerHealthProbeRequestPath); err != nil {
 				return nil, fmt.Errorf("failed to parse annotation %s: %w", consts.ServiceAnnotationLoadBalancerHealthProbeRequestPath, err)
 			}
 		}
@@ -2124,12 +2210,12 @@ func (az *Cloud) buildHealthProbeRulesForPort(annotations map[string]string, por
 		}
 		return nil
 	}
-	numberOfProbes, err := consts.GetInt32HealthProbeConfigOfPortFromK8sSvcAnnotation(annotations, port.Port, consts.HealthProbeParamsNumOfProbe, numOfProbeValidator)
+	numberOfProbes, err := consts.GetInt32HealthProbeConfigOfPortFromK8sSvcAnnotation(serviceManifest.Annotations, port.Port, consts.HealthProbeParamsNumOfProbe, numOfProbeValidator)
 	if err != nil {
 		return nil, fmt.Errorf("failed to parse annotation %s: %w", consts.BuildHealthProbeAnnotationKeyForPort(port.Port, consts.HealthProbeParamsNumOfProbe), err)
 	}
 	if numberOfProbes == nil {
-		if numberOfProbes, err = consts.Getint32ValueFromK8sSvcAnnotation(annotations, consts.ServiceAnnotationLoadBalancerHealthProbeNumOfProbe, numOfProbeValidator); err != nil {
+		if numberOfProbes, err = consts.Getint32ValueFromK8sSvcAnnotation(serviceManifest.Annotations, consts.ServiceAnnotationLoadBalancerHealthProbeNumOfProbe, numOfProbeValidator); err != nil {
 			return nil, fmt.Errorf("failed to parse annotation %s: %w", consts.ServiceAnnotationLoadBalancerHealthProbeNumOfProbe, err)
 		}
 	}
@@ -2150,12 +2236,12 @@ func (az *Cloud) buildHealthProbeRulesForPort(annotations map[string]string, por
 		}
 		return nil
 	}
-	probeInterval, err := consts.GetInt32HealthProbeConfigOfPortFromK8sSvcAnnotation(annotations, port.Port, consts.HealthProbeParamsProbeInterval, probeIntervalValidator)
+	probeInterval, err := consts.GetInt32HealthProbeConfigOfPortFromK8sSvcAnnotation(serviceManifest.Annotations, port.Port, consts.HealthProbeParamsProbeInterval, probeIntervalValidator)
 	if err != nil {
 		return nil, fmt.Errorf("failed to parse annotation %s:%w", consts.BuildHealthProbeAnnotationKeyForPort(port.Port, consts.HealthProbeParamsProbeInterval), err)
 	}
 	if probeInterval == nil {
-		if probeInterval, err = consts.Getint32ValueFromK8sSvcAnnotation(annotations, consts.ServiceAnnotationLoadBalancerHealthProbeInterval, probeIntervalValidator); err != nil {
+		if probeInterval, err = consts.Getint32ValueFromK8sSvcAnnotation(serviceManifest.Annotations, consts.ServiceAnnotationLoadBalancerHealthProbeInterval, probeIntervalValidator); err != nil {
 			return nil, fmt.Errorf("failed to parse annotation %s: %w", consts.ServiceAnnotationLoadBalancerHealthProbeInterval, err)
 		}
 	}
@@ -2170,7 +2256,6 @@ func (az *Cloud) buildHealthProbeRulesForPort(annotations map[string]string, por
 	}
 	properties.IntervalInSeconds = probeInterval
 	properties.NumberOfProbes = numberOfProbes
-	properties.Port = &port.NodePort
 	probe := &network.Probe{
 		Name:                  &lbrule,
 		ProbePropertiesFormat: properties,
@@ -2229,17 +2314,17 @@ func (az *Cloud) getExpectedLBRules(
 		if nodeEndpointHealthprobe == nil {
 			// use user customized health probe rule if any
 			for _, port := range service.Spec.Ports {
-				portprobe, err := az.buildHealthProbeRulesForPort(service.Annotations, port, lbRuleName)
+				portprobe, err := az.buildHealthProbeRulesForPort(service, port, lbRuleName)
 				if err != nil {
 					klog.V(2).ErrorS(err, "error occurred when buildHealthProbeRulesForPort", "service", service.Name, "namespace", service.Namespace,
 						"rule-name", lbRuleName, "port", port.Port)
 					//ignore error because we only need one correct rule
 				}
 				if portprobe != nil {
-					expectedProbes = append(expectedProbes, *portprobe)
 					props.Probe = &network.SubResource{
 						ID: to.StringPtr(az.getLoadBalancerProbeID(lbName, az.getLoadBalancerResourceGroup(), *portprobe.Name)),
 					}
+					expectedProbes = append(expectedProbes, *portprobe)
 					break
 				}
 			}
@@ -2260,7 +2345,16 @@ func (az *Cloud) getExpectedLBRules(
 		for _, port := range service.Spec.Ports {
 			lbRuleName := az.getLoadBalancerRuleName(service, port.Protocol, port.Port)
 			klog.V(2).Infof("getExpectedLBRules lb name (%s) rule name (%s)", lbName, lbRuleName)
-
+			isNoLBRuleRequired, err := consts.IsLBRuleOnK8sServicePortDisabled(service.Annotations, port.Port)
+			if err != nil {
+				err := fmt.Errorf("failed to parse annotation %s: %w", consts.BuildAnnotationKeyForPort(port.Port, consts.PortAnnotationNoLBRule), err)
+				klog.V(2).ErrorS(err, "error occurred when getExpectedLoadBalancingRulePropertiesForPort", "service", service.Name, "namespace", service.Namespace,
+					"rule-name", lbRuleName, "port", port.Port)
+			}
+			if isNoLBRuleRequired {
+				klog.V(2).Infof("getExpectedLBRules lb name (%s) rule name (%s) no lb rule required", lbName, lbRuleName)
+				continue
+			}
 			if port.Protocol == v1.ProtocolSCTP && !(az.useStandardLoadBalancer() && consts.IsK8sServiceUsingInternalLoadBalancer(service)) {
 				return expectedProbes, expectedRules, fmt.Errorf("SCTP is only supported on standard loadbalancer in internal mode")
 			}
@@ -2274,23 +2368,31 @@ func (az *Cloud) getExpectedLBRules(
 				return expectedProbes, expectedRules, fmt.Errorf("error generate lb rule for ha mod loadbalancer. err: %w", err)
 			}
 
-			if nodeEndpointHealthprobe == nil {
-				portprobe, err := az.buildHealthProbeRulesForPort(service.Annotations, port, lbRuleName)
-				if err != nil {
-					klog.V(2).ErrorS(err, "error occurred when buildHealthProbeRulesForPort", "service", service.Name, "namespace", service.Namespace,
-						"rule-name", lbRuleName, "port", port.Port)
-					return expectedProbes, expectedRules, err
-				}
-				if portprobe != nil {
-					expectedProbes = append(expectedProbes, *portprobe)
+			isNoHealthProbeRule, err := consts.IsHealthProbeRuleOnK8sServicePortDisabled(service.Annotations, port.Port)
+			if err != nil {
+				err := fmt.Errorf("failed to parse annotation %s: %w", consts.BuildAnnotationKeyForPort(port.Port, consts.PortAnnotationNoHealthProbeRule), err)
+				klog.V(2).ErrorS(err, "error occurred when buildHealthProbeRulesForPort", "service", service.Name, "namespace", service.Namespace,
+					"rule-name", lbRuleName, "port", port.Port)
+			}
+			if !isNoHealthProbeRule {
+				if nodeEndpointHealthprobe == nil {
+					portprobe, err := az.buildHealthProbeRulesForPort(service, port, lbRuleName)
+					if err != nil {
+						klog.V(2).ErrorS(err, "error occurred when buildHealthProbeRulesForPort", "service", service.Name, "namespace", service.Namespace,
+							"rule-name", lbRuleName, "port", port.Port)
+						return expectedProbes, expectedRules, err
+					}
+					if portprobe != nil {
+						props.Probe = &network.SubResource{
+							ID: to.StringPtr(az.getLoadBalancerProbeID(lbName, az.getLoadBalancerResourceGroup(), *portprobe.Name)),
+						}
+						expectedProbes = append(expectedProbes, *portprobe)
+					}
+				} else {
 					props.Probe = &network.SubResource{
-						ID: to.StringPtr(az.getLoadBalancerProbeID(lbName, az.getLoadBalancerResourceGroup(), *portprobe.Name)),
+						ID: to.StringPtr(az.getLoadBalancerProbeID(lbName, az.getLoadBalancerResourceGroup(), *nodeEndpointHealthprobe.Name)),
 					}
 				}
-			} else {
-				props.Probe = &network.SubResource{
-					ID: to.StringPtr(az.getLoadBalancerProbeID(lbName, az.getLoadBalancerResourceGroup(), *nodeEndpointHealthprobe.Name)),
-				}
 			}
 			if consts.IsK8sServiceDisableLoadBalancerFloatingIP(service) {
 				props.BackendPort = to.Int32Ptr(port.NodePort)
@@ -2300,7 +2402,6 @@ func (az *Cloud) getExpectedLBRules(
 				Name:                              &lbRuleName,
 				LoadBalancingRulePropertiesFormat: props,
 			})
-
 		}
 	}
 
diff --git a/site/content/en/topics/loadbalancer.md b/site/content/en/topics/loadbalancer.md
index 4f3c28507..51c72ae60 100644
--- a/site/content/en/topics/loadbalancer.md
+++ b/site/content/en/topics/loadbalancer.md
@@ -33,6 +33,8 @@ Below is a list of annotations supported for Kubernetes services with type `Load
 | `service.beta.kubernetes.io/azure-load-balancer-health-probe-interval` | Health probe interval | Refer to the detailed docs [here](#custom-load-balancer-health-probe) | v1.21 and later  with out-of-tree cloud provider  |
 | `service.beta.kubernetes.io/azure-load-balancer-health-probe-num-of-probe` | The minimum number of unhealthy responses of health probe  |  Refer to the detailed docs [here](#custom-load-balancer-health-probe) |	v1.21 and later  with out-of-tree cloud provider|
 | `service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path` | Request path of the health probe | Refer to the detailed docs [here](#custom-load-balancer-health-probe) | v1.20 and later  with out-of-tree cloud provider|
+| `service.beta.kubernetes.io/port_{port}_health-probe_protocol` | Health probe protocol | {port} is the port number in the service. Explicit protocol for the health probe for the service port {port}, overriding port.appProtocol if set.  Refer to the detailed docs [here](#custom-load-balancer-health-probe) | v1.21 and later with out-of-tree cloud provider|
+| `service.beta.kubernetes.io/port_{port}_health-probe_port` | Health probe protocol |  port} is the port number in the service. Explicit port for the health probe for the service port {port}, overriding the default value.  Refer to the detailed docs [here](#custom-load-balancer-health-probe) | v1.21 and later with out-of-tree cloud provider|
 | `service.beta.kubernetes.io/port_{port}_health-probe_interval` | Health probe interval |  {port} is port number of service.  Refer to the detailed docs [here](#custom-load-balancer-health-probe) | v1.21 and later  with out-of-tree cloud provider|
 | `service.beta.kubernetes.io/port_{port}_health-probe_num-of-probe` | The minimum number of unhealthy responses of health probe  | {port} is port number of service. Refer to the detailed docs [here](#custom-load-balancer-health-probe) |	v1.21 and later with out-of-tree cloud provider|
 | `service.beta.kubernetes.io/port_{port}_health-probe_request-path` | Request path of the health probe | {port} is port number of service.  Refer to the detailed docs [here](#custom-load-balancer-health-probe) | v1.20 and later with out-of-tree cloud provider|
@@ -189,14 +191,17 @@ Since v1.21, two service annotations `service.beta.kubernetes.io/azure-load-bala
 
 
 ### Custom Load Balancer health probe for port
+Different ports in a service may require different health probe configurations. This could be because of service design (such as a single health endpoint controlling multiple ports), or Kubernetes features like the [MixedProtocolLBService](https://kubernetes.io/docs/concepts/services-networking/service/#load-balancers-with-mixed-protocol-types).
 
-Because [MixedProtocolLBService](https://kubernetes.io/docs/concepts/services-networking/service/#load-balancers-with-mixed-protocol-types) feature is in alpha stage, Ports in one service may have different probe configurations. Following annotations are introduced to customize probe configuration for one port.
+The following annotations can be used to customize probe configuration per service port.
 
-| port specific annotation | global probe annotation |
-| --| -- |
-|service.beta.kubernetes.io/port_{port}_health-probe_request-path|service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path|
-|service.beta.kubernetes.io/port_{port}_health-probe_num-of-probe|service.beta.kubernetes.io/azure-load-balancer-health-probe-num-of-probe|
-|service.beta.kubernetes.io/port_{port}_health-probe_interval    |service.beta.kubernetes.io/azure-load-balancer-health-probe-interval    |
+| port specific annotation | global probe annotation | Usage |
+| --| -- | -- |
+|service.beta.kubernetes.io/port_{port}_health-probe_protocol| N/A (no equivalent globally) | Set the health probe protocol for this service port (e.g. Http, Https, Tcp) |
+|service.beta.kubernetes.io/port_{port}_health-probe_port| N/A (no equivalent globally) | Sets the health probe port for this service port (e.g. 15021) |
+|service.beta.kubernetes.io/port_{port}_health-probe_request-path|service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path| For Http or Https, sets the health probe request path. Defaults to /|
+|service.beta.kubernetes.io/port_{port}_health-probe_num-of-probe|service.beta.kubernetes.io/azure-load-balancer-health-probe-num-of-probe| Number of consecutive probe failures before the port is considered unhealthy|
+|service.beta.kubernetes.io/port_{port}_health-probe_interval    |service.beta.kubernetes.io/azure-load-balancer-health-probe-interval    | The amount of time between probe attempts |
 
 For following manifest, probe rule for port httpsserver is different from the one for httpserver because annoations for port httpsserver are specified.
 
@@ -224,6 +229,38 @@ spec:
       targetPort: 30104
 ```
 
+In this manifest, the https ports use a different health probe endpoint, an HTTP readiness check at port 30000 on /healthz/ready.
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: istio
+  annotations:
+    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
+    service.beta.kubernetes.io/port_443_health-probe_protocol: "http"
+    service.beta.kubernetes.io/port_443_health-probe_port: "30000"
+    service.beta.kubernetes.io/port_443_health-probe_request-path: "/healthz/ready"
+spec:
+  ports:
+    - name: https
+      protocol: TCP
+      port: 443
+      targetPort: 8443
+      appProtocol: https
+  selector:
+    app: istio-ingressgateway
+    gateway: istio-ingressgateway
+    istio: ingressgateway
+  type: LoadBalancer
+  sessionAffinity: None
+  externalTrafficPolicy: Local
+  ipFamilies:
+    - IPv4
+  ipFamilyPolicy: SingleStack
+  allocateLoadBalancerNodePorts: true
+  internalTrafficPolicy: Cluster
+```
+
 ## Configure Load Balancer backend
 
 > This feature is supported since v1.23.0


Test patch:
diff --git a/.golangci-test.yml b/.golangci-test.yml
new file mode 100644
index 000000000..05efbaf64
--- /dev/null
+++ b/.golangci-test.yml
@@ -0,0 +1,26 @@
+run:
+  deadline: 30m
+  concurrency: 2
+  go: '1.18'
+  allow-serial-runners: true
+skip-dirs:
+    - vendor
+linters:
+  disable-all: true
+  enable:
+    - gosec
+  # Run with --fast=false for more extensive checks
+  fast: false
+  govet:
+    # Report about shadowed variables.
+    # Default: false
+    check-shadowing: true
+issues:
+  exclude-rules:
+    - path: tests/e2e
+      linters:
+        - revive
+  #include:
+  #  - EXC0002 # include "missing comments" issues from golint
+  max-issues-per-linter: 0
+  max-same-issues: 0
diff --git a/pkg/consts/helpers_test.go b/pkg/consts/helpers_test.go
index 0763de2fd..c38f5074a 100644
--- a/pkg/consts/helpers_test.go
+++ b/pkg/consts/helpers_test.go
@@ -376,3 +376,39 @@ func TestGetInt32HealthProbeConfigOfPortFromK8sSvcAnnotation(t *testing.T) {
 		})
 	}
 }
+
+func TestBuildAnnotationKeyForPort(t *testing.T) {
+	type args struct {
+		port int32
+		key  PortParams
+	}
+	tests := []struct {
+		name string
+		args args
+		want string
+	}{
+		{
+			name: "no lb rule",
+			args: args{
+				port: 80,
+				key:  PortAnnotationNoLBRule,
+			},
+			want: "service.beta.kubernetes.io/port_80_no_lb_rule",
+		},
+		{
+			name: "no lb rule",
+			args: args{
+				port: 80,
+				key:  PortAnnotationNoHealthProbeRule,
+			},
+			want: "service.beta.kubernetes.io/port_80_no_probe_rule",
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			if got := BuildAnnotationKeyForPort(tt.args.port, tt.args.key); got != tt.want {
+				t.Errorf("BuildAnnotationKeyForPort() = %v, want %v", got, tt.want)
+			}
+		})
+	}
+}
diff --git a/pkg/provider/azure_loadbalancer_test.go b/pkg/provider/azure_loadbalancer_test.go
index 48953b315..2269dd9d7 100644
--- a/pkg/provider/azure_loadbalancer_test.go
+++ b/pkg/provider/azure_loadbalancer_test.go
@@ -2344,7 +2344,7 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 			expectedRules:   getDefaultTestRules(true),
 		},
 		{
-			desc: "getExpectedLBRules should return error when deprecated tcp health probe annotations and protocols are added and config is not valid",
+			desc: "getExpectedLBRules should return error when probe interval * num > 120",
 			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
 				"service.beta.kubernetes.io/port_80_health-probe_interval":             "10",
 				"service.beta.kubernetes.io/port_80_health-probe_num-of-probe":         "20",
@@ -2355,7 +2355,7 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 			expectedErr:     true,
 		},
 		{
-			desc: "getExpectedLBRules should return error when deprecated tcp health probe annotations and protocols are added and config is not valid",
+			desc: "getExpectedLBRules should return error when probe interval * num ==  120",
 			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
 				"service.beta.kubernetes.io/port_80_health-probe_interval":             "10",
 				"service.beta.kubernetes.io/port_80_health-probe_num-of-probe":         "20",
@@ -2374,18 +2374,18 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 			loadBalancerSku: "standard",
 			probeProtocol:   "Https",
 			probePath:       "/healthy1",
-			expectedProbes:  getTestProbes("Https", "/healthy1", to.Int32Ptr(20), to.Int32Ptr(10080), to.Int32Ptr(5)),
+			expectedProbes:  getTestProbes("Https", "/healthy1", to.Int32Ptr(20), to.Int32Ptr(80), to.Int32Ptr(10080), to.Int32Ptr(5)),
 			expectedRules:   getDefaultTestRules(true),
 		},
 		{
-			desc: "getExpectedLBRules should return correct rule when health probe annotations are added,default path should be /healthy",
+			desc: "getExpectedLBRules should return correct rule when health probe annotations are added,default path should be /",
 			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
 				"service.beta.kubernetes.io/port_80_health-probe_interval":     "20",
 				"service.beta.kubernetes.io/port_80_health-probe_num-of-probe": "5",
 			}, false, 80),
 			loadBalancerSku: "standard",
 			probeProtocol:   "Http",
-			expectedProbes:  getTestProbes("Http", "/", to.Int32Ptr(20), to.Int32Ptr(10080), to.Int32Ptr(5)),
+			expectedProbes:  getTestProbes("Http", "/", to.Int32Ptr(20), to.Int32Ptr(80), to.Int32Ptr(10080), to.Int32Ptr(5)),
 			expectedRules:   getDefaultTestRules(true),
 		},
 		{
@@ -2396,7 +2396,7 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 			}, false, 80),
 			loadBalancerSku: "standard",
 			probeProtocol:   "Tcp",
-			expectedProbes:  getTestProbes("Tcp", "", to.Int32Ptr(20), to.Int32Ptr(10080), to.Int32Ptr(5)),
+			expectedProbes:  getTestProbes("Tcp", "", to.Int32Ptr(20), to.Int32Ptr(80), to.Int32Ptr(10080), to.Int32Ptr(5)),
 			expectedRules:   getDefaultTestRules(true),
 		},
 		{
@@ -2448,6 +2448,100 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 			},
 			expectedProbes: getDefaultTestProbes("Tcp", ""),
 		},
+		{
+			desc: "getExpectedLBRules should prioritize port specific probe protocol over defaults",
+			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
+				"service.beta.kubernetes.io/port_80_health-probe_protocol": "HtTp",
+			}, false, 80),
+			expectedRules:  getDefaultTestRules(false),
+			expectedProbes: getDefaultTestProbes("Http", "/"),
+		},
+		{
+			desc: "getExpectedLBRules should prioritize port specific probe protocol over appProtocol",
+			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
+				"service.beta.kubernetes.io/port_80_health-probe_protocol": "HtTp",
+			}, false, 80),
+			probeProtocol:  "Mongodb",
+			expectedRules:  getDefaultTestRules(false),
+			expectedProbes: getDefaultTestProbes("Http", "/"),
+		},
+		{
+			desc: "getExpectedLBRules should prioritize port specific probe protocol over deprecated annotation",
+			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
+				"service.beta.kubernetes.io/port_80_health-probe_protocol":             "HtTpS",
+				"service.beta.kubernetes.io/azure-load-balancer-health-probe-protocol": "TcP",
+			}, false, 80),
+			loadBalancerSku: "standard",
+			probeProtocol:   "Https",
+			expectedRules:   getDefaultTestRules(true),
+			expectedProbes:  getDefaultTestProbes("Https", "/"),
+		},
+		{
+			desc: "getExpectedLBRules should default to Tcp on invalid port specific probe protocol",
+			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
+				"service.beta.kubernetes.io/port_80_health-probe_protocol": "FooBar",
+			}, false, 80),
+			probeProtocol:  "Http",
+			expectedRules:  getDefaultTestRules(false),
+			expectedProbes: getDefaultTestProbes("Tcp", ""),
+		},
+		{
+			desc: "getExpectedLBRules should support customize health probe port in multi-port service",
+			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
+				"service.beta.kubernetes.io/port_8000_health-probe_port": "port-tcp-80",
+			}, false, 80, 8000),
+			expectedRules: []network.LoadBalancingRule{
+				getTestRule(false, 80),
+				getTestRule(false, 8000),
+			},
+			expectedProbes: []network.Probe{
+				getTestProbe("Tcp", "/", to.Int32Ptr(5), to.Int32Ptr(80), to.Int32Ptr(10080), to.Int32Ptr(2)),
+				getTestProbe("Tcp", "/", to.Int32Ptr(5), to.Int32Ptr(8000), to.Int32Ptr(10080), to.Int32Ptr(2)),
+			},
+		},
+		{
+			desc: "getExpectedLBRules should support customize health probe port in multi-port service",
+			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
+				"service.beta.kubernetes.io/port_8000_health-probe_port": "80",
+			}, false, 80, 8000),
+			expectedRules: []network.LoadBalancingRule{
+				getTestRule(false, 80),
+				getTestRule(false, 8000),
+			},
+			expectedProbes: []network.Probe{
+				getTestProbe("Tcp", "/", to.Int32Ptr(5), to.Int32Ptr(80), to.Int32Ptr(10080), to.Int32Ptr(2)),
+				getTestProbe("Tcp", "/", to.Int32Ptr(5), to.Int32Ptr(8000), to.Int32Ptr(10080), to.Int32Ptr(2)),
+			},
+		},
+		{
+			desc: "getExpectedLBRules should not generate probe rule when no health probe rule is specified.",
+			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
+				"service.beta.kubernetes.io/port_8000_no_probe_rule": "true",
+			}, false, 80, 8000),
+			expectedRules: []network.LoadBalancingRule{
+				getTestRule(false, 80),
+				func() network.LoadBalancingRule {
+					rule := getTestRule(false, 8000)
+					rule.Probe = nil
+					return rule
+				}(),
+			},
+			expectedProbes: []network.Probe{
+				getTestProbe("Tcp", "/", to.Int32Ptr(5), to.Int32Ptr(80), to.Int32Ptr(10080), to.Int32Ptr(2)),
+			},
+		},
+		{
+			desc: "getExpectedLBRules should not generate lb rule and health probe rule when no lb rule is specified.",
+			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
+				"service.beta.kubernetes.io/port_8000_no_lb_rule": "true",
+			}, false, 80, 8000),
+			expectedRules: []network.LoadBalancingRule{
+				getTestRule(false, 80),
+			},
+			expectedProbes: []network.Probe{
+				getTestProbe("Tcp", "/", to.Int32Ptr(5), to.Int32Ptr(80), to.Int32Ptr(10080), to.Int32Ptr(2)),
+			},
+		},
 	}
 	rules := getDefaultTestRules(true)
 	rules[0].IdleTimeoutInMinutes = to.Int32Ptr(5)
@@ -2469,7 +2563,7 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 		}, false, 80),
 		loadBalancerSku: "standard",
 		probeProtocol:   "Tcp",
-		expectedProbes:  getTestProbes("Tcp", "", to.Int32Ptr(10), to.Int32Ptr(10080), to.Int32Ptr(10)),
+		expectedProbes:  getTestProbes("Tcp", "", to.Int32Ptr(10), to.Int32Ptr(80), to.Int32Ptr(10080), to.Int32Ptr(10)),
 		expectedRules:   rules,
 	})
 	rules1 := []network.LoadBalancingRule{
@@ -2480,14 +2574,19 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 	rules1[0].Probe.ID = to.StringPtr("/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/loadBalancers/lbname/probes/atest1-TCP-34567")
 	rules1[1].Probe.ID = to.StringPtr("/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/loadBalancers/lbname/probes/atest1-TCP-34567")
 	rules1[2].Probe.ID = to.StringPtr("/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/loadBalancers/lbname/probes/atest1-TCP-34567")
+
+	// When the service spec externalTrafficPolicy is Local all of these annotations should be ignored
 	svc := getTestService("test1", v1.ProtocolTCP, map[string]string{
-		"service.beta.kubernetes.io/port_80_health-probe_interval":     "10",
-		"service.beta.kubernetes.io/port_80_health-probe_num-of-probe": "10",
+		consts.ServiceAnnotationLoadBalancerHealthProbeProtocol:                                "tcp",
+		consts.ServiceAnnotationLoadBalancerHealthProbeRequestPath:                             "/broken/global/path",
+		consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsProbeInterval): "10",
+		consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsProtocol):      "https",
+		consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsRequestPath):   "/broken/local/path",
+		consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsNumOfProbe):    "10",
 	}, false, 80, 443, 421)
 	svc.Spec.ExternalTrafficPolicy = v1.ServiceExternalTrafficPolicyTypeLocal
 	svc.Spec.HealthCheckNodePort = 34567
-	probes := getTestProbes("Http", "/healthz", to.Int32Ptr(5), to.Int32Ptr(34567), to.Int32Ptr(2))
-	probes[0].Name = to.StringPtr("atest1-TCP-34567")
+	probes := getTestProbes("Http", "/healthz", to.Int32Ptr(5), to.Int32Ptr(34567), to.Int32Ptr(34567), to.Int32Ptr(2))
 	testCases = append(testCases, struct {
 		desc            string
 		service         v1.Service
@@ -2498,7 +2597,7 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 		expectedRules   []network.LoadBalancingRule
 		expectedErr     bool
 	}{
-		desc:            "getExpectedLBRules should expected rules when externaltrafficpolicy is local",
+		desc:            "getExpectedLBRules should expected rules when externalTrafficPolicy is local",
 		service:         svc,
 		loadBalancerSku: "standard",
 		probeProtocol:   "Http",
@@ -2528,18 +2627,19 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 		}
 	}
 }
-func getTestProbes(protocol, path string, interval, port, numOfProbe *int32) []network.Probe {
+
+func getTestProbes(protocol, path string, interval, servicePort, probePort, numOfProbe *int32) []network.Probe {
 	return []network.Probe{
-		getTestProbe(protocol, path, interval, port, numOfProbe),
+		getTestProbe(protocol, path, interval, servicePort, probePort, numOfProbe),
 	}
 }
 
-func getTestProbe(protocol, path string, interval, port, numOfProbe *int32) network.Probe {
+func getTestProbe(protocol, path string, interval, servicePort, probePort, numOfProbe *int32) network.Probe {
 	expectedProbes := network.Probe{
-		Name: to.StringPtr(fmt.Sprintf("atest1-TCP-%d", *port-10000)),
+		Name: to.StringPtr(fmt.Sprintf("atest1-TCP-%d", *servicePort)),
 		ProbePropertiesFormat: &network.ProbePropertiesFormat{
 			Protocol:          network.ProbeProtocol(protocol),
-			Port:              port,
+			Port:              probePort,
 			IntervalInSeconds: interval,
 			NumberOfProbes:    numOfProbe,
 		},
@@ -2550,7 +2650,7 @@ func getTestProbe(protocol, path string, interval, port, numOfProbe *int32) netw
 	return expectedProbes
 }
 func getDefaultTestProbes(protocol, path string) []network.Probe {
-	return getTestProbes(protocol, path, to.Int32Ptr(5), to.Int32Ptr(10080), to.Int32Ptr(2))
+	return getTestProbes(protocol, path, to.Int32Ptr(5), to.Int32Ptr(80), to.Int32Ptr(10080), to.Int32Ptr(2))
 }
 
 func getDefaultTestRules(enableTCPReset bool) []network.LoadBalancingRule {
diff --git a/tests/e2e/network/service_annotations.go b/tests/e2e/network/service_annotations.go
index a7373e663..4d1137666 100644
--- a/tests/e2e/network/service_annotations.go
+++ b/tests/e2e/network/service_annotations.go
@@ -23,6 +23,7 @@ import (
 	"os"
 	"reflect"
 	"regexp"
+	"strconv"
 	"strings"
 	"time"
 
@@ -53,8 +54,9 @@ var (
 )
 
 const (
-	serverPort  = 80
-	testingPort = 81
+	serverPort             = 80
+	alterNativeServicePort = 8080
+	testingPort            = 81
 )
 
 var _ = Describe("Service with annotation", Label(utils.TestSuiteLabelServiceAnnotation), func() {
@@ -71,10 +73,7 @@ var _ = Describe("Service with annotation", Label(utils.TestSuiteLabelServiceAnn
 	labels := map[string]string{
 		"app": serviceName,
 	}
-	ports := []v1.ServicePort{{
-		Port:       serverPort,
-		TargetPort: intstr.FromInt(serverPort),
-	}}
+	var ports []v1.ServicePort
 
 	BeforeEach(func() {
 		var err error
@@ -96,6 +95,12 @@ var _ = Describe("Service with annotation", Label(utils.TestSuiteLabelServiceAnn
 		utils.Logf("Waiting for backend pods to be ready")
 		err = utils.WaitPodsToBeReady(cs, ns.Name)
 		Expect(err).NotTo(HaveOccurred())
+
+		ports = []v1.ServicePort{{
+			Name:       "http",
+			Port:       serverPort,
+			TargetPort: intstr.FromInt(serverPort),
+		}}
 	})
 
 	AfterEach(func() {
@@ -598,6 +603,70 @@ var _ = Describe("Service with annotation", Label(utils.TestSuiteLabelServiceAnn
 		Expect(targetProbes[0].Protocol).To(Equal(network.ProbeProtocolHTTP))
 	})
 
+	It("should generate health probe configs in multi-port scenario", func() {
+		By("Creating a service with health probe annotations")
+		annotation := map[string]string{
+			consts.ServiceAnnotationLoadBalancerHealthProbeNumOfProbe:                                      "5",
+			consts.BuildHealthProbeAnnotationKeyForPort(serverPort, consts.HealthProbeParamsNumOfProbe):    "3",
+			consts.ServiceAnnotationLoadBalancerHealthProbeInterval:                                        "15",
+			consts.BuildHealthProbeAnnotationKeyForPort(serverPort, consts.HealthProbeParamsProbeInterval): "10",
+			consts.BuildHealthProbeAnnotationKeyForPort(serverPort, consts.HealthProbeParamsPort):          strconv.Itoa(alterNativeServicePort),
+			consts.ServiceAnnotationLoadBalancerHealthProbeProtocol:                                        "Http",
+			consts.ServiceAnnotationLoadBalancerHealthProbeRequestPath:                                     "/",
+			consts.BuildAnnotationKeyForPort(alterNativeServicePort, consts.PortAnnotationNoLBRule):        "true",
+		}
+		ports = append(ports, v1.ServicePort{
+			Name:       "port2",
+			Port:       alterNativeServicePort,
+			TargetPort: intstr.FromInt(serverPort),
+		})
+
+		// create service with given annotation and wait it to expose
+		publicIP := createAndExposeDefaultServiceWithAnnotation(cs, serviceName, ns.Name, labels, annotation, ports)
+		defer func() {
+			By("Cleaning up service")
+			err := utils.DeleteService(cs, ns.Name, serviceName)
+			Expect(err).NotTo(HaveOccurred())
+		}()
+		pipFrontendConfigID := getPIPFrontendConfigurationID(tc, publicIP, tc.GetResourceGroup(), "")
+		pipFrontendConfigIDSplit := strings.Split(pipFrontendConfigID, "/")
+		Expect(len(pipFrontendConfigIDSplit)).NotTo(Equal(0))
+
+		var lb *network.LoadBalancer
+		var targetProbes []*network.Probe
+		//wait for backend update
+		err := wait.PollImmediate(5*time.Second, 60*time.Second, func() (bool, error) {
+			lb = getAzureLoadBalancerFromPIP(tc, publicIP, tc.GetResourceGroup(), "")
+			targetProbes = []*network.Probe{}
+			for i := range *lb.LoadBalancerPropertiesFormat.Probes {
+				probe := (*lb.LoadBalancerPropertiesFormat.Probes)[i]
+				utils.Logf("One probe of LB is %q", *probe.Name)
+				targetProbes = append(targetProbes, &probe)
+			}
+			return len(targetProbes) == 1, nil
+		})
+		Expect(err).NotTo(HaveOccurred())
+
+		By("Validating health probe configs")
+		var numberOfProbes *int32
+		var intervalInSeconds *int32
+		for _, probe := range targetProbes {
+			if probe.NumberOfProbes != nil {
+				numberOfProbes = probe.NumberOfProbes
+			}
+			if probe.IntervalInSeconds != nil {
+				intervalInSeconds = probe.IntervalInSeconds
+			}
+		}
+		utils.Logf("Validating health probe config numberOfProbes")
+		Expect(*numberOfProbes).To(Equal(int32(3)))
+		utils.Logf("Validating health probe config intervalInSeconds")
+		Expect(*intervalInSeconds).To(Equal(int32(10)))
+		utils.Logf("Validating health probe config protocol")
+		Expect((len(targetProbes))).To(Equal(1))
+		Expect(targetProbes[0].Protocol).To(Equal(network.ProbeProtocolHTTP))
+	})
+
 	// Check if the following annotations are correctly set with Service LB IP
 	// service.beta.kubernetes.io/azure-load-balancer-ipv4 or service.beta.kubernetes.io/azure-load-balancer-ipv6
 	It("should support service annotation 'service.beta.kubernetes.io/azure-load-balancer-ip'", func() {