Repository: kubernetes-sigs/cloud-provider-azure
Pull request #4762 (2023-10-10 09:01:11)

feat: Add annotation `service.beta.kubernetes.io/azure-allowed-ip-ranges`
<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide/first-contribution.md#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?

<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature
/kind design

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->
/kind feature
#### What this PR does / why we need it:

This PR introduces a new annotation `service.beta.kubernetes.io/azure-allowed-ip-ranges` to manage the LoadBalancer service access. Previously, it was impossible to restrict access using both IP ranges and service tags (trying to use `spec.loadBalancerSourceRanges` and `service.beta.kubernetes.io/azure-service-tags` together didn’t work as one might think). With these changes, we can use `service.beta.kubernetes.io/azure-allowed-ip-ranges` and `service.beta.kubernetes.io/azure-service-tags` simultaneously.

Use cases:
##### limit access with both IP ranges and service tags
``` yaml
apiVersion: v1
kind: Service
metadata:
  name: test-lb
  annotations:
    service.beta.kubernetes.io/azure-allowed-ip-ranges: "192.168.0.1/32,10.20.0.0/16"
    service.beta.kubernetes.io/azure-allowed-service-tags: "AzureCloud"
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: nginx
```
<img width="1208" alt="image" src="https://github.com/kubernetes-sigs/cloud-provider-azure/assets/9134703/fa85bfd8-fe0b-4932-b3e3-75b5a83b1940">
As expected, three NSG rules will be created. Any source service with the IPs mentioned above, or those tagged with AzureCloud, will be able to access this LoadBalancer service.

##### specify both `spec.loadBalancerSourceRange` and the new annotation
``` yaml
apiVersion: v1
kind: Service
metadata:
  name: test-lb
  annotations:
    service.beta.kubernetes.io/azure-allowed-ip-ranges: "192.168.0.1/32,10.20.0.0/16"
spec:
  type: LoadBalancer
  loadBalancerSourceRanges:
    - 1.1.1.1/32
  ports:
  - port: 80
  selector:
    app: nginx
```
<img width="1512" alt="image" src="https://github.com/kubernetes-sigs/cloud-provider-azure/assets/9134703/d8d14595-5f23-4453-98ce-08a17cda04a1">
It doesn’t work, as expected.

##### specify `spec.loadBalancerSourceRange` and service tags annotation
``` yaml
apiVersion: v1
kind: Service
metadata:
  name: test-lb
  annotations:
    service.beta.kubernetes.io/azure-allowed-service-tags: "AzureCloud"
spec:
  type: LoadBalancer
  loadBalancerSourceRanges:
    - 1.1.1.1/32
  ports:
  - port: 80
  selector:
    app: nginx
```
<img width="1511" alt="image" src="https://github.com/kubernetes-sigs/cloud-provider-azure/assets/9134703/53c8c27d-048a-46dc-82c6-0673aa303f36">
Rather than blocking the process of creation or modification, it will generate a warning event.

#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes #

#### Special notes for your reviewer:

#### Does this PR introduce a user-facing change?
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://git.k8s.io/community/contributors/guide/release-notes.md
-->
```release-note
Introduce the annotation `service.beta.kubernetes.io/azure-allowed-ip-ranges` to manage the LoadBalancer service access.
```

#### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

<!--
This section can be blank if this pull request does not require a release note.

When adding links which point to resources within git repositories, like
KEPs or supporting documentation, please reference a specific commit and avoid
linking directly to the master branch. This ensures that links reference a
specific point in time, rather than a document that may change over time.

See here for guidance on getting permanent links to files: https://help.github.com/en/articles/getting-permanent-links-to-files

Please use the following format for linking documentation:
- [KEP]: <link>
- [Usage]: <link>
- [Other doc]: <link>
-->
```docs

```


k8s-ci-robot: Hi @lodrem. Thanks for your PR.

I'm waiting for a [kubernetes-sigs](https://github.com/orgs/kubernetes-sigs/people) member to verify that this patch is reasonable to test. If it is, they should reply with `/ok-to-test` on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should [join the org](https://git.k8s.io/community/community-membership.md#member) to skip this step.

Once the patch is verified, the new status will be reflected by the `ok-to-test` label.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>

k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/4762#" title="Author self-approved">lodrem</a>*
**Once this PR has been reviewed and has the lgtm label**, please assign [jwtty](https://github.com/jwtty) for approval. For more information see [the Kubernetes Code Review Process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process).

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["jwtty"]} -->
MartinForReal: /ok-to-test
k8s-ci-robot: @lodrem: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-check | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-check/1711686571754786816) | true | `/test pull-cloud-provider-azure-check`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-check | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-check/1711686571754786816) | true | `/test pull-cloud-provider-azure-check`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1711686572098719744) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-check | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-check/1711686571754786816) | true | `/test pull-cloud-provider-azure-check`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1711686572098719744) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1711686572048388096) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-check | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-check/1711686571754786816) | true | `/test pull-cloud-provider-azure-check`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1711686572098719744) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1711686572048388096) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1711686571876421632) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-check | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-check/1711686571754786816) | true | `/test pull-cloud-provider-azure-check`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1711686572098719744) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1711686572048388096) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1711686571876421632) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-capz | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-capz/1711686572002250752) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-check | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-check/1711686571754786816) | true | `/test pull-cloud-provider-azure-check`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1711686572098719744) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1711686572048388096) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1711686571876421632) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-capz | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-capz/1711686572002250752) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-capz | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1711686571821895680) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1711686572098719744) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1711686572048388096) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1711686571876421632) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-capz | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-capz/1711686572002250752) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-capz | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1711686571821895680) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`
pull-cloud-provider-azure-check | 75d17c53df9356df7b5cd4f681a761a472361b46 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-check/1711979678476013568) | true | `/test pull-cloud-provider-azure-check`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1711686572098719744) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1711686571876421632) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | f881a5f51f8453be60bc467268d741f0b1bdced0 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1711980663663497216) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 5b48e7ed3e3067cb86b95b54794b0632127765f5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1711686571876421632) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | f881a5f51f8453be60bc467268d741f0b1bdced0 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1711980663663497216) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | f881a5f51f8453be60bc467268d741f0b1bdced0 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1711980663722217472) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | f881a5f51f8453be60bc467268d741f0b1bdced0 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1711980663663497216) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | f881a5f51f8453be60bc467268d741f0b1bdced0 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1711980663722217472) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | f881a5f51f8453be60bc467268d741f0b1bdced0 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1711980663495725056) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | f881a5f51f8453be60bc467268d741f0b1bdced0 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1711980663722217472) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | f881a5f51f8453be60bc467268d741f0b1bdced0 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1711980663495725056) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 465b7c47d9ae26914f46a12d1237e83ac0544a9f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1712029479125127168) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | f881a5f51f8453be60bc467268d741f0b1bdced0 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1711980663495725056) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 465b7c47d9ae26914f46a12d1237e83ac0544a9f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1712029479125127168) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 465b7c47d9ae26914f46a12d1237e83ac0544a9f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1712029479175458816) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 465b7c47d9ae26914f46a12d1237e83ac0544a9f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1712029479125127168) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 465b7c47d9ae26914f46a12d1237e83ac0544a9f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1712029479175458816) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 465b7c47d9ae26914f46a12d1237e83ac0544a9f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1712029478969937920) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 465b7c47d9ae26914f46a12d1237e83ac0544a9f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1712029479125127168) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 465b7c47d9ae26914f46a12d1237e83ac0544a9f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1712029479175458816) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 465b7c47d9ae26914f46a12d1237e83ac0544a9f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1712029478969937920) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-capz | 465b7c47d9ae26914f46a12d1237e83ac0544a9f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1712029478911217664) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 465b7c47d9ae26914f46a12d1237e83ac0544a9f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1712029479125127168) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 465b7c47d9ae26914f46a12d1237e83ac0544a9f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1712029479175458816) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 465b7c47d9ae26914f46a12d1237e83ac0544a9f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1712029478969937920) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-capz | 465b7c47d9ae26914f46a12d1237e83ac0544a9f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1712029478911217664) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`
pull-cloud-provider-azure-e2e-ccm-capz | 465b7c47d9ae26914f46a12d1237e83ac0544a9f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-capz/1712029479070601216) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 465b7c47d9ae26914f46a12d1237e83ac0544a9f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1712029478969937920) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-capz | 465b7c47d9ae26914f46a12d1237e83ac0544a9f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1712029478911217664) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`
pull-cloud-provider-azure-e2e-ccm-capz | 465b7c47d9ae26914f46a12d1237e83ac0544a9f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-capz/1712029479070601216) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 5c9a50970ee033ee51bf641793c44a5731d01979 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1712120655576567808) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 5c9a50970ee033ee51bf641793c44a5731d01979 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1712120655639482368) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-capz | 465b7c47d9ae26914f46a12d1237e83ac0544a9f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1712029478911217664) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`
pull-cloud-provider-azure-e2e-ccm-capz | 465b7c47d9ae26914f46a12d1237e83ac0544a9f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-capz/1712029479070601216) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 5c9a50970ee033ee51bf641793c44a5731d01979 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1712120655576567808) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 5c9a50970ee033ee51bf641793c44a5731d01979 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1712120655639482368) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 5c9a50970ee033ee51bf641793c44a5731d01979 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1712120655392018432) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-capz | 465b7c47d9ae26914f46a12d1237e83ac0544a9f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-capz/1712029479070601216) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 5c9a50970ee033ee51bf641793c44a5731d01979 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1712120655576567808) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 5c9a50970ee033ee51bf641793c44a5731d01979 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1712120655639482368) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 5c9a50970ee033ee51bf641793c44a5731d01979 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1712120655392018432) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-capz | 5c9a50970ee033ee51bf641793c44a5731d01979 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1712120655320715264) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 5c9a50970ee033ee51bf641793c44a5731d01979 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1712120655576567808) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 5c9a50970ee033ee51bf641793c44a5731d01979 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1712120655639482368) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 5c9a50970ee033ee51bf641793c44a5731d01979 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1712120655392018432) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-capz | 5c9a50970ee033ee51bf641793c44a5731d01979 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1712120655320715264) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`
pull-cloud-provider-azure-e2e-ccm-capz | 5c9a50970ee033ee51bf641793c44a5731d01979 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-capz/1712120655509458944) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 5c9a50970ee033ee51bf641793c44a5731d01979 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1712120655576567808) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 5c9a50970ee033ee51bf641793c44a5731d01979 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1712120655639482368) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 5c9a50970ee033ee51bf641793c44a5731d01979 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1712120655392018432) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-capz | 5c9a50970ee033ee51bf641793c44a5731d01979 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1712120655320715264) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`
pull-cloud-provider-azure-e2e-ccm-capz | 5c9a50970ee033ee51bf641793c44a5731d01979 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-capz/1712120655509458944) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`
pull-cloud-provider-azure-e2e-capz | 8786938ca8e71a8f07133470a3ab809b65462ed8 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-capz/1712290188845125632) | true | `/test pull-cloud-provider-azure-e2e-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 5c9a50970ee033ee51bf641793c44a5731d01979 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1712120655392018432) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-capz | 5c9a50970ee033ee51bf641793c44a5731d01979 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-capz/1712120655509458944) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`
pull-cloud-provider-azure-e2e-capz | 8786938ca8e71a8f07133470a3ab809b65462ed8 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-capz/1712290188845125632) | true | `/test pull-cloud-provider-azure-e2e-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-capz | 8786938ca8e71a8f07133470a3ab809b65462ed8 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1712290187117072384) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-capz | 8786938ca8e71a8f07133470a3ab809b65462ed8 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-capz/1712290188845125632) | true | `/test pull-cloud-provider-azure-e2e-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-capz | 8786938ca8e71a8f07133470a3ab809b65462ed8 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1712290187117072384) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`
pull-cloud-provider-azure-e2e-ccm-capz | 8786938ca8e71a8f07133470a3ab809b65462ed8 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-capz/1712290189633654784) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
lodrem: /retest
k8s-ci-robot: @lodrem: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1719140274191470592) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1719140274191470592) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1719140274300522496) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1719140274191470592) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1719140274300522496) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1719140274124361728) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1719140274191470592) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1719140274300522496) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1719140274124361728) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1719140274355048448) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
nilo19: /retest
nilo19: /lgtm
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1719140274191470592) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1719140274300522496) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1719140274124361728) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1719536271879049216) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1719140274191470592) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1719536271879049216) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1719536271824523264) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1719536271715471360) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1719536271879049216) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1719536271824523264) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1719536271715471360) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1719536271765803008) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: New changes are detected. LGTM label has been removed.
k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/4762#" title="Author self-approved">lodrem</a>*
**Once this PR has been reviewed and has the lgtm label**, please ask for approval from [nilo19](https://github.com/nilo19). For more information see [the Kubernetes Code Review Process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process).

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["nilo19"]} -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1719536271824523264) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1719536271715471360) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1719536271765803008) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | f9a9a2ab24b844cbf5e91545b695d9987fde1238 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1719594449446637568) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1719536271824523264) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1719536271715471360) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | f9a9a2ab24b844cbf5e91545b695d9987fde1238 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1719594449446637568) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | f9a9a2ab24b844cbf5e91545b695d9987fde1238 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1719594449215950848) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1719536271715471360) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | f9a9a2ab24b844cbf5e91545b695d9987fde1238 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1719594449446637568) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | f9a9a2ab24b844cbf5e91545b695d9987fde1238 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1719594449215950848) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | f9a9a2ab24b844cbf5e91545b695d9987fde1238 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1719594449387917312) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-capz | 0efe5f84ccba93817e4fcffb8a87f393f0c80483 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1719536271715471360) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | f9a9a2ab24b844cbf5e91545b695d9987fde1238 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1719594449446637568) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | f9a9a2ab24b844cbf5e91545b695d9987fde1238 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1719594449215950848) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | f9a9a2ab24b844cbf5e91545b695d9987fde1238 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1719594449387917312) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-capz | f9a9a2ab24b844cbf5e91545b695d9987fde1238 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-capz/1719594449316614144) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-capz/1719926668430151680) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
feiskyer: /retest
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-capz/1719926668430151680) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1719926668539203584) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-capz/1719926668430151680) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1719926668539203584) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1719926668480483328) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-capz/1719926668430151680) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1719926668539203584) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1719926668480483328) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1719926668262379520) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-capz/1719926668430151680) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1719926668539203584) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1719926668480483328) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1719926668262379520) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1719926668316905472) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1719926668539203584) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1719926668480483328) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1719926668262379520) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1719926668316905472) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-capz/1719940222294167552) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1719926668539203584) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1719926668480483328) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-capz/1719926668262379520) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz`
pull-cloud-provider-azure-e2e-ccm-vmssflex-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmssflex-capz/1719926668316905472) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmssflex-capz`
pull-cloud-provider-azure-e2e-ccm-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-capz/1719940222294167552) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`
pull-cloud-provider-azure-e2e-capz | 7f24c51cd0bc33798de98e8a92f1fff20c7901d1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-capz/1719926668375625728) | true | `/test pull-cloud-provider-azure-e2e-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
phealy: > <img alt="image" width="1208" src="https://user-images.githubusercontent.com/9134703/274150690-fa85bfd8-fe0b-4932-b3e3-75b5a83b1940.png">

@lodrem Security rules support putting multiple CIDR blocks in a single rule source entry - can we combine rules 500 and 501 into a single rule with the source list of `192.168.0.1/32,10.20.0.0/16`?

[From the NSG docs](https://learn.microsoft.com/en-us/azure/virtual-network/network-security-groups-overview#:~:text=The%20ability%20to%20specify%20multiple%20individual%20IP%20addresses%20and%20ranges%20(you%20can%27t%20specify%20multiple%20service%20tags%20or%20application%20groups)%20in%20a%20rule%20is%20referred%20to%20as%20augmented%20security%20rules):
> The ability to specify multiple individual IP addresses and ranges (you can't specify multiple service tags or application groups) in a rule is referred to as [augmented security rules](https://learn.microsoft.com/en-us/azure/virtual-network/network-security-groups-overview#augmented-security-rules).

I believe we can save on rule count if `service.beta.kubernetes.io/azure-allowed-ip-ranges` maps to only 1 rule. `service.beta.kubernetes.io/azure-allowed-service-tags` still has create one rule per tag because NSG rules don't support combining service tag entries.

lodrem: > @lodrem Security rules support putting multiple CIDR blocks in a single rule source entry - can we combine rules 500 and 501 into a single rule with the source list of 192.168.0.1/32,10.20.0.0/16?

@phealy Yes, but this is a big change since we used to use the **1 CIDR 1 rule** before this PR. I think we should make improvements to this pattern in a separate PR later on.
phealy: > @phealy Yes, but this is a big change since we used to use the **1 CIDR 1 rule** before this PR. I think we should make improvements to this pattern in a separate PR later on.

Good call. I didn't realize we were doing that for `loadBalancerSourceRanges` as well - I thought we were regressing a bit here because for some reason I thought we were combining previously. Splitting that to a separate PR makes perfect sense; then we can fix it for both.

Thanks!
yanrez: > Good call. I didn't realize we were doing that for `loadBalancerSourceRanges` as well - I thought we were regressing a bit here because for some reason I thought we were combining previously. Splitting that to a separate PR makes perfect sense; then we can fix it for both.

This is quite important to improve, rule count limits have been an issue in our use cases already, it would be very useful to see them combined as much as possible.

feiskyer: > I believe we can save on rule count if service.beta.kubernetes.io/azure-allowed-ip-ranges maps to only 1 rule. service.beta.kubernetes.io/azure-allowed-service-tags still has create one rule per tag because NSG rules don't support combining service tag entries.

@lodrem  could you create a new issue to track this improvement?
lodrem: https://github.com/kubernetes-sigs/cloud-provider-azure/issues/4713
feiskyer: /lgtm
/approve
k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/4762#issuecomment-1798371757" title="Approved">feiskyer</a>*, *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/4762#" title="Author self-approved">lodrem</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/master/OWNERS)~~ [feiskyer]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
k8s-ci-robot: @lodrem: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz | 33d8cc1ac497bcd0fa3d9b6d0f634df056449c02 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz/1721877794251608064) | unknown | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: New changes are detected. LGTM label has been removed.
k8s-ci-robot: @lodrem: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 1db9da097e732c49c2e612af9b090a0e2845dd90 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1722067744314101760) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
lodrem: /retest
k8s-ci-robot: @lodrem: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | 1db9da097e732c49c2e612af9b090a0e2845dd90 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1722074731571580928) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
lodrem: /retest
k8s-ci-robot: @lodrem: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-capz | 1db9da097e732c49c2e612af9b090a0e2845dd90 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-capz/1722067744213438464) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
lodrem: /retest
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-capz | 1db9da097e732c49c2e612af9b090a0e2845dd90 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-capz/1722067744213438464) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | a23284e076a0424fa71854b8f1f20bb5d6704759 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1722177585766469632) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
lodrem: /retest
k8s-ci-robot: @lodrem: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-capz | 1db9da097e732c49c2e612af9b090a0e2845dd90 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-capz/1722067744213438464) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz`
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz | a23284e076a0424fa71854b8f1f20bb5d6704759 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz/1722177585766469632) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz`
pull-cloud-provider-azure-e2e-capz | a23284e076a0424fa71854b8f1f20bb5d6704759 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/4762/pull-cloud-provider-azure-e2e-capz/1722177585590308864) | true | `/test pull-cloud-provider-azure-e2e-capz`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=4762). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alodrem). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->

Patch:
diff --git a/pkg/consts/consts.go b/pkg/consts/consts.go
index 3f40566f9..98c401fb2 100644
--- a/pkg/consts/consts.go
+++ b/pkg/consts/consts.go
@@ -282,10 +282,13 @@ const (
 	// ServiceAnnotationIPTagsForPublicIP specifies the iptags used when dynamically creating a public ip
 	ServiceAnnotationIPTagsForPublicIP = "service.beta.kubernetes.io/azure-pip-ip-tags"
 
-	// ServiceAnnotationAllowedServiceTag is the annotation used on the service
+	// ServiceAnnotationAllowedServiceTags is the annotation used on the service
 	// to specify a list of allowed service tags separated by comma
 	// Refer https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#service-tags for all supported service tags.
-	ServiceAnnotationAllowedServiceTag = "service.beta.kubernetes.io/azure-allowed-service-tags"
+	ServiceAnnotationAllowedServiceTags = "service.beta.kubernetes.io/azure-allowed-service-tags"
+
+	// ServiceAnnotationAllowedIPRanges is the annotation used on the service
+	ServiceAnnotationAllowedIPRanges = "service.beta.kubernetes.io/azure-allowed-ip-ranges"
 
 	// ServiceAnnotationDenyAllExceptLoadBalancerSourceRanges  denies all traffic to the load balancer except those
 	// within the service.Spec.LoadBalancerSourceRanges. Ref: https://github.com/kubernetes-sigs/cloud-provider-azure/issues/374.
diff --git a/pkg/provider/azure_loadbalancer.go b/pkg/provider/azure_loadbalancer.go
index 6a41443a1..8301937b3 100644
--- a/pkg/provider/azure_loadbalancer.go
+++ b/pkg/provider/azure_loadbalancer.go
@@ -41,13 +41,13 @@ import (
 	cloudprovider "k8s.io/cloud-provider"
 	servicehelpers "k8s.io/cloud-provider/service/helpers"
 	"k8s.io/klog/v2"
-	utilnet "k8s.io/utils/net"
 	"k8s.io/utils/pointer"
 	"k8s.io/utils/strings/slices"
 
 	azcache "sigs.k8s.io/cloud-provider-azure/pkg/cache"
 	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
 	"sigs.k8s.io/cloud-provider-azure/pkg/metrics"
+	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer"
 	"sigs.k8s.io/cloud-provider-azure/pkg/retry"
 )
 
@@ -2855,36 +2855,50 @@ func (az *Cloud) reconcileSecurityGroup(clusterName string, service *v1.Service,
 		}
 	}
 
-	sourceRanges, err := servicehelpers.GetLoadBalancerSourceRanges(service)
+	accessControl, err := loadbalancer.NewAccessControl(service)
 	if err != nil {
+		klog.ErrorS(err, "Failed to parse access control configuration for service", "service", service.Name)
 		return nil, err
 	}
-	serviceTags := getServiceTags(service)
-	if len(serviceTags) != 0 {
-		delete(sourceRanges, consts.DefaultLoadBalancerSourceRanges)
-	}
 
-	sourceAddressPrefixes := map[bool][]string{}
-	if (sourceRanges == nil || servicehelpers.IsAllowAll(sourceRanges)) && len(serviceTags) == 0 {
-		if !requiresInternalLoadBalancer(service) || len(service.Spec.LoadBalancerSourceRanges) > 0 {
-			sourceAddressPrefixes[false] = []string{"Internet"}
-			sourceAddressPrefixes[true] = []string{"Internet"}
-		}
-	} else {
-		for _, ip := range sourceRanges {
-			if ip == nil {
-				continue
-			}
-			isIPv6 := net.ParseIP(ip.IP.String()).To4() == nil
-			sourceAddressPrefixes[isIPv6] = append(sourceAddressPrefixes[isIPv6], ip.String())
+	var (
+		sourceRanges          = accessControl.SourceRanges()
+		allowedServiceTags    = accessControl.AllowedServiceTags()
+		allowedIPRanges       = accessControl.AllowedIPRanges()
+		sourceAddressPrefixes = map[bool][]string{
+			false: accessControl.IPV4Sources(),
+			true:  accessControl.IPV6Sources(),
 		}
-		sourceAddressPrefixes[false] = append(sourceAddressPrefixes[false], serviceTags...)
-		sourceAddressPrefixes[true] = append(sourceAddressPrefixes[true], serviceTags...)
+	)
+
+	if len(sourceRanges) != 0 && len(allowedIPRanges) != 0 {
+		// Block the service and return error if both of spec.loadBalancerSourceRanges and annotation are specified
+		klog.Errorf("Service %s is using both of spec.loadBalancerSourceRanges and annotation %s.", service.Name, consts.ServiceAnnotationAllowedIPRanges)
+		return nil, fmt.Errorf(
+			"both of spec.loadBalancerSourceRanges and annotation %s are specified for service %s, which is not allowed",
+			consts.ServiceAnnotationAllowedIPRanges, service.Name,
+		)
+	}
+	if len(sourceRanges) != 0 && len(allowedServiceTags) != 0 {
+		// Suggesting to use aks custom annotation instead of spec.loadBalancerSourceRanges
+		klog.Warningf(
+			"Service %s is using both of spec.loadBalancerSourceRanges and annotation %s.",
+			service.Name, consts.ServiceAnnotationAllowedServiceTags,
+		)
+		az.Event(service, v1.EventTypeWarning, "ConflictConfiguration", fmt.Sprintf(
+			"Please use annotation %s instead of spec.loadBalancerSourceRanges while using %s annotation at the same time.",
+			consts.ServiceAnnotationAllowedIPRanges, consts.ServiceAnnotationAllowedServiceTags,
+		))
 	}
 
-	expectedSecurityRules := []network.SecurityRule{}
+	var expectedSecurityRules []network.SecurityRule
 	handleSecurityRules := func(isIPv6 bool) error {
-		expectedSecurityRulesSingleStack, err := az.getExpectedSecurityRules(wantLb, ports, sourceAddressPrefixes[isIPv6], service, destinationIPAddresses[isIPv6], sourceRanges, backendIPAddresses[isIPv6], disableFloatingIP, isIPv6)
+		expectedSecurityRulesSingleStack, err := az.getExpectedSecurityRules(
+			wantLb, ports,
+			sourceAddressPrefixes[isIPv6], service,
+			destinationIPAddresses[isIPv6], sourceRanges,
+			backendIPAddresses[isIPv6], disableFloatingIP, isIPv6,
+		)
 		expectedSecurityRules = append(expectedSecurityRules, expectedSecurityRulesSingleStack...)
 		return err
 	}
@@ -3068,7 +3082,16 @@ func (az *Cloud) reconcileSecurityRules(sg network.SecurityGroup,
 	return dirtySg, updatedRules, nil
 }
 
-func (az *Cloud) getExpectedSecurityRules(wantLb bool, ports []v1.ServicePort, sourceAddressPrefixes []string, service *v1.Service, destinationIPAddresses []string, sourceRanges utilnet.IPNetSet, backendIPAddresses []string, disableFloatingIP, isIPv6 bool) ([]network.SecurityRule, error) {
+func (az *Cloud) getExpectedSecurityRules(
+	wantLb bool,
+	ports []v1.ServicePort,
+	sourceAddressPrefixes []string,
+	service *v1.Service,
+	destinationIPAddresses []string,
+	sourceRanges []netip.Prefix,
+	backendIPAddresses []string,
+	disableFloatingIP, isIPv6 bool,
+) ([]network.SecurityRule, error) {
 	expectedSecurityRules := []network.SecurityRule{}
 
 	if wantLb {
@@ -3111,7 +3134,7 @@ func (az *Cloud) getExpectedSecurityRules(wantLb bool, ports []v1.ServicePort, s
 		}
 
 		shouldAddDenyRule := false
-		if len(sourceRanges) > 0 && !servicehelpers.IsAllowAll(sourceRanges) {
+		if len(sourceRanges) > 0 && !loadbalancer.IsCIDRsAllowAll(sourceRanges) {
 			if v, ok := service.Annotations[consts.ServiceAnnotationDenyAllExceptLoadBalancerSourceRanges]; ok && strings.EqualFold(v, consts.TrueAnnotationValue) {
 				shouldAddDenyRule = true
 			}
@@ -3870,7 +3893,7 @@ func getServiceTags(service *v1.Service) []string {
 		return nil
 	}
 
-	if serviceTags, found := service.Annotations[consts.ServiceAnnotationAllowedServiceTag]; found {
+	if serviceTags, found := service.Annotations[consts.ServiceAnnotationAllowedServiceTags]; found {
 		result := []string{}
 		tags := strings.Split(strings.TrimSpace(serviceTags), ",")
 		for _, tag := range tags {
diff --git a/pkg/provider/loadbalancer/accesscontrol.go b/pkg/provider/loadbalancer/accesscontrol.go
new file mode 100644
index 000000000..c71758469
--- /dev/null
+++ b/pkg/provider/loadbalancer/accesscontrol.go
@@ -0,0 +1,202 @@
+/*
+Copyright 2023 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package loadbalancer
+
+import (
+	"fmt"
+	"net/netip"
+	"strings"
+
+	v1 "k8s.io/api/core/v1"
+
+	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
+)
+
+// IsInternal returns true if the given service is internal load balancer.
+func IsInternal(svc *v1.Service) bool {
+	value, found := svc.Annotations[consts.ServiceAnnotationLoadBalancerInternal]
+	return found && strings.ToLower(value) == "true"
+}
+
+// IsExternal returns true if the given service is external load balancer.
+func IsExternal(svc *v1.Service) bool {
+	return !IsInternal(svc)
+}
+
+// AllowedServiceTags returns the allowed service tags configured by user through AKS custom annotation.
+func AllowedServiceTags(svc *v1.Service) ([]string, error) {
+	const Sep = ","
+
+	value, found := svc.Annotations[consts.ServiceAnnotationAllowedServiceTags]
+	if !found {
+		return nil, nil
+	}
+
+	return strings.Split(strings.TrimSpace(value), Sep), nil
+}
+
+// AllowedIPRanges returns the allowed IP ranges configured by user through AKS custom annotation.
+func AllowedIPRanges(svc *v1.Service) ([]netip.Prefix, error) {
+	const Sep = ","
+
+	value, found := svc.Annotations[consts.ServiceAnnotationAllowedIPRanges]
+	if !found {
+		return nil, nil
+	}
+
+	rv, err := ParseCIDRs(strings.Split(strings.TrimSpace(value), Sep))
+	if err != nil {
+		return nil, fmt.Errorf("invalid service annotation %s:%s: %w", consts.ServiceAnnotationAllowedIPRanges, value, err)
+	}
+
+	return rv, nil
+}
+
+// SourceRanges returns the allowed IP ranges configured by user through `spec.LoadBalancerSourceRanges` and standard annotation.
+// If `spec.LoadBalancerSourceRanges` is not set, it will try to parse the annotation.
+func SourceRanges(svc *v1.Service) ([]netip.Prefix, error) {
+	if len(svc.Spec.LoadBalancerSourceRanges) > 0 {
+		rv, err := ParseCIDRs(svc.Spec.LoadBalancerSourceRanges)
+		if err != nil {
+			return nil, fmt.Errorf("invalid service.Spec.LoadBalancerSourceRanges [%v]: %w", svc.Spec.LoadBalancerSourceRanges, err)
+		}
+		return rv, nil
+	}
+
+	const Sep = ","
+	value, found := svc.Annotations[v1.AnnotationLoadBalancerSourceRangesKey]
+	if !found {
+		return nil, nil
+	}
+	rv, err := ParseCIDRs(strings.Split(strings.TrimSpace(value), Sep))
+	if err != nil {
+		return nil, fmt.Errorf("invalid service annotation %s:%s: %w", v1.AnnotationLoadBalancerSourceRangesKey, value, err)
+	}
+	return rv, nil
+}
+
+type AccessControl struct {
+	svc *v1.Service
+
+	// immutable redundant states.
+	sourceRanges       []netip.Prefix
+	allowedIPRanges    []netip.Prefix
+	allowedServiceTags []string
+}
+
+func NewAccessControl(svc *v1.Service) (*AccessControl, error) {
+	sourceRanges, err := SourceRanges(svc)
+	if err != nil {
+		return nil, err
+	}
+	allowedIPRanges, err := AllowedIPRanges(svc)
+	if err != nil {
+		return nil, err
+	}
+	allowedServiceTags, err := AllowedServiceTags(svc)
+	if err != nil {
+		return nil, err
+	}
+
+	return &AccessControl{
+		svc:                svc,
+		sourceRanges:       sourceRanges,
+		allowedIPRanges:    allowedIPRanges,
+		allowedServiceTags: allowedServiceTags,
+	}, nil
+}
+
+// SourceRanges returns the allowed IP ranges configured by user through `spec.LoadBalancerSourceRanges` and standard annotation.
+func (ac *AccessControl) SourceRanges() []netip.Prefix {
+	return ac.sourceRanges
+}
+
+// AllowedIPRanges returns the allowed IP ranges configured by user through AKS custom annotation.
+func (ac *AccessControl) AllowedIPRanges() []netip.Prefix {
+	return ac.allowedIPRanges
+}
+
+// AllowedServiceTags returns the allowed service tags configured by user through AKS custom annotation.
+func (ac *AccessControl) AllowedServiceTags() []string {
+	return ac.allowedServiceTags
+}
+
+// IsAllowFromInternet returns true if the given service is allowed to be accessed from internet.
+// To be specific,
+// 1. For all types of LB, it returns false if the given service is specified with `service tags` or `not allowed all IP ranges`.
+// 2. For internal LB, it returns true iff the given service is explicitly specified with `allowed all IP ranges`. Refer: https://github.com/kubernetes-sigs/cloud-provider-azure/issues/698
+func (ac *AccessControl) IsAllowFromInternet() bool {
+	if len(ac.allowedServiceTags) > 0 {
+		return false
+	}
+	if len(ac.sourceRanges) > 0 && !IsCIDRsAllowAll(ac.sourceRanges) {
+		return false
+	}
+	if len(ac.allowedIPRanges) > 0 && !IsCIDRsAllowAll(ac.allowedIPRanges) {
+		return false
+	}
+	if IsExternal(ac.svc) {
+		return true
+	}
+	// Internal LB with explicit allowedAll IP ranges is allowed to be accessed from internet.
+	return len(ac.allowedIPRanges) > 0 || len(ac.sourceRanges) > 0
+}
+
+// IPV4Sources returns the allowed sources for IPv4.
+func (ac *AccessControl) IPV4Sources() []string {
+	var rv []string
+
+	if ac.IsAllowFromInternet() {
+		rv = append(rv, "Internet")
+	}
+	for _, cidr := range ac.sourceRanges {
+		if cidr.Addr().Is4() {
+			rv = append(rv, cidr.String())
+		}
+	}
+	for _, cidr := range ac.allowedIPRanges {
+		if cidr.Addr().Is4() {
+			rv = append(rv, cidr.String())
+		}
+	}
+	rv = append(rv, ac.allowedServiceTags...)
+
+	return rv
+}
+
+// IPV6Sources returns the allowed sources for IPv6.
+func (ac *AccessControl) IPV6Sources() []string {
+	var (
+		rv []string
+	)
+	if ac.IsAllowFromInternet() {
+		rv = append(rv, "Internet")
+	}
+	for _, cidr := range ac.sourceRanges {
+		if cidr.Addr().Is6() {
+			rv = append(rv, cidr.String())
+		}
+	}
+	for _, cidr := range ac.allowedIPRanges {
+		if cidr.Addr().Is6() {
+			rv = append(rv, cidr.String())
+		}
+	}
+	rv = append(rv, ac.allowedServiceTags...)
+
+	return rv
+}
diff --git a/pkg/provider/loadbalancer/netip.go b/pkg/provider/loadbalancer/netip.go
new file mode 100644
index 000000000..c7e4632b6
--- /dev/null
+++ b/pkg/provider/loadbalancer/netip.go
@@ -0,0 +1,50 @@
+/*
+Copyright 2023 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package loadbalancer
+
+import (
+	"fmt"
+	"net/netip"
+)
+
+const (
+	IPv4AllowedAll = "0.0.0.0/0"
+	IPv6AllowedAll = "::/0"
+)
+
+// IsCIDRsAllowAll return true if the given IP Ranges covers all IPs.
+// It returns false if the given IP Ranges is empty.
+func IsCIDRsAllowAll(cidrs []netip.Prefix) bool {
+	for _, cidr := range cidrs {
+		if cidr.String() == IPv4AllowedAll || cidr.String() == IPv6AllowedAll {
+			return true
+		}
+	}
+	return false
+}
+
+func ParseCIDRs(parts []string) ([]netip.Prefix, error) {
+	var rv []netip.Prefix
+	for _, part := range parts {
+		prefix, err := netip.ParsePrefix(part)
+		if err != nil {
+			return nil, fmt.Errorf("invalid IP range %s: %w", part, err)
+		}
+		rv = append(rv, prefix)
+	}
+	return rv, nil
+}


Test patch:
diff --git a/pkg/provider/azure_loadbalancer_test.go b/pkg/provider/azure_loadbalancer_test.go
index c864f5572..2556f26df 100644
--- a/pkg/provider/azure_loadbalancer_test.go
+++ b/pkg/provider/azure_loadbalancer_test.go
@@ -1622,7 +1622,7 @@ func TestGetServiceTags(t *testing.T) {
 			service: &v1.Service{
 				ObjectMeta: metav1.ObjectMeta{
 					Annotations: map[string]string{
-						consts.ServiceAnnotationAllowedServiceTag: "tag1",
+						consts.ServiceAnnotationAllowedServiceTags: "tag1",
 					},
 				},
 			},
@@ -1633,7 +1633,7 @@ func TestGetServiceTags(t *testing.T) {
 			service: &v1.Service{
 				ObjectMeta: metav1.ObjectMeta{
 					Annotations: map[string]string{
-						consts.ServiceAnnotationAllowedServiceTag: "tag1, tag2",
+						consts.ServiceAnnotationAllowedServiceTags: "tag1, tag2",
 					},
 				},
 			},
@@ -1644,7 +1644,7 @@ func TestGetServiceTags(t *testing.T) {
 			service: &v1.Service{
 				ObjectMeta: metav1.ObjectMeta{
 					Annotations: map[string]string{
-						consts.ServiceAnnotationAllowedServiceTag: ", tag1, ",
+						consts.ServiceAnnotationAllowedServiceTags: ", tag1, ",
 					},
 				},
 			},
@@ -4341,7 +4341,7 @@ func TestReconcileSecurityGroupCommon(t *testing.T) {
 		},
 		{
 			desc:    "reconcileSecurityGroup shall not create unwanted security rules if there is service tags",
-			service: getTestService("test1", v1.ProtocolTCP, map[string]string{consts.ServiceAnnotationAllowedServiceTag: "tag"}, false, 80),
+			service: getTestService("test1", v1.ProtocolTCP, map[string]string{consts.ServiceAnnotationAllowedServiceTags: "tag"}, false, 80),
 			wantLb:  true,
 			lbIPs:   &[]string{"1.1.1.1"},
 			existingSgs: map[string]network.SecurityGroup{"nsg": {
@@ -4551,6 +4551,316 @@ func TestReconcileSecurityGroupCommon(t *testing.T) {
 				},
 			},
 		},
+		{
+			desc: "reconcileSecurityGroup shall create sgs while allowedIPRanges annotation is set for IPv4",
+			service: getTestService("svc", v1.ProtocolTCP, map[string]string{
+				consts.ServiceAnnotationAllowedIPRanges: "10.10.10.0/24,192.168.0.1/32",
+			}, false, 80),
+			existingSgs: map[string]network.SecurityGroup{"nsg": {
+				Name:                          pointer.String("nsg"),
+				SecurityGroupPropertiesFormat: &network.SecurityGroupPropertiesFormat{},
+			}},
+			lbIPs:  &[]string{"10.0.0.1", "10.0.0.2"},
+			wantLb: true,
+			expectedSg: &network.SecurityGroup{
+				Name: pointer.String("nsg"),
+				SecurityGroupPropertiesFormat: &network.SecurityGroupPropertiesFormat{
+					SecurityRules: &[]network.SecurityRule{
+						{
+							Name: pointer.String("asvc-TCP-80-10.10.10.0_24"),
+							SecurityRulePropertiesFormat: &network.SecurityRulePropertiesFormat{
+								Protocol:                   network.SecurityRuleProtocol("Tcp"),
+								SourcePortRange:            pointer.String("*"),
+								DestinationPortRange:       pointer.String(strconv.Itoa(80)),
+								SourceAddressPrefix:        pointer.String("10.10.10.0/24"),
+								DestinationAddressPrefixes: &([]string{"10.0.0.1", "10.0.0.2"}),
+								Access:                     network.SecurityRuleAccessAllow,
+								Priority:                   pointer.Int32(500),
+								Direction:                  network.SecurityRuleDirection("Inbound"),
+							},
+						},
+						{
+							Name: pointer.String("asvc-TCP-80-192.168.0.1_32"),
+							SecurityRulePropertiesFormat: &network.SecurityRulePropertiesFormat{
+								Protocol:                   network.SecurityRuleProtocol("Tcp"),
+								SourcePortRange:            pointer.String("*"),
+								DestinationPortRange:       pointer.String(strconv.Itoa(80)),
+								SourceAddressPrefix:        pointer.String("192.168.0.1/32"),
+								DestinationAddressPrefixes: &([]string{"10.0.0.1", "10.0.0.2"}),
+								Access:                     network.SecurityRuleAccessAllow,
+								Priority:                   pointer.Int32(501),
+								Direction:                  network.SecurityRuleDirection("Inbound"),
+							},
+						},
+					},
+				},
+			},
+		},
+		{
+			desc: "reconcileSecurityGroup shall create sgs while allowedIPRanges annotation is set for IPv6",
+			service: getTestService("svc", v1.ProtocolTCP, map[string]string{
+				consts.ServiceAnnotationAllowedIPRanges: "2001:0db8:85a3::/64,2607:f0d0:1002:0051::/64",
+			}, true, 80),
+			existingSgs: map[string]network.SecurityGroup{"nsg": {
+				Name:                          pointer.String("nsg"),
+				SecurityGroupPropertiesFormat: &network.SecurityGroupPropertiesFormat{},
+			}},
+			lbIPs:  &[]string{"10.0.0.1", "10.0.0.2", "fd12:3456:789a:1::1"},
+			wantLb: true,
+			expectedSg: &network.SecurityGroup{
+				Name: pointer.String("nsg"),
+				SecurityGroupPropertiesFormat: &network.SecurityGroupPropertiesFormat{
+					SecurityRules: &[]network.SecurityRule{
+						{
+							Name: pointer.String("asvc-TCP-80-2001.db8.85a3.._64"),
+							SecurityRulePropertiesFormat: &network.SecurityRulePropertiesFormat{
+								Protocol:                 network.SecurityRuleProtocol("Tcp"),
+								SourcePortRange:          pointer.String("*"),
+								DestinationPortRange:     pointer.String(strconv.Itoa(80)),
+								SourceAddressPrefix:      pointer.String("2001:db8:85a3::/64"),
+								DestinationAddressPrefix: pointer.String("fd12:3456:789a:1::1"),
+								Access:                   network.SecurityRuleAccessAllow,
+								Priority:                 pointer.Int32(500),
+								Direction:                network.SecurityRuleDirection("Inbound"),
+							},
+						},
+						{
+							Name: pointer.String("asvc-TCP-80-2607.f0d0.1002.51.._64"),
+							SecurityRulePropertiesFormat: &network.SecurityRulePropertiesFormat{
+								Protocol:                 network.SecurityRuleProtocol("Tcp"),
+								SourcePortRange:          pointer.String("*"),
+								DestinationPortRange:     pointer.String(strconv.Itoa(80)),
+								SourceAddressPrefix:      pointer.String("2607:f0d0:1002:51::/64"),
+								DestinationAddressPrefix: pointer.String("fd12:3456:789a:1::1"),
+								Access:                   network.SecurityRuleAccessAllow,
+								Priority:                 pointer.Int32(501),
+								Direction:                network.SecurityRuleDirection("Inbound"),
+							},
+						},
+					},
+				},
+			},
+		},
+		{
+			desc: "reconcileSecurityGroup shall create sgs while allowedIPRanges annotation is set for both IPv4 and IPv6",
+			service: getTestServiceDualStack("svc", v1.ProtocolTCP, map[string]string{
+				consts.ServiceAnnotationAllowedIPRanges: "10.10.10.0/24,2607:f0d0:1002:0051::/64",
+			}, 80),
+			existingSgs: map[string]network.SecurityGroup{"nsg": {
+				Name:                          pointer.String("nsg"),
+				SecurityGroupPropertiesFormat: &network.SecurityGroupPropertiesFormat{},
+			}},
+			lbIPs:  &[]string{"10.0.0.1", "10.0.0.2", "fd12:3456:789a:1::1"},
+			wantLb: true,
+			expectedSg: &network.SecurityGroup{
+				Name: pointer.String("nsg"),
+				SecurityGroupPropertiesFormat: &network.SecurityGroupPropertiesFormat{
+					SecurityRules: &[]network.SecurityRule{
+						{
+							Name: pointer.String("asvc-TCP-80-10.10.10.0_24"),
+							SecurityRulePropertiesFormat: &network.SecurityRulePropertiesFormat{
+								Protocol:                   network.SecurityRuleProtocol("Tcp"),
+								SourcePortRange:            pointer.String("*"),
+								DestinationPortRange:       pointer.String(strconv.Itoa(80)),
+								SourceAddressPrefix:        pointer.String("10.10.10.0/24"),
+								DestinationAddressPrefixes: &([]string{"10.0.0.1", "10.0.0.2"}),
+								Access:                     network.SecurityRuleAccessAllow,
+								Priority:                   pointer.Int32(500),
+								Direction:                  network.SecurityRuleDirection("Inbound"),
+							},
+						},
+						{
+							Name: pointer.String("asvc-TCP-80-2607.f0d0.1002.51.._64-IPv6"),
+							SecurityRulePropertiesFormat: &network.SecurityRulePropertiesFormat{
+								Protocol:                 network.SecurityRuleProtocol("Tcp"),
+								SourcePortRange:          pointer.String("*"),
+								DestinationPortRange:     pointer.String(strconv.Itoa(80)),
+								SourceAddressPrefix:      pointer.String("2607:f0d0:1002:51::/64"),
+								DestinationAddressPrefix: pointer.String("fd12:3456:789a:1::1"),
+								Access:                   network.SecurityRuleAccessAllow,
+								Priority:                 pointer.Int32(501),
+								Direction:                network.SecurityRuleDirection("Inbound"),
+							},
+						},
+					},
+				},
+			},
+		},
+		{
+			desc: "reconcileSecurityGroup shall create sgs while allowedIPRanges and serviceTags annotation is set",
+			service: getTestService("svc", v1.ProtocolTCP, map[string]string{
+				consts.ServiceAnnotationAllowedIPRanges:    "10.10.10.0/24,192.168.0.1/32",
+				consts.ServiceAnnotationAllowedServiceTags: "foo,bar",
+			}, false, 80),
+			existingSgs: map[string]network.SecurityGroup{"nsg": {
+				Name:                          pointer.String("nsg"),
+				SecurityGroupPropertiesFormat: &network.SecurityGroupPropertiesFormat{},
+			}},
+			lbIPs:  &[]string{"10.0.0.1", "10.0.0.2"},
+			wantLb: true,
+			expectedSg: &network.SecurityGroup{
+				Name: pointer.String("nsg"),
+				SecurityGroupPropertiesFormat: &network.SecurityGroupPropertiesFormat{
+					SecurityRules: &[]network.SecurityRule{
+						{
+							Name: pointer.String("asvc-TCP-80-10.10.10.0_24"),
+							SecurityRulePropertiesFormat: &network.SecurityRulePropertiesFormat{
+								Protocol:                   network.SecurityRuleProtocol("Tcp"),
+								SourcePortRange:            pointer.String("*"),
+								DestinationPortRange:       pointer.String(strconv.Itoa(80)),
+								SourceAddressPrefix:        pointer.String("10.10.10.0/24"),
+								DestinationAddressPrefixes: &([]string{"10.0.0.1", "10.0.0.2"}),
+								Access:                     network.SecurityRuleAccessAllow,
+								Priority:                   pointer.Int32(500),
+								Direction:                  network.SecurityRuleDirection("Inbound"),
+							},
+						},
+						{
+							Name: pointer.String("asvc-TCP-80-192.168.0.1_32"),
+							SecurityRulePropertiesFormat: &network.SecurityRulePropertiesFormat{
+								Protocol:                   network.SecurityRuleProtocol("Tcp"),
+								SourcePortRange:            pointer.String("*"),
+								DestinationPortRange:       pointer.String(strconv.Itoa(80)),
+								SourceAddressPrefix:        pointer.String("192.168.0.1/32"),
+								DestinationAddressPrefixes: &([]string{"10.0.0.1", "10.0.0.2"}),
+								Access:                     network.SecurityRuleAccessAllow,
+								Priority:                   pointer.Int32(501),
+								Direction:                  network.SecurityRuleDirection("Inbound"),
+							},
+						},
+						{
+							Name: pointer.String("asvc-TCP-80-foo"),
+							SecurityRulePropertiesFormat: &network.SecurityRulePropertiesFormat{
+								Protocol:                   network.SecurityRuleProtocol("Tcp"),
+								SourcePortRange:            pointer.String("*"),
+								DestinationPortRange:       pointer.String(strconv.Itoa(80)),
+								SourceAddressPrefix:        pointer.String("foo"),
+								DestinationAddressPrefixes: &([]string{"10.0.0.1", "10.0.0.2"}),
+								Access:                     network.SecurityRuleAccessAllow,
+								Priority:                   pointer.Int32(502),
+								Direction:                  network.SecurityRuleDirection("Inbound"),
+							},
+						},
+						{
+							Name: pointer.String("asvc-TCP-80-bar"),
+							SecurityRulePropertiesFormat: &network.SecurityRulePropertiesFormat{
+								Protocol:                   network.SecurityRuleProtocol("Tcp"),
+								SourcePortRange:            pointer.String("*"),
+								DestinationPortRange:       pointer.String(strconv.Itoa(80)),
+								SourceAddressPrefix:        pointer.String("bar"),
+								DestinationAddressPrefixes: &([]string{"10.0.0.1", "10.0.0.2"}),
+								Access:                     network.SecurityRuleAccessAllow,
+								Priority:                   pointer.Int32(503),
+								Direction:                  network.SecurityRuleDirection("Inbound"),
+							},
+						},
+					},
+				},
+			},
+		},
+		{
+			desc: "reconcileSecurityGroup shall create/update/delete sgs while allowedIPRanges and serviceTags annotation is set",
+			service: getTestService("svc", v1.ProtocolTCP, map[string]string{
+				consts.ServiceAnnotationAllowedIPRanges:    "10.10.10.0/24,192.168.0.1/32",
+				consts.ServiceAnnotationAllowedServiceTags: "foo,bar",
+			}, false, 80),
+			existingSgs: map[string]network.SecurityGroup{"nsg": {
+				Name: pointer.String("nsg"),
+				SecurityGroupPropertiesFormat: &network.SecurityGroupPropertiesFormat{
+					SecurityRules: &[]network.SecurityRule{
+						{
+							// will be kept
+							Name: pointer.String("asvc-TCP-80-192.168.0.1_32"),
+							SecurityRulePropertiesFormat: &network.SecurityRulePropertiesFormat{
+								Protocol:                   network.SecurityRuleProtocol("Tcp"),
+								SourcePortRange:            pointer.String("*"),
+								DestinationPortRange:       pointer.String(strconv.Itoa(80)),
+								SourceAddressPrefix:        pointer.String("192.168.0.1/32"),
+								DestinationAddressPrefixes: &([]string{"10.0.0.1", "10.0.0.2"}),
+								Access:                     network.SecurityRuleAccessAllow,
+								Priority:                   pointer.Int32(500),
+								Direction:                  network.SecurityRuleDirection("Inbound"),
+							},
+						},
+						{
+							// will be removed: no longer in allowedIPRanges
+							Name: pointer.String("asvc-TCP-80-192.168.0.5_32"),
+							SecurityRulePropertiesFormat: &network.SecurityRulePropertiesFormat{
+								Protocol:                   network.SecurityRuleProtocol("Tcp"),
+								SourcePortRange:            pointer.String("*"),
+								DestinationPortRange:       pointer.String(strconv.Itoa(80)),
+								SourceAddressPrefix:        pointer.String("192.168.0.5/32"),
+								DestinationAddressPrefixes: &([]string{"10.0.0.1", "10.0.0.2"}),
+								Access:                     network.SecurityRuleAccessAllow,
+								Priority:                   pointer.Int32(501),
+								Direction:                  network.SecurityRuleDirection("Inbound"),
+							},
+						},
+					},
+				},
+			}},
+			lbIPs:  &[]string{"10.0.0.1", "10.0.0.2"},
+			wantLb: true,
+			expectedSg: &network.SecurityGroup{
+				Name: pointer.String("nsg"),
+				SecurityGroupPropertiesFormat: &network.SecurityGroupPropertiesFormat{
+					SecurityRules: &[]network.SecurityRule{
+						{
+							Name: pointer.String("asvc-TCP-80-192.168.0.1_32"),
+							SecurityRulePropertiesFormat: &network.SecurityRulePropertiesFormat{
+								Protocol:                   network.SecurityRuleProtocol("Tcp"),
+								SourcePortRange:            pointer.String("*"),
+								DestinationPortRange:       pointer.String(strconv.Itoa(80)),
+								SourceAddressPrefix:        pointer.String("192.168.0.1/32"),
+								DestinationAddressPrefixes: &([]string{"10.0.0.1", "10.0.0.2"}),
+								Access:                     network.SecurityRuleAccessAllow,
+								Priority:                   pointer.Int32(500),
+								Direction:                  network.SecurityRuleDirection("Inbound"),
+							},
+						},
+						{
+							Name: pointer.String("asvc-TCP-80-10.10.10.0_24"),
+							SecurityRulePropertiesFormat: &network.SecurityRulePropertiesFormat{
+								Protocol:                   network.SecurityRuleProtocol("Tcp"),
+								SourcePortRange:            pointer.String("*"),
+								DestinationPortRange:       pointer.String(strconv.Itoa(80)),
+								SourceAddressPrefix:        pointer.String("10.10.10.0/24"),
+								DestinationAddressPrefixes: &([]string{"10.0.0.1", "10.0.0.2"}),
+								Access:                     network.SecurityRuleAccessAllow,
+								Priority:                   pointer.Int32(501),
+								Direction:                  network.SecurityRuleDirection("Inbound"),
+							},
+						},
+						{
+							Name: pointer.String("asvc-TCP-80-foo"),
+							SecurityRulePropertiesFormat: &network.SecurityRulePropertiesFormat{
+								Protocol:                   network.SecurityRuleProtocol("Tcp"),
+								SourcePortRange:            pointer.String("*"),
+								DestinationPortRange:       pointer.String(strconv.Itoa(80)),
+								SourceAddressPrefix:        pointer.String("foo"),
+								DestinationAddressPrefixes: &([]string{"10.0.0.1", "10.0.0.2"}),
+								Access:                     network.SecurityRuleAccessAllow,
+								Priority:                   pointer.Int32(502),
+								Direction:                  network.SecurityRuleDirection("Inbound"),
+							},
+						},
+						{
+							Name: pointer.String("asvc-TCP-80-bar"),
+							SecurityRulePropertiesFormat: &network.SecurityRulePropertiesFormat{
+								Protocol:                   network.SecurityRuleProtocol("Tcp"),
+								SourcePortRange:            pointer.String("*"),
+								DestinationPortRange:       pointer.String(strconv.Itoa(80)),
+								SourceAddressPrefix:        pointer.String("bar"),
+								DestinationAddressPrefixes: &([]string{"10.0.0.1", "10.0.0.2"}),
+								Access:                     network.SecurityRuleAccessAllow,
+								Priority:                   pointer.Int32(503),
+								Direction:                  network.SecurityRuleDirection("Inbound"),
+							},
+						},
+					},
+				},
+			},
+		},
 	}
 
 	for _, test := range testCases {
@@ -4635,6 +4945,102 @@ func TestReconcileSecurityGroupLoadBalancerSourceRanges(t *testing.T) {
 	assert.Equal(t, expectedSg, *sg)
 }
 
+func TestReconcileSecurityGroup(t *testing.T) {
+	var (
+		clusterName = "testCluster"
+		lbIPs       = []string{"10.0.0.1", "10.0.0.2"}
+		lbName      = "lb-name"
+	)
+
+	t.Run("with spec.loadBalancerSourceRanges and IPRanges annotation specified", func(t *testing.T) {
+		var (
+			ctrl = gomock.NewController(t)
+			az   = GetTestCloud(ctrl)
+			svc  = v1.Service{
+				Spec: v1.ServiceSpec{
+					Type:                     v1.ServiceTypeLoadBalancer,
+					LoadBalancerSourceRanges: []string{"10.10.10.0/24"},
+					IPFamilies: []v1.IPFamily{
+						v1.IPv4Protocol,
+						v1.IPv6Protocol,
+					},
+					Ports: []v1.ServicePort{
+						{
+							Name:     "http",
+							Port:     80,
+							Protocol: v1.ProtocolTCP,
+						},
+					},
+				},
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test-svc",
+					Annotations: map[string]string{
+						consts.ServiceAnnotationAllowedIPRanges: "192.168.0.1/32",
+					},
+				},
+			}
+		)
+		defer ctrl.Finish()
+
+		mockSGClient := az.SecurityGroupsClient.(*mocksecuritygroupclient.MockInterface)
+		mockSGClient.EXPECT().Get(gomock.Any(), az.ResourceGroup, gomock.Any(), gomock.Any()).Return(network.SecurityGroup{
+			Name: pointer.String("nsg"),
+			SecurityGroupPropertiesFormat: &network.SecurityGroupPropertiesFormat{
+				SecurityRules: &[]network.SecurityRule{},
+			},
+		}, nil)
+		_, err := az.reconcileSecurityGroup(clusterName, &svc, &lbIPs, &lbName, true)
+		assert.Error(t, err)
+		assert.EqualError(t, err, fmt.Sprintf(
+			"both of spec.loadBalancerSourceRanges and annotation %s are specified for service %s, which is not allowed",
+			consts.ServiceAnnotationAllowedIPRanges, svc.Name,
+		))
+	})
+
+	t.Run("with spec.loadBalancerSourceRanges and ServiceTags annotation specified", func(t *testing.T) {
+		var (
+			ctrl = gomock.NewController(t)
+			az   = GetTestCloud(ctrl)
+			svc  = v1.Service{
+				Spec: v1.ServiceSpec{
+					Type:                     v1.ServiceTypeLoadBalancer,
+					LoadBalancerSourceRanges: []string{"10.10.10.0/24"},
+					IPFamilies: []v1.IPFamily{
+						v1.IPv4Protocol,
+						v1.IPv6Protocol,
+					},
+					Ports: []v1.ServicePort{
+						{
+							Name:     "http",
+							Port:     80,
+							Protocol: v1.ProtocolTCP,
+						},
+					},
+				},
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test-svc",
+					Annotations: map[string]string{
+						consts.ServiceAnnotationAllowedServiceTags: "foo,bar",
+					},
+				},
+			}
+		)
+		defer ctrl.Finish()
+
+		mockSGClient := az.SecurityGroupsClient.(*mocksecuritygroupclient.MockInterface)
+		mockSGClient.EXPECT().Get(gomock.Any(), az.ResourceGroup, gomock.Any(), gomock.Any()).Return(network.SecurityGroup{
+			Name: pointer.String("nsg"),
+			SecurityGroupPropertiesFormat: &network.SecurityGroupPropertiesFormat{
+				SecurityRules: &[]network.SecurityRule{},
+			},
+		}, nil)
+		mockSGClient.EXPECT().CreateOrUpdate(gomock.Any(), az.ResourceGroup, gomock.Any(), gomock.Any(), gomock.Any()).Return(nil)
+
+		_, err := az.reconcileSecurityGroup(clusterName, &svc, &lbIPs, &lbName, true)
+		assert.NoError(t, err)
+	})
+}
+
 func TestSafeDeletePublicIP(t *testing.T) {
 	ctrl := gomock.NewController(t)
 	defer ctrl.Finish()
diff --git a/pkg/provider/loadbalancer/accesscontrol_test.go b/pkg/provider/loadbalancer/accesscontrol_test.go
new file mode 100644
index 000000000..e21d564f8
--- /dev/null
+++ b/pkg/provider/loadbalancer/accesscontrol_test.go
@@ -0,0 +1,484 @@
+/*
+Copyright 2023 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package loadbalancer
+
+import (
+	"net/netip"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	v1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+
+	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
+)
+
+func TestIsInternal(t *testing.T) {
+	{
+		svc := v1.Service{
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{
+					consts.ServiceAnnotationLoadBalancerInternal: "true",
+				},
+			},
+		}
+		assert.True(t, IsInternal(&svc))
+	}
+	{
+		svc := v1.Service{
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{
+					consts.ServiceAnnotationLoadBalancerInternal: "TRUE",
+				},
+			},
+		}
+		assert.True(t, IsInternal(&svc))
+	}
+	{
+		svc := v1.Service{
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{
+					consts.ServiceAnnotationLoadBalancerInternal: "foobar",
+				},
+			},
+		}
+		assert.False(t, IsInternal(&svc))
+	}
+	{
+		svc := v1.Service{
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{},
+			},
+		}
+		assert.False(t, IsInternal(&svc))
+	}
+}
+
+func TestIsExternal(t *testing.T) {
+	{
+		svc := v1.Service{
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{
+					consts.ServiceAnnotationLoadBalancerInternal: "true",
+				},
+			},
+		}
+		assert.False(t, IsExternal(&svc))
+	}
+	{
+		svc := v1.Service{
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{
+					consts.ServiceAnnotationLoadBalancerInternal: "TRUE",
+				},
+			},
+		}
+		assert.False(t, IsExternal(&svc))
+	}
+	{
+		svc := v1.Service{
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{
+					consts.ServiceAnnotationLoadBalancerInternal: "foobar",
+				},
+			},
+		}
+		assert.True(t, IsExternal(&svc))
+	}
+	{
+		svc := v1.Service{
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{},
+			},
+		}
+		assert.True(t, IsExternal(&svc))
+	}
+}
+
+func TestAllowedServiceTags(t *testing.T) {
+	t.Run("no annotation", func(t *testing.T) {
+		actual, err := AllowedServiceTags(&v1.Service{
+			Spec: v1.ServiceSpec{
+				Type: v1.ServiceTypeLoadBalancer,
+			},
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{},
+			},
+		})
+		assert.NoError(t, err)
+		assert.Empty(t, actual)
+	})
+	t.Run("with 1 service tag", func(t *testing.T) {
+		actual, err := AllowedServiceTags(&v1.Service{
+			Spec: v1.ServiceSpec{
+				Type: v1.ServiceTypeLoadBalancer,
+			},
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{
+					consts.ServiceAnnotationAllowedServiceTags: "Microsoft.ContainerInstance/containerGroups",
+				},
+			},
+		})
+		assert.NoError(t, err)
+		assert.Equal(t, []string{"Microsoft.ContainerInstance/containerGroups"}, actual)
+	})
+	t.Run("with multiple service tags", func(t *testing.T) {
+		actual, err := AllowedServiceTags(&v1.Service{
+			Spec: v1.ServiceSpec{
+				Type: v1.ServiceTypeLoadBalancer,
+			},
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{
+					consts.ServiceAnnotationAllowedServiceTags: "Microsoft.ContainerInstance/containerGroups,foo,bar",
+				},
+			},
+		})
+		assert.NoError(t, err)
+		assert.Equal(t, []string{"Microsoft.ContainerInstance/containerGroups", "foo", "bar"}, actual)
+	})
+}
+
+func TestAllowedIPRanges(t *testing.T) {
+	t.Run("no annotation", func(t *testing.T) {
+		actual, err := AllowedIPRanges(&v1.Service{
+			Spec: v1.ServiceSpec{
+				Type: v1.ServiceTypeLoadBalancer,
+			},
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{},
+			},
+		})
+		assert.NoError(t, err)
+		assert.Empty(t, actual)
+	})
+	t.Run("with 1 IPv4 range", func(t *testing.T) {
+		actual, err := AllowedIPRanges(&v1.Service{
+			Spec: v1.ServiceSpec{
+				Type: v1.ServiceTypeLoadBalancer,
+			},
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{
+					consts.ServiceAnnotationAllowedIPRanges: "10.10.0.0/24",
+				},
+			},
+		})
+		assert.NoError(t, err)
+		assert.Equal(t, []netip.Prefix{netip.MustParsePrefix("10.10.0.0/24")}, actual)
+	})
+	t.Run("with 1 IPv6 range", func(t *testing.T) {
+		actual, err := AllowedIPRanges(&v1.Service{
+			Spec: v1.ServiceSpec{
+				Type: v1.ServiceTypeLoadBalancer,
+			},
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{
+					consts.ServiceAnnotationAllowedIPRanges: "2001:db8::/32",
+				},
+			},
+		})
+		assert.NoError(t, err)
+		assert.Equal(t, []netip.Prefix{netip.MustParsePrefix("2001:db8::/32")}, actual)
+	})
+	t.Run("with multiple IP ranges", func(t *testing.T) {
+		actual, err := AllowedIPRanges(&v1.Service{
+			Spec: v1.ServiceSpec{
+				Type: v1.ServiceTypeLoadBalancer,
+			},
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{
+					consts.ServiceAnnotationAllowedIPRanges: "10.10.0.0/24,2001:db8::/32",
+				},
+			},
+		})
+		assert.NoError(t, err)
+		assert.Equal(t, []netip.Prefix{
+			netip.MustParsePrefix("10.10.0.0/24"),
+			netip.MustParsePrefix("2001:db8::/32"),
+		}, actual)
+	})
+	t.Run("with invalid IP range", func(t *testing.T) {
+		_, err := AllowedIPRanges(&v1.Service{
+			Spec: v1.ServiceSpec{
+				Type: v1.ServiceTypeLoadBalancer,
+			},
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{
+					consts.ServiceAnnotationAllowedIPRanges: "foobar",
+				},
+			},
+		})
+		assert.Error(t, err)
+	})
+}
+
+func TestSourceRanges(t *testing.T) {
+	t.Run("not specified in spec", func(t *testing.T) {
+		actual, err := SourceRanges(&v1.Service{
+			Spec: v1.ServiceSpec{
+				Type: v1.ServiceTypeLoadBalancer,
+			},
+		})
+		assert.NoError(t, err)
+		assert.Empty(t, actual)
+	})
+	t.Run("specified in spec", func(t *testing.T) {
+		actual, err := SourceRanges(&v1.Service{
+			Spec: v1.ServiceSpec{
+				Type:                     v1.ServiceTypeLoadBalancer,
+				LoadBalancerSourceRanges: []string{"10.10.0.0/24", "2001:db8::/32"},
+			},
+		})
+		assert.NoError(t, err)
+		assert.Equal(t, []netip.Prefix{
+			netip.MustParsePrefix("10.10.0.0/24"),
+			netip.MustParsePrefix("2001:db8::/32"),
+		}, actual)
+	})
+	t.Run("specified in annotation", func(t *testing.T) {
+		actual, err := SourceRanges(&v1.Service{
+			Spec: v1.ServiceSpec{
+				Type: v1.ServiceTypeLoadBalancer,
+			},
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{
+					v1.AnnotationLoadBalancerSourceRangesKey: "10.10.0.0/24,2001:db8::/32",
+				},
+			},
+		})
+		assert.NoError(t, err)
+		assert.Equal(t, []netip.Prefix{
+			netip.MustParsePrefix("10.10.0.0/24"),
+			netip.MustParsePrefix("2001:db8::/32"),
+		}, actual)
+	})
+	t.Run("specified in both spec and annotation", func(t *testing.T) {
+		actual, err := SourceRanges(&v1.Service{
+			Spec: v1.ServiceSpec{
+				Type:                     v1.ServiceTypeLoadBalancer,
+				LoadBalancerSourceRanges: []string{"10.10.0.0/24"},
+			},
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{
+					v1.AnnotationLoadBalancerSourceRangesKey: "2001:db8::/32",
+				},
+			},
+		})
+		assert.NoError(t, err)
+		assert.Equal(t, []netip.Prefix{
+			netip.MustParsePrefix("10.10.0.0/24"),
+		}, actual, "spec should take precedence over annotation")
+	})
+	t.Run("with invalid IP range", func(t *testing.T) {
+		_, err := SourceRanges(&v1.Service{
+			Spec: v1.ServiceSpec{
+				Type:                     v1.ServiceTypeLoadBalancer,
+				LoadBalancerSourceRanges: []string{"foobar"},
+			},
+		})
+		assert.Error(t, err)
+	})
+}
+
+func TestAccessControl_IsAllowFromInternet(t *testing.T) {
+	t.Run("external LB", func(t *testing.T) {
+		svc := v1.Service{
+			Spec: v1.ServiceSpec{
+				Type: v1.ServiceTypeLoadBalancer,
+			},
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{},
+			},
+		}
+		t.Run("default", func(t *testing.T) {
+			ac, err := NewAccessControl(&svc)
+			assert.NoError(t, err)
+			assert.True(t, ac.IsAllowFromInternet())
+		})
+		t.Run("not allowed from all", func(t *testing.T) {
+			svc.Spec.LoadBalancerSourceRanges = []string{"10.10.10.0/24"}
+			ac, err := NewAccessControl(&svc)
+			assert.NoError(t, err)
+			assert.False(t, ac.IsAllowFromInternet())
+		})
+		t.Run("allowed from all", func(t *testing.T) {
+			svc.Spec.LoadBalancerSourceRanges = []string{"0.0.0.0/0"}
+			ac, err := NewAccessControl(&svc)
+			assert.NoError(t, err)
+			assert.True(t, ac.IsAllowFromInternet())
+		})
+	})
+
+	t.Run("internal LB", func(t *testing.T) {
+		svc := v1.Service{
+			Spec: v1.ServiceSpec{
+				Type: v1.ServiceTypeLoadBalancer,
+			},
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{
+					consts.ServiceAnnotationLoadBalancerInternal: "true",
+				},
+			},
+		}
+		t.Run("default", func(t *testing.T) {
+			ac, err := NewAccessControl(&svc)
+			assert.NoError(t, err)
+			assert.False(t, ac.IsAllowFromInternet())
+		})
+		t.Run("not allowed from all", func(t *testing.T) {
+			svc.Spec.LoadBalancerSourceRanges = []string{"10.10.10.0/24"}
+			ac, err := NewAccessControl(&svc)
+			assert.NoError(t, err)
+			assert.False(t, ac.IsAllowFromInternet())
+		})
+		t.Run("allowed from all", func(t *testing.T) {
+			svc.Spec.LoadBalancerSourceRanges = []string{"0.0.0.0/0"}
+			ac, err := NewAccessControl(&svc)
+			assert.NoError(t, err)
+			assert.True(t, ac.IsAllowFromInternet())
+		})
+	})
+}
+
+func TestAccessControl_IPV4Sources(t *testing.T) {
+	t.Run("external LB", func(t *testing.T) {
+		svc := v1.Service{
+			Spec: v1.ServiceSpec{
+				Type: v1.ServiceTypeLoadBalancer,
+			},
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{
+					consts.ServiceAnnotationAllowedIPRanges:    "10.10.10.0/24,192.168.0.1/32,2001:db8::/32,2002:db8::/32",
+					consts.ServiceAnnotationAllowedServiceTags: "foo,bar",
+				},
+			},
+		}
+		ac, err := NewAccessControl(&svc)
+		assert.NoError(t, err)
+		assert.Equal(t, []string{
+			"10.10.10.0/24",
+			"192.168.0.1/32",
+			"foo",
+			"bar",
+		}, ac.IPV4Sources())
+	})
+	t.Run("internal LB with Internet access", func(t *testing.T) {
+		svc := v1.Service{
+			Spec: v1.ServiceSpec{
+				Type: v1.ServiceTypeLoadBalancer,
+			},
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{
+					consts.ServiceAnnotationLoadBalancerInternal: "true",
+					consts.ServiceAnnotationAllowedIPRanges:      "0.0.0.0/0",
+				},
+			},
+		}
+		ac, err := NewAccessControl(&svc)
+		assert.NoError(t, err)
+		assert.Equal(t, []string{
+			"Internet",
+			"0.0.0.0/0",
+		}, ac.IPV4Sources())
+	})
+	t.Run("internal LB without Internet access", func(t *testing.T) {
+		svc := v1.Service{
+			Spec: v1.ServiceSpec{
+				Type: v1.ServiceTypeLoadBalancer,
+			},
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{
+					consts.ServiceAnnotationLoadBalancerInternal: "true",
+					consts.ServiceAnnotationAllowedIPRanges:      "10.10.10.0/24,192.168.0.1/32,2001:db8::/32,2002:db8::/32",
+					consts.ServiceAnnotationAllowedServiceTags:   "foo,bar",
+				},
+			},
+		}
+		ac, err := NewAccessControl(&svc)
+		assert.NoError(t, err)
+		assert.Equal(t, []string{
+			"10.10.10.0/24",
+			"192.168.0.1/32",
+			"foo",
+			"bar",
+		}, ac.IPV4Sources())
+	})
+}
+
+func TestAccessControl_IPV6Sources(t *testing.T) {
+	t.Run("external LB", func(t *testing.T) {
+		svc := v1.Service{
+			Spec: v1.ServiceSpec{
+				Type: v1.ServiceTypeLoadBalancer,
+			},
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{
+					consts.ServiceAnnotationAllowedIPRanges:    "10.10.10.0/24,192.168.0.1/32,2001:db8::/32,2002:db8::/32",
+					consts.ServiceAnnotationAllowedServiceTags: "foo,bar",
+				},
+			},
+		}
+		ac, err := NewAccessControl(&svc)
+		assert.NoError(t, err)
+		assert.Equal(t, []string{
+			"2001:db8::/32",
+			"2002:db8::/32",
+			"foo",
+			"bar",
+		}, ac.IPV6Sources())
+	})
+	t.Run("internal LB with Internet access", func(t *testing.T) {
+		svc := v1.Service{
+			Spec: v1.ServiceSpec{
+				Type: v1.ServiceTypeLoadBalancer,
+			},
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{
+					consts.ServiceAnnotationLoadBalancerInternal: "true",
+					consts.ServiceAnnotationAllowedIPRanges:      "::/0",
+				},
+			},
+		}
+		ac, err := NewAccessControl(&svc)
+		assert.NoError(t, err)
+		assert.Equal(t, []string{
+			"Internet",
+			"::/0",
+		}, ac.IPV6Sources())
+	})
+	t.Run("internal LB without Internet access", func(t *testing.T) {
+		svc := v1.Service{
+			Spec: v1.ServiceSpec{
+				Type: v1.ServiceTypeLoadBalancer,
+			},
+			ObjectMeta: metav1.ObjectMeta{
+				Annotations: map[string]string{
+					consts.ServiceAnnotationLoadBalancerInternal: "true",
+					consts.ServiceAnnotationAllowedIPRanges:      "10.10.10.0/24,192.168.0.1/32,2001:db8::/32,2002:db8::/32",
+					consts.ServiceAnnotationAllowedServiceTags:   "foo,bar",
+				},
+			},
+		}
+		ac, err := NewAccessControl(&svc)
+		assert.NoError(t, err)
+		assert.Equal(t, []string{
+			"2001:db8::/32",
+			"2002:db8::/32",
+			"foo",
+			"bar",
+		}, ac.IPV6Sources())
+	})
+}
diff --git a/pkg/provider/loadbalancer/netip_test.go b/pkg/provider/loadbalancer/netip_test.go
new file mode 100644
index 000000000..c989eb277
--- /dev/null
+++ b/pkg/provider/loadbalancer/netip_test.go
@@ -0,0 +1,96 @@
+/*
+Copyright 2023 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package loadbalancer
+
+import (
+	"net/netip"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestIsAllowAll(t *testing.T) {
+	assert.False(t, IsCIDRsAllowAll([]netip.Prefix{}))
+	assert.True(t, IsCIDRsAllowAll([]netip.Prefix{
+		netip.MustParsePrefix(IPv4AllowedAll),
+	}))
+	assert.True(t, IsCIDRsAllowAll([]netip.Prefix{
+		netip.MustParsePrefix(IPv6AllowedAll),
+	}))
+	assert.True(t, IsCIDRsAllowAll([]netip.Prefix{
+		netip.MustParsePrefix("1.1.1.1/32"),
+		netip.MustParsePrefix(IPv4AllowedAll),
+	}))
+	assert.True(t, IsCIDRsAllowAll([]netip.Prefix{
+		netip.MustParsePrefix("1.1.1.1/32"),
+		netip.MustParsePrefix(IPv6AllowedAll),
+	}))
+	assert.False(t, IsCIDRsAllowAll([]netip.Prefix{
+		netip.MustParsePrefix("1.1.1.1/32"),
+	}))
+}
+
+func TestParseCIDRs(t *testing.T) {
+	t.Run("empty", func(t *testing.T) {
+		actual, err := ParseCIDRs([]string{})
+		assert.NoError(t, err)
+		assert.Empty(t, actual)
+	})
+	t.Run("1 ipv4 cidr", func(t *testing.T) {
+		actual, err := ParseCIDRs([]string{
+			"10.10.10.0/24",
+		})
+		assert.NoError(t, err)
+		assert.Equal(t, []netip.Prefix{
+			netip.MustParsePrefix("10.10.10.0/24"),
+		}, actual)
+	})
+	t.Run("1 ipv6 cidr", func(t *testing.T) {
+		actual, err := ParseCIDRs([]string{
+			"2001:db8::/32",
+		})
+		assert.NoError(t, err)
+		assert.Equal(t, []netip.Prefix{
+			netip.MustParsePrefix("2001:db8::/32"),
+		}, actual)
+	})
+	t.Run("multiple cidrs", func(t *testing.T) {
+		actual, err := ParseCIDRs([]string{
+			"10.10.10.0/24",
+			"2001:db8::/32",
+		})
+		assert.NoError(t, err)
+		assert.Equal(t, []netip.Prefix{
+			netip.MustParsePrefix("10.10.10.0/24"),
+			netip.MustParsePrefix("2001:db8::/32"),
+		}, actual)
+	})
+	t.Run("invalid cidr", func(t *testing.T) {
+		{
+			_, err := ParseCIDRs([]string{""})
+			assert.Error(t, err)
+		}
+		{
+			_, err := ParseCIDRs([]string{"foo"})
+			assert.Error(t, err)
+		}
+		{
+			_, err := ParseCIDRs([]string{"10.10.10.0/24", "foo"})
+			assert.Error(t, err)
+		}
+	})
+}
diff --git a/tests/e2e/network/network_security_group.go b/tests/e2e/network/network_security_group.go
index 297b52b0e..ee560fc4a 100644
--- a/tests/e2e/network/network_security_group.go
+++ b/tests/e2e/network/network_security_group.go
@@ -195,7 +195,7 @@ var _ = Describe("Network security group", Label(utils.TestSuiteLabelNSG), func(
 	It("can set source IP prefixes automatically according to corresponding service tag", func() {
 		By("Creating service and wait it to expose")
 		annotation := map[string]string{
-			consts.ServiceAnnotationAllowedServiceTag: "AzureCloud",
+			consts.ServiceAnnotationAllowedServiceTags: "AzureCloud",
 		}
 		utils.Logf("Creating service " + serviceName + " in namespace " + ns.Name)
 		service := utils.CreateLoadBalancerServiceManifest(serviceName, annotation, labels, ns.Name, ports)
@@ -379,6 +379,74 @@ var _ = Describe("Network security group", Label(utils.TestSuiteLabelNSG), func(
 			}
 		}
 	})
+
+	It("should support service annotation `service.beta.kubernetes.io/azure-allowed-ip-ranges`", func() {
+
+		// isSuperSet returns true if set1 is a superset of set2.
+		isSuperSet := func(set1, set2 []string) bool {
+			s1 := map[string]bool{}
+			for _, s := range set1 {
+				s1[s] = true
+			}
+			for _, s := range set2 {
+				if !s1[s] {
+					return false
+				}
+			}
+			return true
+		}
+
+		allowedIPRanges := []string{
+			"10.20.0.0/16",
+			"192.168.0.1/32",
+		}
+
+		ipFamilyPolicy := v1.IPFamilyPolicyPreferDualStack
+		svc := v1.Service{
+			Spec: v1.ServiceSpec{
+				Ports:          ports,
+				Type:           v1.ServiceTypeLoadBalancer,
+				Selector:       labels,
+				IPFamilyPolicy: &ipFamilyPolicy,
+			},
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      serviceName,
+				Namespace: ns.Name,
+				Annotations: map[string]string{
+					consts.ServiceAnnotationAllowedIPRanges: strings.Join(allowedIPRanges, ","),
+				},
+				Labels: labels,
+			},
+		}
+
+		By("Creating load balancer service with allowed IP ranges")
+		_, err := cs.CoreV1().Services(ns.Name).Create(context.Background(), &svc, metav1.CreateOptions{})
+		Expect(err).NotTo(HaveOccurred())
+
+		By("Waiting for the service to be exposed")
+		_, err = utils.WaitServiceExposure(cs, ns.Name, serviceName, []*string{})
+		Expect(err).NotTo(HaveOccurred())
+
+		By("Validating if the corresponding IP prefix existing in nsg")
+		nsgs, err := tc.GetClusterSecurityGroups()
+		Expect(err).NotTo(HaveOccurred())
+
+		var sources []string
+
+		for _, nsg := range nsgs {
+
+			rules := nsg.Properties.SecurityRules
+			if rules == nil {
+				continue
+			}
+			for _, rule := range rules {
+				if rule.Properties.SourceAddressPrefix != nil {
+					sources = append(sources, *rule.Properties.SourceAddressPrefix)
+				}
+			}
+		}
+		Expect(isSuperSet(sources, allowedIPRanges)).To(BeTrue(), "Expected %v to be a superset of %v", sources, allowedIPRanges)
+	})
 })
 
 func validateUnsharedSecurityRuleExists(nsgs []*aznetwork.SecurityGroup, ips []*string, port string) bool {