Repository: kubernetes-sigs/cloud-provider-azure
Pull request #7343 (2024-10-16 23:55:18)

[release-1.29] feat: Lock updates on azure resources when other component is doing t…
…he same thing.

This PR utilizes a lease in each service reconciliation to prevent race conditions where cloud provider and others are updating the same azure resources.

<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide/first-contribution.md#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?

<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature
/kind design

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->

/kind feature

#### What this PR does / why we need it:

This PR utilizes a lease in each service reconciliation to prevent race conditions where cloud provider and others are updating the same azure resources. If the lease has not expired yet and is held by other component, the service reconciliation will be aborted and retried exponentially.

#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes #

#### Special notes for your reviewer:

#### Does this PR introduce a user-facing change?
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://git.k8s.io/community/contributors/guide/release-notes.md
-->
```release-note
feat: Lock updates on azure resources when other component is doing the same thing.

This PR utilizes a lease in each service reconciliation to prevent race conditions where cloud provider and others are updating the same azure resources.
```

#### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

<!--
This section can be blank if this pull request does not require a release note.

When adding links which point to resources within git repositories, like
KEPs or supporting documentation, please reference a specific commit and avoid
linking directly to the master branch. This ensures that links reference a
specific point in time, rather than a document that may change over time.

See here for guidance on getting permanent links to files: https://help.github.com/en/articles/getting-permanent-links-to-files

Please use the following format for linking documentation:
- [KEP]: <link>
- [Usage]: <link>
- [Other doc]: <link>
-->
```docs

```


k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/7343#" title="Author self-approved">nilo19</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/release-1.29/OWNERS)~~ [nilo19]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
k8s-ci-robot: @nilo19: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz-1-29 | 1d836a666df00b3c7f657ef1fcb937b717e04f12 | [link](https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/7343/pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz-1-29/1846759194254905344) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-multi-slb-capz-1-29`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=7343). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Anilo19). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->

Patch:
diff --git a/Dockerfile b/Dockerfile
index 80e961757..5123d1b8e 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -14,7 +14,7 @@
 
 # syntax=docker/dockerfile:1
 
-FROM --platform=linux/amd64 golang:1.23-bullseye@sha256:ebaf58d74f200c27dce0181203a1347e41b5c159c9aa0517a95bec594aabbde4 AS builder
+FROM --platform=linux/amd64 golang:1.23-bullseye@sha256:48ac5022f9740543cac0eba41a4e37b721073a0103349e416678e0142a53b49a AS builder
 
 ARG ENABLE_GIT_COMMAND=true
 ARG ARCH=amd64
diff --git a/cloud-node-manager.Dockerfile b/cloud-node-manager.Dockerfile
index 67e390a07..061a03240 100644
--- a/cloud-node-manager.Dockerfile
+++ b/cloud-node-manager.Dockerfile
@@ -14,7 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM --platform=linux/amd64 golang:1.23-bullseye@sha256:ebaf58d74f200c27dce0181203a1347e41b5c159c9aa0517a95bec594aabbde4 AS builder
+FROM --platform=linux/amd64 golang:1.23-bullseye@sha256:48ac5022f9740543cac0eba41a4e37b721073a0103349e416678e0142a53b49a AS builder
 
 ARG ENABLE_GIT_COMMAND=true
 ARG ARCH=amd64
diff --git a/cmd/cloud-controller-manager/app/controllermanager.go b/cmd/cloud-controller-manager/app/controllermanager.go
index 07b2ba65e..7bc8016dd 100644
--- a/cmd/cloud-controller-manager/app/controllermanager.go
+++ b/cmd/cloud-controller-manager/app/controllermanager.go
@@ -350,7 +350,7 @@ func Run(ctx context.Context, c *cloudcontrollerconfig.CompletedConfig, h *contr
 	)
 
 	if c.ComponentConfig.KubeCloudShared.CloudProvider.CloudConfigFile != "" {
-		cloud, err = provider.NewCloudFromConfigFile(ctx, c.ComponentConfig.KubeCloudShared.CloudProvider.CloudConfigFile, true)
+		cloud, err = provider.NewCloudFromConfigFile(ctx, c.ClientBuilder, c.ComponentConfig.KubeCloudShared.CloudProvider.CloudConfigFile, true)
 		if err != nil {
 			klog.Fatalf("Cloud provider azure could not be initialized: %v", err)
 		}
diff --git a/go.sum b/go.sum
index 7eb9a90ab..fc6ee863d 100644
--- a/go.sum
+++ b/go.sum
@@ -1,3 +1,5 @@
+cel.dev/expr v0.15.0/go.mod h1:TRSuuV7DlVCE/uwv5QbAiW/v8l5O8C4eEPHeu7gf7Sg=
+cloud.google.com/go/compute/metadata v0.3.0/go.mod h1:zFmK7XCadkQkj6TtorcaGlCW1hT1fIilQDwofLpJ20k=
 github.com/Azure/azure-kusto-go v0.16.1 h1:vCBWcQghmC1qIErUUgVNWHxGhZVStu1U/hki6iBA14k=
 github.com/Azure/azure-kusto-go v0.16.1/go.mod h1:9F2zvXH8B6eWzgI1S4k1ZXAIufnBZ1bv1cW1kB1n3D0=
 github.com/Azure/azure-pipeline-go v0.1.8/go.mod h1:XA1kFWRVhSK+KNFiOhfv83Fv8L9achrP7OxIzeTn1Yg=
@@ -79,8 +81,12 @@ github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 h1:XHOnouVk1mx
 github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2/go.mod h1:wP83P5OoQ5p6ip3ScPr0BAq0BvuPAvacpEuSzyouqAI=
 github.com/NYTimes/gziphandler v1.1.1 h1:ZUDjpQae29j0ryrS0u/B8HZfJBtBQHjqw2rQ2cqUQ3I=
 github.com/NYTimes/gziphandler v1.1.1/go.mod h1:n/CVRwUEOgIxrgPvAQhUUr9oeUtvrhMomdKFjzJNB0c=
+github.com/alecthomas/kingpin/v2 v2.4.0/go.mod h1:0gyi0zQnjuFk8xrkNKamJoyUo382HRL7ATRpFZCw6tE=
+github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137/go.mod h1:OMCwj8VM1Kc9e19TLln2VL61YJF0x1XFtfdL4JdbSyE=
+github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY=
 github.com/antlr/antlr4/runtime/Go/antlr/v4 v4.0.0-20230305170008-8188dc5388df h1:7RFfzj4SSt6nnvCPbCqijJi1nWCd+TqAT3bYCStRC18=
 github.com/antlr/antlr4/runtime/Go/antlr/v4 v4.0.0-20230305170008-8188dc5388df/go.mod h1:pSwJ0fSY5KhvocuWSx4fz3BA8OrA1bQn+K1Eli3BRwM=
+github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs=
 github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a h1:idn718Q4B6AGu/h5Sxe66HYVdqdGu2l9Iebqhi/AEoA=
 github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
 github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
@@ -89,8 +95,12 @@ github.com/blang/semver/v4 v4.0.0 h1:1PFHFE6yCCTv8C1TeyNNarDzntLi7wMI5i/pzqYIsAM
 github.com/blang/semver/v4 v4.0.0/go.mod h1:IbckMUScFkM3pff0VJDNKRiT6TG/YpiHIM2yvyW5YoQ=
 github.com/cenkalti/backoff/v4 v4.3.0 h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK343L8=
 github.com/cenkalti/backoff/v4 v4.3.0/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE=
+github.com/census-instrumentation/opencensus-proto v0.4.1/go.mod h1:4T9NM4+4Vw91VeyqjLS6ao50K5bOcLKN6Q42XnYaRYw=
 github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
 github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
+github.com/chzyer/readline v1.5.1/go.mod h1:Eh+b79XXUwfKfcPLepksvw2tcLE/Ct21YObkaSkeBlk=
+github.com/cncf/xds/go v0.0.0-20240423153145-555b57ec207b/go.mod h1:W+zGtBO5Y1IgJhy4+A9GOqVhqLpfZi+vwmdNXUehLA8=
+github.com/coreos/go-oidc v2.2.1+incompatible/go.mod h1:CgnwVTmzoESiwO9qyAFEMiHoZ1nMCKZlZ9V6mm3/LKc=
 github.com/coreos/go-semver v0.3.1 h1:yi21YpKnrx1gt5R+la8n5WgS0kCrsPp33dmEyHReZr4=
 github.com/coreos/go-semver v0.3.1/go.mod h1:irMmmIw/7yzSRPWryHsK7EYSg09caPQL03VsM8rvUec=
 github.com/coreos/go-systemd/v22 v22.5.0 h1:RrqgGjYQKalulkV8NGVIfkXQf6YYmOyiJKk8iXXhfZs=
@@ -111,12 +121,17 @@ github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkp
 github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
 github.com/emicklei/go-restful/v3 v3.11.0 h1:rAQeMHw1c7zTmncogyy8VvRZwtkmkZ4FxERmMY4rD+g=
 github.com/emicklei/go-restful/v3 v3.11.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc=
+github.com/envoyproxy/go-control-plane v0.12.0/go.mod h1:ZBTaoJ23lqITozF0M6G4/IragXCQKCnYbmlmtHvwRG0=
+github.com/envoyproxy/protoc-gen-validate v1.0.4/go.mod h1:qys6tmnRsYrQqIhm2bvKZH4Blx/1gTIZ2UKVY1M+Yew=
 github.com/evanphx/json-patch v5.9.0+incompatible h1:fBXyNpNMuTTDdquAq/uisOr2lShz4oaXpDTX2bLe7ls=
 github.com/evanphx/json-patch v5.9.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk=
 github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
 github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
 github.com/fsnotify/fsnotify v1.7.0 h1:8JEhPFa5W2WU7YfeZzPNqzMP6Lwt7L2715Ggo0nosvA=
 github.com/fsnotify/fsnotify v1.7.0/go.mod h1:40Bi/Hjc2AVfZrqy+aj+yEI+/bRxZnMJyTJwOpGvigM=
+github.com/fxamacker/cbor/v2 v2.7.0/go.mod h1:pxXPTn3joSm21Gbwsv0w9OSA2y1HFR9qXEeXQVeNoDQ=
+github.com/go-kit/log v0.2.1/go.mod h1:NwTd00d/i8cPZ3xOwwiv2PO5MOcx78fFErGNcVmBjv0=
+github.com/go-logfmt/logfmt v0.5.1/go.mod h1:WYhtIu8zTZfxdn5+rREduYbwxfcBr/Vr6KEVveWlfTs=
 github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
 github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
 github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
@@ -131,6 +146,7 @@ github.com/go-openapi/jsonreference v0.20.2/go.mod h1:Bl1zwGIM8/wsvqjsOQLJ/SH+En
 github.com/go-openapi/swag v0.22.3/go.mod h1:UzaqsxGiab7freDnrUUra0MwWfN/q7tE4j+VcZ0yl14=
 github.com/go-openapi/swag v0.22.4 h1:QLMzNJnMGPRNDCbySlcj1x01tzU8/9LTTL9hZZZogBU=
 github.com/go-openapi/swag v0.22.4/go.mod h1:UzaqsxGiab7freDnrUUra0MwWfN/q7tE4j+VcZ0yl14=
+github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572/go.mod h1:9Pwr4B2jHnOSGXyyzV8ROjYa2ojvAY6HCGYYfMoC3Ls=
 github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI=
 github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8=
 github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
@@ -143,6 +159,7 @@ github.com/golang-jwt/jwt/v4 v4.5.0 h1:7cYmW1XlMY7h7ii7UhUyChSgS5wUJEnm9uZVTGqOW
 github.com/golang-jwt/jwt/v4 v4.5.0/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
 github.com/golang-jwt/jwt/v5 v5.2.1 h1:OuVbFODueb089Lh128TAcimifWaLhJwVflnrgM17wHk=
 github.com/golang-jwt/jwt/v5 v5.2.1/go.mod h1:pqrtFR0X4osieyHYxtmOUWsAWrfe1Q5UVIyoH402zdk=
+github.com/golang/glog v1.2.1/go.mod h1:6AhwSGph0fcJtXVM/PEHPqZlFeoLxhs7/t5UDAwmO+w=
 github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da h1:oI5xCqsCo564l8iNU+DwB5epxmsaqB+rhGL0m5jtYqE=
 github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
 github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
@@ -165,6 +182,7 @@ github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
 github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/gorilla/websocket v1.5.0 h1:PPwGk2jz7EePpoHN/+ClbZu8SPxiqlu12wZP/3sWmnc=
 github.com/gorilla/websocket v1.5.0/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
+github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA=
 github.com/grpc-ecosystem/go-grpc-middleware v1.3.0 h1:+9834+KizmvFV7pXQGSXQTsaWhq2GjuNUt0aUU0YBYw=
 github.com/grpc-ecosystem/go-grpc-middleware v1.3.0/go.mod h1:z0ButlSOZa5vEBq9m2m2hlwIgKw+rp3sdCBRoJY+30Y=
 github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 h1:Ovs26xHkKqVztRpIrF/92BcuyuQ/YW4NSIpoGtfXNho=
@@ -173,6 +191,7 @@ github.com/grpc-ecosystem/grpc-gateway v1.16.0 h1:gmcG1KaJ57LophUzW0Hy8NmPhnMZb4
 github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFbcEtjT1g+wF4CSlocrBnw=
 github.com/grpc-ecosystem/grpc-gateway/v2 v2.22.0 h1:asbCHRVmodnJTuQ3qamDwqVOIjwqUPTYmYuemVOx+Ys=
 github.com/grpc-ecosystem/grpc-gateway/v2 v2.22.0/go.mod h1:ggCgvZ2r7uOoQjOyu2Y1NhHmEPPzzuhWgcza5M1Ji1I=
+github.com/ianlancetaylor/demangle v0.0.0-20240312041847-bd984b5ce465/go.mod h1:gx7rwoVhcfuVKG5uya9Hs3Sxj7EIvldVofAWIUtGouw=
 github.com/imdario/mergo v0.3.6 h1:xTNEAn+kxVO7dTZGu0CegyqKZmoWFI0rF8UxjlB2d28=
 github.com/imdario/mergo v0.3.6/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
 github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
@@ -181,8 +200,10 @@ github.com/jonboulle/clockwork v0.2.2 h1:UOGuzwb1PwsrDAObMuhUnj0p5ULPj8V/xJ7Kx9q
 github.com/jonboulle/clockwork v0.2.2/go.mod h1:Pkfl5aHPm1nk2H9h0bjmnJD/BcgbGXUBGnn1kMkgxc8=
 github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY=
 github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
+github.com/jpillora/backoff v1.0.0/go.mod h1:J/6gKK9jxlEcS3zixgDgUAsiuZ7yrSoa/FX5e0EB2j4=
 github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
 github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
+github.com/julienschmidt/httprouter v1.3.0/go.mod h1:JR6WtHb+2LUe8TCKY3cZOxFyyO8IZAc4RVcycCCAKdM=
 github.com/keybase/go-keychain v0.0.0-20231219164618-57a3676c3af6 h1:IsMZxCuZqKuao2vNdfD82fjjgPLfyHLpR41Z88viRWs=
 github.com/keybase/go-keychain v0.0.0-20231219164618-57a3676c3af6/go.mod h1:3VeWNIJaW+O5xpRQbPp0Ybqu1vJd/pm7s2F473HRrkw=
 github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
@@ -203,6 +224,8 @@ github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJ
 github.com/mattn/go-ieproxy v0.0.1/go.mod h1:pYabZ6IHcRpFh7vIaLfK7rdcWgFEb3SFJ6/gNWuh88E=
 github.com/mattn/go-ieproxy v0.0.11 h1:MQ/5BuGSgDAHZOJe6YY80IF2UVCfGkwfo6AeD7HtHYo=
 github.com/mattn/go-ieproxy v0.0.11/go.mod h1:/NsJd+kxZBmjMc5hrJCKMbP57B84rvq9BiDRbtO9AS0=
+github.com/matttproud/golang_protobuf_extensions v1.0.4/go.mod h1:BSXmuO+STAnVfrANrmjBb36TMTDstsz7MSK+HVaYKv4=
+github.com/moby/spdystream v0.2.0/go.mod h1:f7i0iNDQJ059oMTcWxx8MA/zKFIuD/lY+0GqbN2Wy8c=
 github.com/moby/term v0.0.0-20221205130635-1aeaba878587 h1:HfkjXDfhgVaN5rmueG8cL8KKeFNecRCXFhaJ2qZ5SKA=
 github.com/moby/term v0.0.0-20221205130635-1aeaba878587/go.mod h1:8FzsFHVUBGZdbDsJw/ot+X+d5HLUbvklYLJ9uGfcI3Y=
 github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
@@ -210,12 +233,16 @@ github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w
 github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
 github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
 github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
+github.com/montanaflynn/stats v0.7.0/go.mod h1:etXPPgVO6n31NxCd9KQUMvCM+ve0ruNzt6R8Bnaayow=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
+github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
+github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+o7JKHSa8/e818NopupXU1YMK5fe1lsApnBw=
 github.com/onsi/ginkgo/v2 v2.20.2 h1:7NVCeyIWROIAheY21RLS+3j2bb52W0W82tkberYytp4=
 github.com/onsi/ginkgo/v2 v2.20.2/go.mod h1:K9gyxPIlb+aIvnZ8bd9Ak+YP18w3APlR+5coaZoE2ag=
 github.com/onsi/gomega v1.34.2 h1:pNCwDkzrsv7MS9kpaQvVb1aVLahQXyJ/Tv5oAZMI3i8=
 github.com/onsi/gomega v1.34.2/go.mod h1:v1xfxRgk0KIsG+QOdm7p8UosrOzPYRo60fd3B/1Dukc=
+github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU=
 github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c h1:+mdjkGKdHQG3305AYmdv1U2eRNDiU2ErMBj1gwrq8eQ=
 github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c/go.mod h1:7rwL4CYBLnjLxUqIJNnCWiEdr3bn6IUYi15bNlnbCCU=
 github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
@@ -223,6 +250,7 @@ github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINE
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
 github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/pquerna/cachecontrol v0.1.0/go.mod h1:NrUG3Z7Rdu85UNR3vm7SOsl1nFIeSiQnrHV5K9mBcUI=
 github.com/prometheus/client_golang v1.20.5 h1:cxppBPuYhUnsO6yo/aoRol4L7q7UFfdm+bR9r+8l63Y=
 github.com/prometheus/client_golang v1.20.5/go.mod h1:PIEt8X02hGcP8JWbeHyeZ53Y/jReSnHgO035n//V5WE=
 github.com/prometheus/client_model v0.6.1 h1:ZKSh/rekM+n3CeS952MLRAdFwIKqeY8b62p8ais2e9E=
@@ -233,6 +261,7 @@ github.com/prometheus/procfs v0.15.1 h1:YagwOFzUgYfKKHX6Dr+sHT7km/hxC76UB0leargg
 github.com/prometheus/procfs v0.15.1/go.mod h1:fB45yRUv8NstnjriLhBQLuOUt+WW4BsoGhij/e3PBqk=
 github.com/redis/go-redis/v9 v9.6.1 h1:HHDteefn6ZkTtY5fGUE8tj8uy85AHk6zP7CpzIAM0y4=
 github.com/redis/go-redis/v9 v9.6.1/go.mod h1:0C0c6ycQsdpVNQpxb1njEQIqkx5UcsM8FJCQLgE9+RA=
+github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
 github.com/rogpeppe/go-internal v1.12.0 h1:exVL4IDcn6na9z1rAb56Vxr+CgyK3nn3O+epU5NdKM8=
 github.com/rogpeppe/go-internal v1.12.0/go.mod h1:E+RYuTGaKKdloAfM02xzb0FW3Paa99yedzYV+kq4uf4=
 github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
@@ -253,6 +282,7 @@ github.com/stoewer/go-strcase v1.2.0/go.mod h1:IBiWB2sKIp3wVVQ3Y035++gc+knqhUQag
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
 github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
+github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
 github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
 github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
 github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
@@ -265,6 +295,8 @@ github.com/tj/assert v0.0.3 h1:Df/BlaZ20mq6kuai7f5z2TvPFiwC3xaWJSDQNiIS3Rk=
 github.com/tj/assert v0.0.3/go.mod h1:Ne6X72Q+TB1AteidzQncjw9PabbMp4PBMZ1k+vd1Pvk=
 github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 h1:6fotK7otjonDflCTK0BCfls4SPy3NcCVb5dqqmbRknE=
 github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75/go.mod h1:KO6IkyS8Y3j8OdNO85qEYBsRPuteD+YciPomcXdrMnk=
+github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg=
+github.com/xhit/go-str2duration/v2 v2.1.0/go.mod h1:ohY8p+0f07DiV6Em5LKB0s2YpLtXVyJfNt1+BlmyAsU=
 github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2 h1:eY9dn8+vbi4tKz5Qo6v2eYzo7kUS51QINcR5jNpbZS8=
 github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
 github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
@@ -308,6 +340,8 @@ go.opentelemetry.io/otel/trace v1.31.0 h1:ffjsj1aRouKewfr85U2aGagJ46+MvodynlQ1HY
 go.opentelemetry.io/otel/trace v1.31.0/go.mod h1:TXZkRk7SM2ZQLtR6eoAWQFIHPvzQ06FJAsO1tJg480A=
 go.opentelemetry.io/proto/otlp v1.3.1 h1:TrMUixzpM0yuc/znrFTP9MMRh8trP93mkCiDVeXrui0=
 go.opentelemetry.io/proto/otlp v1.3.1/go.mod h1:0X1WI4de4ZsLrrJNLAQbFeLCm3T7yBkR0XqQ7niQU+8=
+go.uber.org/atomic v1.7.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
+go.uber.org/atomic v1.10.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
 go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
 go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
 go.uber.org/mock v0.4.0 h1:VcM4ZOtdbR4f6VXfiOpwpVJDL6lCReaZ6mw31wqh7KU=
@@ -331,6 +365,7 @@ golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
 golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
 golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
 golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
+golang.org/x/mod v0.20.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
 golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
 golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20191112182307-2180aed22343/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
@@ -368,6 +403,7 @@ golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.15.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
 golang.org/x/sys v0.26.0 h1:KHjCJyddX0LoSTb3J+vWpupP9p0oznkqVk/IfjymZbo=
 golang.org/x/sys v0.26.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
+golang.org/x/telemetry v0.0.0-20240521205824-bda55230c457/go.mod h1:pRgIJT+bRLFKnoM1ldnzKoxTIn14Yxz928LQRYYgIN0=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
@@ -399,6 +435,8 @@ golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8T
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8=
+google.golang.org/appengine v1.6.7/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
 google.golang.org/genproto v0.0.0-20230822172742-b8732ec3820d h1:VBu5YqKPv6XiJ199exd8Br+Aetz+o08F+PLMnwJQHAY=
 google.golang.org/genproto v0.0.0-20230822172742-b8732ec3820d/go.mod h1:yZTlhN0tQnXo3h00fuXNCxJdLdIdnVFVBaRJ5LWBbw4=
 google.golang.org/genproto/googleapis/api v0.0.0-20240822170219-fc7c04adadcd h1:BBOTEWLuuEGQy9n1y9MhVJ9Qt0BDu21X8qZs71/uPZo=
@@ -415,10 +453,12 @@ gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntN
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
 gopkg.in/dnaeon/go-vcr.v3 v3.2.0 h1:Rltp0Vf+Aq0u4rQXgmXgtgoRDStTnFN83cWgSGSoRzM=
 gopkg.in/dnaeon/go-vcr.v3 v3.2.0/go.mod h1:2IMOnnlx9I6u9x+YBsM3tAMx6AlOxnJ0pWxQAzZ79Ag=
+gopkg.in/evanphx/json-patch.v4 v4.12.0/go.mod h1:p8EYWUEYMpynmqDbY58zCKCFZw8pRWMG4EsWvDvM72M=
 gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc=
 gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
 gopkg.in/natefinch/lumberjack.v2 v2.2.1 h1:bBRl1b0OH9s/DuPhuXpNl+VtCaJXFZ5/uEFST95x9zc=
 gopkg.in/natefinch/lumberjack.v2 v2.2.1/go.mod h1:YD8tP3GAjkrDg1eZH7EGmyESg/lsYskCTPBJVb9jqSc=
+gopkg.in/square/go-jose.v2 v2.6.0/go.mod h1:M9dMgbHiYLoDGQrXy7OpJDJWiKiU//h+vD76mk0e1AI=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
@@ -442,6 +482,8 @@ k8s.io/component-helpers v0.29.9 h1:+3dLb8nHWPeNwrL6whkpvKv46o90EW5O6Aju/hLQW70=
 k8s.io/component-helpers v0.29.9/go.mod h1:sUtoToDUdZDku0Kj4i3dRWyb1jmZLxXO2cvWvD27A40=
 k8s.io/controller-manager v0.29.9 h1:hKGqtTf7IDmN5csLVgf6VUXPJqosajAnMsqw2jCDh6s=
 k8s.io/controller-manager v0.29.9/go.mod h1:OZVIb2E9QFoSaiaVt4DFOsH/kZpCaEgFSYr/mCF9zuk=
+k8s.io/cri-api v0.29.9/go.mod h1:A6pdbjzML2xi9B0Clqn5qt1HJ3Ik12x2j+jv/TkqjRE=
+k8s.io/gengo/v2 v2.0.0-20240228010128-51d4e06bde70/go.mod h1:VH3AT8AaQOqiGjMF9p0/IM1Dj+82ZwjfxUP1IxaHE+8=
 k8s.io/klog/v2 v2.130.1 h1:n9Xl7H1Xvksem4KFG4PYbdQCQxqc/tTUyrgXaOhHSzk=
 k8s.io/klog/v2 v2.130.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE=
 k8s.io/kms v0.29.9 h1:7B9VGhFAzNfBmfXeQ0Xi+OzIcNPw/xazPnLm/J3vNT4=
diff --git a/pkg/consts/consts.go b/pkg/consts/consts.go
index c2c14e520..df83dd72c 100644
--- a/pkg/consts/consts.go
+++ b/pkg/consts/consts.go
@@ -590,3 +590,16 @@ const (
 	VMPowerStateDeallocating = "deallocating"
 	VMPowerStateUnknown      = "unknown"
 )
+
+// Azure resource lock
+const (
+	AzureResourceLockHolderNameCloudControllerManager = "cloud-controller-manager"
+	AzureResourceLockLeaseName                        = "aks-managed-resource-locker"
+	AzureResourceLockLeaseNamespace                   = "kube-system"
+	AzureResourceLockLeaseDuration                    = int32(15 * 60)
+	AzureResourceLockPreviousHolderNameAnnotation     = "aks-managed-resource-locker-previous-holder"
+
+	AzureResourceLockFailedToLockErrorTemplate                = "%s failed due to fail to lock azure resources. This may because another component is trying to update azure resources, e.g., load balancers. This will be automatically retried by cloud provider exponentially: %w"
+	AzureResourceLockFailedToUnlockErrorTemplate              = "%s failed due to fail to unlock azure resources. This will be automatically retried by cloud provider exponentially, but can also be manually unlocked by removing the holder of the lease '%s/%s'. Before doing this, please be aware that this could lead to unexpected issues: %w"
+	AzureResourceLockFailedToReconcileWithUnlockErrorTemplate = "%s failed due to %s, and when unlocking azure resources another error happened: %w"
+)
diff --git a/pkg/node/node.go b/pkg/node/node.go
index fe5a2a549..59d7a6c8a 100644
--- a/pkg/node/node.go
+++ b/pkg/node/node.go
@@ -34,7 +34,7 @@ type IMDSNodeProvider struct {
 
 // NewIMDSNodeProvider creates a new IMDSNodeProvider.
 func NewIMDSNodeProvider(ctx context.Context) *IMDSNodeProvider {
-	az, err := azureprovider.NewCloud(ctx, &azureprovider.Config{
+	az, err := azureprovider.NewCloud(ctx, nil, &azureprovider.Config{
 		UseInstanceMetadata: true,
 		VMType:              "vmss",
 	}, false)
diff --git a/pkg/node/nodearm.go b/pkg/node/nodearm.go
index 95e26bbf1..9c4f3306b 100644
--- a/pkg/node/nodearm.go
+++ b/pkg/node/nodearm.go
@@ -37,7 +37,7 @@ type ARMNodeProvider struct {
 func NewARMNodeProvider(ctx context.Context, cloudConfigFilePath string) *ARMNodeProvider {
 	var err error
 	var az cloudprovider.Interface
-	az, err = azureprovider.NewCloudFromConfigFile(ctx, cloudConfigFilePath, false)
+	az, err = azureprovider.NewCloudFromConfigFile(ctx, nil, cloudConfigFilePath, false)
 	if err != nil {
 		klog.Fatalf("Failed to initialize Azure cloud provider: %v", err)
 	}
diff --git a/pkg/provider/azure.go b/pkg/provider/azure.go
index b27709c2f..cca9023b5 100644
--- a/pkg/provider/azure.go
+++ b/pkg/provider/azure.go
@@ -455,10 +455,12 @@ type Cloud struct {
 	multipleStandardLoadBalancersActiveNodesLock    sync.Mutex
 	localServiceNameToServiceInfoMap                sync.Map
 	endpointSlicesCache                             sync.Map
+
+	azureResourceLocker *AzureResourceLocker
 }
 
 // NewCloud returns a Cloud with initialized clients
-func NewCloud(ctx context.Context, config *Config, callFromCCM bool) (cloudprovider.Interface, error) {
+func NewCloud(ctx context.Context, clientBuilder cloudprovider.ControllerClientBuilder, config *Config, callFromCCM bool) (cloudprovider.Interface, error) {
 	az := &Cloud{
 		nodeNames:                  utilsets.NewString(),
 		nodeZones:                  map[string]*utilsets.IgnoreCaseSet{},
@@ -477,10 +479,22 @@ func NewCloud(ctx context.Context, config *Config, callFromCCM bool) (cloudprovi
 
 	az.ipv6DualStackEnabled = true
 	az.lockMap = newLockMap()
+
+	if clientBuilder != nil {
+		az.KubeClient = clientBuilder.ClientOrDie("azure-cloud-provider")
+	}
+	az.azureResourceLocker = NewAzureResourceLocker(
+		az,
+		consts.AzureResourceLockHolderNameCloudControllerManager,
+		consts.AzureResourceLockLeaseName,
+		consts.AzureResourceLockLeaseNamespace,
+		consts.AzureResourceLockLeaseDuration,
+	)
+
 	return az, nil
 }
 
-func NewCloudFromConfigFile(ctx context.Context, configFilePath string, calFromCCM bool) (cloudprovider.Interface, error) {
+func NewCloudFromConfigFile(ctx context.Context, clientBuilder cloudprovider.ControllerClientBuilder, configFilePath string, calFromCCM bool) (cloudprovider.Interface, error) {
 	var (
 		cloud cloudprovider.Interface
 		err   error
@@ -501,8 +515,7 @@ func NewCloudFromConfigFile(ctx context.Context, configFilePath string, calFromC
 			klog.Fatalf("Failed to parse Azure cloud provider config: %v", err)
 		}
 	}
-	cloud, err = NewCloud(ctx, configValue, calFromCCM && configFilePath != "")
-
+	cloud, err = NewCloud(ctx, clientBuilder, configValue, calFromCCM && configFilePath != "")
 	if err != nil {
 		return nil, fmt.Errorf("could not init cloud provider azure: %w", err)
 	}
@@ -525,7 +538,7 @@ func NewCloudFromSecret(ctx context.Context, clientBuilder cloudprovider.Control
 	if err != nil {
 		return nil, fmt.Errorf("NewCloudFromSecret: failed to get config from secret %s/%s: %w", secretNamespace, secretName, err)
 	}
-	az, err := NewCloud(ctx, config, true)
+	az, err := NewCloud(ctx, clientBuilder, config, true)
 	if err != nil {
 		return nil, fmt.Errorf("NewCloudFromSecret: failed to initialize cloud from secret %s/%s: %w", secretNamespace, secretName, err)
 	}
@@ -953,7 +966,8 @@ func (az *Cloud) setCloudProviderBackoffDefaults(config *Config) wait.Backoff {
 func (az *Cloud) configAzureClients(
 	servicePrincipalToken *adal.ServicePrincipalToken,
 	multiTenantOAuthTokenProvider adal.MultitenantOAuthTokenProvider,
-	networkResourceServicePrincipalToken adal.OAuthTokenProvider) {
+	networkResourceServicePrincipalToken adal.OAuthTokenProvider,
+) {
 	azClientConfig := az.getAzureClientConfig(servicePrincipalToken)
 
 	// Prepare AzureClientConfig for all azure clients
@@ -1529,5 +1543,4 @@ func (az *Cloud) GetNodeVMSet(nodeName types.NodeName, crt azcache.AzureCacheRea
 
 	// 5. Node is managed by vmss
 	return ss, nil
-
 }
diff --git a/pkg/provider/azure_loadbalancer.go b/pkg/provider/azure_loadbalancer.go
index ad0983649..595120f22 100644
--- a/pkg/provider/azure_loadbalancer.go
+++ b/pkg/provider/azure_loadbalancer.go
@@ -202,13 +202,12 @@ func (az *Cloud) reconcileService(ctx context.Context, clusterName string, servi
 // load balancer is not ready yet (e.g., it is still being provisioned) and
 // polling at a fixed rate is preferred over backing off exponentially in
 // order to minimize latency.
-func (az *Cloud) EnsureLoadBalancer(ctx context.Context, clusterName string, service *v1.Service, nodes []*v1.Node) (*v1.LoadBalancerStatus, error) {
+func (az *Cloud) EnsureLoadBalancer(ctx context.Context, clusterName string, service *v1.Service, nodes []*v1.Node) (lbStatus *v1.LoadBalancerStatus, err error) {
 	// When a client updates the internal load balancer annotation,
 	// the service may be switched from an internal LB to a public one, or vice versa.
 	// Here we'll firstly ensure service do not lie in the opposite LB.
 	const Operation = "EnsureLoadBalancer"
 
-	var err error
 	ctx, span := trace.BeginReconcile(ctx, trace.DefaultTracer(), Operation, attributes.FeatureOfService(service)...)
 	defer func() { span.Observe(ctx, err) }()
 
@@ -223,6 +222,41 @@ func (az *Cloud) EnsureLoadBalancer(ctx context.Context, clusterName string, ser
 		isOperationSucceeded = false
 	)
 
+	if az.azureResourceLocker != nil {
+		err = az.azureResourceLocker.Lock(ctx)
+		if err != nil {
+			logger.Error(err, "failed to lock azure resources")
+			return nil, fmt.Errorf(
+				consts.AzureResourceLockFailedToLockErrorTemplate,
+				"EnsureLoadBalancer",
+				err,
+			)
+		}
+
+		defer func() {
+			unlockErr := az.azureResourceLocker.Unlock(ctx)
+			if unlockErr != nil {
+				unlockErr = fmt.Errorf(
+					consts.AzureResourceLockFailedToUnlockErrorTemplate,
+					"EnsureLoadBalancer",
+					consts.AzureResourceLockLeaseNamespace,
+					consts.AzureResourceLockLeaseName,
+					unlockErr,
+				)
+			}
+			if err == nil {
+				err = unlockErr
+			} else if unlockErr != nil {
+				err = fmt.Errorf(
+					consts.AzureResourceLockFailedToReconcileWithUnlockErrorTemplate,
+					"EnsureLoadBalancer",
+					err,
+					unlockErr,
+				)
+			}
+		}()
+	}
+
 	logger.V(5).Info("Starting", "service-spec", log.ValueAsMap(service))
 
 	defer func() {
@@ -234,7 +268,7 @@ func (az *Cloud) EnsureLoadBalancer(ctx context.Context, clusterName string, ser
 		}
 	}()
 
-	lbStatus, err := az.reconcileService(ctx, clusterName, service, nodes)
+	lbStatus, err = az.reconcileService(ctx, clusterName, service, nodes)
 	if err != nil {
 		return nil, err
 	}
@@ -315,6 +349,41 @@ func (az *Cloud) UpdateLoadBalancer(ctx context.Context, clusterName string, ser
 		return nil
 	}
 
+	if az.azureResourceLocker != nil {
+		err = az.azureResourceLocker.Lock(ctx)
+		if err != nil {
+			logger.Error(err, "failed to lock azure resources")
+			return fmt.Errorf(
+				consts.AzureResourceLockFailedToLockErrorTemplate,
+				"UpdateLoadBalancer",
+				err,
+			)
+		}
+
+		defer func() {
+			unlockErr := az.azureResourceLocker.Unlock(ctx)
+			if unlockErr != nil {
+				unlockErr = fmt.Errorf(
+					consts.AzureResourceLockFailedToUnlockErrorTemplate,
+					"UpdateLoadBalancer",
+					consts.AzureResourceLockLeaseNamespace,
+					consts.AzureResourceLockLeaseName,
+					unlockErr,
+				)
+			}
+			if err == nil {
+				err = unlockErr
+			} else if unlockErr != nil {
+				err = fmt.Errorf(
+					consts.AzureResourceLockFailedToReconcileWithUnlockErrorTemplate,
+					"UpdateLoadBalancer",
+					err,
+					unlockErr,
+				)
+			}
+		}()
+	}
+
 	_, err = az.reconcileService(ctx, clusterName, service, nodes)
 	if err != nil {
 		return err
@@ -332,10 +401,9 @@ func (az *Cloud) UpdateLoadBalancer(ctx context.Context, clusterName string, ser
 // doesn't exist even if some part of it is still laying around.
 // Implementations must treat the *v1.Service parameter as read-only and not modify it.
 // Parameter 'clusterName' is the name of the cluster as presented to kube-controller-manager
-func (az *Cloud) EnsureLoadBalancerDeleted(ctx context.Context, clusterName string, service *v1.Service) error {
+func (az *Cloud) EnsureLoadBalancerDeleted(ctx context.Context, clusterName string, service *v1.Service) (err error) {
 	const Operation = "EnsureLoadBalancerDeleted"
 
-	var err error
 	ctx, span := trace.BeginReconcile(ctx, trace.DefaultTracer(), Operation, attributes.FeatureOfService(service)...)
 	defer func() { span.Observe(ctx, err) }()
 
@@ -350,6 +418,40 @@ func (az *Cloud) EnsureLoadBalancerDeleted(ctx context.Context, clusterName stri
 		isOperationSucceeded = false
 	)
 	ctx = log.NewContext(ctx, logger)
+	if az.azureResourceLocker != nil {
+		err = az.azureResourceLocker.Lock(ctx)
+		if err != nil {
+			logger.Error(err, "failed to lock azure resources")
+			return fmt.Errorf(
+				consts.AzureResourceLockFailedToLockErrorTemplate,
+				"EnsureLoadBalancerDeleted",
+				err,
+			)
+		}
+
+		defer func() {
+			unlockErr := az.azureResourceLocker.Unlock(ctx)
+			if unlockErr != nil {
+				unlockErr = fmt.Errorf(
+					consts.AzureResourceLockFailedToUnlockErrorTemplate,
+					"EnsureLoadBalancerDeleted",
+					consts.AzureResourceLockLeaseNamespace,
+					consts.AzureResourceLockLeaseName,
+					unlockErr,
+				)
+			}
+			if err == nil {
+				err = unlockErr
+			} else if unlockErr != nil {
+				err = fmt.Errorf(
+					consts.AzureResourceLockFailedToReconcileWithUnlockErrorTemplate,
+					"EnsureLoadBalancerDeleted",
+					err,
+					unlockErr,
+				)
+			}
+		}()
+	}
 
 	logger.V(5).Info("Starting", "service-spec", log.ValueAsMap(service))
 	defer func() {
@@ -2209,8 +2311,7 @@ func (az *Cloud) recordExistingNodesOnLoadBalancers(clusterName string, lbs *[]n
 					if strings.EqualFold(trimSuffixIgnoreCase(
 						lbName, consts.InternalLoadBalancerNameSuffix,
 					), multiSLBConfig.Name) {
-						az.MultipleStandardLoadBalancerConfigurations[i].ActiveNodes =
-							utilsets.SafeInsert(multiSLBConfig.ActiveNodes, nodeNames...)
+						az.MultipleStandardLoadBalancerConfigurations[i].ActiveNodes = utilsets.SafeInsert(multiSLBConfig.ActiveNodes, nodeNames...)
 					}
 				}
 			}
@@ -2335,7 +2436,8 @@ func (az *Cloud) reconcileFrontendIPConfigs(clusterName string,
 	lb *network.LoadBalancer,
 	status *v1.LoadBalancerStatus,
 	wantLb bool,
-	lbFrontendIPConfigNames map[bool]string) ([]*network.FrontendIPConfiguration, []network.FrontendIPConfiguration, bool, error) {
+	lbFrontendIPConfigNames map[bool]string,
+) ([]*network.FrontendIPConfiguration, []network.FrontendIPConfiguration, bool, error) {
 	var err error
 	lbName := *lb.Name
 	serviceName := getServiceName(service)
@@ -2541,7 +2643,8 @@ func (az *Cloud) getFrontendZones(
 	fipConfig *network.FrontendIPConfiguration,
 	previousZone *[]string,
 	isFipChanged bool,
-	serviceName, lbFrontendIPConfigName string) error {
+	serviceName, lbFrontendIPConfigName string,
+) error {
 	if !isFipChanged { // fetch zone information from API for new frontends
 		// only add zone information for new internal frontend IP configurations for standard load balancer not deployed to an edge zone.
 		location := az.Location
@@ -2673,8 +2776,8 @@ func (az *Cloud) getExpectedLBRules(
 	lbFrontendIPConfigID string,
 	lbBackendPoolID string,
 	lbName string,
-	isIPv6 bool) ([]network.Probe, []network.LoadBalancingRule, error) {
-
+	isIPv6 bool,
+) ([]network.Probe, []network.LoadBalancingRule, error) {
 	var expectedRules []network.LoadBalancingRule
 	var expectedProbes []network.Probe
 
@@ -2723,7 +2826,7 @@ func (az *Cloud) getExpectedLBRules(
 		if err != nil {
 			return nil, nil, fmt.Errorf("error generate lb rule for ha mod loadbalancer. err: %w", err)
 		}
-		//Here we need to find one health probe rule for the HA lb rule.
+		// Here we need to find one health probe rule for the HA lb rule.
 		if nodeEndpointHealthprobe == nil {
 			// use user customized health probe rule if any
 			for _, port := range service.Spec.Ports {
@@ -2731,7 +2834,7 @@ func (az *Cloud) getExpectedLBRules(
 				if err != nil {
 					klog.V(2).ErrorS(err, "error occurred when buildHealthProbeRulesForPort", "service", service.Name, "namespace", service.Namespace,
 						"rule-name", lbRuleName, "port", port.Port)
-					//ignore error because we only need one correct rule
+					// ignore error because we only need one correct rule
 				}
 				if portprobe != nil {
 					props.Probe = &network.SubResource{
@@ -2828,7 +2931,8 @@ func (az *Cloud) getExpectedLBRules(
 func (az *Cloud) getExpectedLoadBalancingRulePropertiesForPort(
 	service *v1.Service,
 	lbFrontendIPConfigID string,
-	lbBackendPoolID string, servicePort v1.ServicePort, transportProto network.TransportProtocol) (*network.LoadBalancingRulePropertiesFormat, error) {
+	lbBackendPoolID string, servicePort v1.ServicePort, transportProto network.TransportProtocol,
+) (*network.LoadBalancingRulePropertiesFormat, error) {
 	var err error
 
 	loadDistribution := network.LoadDistributionDefault
@@ -2884,7 +2988,8 @@ func (az *Cloud) getExpectedLoadBalancingRulePropertiesForPort(
 func (az *Cloud) getExpectedHAModeLoadBalancingRuleProperties(
 	service *v1.Service,
 	lbFrontendIPConfigID string,
-	lbBackendPoolID string) (*network.LoadBalancingRulePropertiesFormat, error) {
+	lbBackendPoolID string,
+) (*network.LoadBalancingRulePropertiesFormat, error) {
 	props, err := az.getExpectedLoadBalancingRulePropertiesForPort(service, lbFrontendIPConfigID, lbBackendPoolID, v1.ServicePort{}, network.TransportProtocolAll)
 	if err != nil {
 		return nil, fmt.Errorf("error generate lb rule for ha mod loadbalancer. err: %w", err)
diff --git a/pkg/provider/azure_lock.go b/pkg/provider/azure_lock.go
new file mode 100644
index 000000000..12a325584
--- /dev/null
+++ b/pkg/provider/azure_lock.go
@@ -0,0 +1,244 @@
+/*
+Copyright 2024 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package provider
+
+import (
+	"context"
+	"errors"
+	"strings"
+	"time"
+
+	v1 "k8s.io/api/coordination/v1"
+	k8sapierrors "k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/client-go/kubernetes"
+	"k8s.io/klog/v2"
+	"k8s.io/utils/ptr"
+
+	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
+)
+
+type AzureResourceLocker struct {
+	*Cloud
+	holder               string
+	leaseName            string
+	leaseNamespace       string
+	leaseDurationSeconds int32
+}
+
+func NewAzureResourceLocker(
+	cloud *Cloud,
+	holder, leaseName, leaseNamespace string,
+	leaseDurationSeconds int32,
+) *AzureResourceLocker {
+	return &AzureResourceLocker{
+		Cloud:                cloud,
+		holder:               holder,
+		leaseName:            leaseName,
+		leaseNamespace:       leaseNamespace,
+		leaseDurationSeconds: leaseDurationSeconds,
+	}
+}
+
+// Lock creates a lease if it does not exist and acquires the lease.
+// If the lease has not expired yet and is held by another holder, it will return an error.
+func (l *AzureResourceLocker) Lock(ctx context.Context) error {
+	if err := createLeaseIfNotExists(
+		ctx, l.leaseNamespace, l.leaseName, l.leaseDurationSeconds, l.KubeClient,
+	); err != nil {
+		return err
+	}
+	if err := l.acquireLease(
+		ctx, l.KubeClient, l.holder, l.leaseNamespace, l.leaseName); err != nil {
+		return err
+	}
+
+	return nil
+}
+
+// Unlock releases the lease if needed.
+func (l *AzureResourceLocker) Unlock(ctx context.Context) error {
+	if err := releaseLease(
+		ctx, l.KubeClient, l.leaseNamespace, l.leaseName, l.holder,
+	); err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func createLeaseIfNotExists(
+	ctx context.Context,
+	leaseNamespace, leaseName string,
+	leaseDurationSeconds int32,
+	clientset kubernetes.Interface,
+) error {
+	logger := klog.Background().WithName("createLeaseIfNotExists").
+		WithValues("leaseNamespace", leaseNamespace, "leaseName", leaseName,
+			"leaseDurationSeconds", leaseDurationSeconds).V(4)
+
+	lease := &v1.Lease{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      leaseName,
+			Namespace: leaseNamespace,
+		},
+		Spec: v1.LeaseSpec{
+			LeaseDurationSeconds: ptr.To(leaseDurationSeconds),
+		},
+	}
+
+	_, err := clientset.CoordinationV1().Leases(leaseNamespace).Get(ctx, leaseName, metav1.GetOptions{})
+	if err != nil {
+		if !k8sapierrors.IsNotFound(err) {
+			logger.Error(err, "failed to get lease")
+			return err
+		}
+		// Lease does not exist, create it
+		_, err = clientset.CoordinationV1().Leases(leaseNamespace).Create(ctx, lease, metav1.CreateOptions{})
+		if err != nil {
+			logger.Error(err, "failed to create lease")
+			return err
+		}
+		logger.Info("Successfully created lease.")
+	} else {
+		logger.Info("Lease already exists.")
+	}
+
+	return nil
+}
+
+func (l *AzureResourceLocker) acquireLease(
+	ctx context.Context,
+	clientset kubernetes.Interface,
+	holder, leaseNamespace, leaseName string,
+) error {
+	logger := klog.Background().WithName("acquireLease").
+		WithValues("holder", holder, "leaseNamespace", leaseNamespace, "leaseName", leaseName).V(4)
+
+	lease, err := clientset.CoordinationV1().Leases(leaseNamespace).Get(ctx, leaseName, metav1.GetOptions{})
+	if err != nil {
+		logger.Error(err, "failed to get lease")
+		return err
+	}
+
+	if !hasExpired(lease) {
+		// If the lease has not expired, but the previous holder is the same,
+		// it means the same previous operation failed and the releaseLease
+		// also failed. In this case we should continue acquiring the lease.
+		// This should be a rare case, otherwise we should not acquire the lease.
+		prevHolder := ptr.Deref(lease.Spec.HolderIdentity, "")
+		if !strings.EqualFold(prevHolder, holder) {
+			errMsg := "Lease has not expired yet, another component such as aks rp may be processing another request. This would be automatically recovered after the lease expires."
+			err := errors.New(errMsg)
+			logger.Error(err, "failed to acquire lease")
+			return err
+		}
+	}
+
+	lease.Spec.HolderIdentity = ptr.To(holder)
+	now := &metav1.MicroTime{Time: time.Now()}
+	lease.Spec.AcquireTime = now
+	lease.Spec.RenewTime = now
+	_, err = clientset.CoordinationV1().
+		Leases(leaseNamespace).Update(ctx, lease, metav1.UpdateOptions{})
+	if err != nil {
+		logger.Error(err, "failed to acquire lease")
+		return err
+	}
+
+	// invalidate caches if the previous holder was not the cloud controller manager
+	if lease.Annotations != nil &&
+		!strings.EqualFold(
+			lease.Annotations[consts.AzureResourceLockPreviousHolderNameAnnotation],
+			consts.AzureResourceLockHolderNameCloudControllerManager,
+		) {
+		l.lbCache, err = l.newLBCache()
+		if err != nil {
+			return err
+		}
+		if err := l.VMSet.RefreshCaches(ctx); err != nil {
+			return err
+		}
+	}
+
+	logger.Info("Successfully acquired lease.")
+	return nil
+}
+
+func hasExpired(lease *v1.Lease) bool {
+	if ptr.Deref(lease.Spec.HolderIdentity, "") == "" {
+		return true // No holder, hence expired
+	}
+
+	// Calculate the expiration time
+	leaseDuration := time.Duration(ptr.Deref(lease.Spec.LeaseDurationSeconds, 0)) * time.Second
+	var expirationTime time.Time
+	if lease.Spec.RenewTime != nil {
+		expirationTime = lease.Spec.RenewTime.Add(leaseDuration)
+	}
+
+	return time.Now().After(expirationTime)
+}
+
+func releaseLease(
+	ctx context.Context,
+	clientset kubernetes.Interface,
+	leaseNamespace, leaseName, holder string,
+) error {
+	logger := klog.Background().WithName("releaseLease").
+		WithValues("leaseNamespace", leaseNamespace, "leaseName", leaseName, "holder", holder).V(4)
+
+	lease, err := clientset.CoordinationV1().
+		Leases(leaseNamespace).Get(ctx, leaseName, metav1.GetOptions{})
+	if err != nil {
+		if k8sapierrors.IsNotFound(err) {
+			logger.Info("lease does not exist, no need to release it.")
+			return nil
+		}
+		logger.Error(err, "failed to get lease")
+		return err
+	}
+
+	prevHolder := ptr.Deref(lease.Spec.HolderIdentity, "")
+	if !strings.EqualFold(prevHolder, holder) {
+		logger.Info(
+			"%s is holding the lease instead of %s, no need to release it.",
+			prevHolder, holder,
+		)
+		return nil
+	}
+
+	if hasExpired(lease) {
+		logger.Info("lease has already expired, no need to release it.")
+		return nil
+	}
+
+	lease.Spec.HolderIdentity = ptr.To("")
+	if lease.Annotations == nil {
+		lease.Annotations = make(map[string]string)
+	}
+	lease.Annotations[consts.AzureResourceLockPreviousHolderNameAnnotation] = consts.AzureResourceLockHolderNameCloudControllerManager
+	_, err = clientset.CoordinationV1().
+		Leases(leaseNamespace).Update(ctx, lease, metav1.UpdateOptions{})
+	if err != nil {
+		logger.Error(err, "failed to release lease")
+		return err
+	}
+
+	logger.Info("Successfully released lease.")
+	return nil
+}
diff --git a/pkg/provider/azure_mock_vmsets.go b/pkg/provider/azure_mock_vmsets.go
index 920f11db1..cff17fc44 100644
--- a/pkg/provider/azure_mock_vmsets.go
+++ b/pkg/provider/azure_mock_vmsets.go
@@ -16,11 +16,11 @@
 //
 
 // Code generated by MockGen. DO NOT EDIT.
-// Source: azure_vmsets.go
+// Source: /Users/niqi/go/src/sigs.k8s.io/cloud-provider-azure/pkg/provider/azure_vmsets.go
 //
 // Generated by this command:
 //
-//	mockgen -package provider -source azure_vmsets.go -self_package sigs.k8s.io/cloud-provider-azure/pkg/provider -copyright_file ../../hack/boilerplate/boilerplate.generatego.txt
+//	mockgen -destination=/Users/niqi/go/src/sigs.k8s.io/cloud-provider-azure/pkg/provider/azure_mock_vmsets.go -source=/Users/niqi/go/src/sigs.k8s.io/cloud-provider-azure/pkg/provider/azure_vmsets.go -package=provider VMSet
 //
 
 // Package provider is a generated GoMock package.
@@ -30,15 +30,14 @@ import (
 	context "context"
 	reflect "reflect"
 
-	v5 "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v6"
+	armcompute "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v6"
 	compute "github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2022-08-01/compute"
 	network "github.com/Azure/azure-sdk-for-go/services/network/mgmt/2022-07-01/network"
 	azure "github.com/Azure/go-autorest/autorest/azure"
 	gomock "go.uber.org/mock/gomock"
 	v1 "k8s.io/api/core/v1"
 	types "k8s.io/apimachinery/pkg/types"
-	cloud_provider "k8s.io/cloud-provider"
-
+	cloudprovider "k8s.io/cloud-provider"
 	cache "sigs.k8s.io/cloud-provider-azure/pkg/cache"
 )
 
@@ -185,10 +184,10 @@ func (mr *MockVMSetMockRecorder) GetAgentPoolVMSetNames(nodes any) *gomock.Call
 }
 
 // GetDataDisks mocks base method.
-func (m *MockVMSet) GetDataDisks(nodeName types.NodeName, crt cache.AzureCacheReadType) ([]*v5.DataDisk, *string, error) {
+func (m *MockVMSet) GetDataDisks(nodeName types.NodeName, crt cache.AzureCacheReadType) ([]*armcompute.DataDisk, *string, error) {
 	m.ctrl.T.Helper()
 	ret := m.ctrl.Call(m, "GetDataDisks", nodeName, crt)
-	ret0, _ := ret[0].([]*v5.DataDisk)
+	ret0, _ := ret[0].([]*armcompute.DataDisk)
 	ret1, _ := ret[1].(*string)
 	ret2, _ := ret[2].(error)
 	return ret0, ret1, ret2
@@ -398,10 +397,10 @@ func (mr *MockVMSetMockRecorder) GetVMSetNames(service, nodes any) *gomock.Call
 }
 
 // GetZoneByNodeName mocks base method.
-func (m *MockVMSet) GetZoneByNodeName(name string) (cloud_provider.Zone, error) {
+func (m *MockVMSet) GetZoneByNodeName(name string) (cloudprovider.Zone, error) {
 	m.ctrl.T.Helper()
 	ret := m.ctrl.Call(m, "GetZoneByNodeName", name)
-	ret0, _ := ret[0].(cloud_provider.Zone)
+	ret0, _ := ret[0].(cloudprovider.Zone)
 	ret1, _ := ret[1].(error)
 	return ret0, ret1
 }
@@ -412,6 +411,20 @@ func (mr *MockVMSetMockRecorder) GetZoneByNodeName(name any) *gomock.Call {
 	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetZoneByNodeName", reflect.TypeOf((*MockVMSet)(nil).GetZoneByNodeName), name)
 }
 
+// RefreshCaches mocks base method.
+func (m *MockVMSet) RefreshCaches(ctx context.Context) error {
+	m.ctrl.T.Helper()
+	ret := m.ctrl.Call(m, "RefreshCaches", ctx)
+	ret0, _ := ret[0].(error)
+	return ret0
+}
+
+// RefreshCaches indicates an expected call of RefreshCaches.
+func (mr *MockVMSetMockRecorder) RefreshCaches(ctx any) *gomock.Call {
+	mr.mock.ctrl.T.Helper()
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "RefreshCaches", reflect.TypeOf((*MockVMSet)(nil).RefreshCaches), ctx)
+}
+
 // UpdateVM mocks base method.
 func (m *MockVMSet) UpdateVM(ctx context.Context, nodeName types.NodeName) error {
 	m.ctrl.T.Helper()
diff --git a/pkg/provider/azure_standard.go b/pkg/provider/azure_standard.go
index 7c636f305..41823591c 100644
--- a/pkg/provider/azure_standard.go
+++ b/pkg/provider/azure_standard.go
@@ -190,7 +190,6 @@ func getProtocolsFromKubernetesProtocol(protocol v1.Protocol) (*network.Transpor
 	default:
 		return &transportProto, &securityProto, &probeProto, fmt.Errorf("only TCP, UDP and SCTP are supported for Azure LoadBalancers")
 	}
-
 }
 
 // This returns the full identifier of the primary NIC for the given VM.
@@ -447,15 +446,24 @@ func (as *availabilitySet) newVMASCache() (azcache.Resource, error) {
 	return azcache.NewTimedCache(time.Duration(as.Config.AvailabilitySetsCacheTTLInSeconds)*time.Second, getter, as.Cloud.Config.DisableAPICallCache)
 }
 
+// RefreshCaches invalidates and renew all related caches.
+func (as *availabilitySet) RefreshCaches(_ context.Context) error {
+	var err error
+	as.vmasCache, err = as.newVMASCache()
+	if err != nil {
+		klog.Errorf("as.RefreshCaches: failed to create or refresh VMAS cache: %s", err)
+		return err
+	}
+	return nil
+}
+
 // newStandardSet creates a new availabilitySet.
 func newAvailabilitySet(az *Cloud) (VMSet, error) {
 	as := &availabilitySet{
 		Cloud: az,
 	}
 
-	var err error
-	as.vmasCache, err = as.newVMASCache()
-	if err != nil {
+	if err := as.RefreshCaches(context.Background()); err != nil {
 		return nil, err
 	}
 
@@ -1011,7 +1019,6 @@ func (as *availabilitySet) EnsureBackendPoolDeleted(service *v1.Service, backend
 				}
 			}
 		}
-
 	}
 	nicUpdaters := make([]func() error, 0)
 	allErrs := make([]error, 0)
diff --git a/pkg/provider/azure_vmsets.go b/pkg/provider/azure_vmsets.go
index 88b016fd0..b7f95e040 100644
--- a/pkg/provider/azure_vmsets.go
+++ b/pkg/provider/azure_vmsets.go
@@ -73,7 +73,7 @@ type VMSet interface {
 	EnsureHostInPool(service *v1.Service, nodeName types.NodeName, backendPoolID string, vmSetName string) (string, string, string, *compute.VirtualMachineScaleSetVM, error)
 	// EnsureBackendPoolDeleted ensures the loadBalancer backendAddressPools deleted from the specified nodes.
 	EnsureBackendPoolDeleted(service *v1.Service, backendPoolIDs []string, vmSetName string, backendAddressPools *[]network.BackendAddressPool, deleteFromVMSet bool) (bool, error)
-	//EnsureBackendPoolDeletedFromVMSets ensures the loadBalancer backendAddressPools deleted from the specified VMSS/VMAS
+	// EnsureBackendPoolDeletedFromVMSets ensures the loadBalancer backendAddressPools deleted from the specified VMSS/VMAS
 	EnsureBackendPoolDeletedFromVMSets(vmSetNamesMap map[string]bool, backendPoolIDs []string) error
 
 	// AttachDisk attaches a disk to vm
@@ -112,6 +112,9 @@ type VMSet interface {
 
 	// DeleteCacheForNode removes the node entry from cache.
 	DeleteCacheForNode(nodeName string) error
+
+	// RefreshCaches invalidates and renew all related caches.
+	RefreshCaches(ctx context.Context) error
 }
 
 // AttachDiskOptions attach disk options
diff --git a/pkg/provider/azure_vmss.go b/pkg/provider/azure_vmss.go
index 59d5fbbde..c1a3f599c 100644
--- a/pkg/provider/azure_vmss.go
+++ b/pkg/provider/azure_vmss.go
@@ -106,6 +106,34 @@ type ScaleSet struct {
 	lockMap *LockMap
 }
 
+// RefreshCaches invalidates and renew all related caches.
+func (ss *ScaleSet) RefreshCaches(ctx context.Context) error {
+	logger := klog.Background().WithName("ss.RefreshCaches")
+
+	var err error
+	ss.vmssCache, err = ss.newVMSSCache(ctx)
+	if err != nil {
+		logger.Error(err, "failed to create or refresh vmss cache")
+		return err
+	}
+
+	if !ss.DisableAvailabilitySetNodes || ss.EnableVmssFlexNodes {
+		ss.nonVmssUniformNodesCache, err = ss.newNonVmssUniformNodesCache()
+		if err != nil {
+			logger.Error(err, "failed to create or refresh nonVmssUniformNodes cache")
+			return err
+		}
+	}
+
+	ss.vmssVMCache, err = ss.newVMSSVirtualMachinesCache()
+	if err != nil {
+		logger.Error(err, "failed to create or refresh vmssVM cache")
+		return err
+	}
+
+	return nil
+}
+
 // newScaleSet creates a new ScaleSet.
 func newScaleSet(ctx context.Context, az *Cloud) (VMSet, error) {
 	if az.Config.VmssVirtualMachinesCacheTTLInSeconds == 0 {
@@ -129,20 +157,7 @@ func newScaleSet(ctx context.Context, az *Cloud) (VMSet, error) {
 		lockMap:         newLockMap(),
 	}
 
-	if !ss.DisableAvailabilitySetNodes || ss.EnableVmssFlexNodes {
-		ss.nonVmssUniformNodesCache, err = ss.newNonVmssUniformNodesCache()
-		if err != nil {
-			return nil, err
-		}
-	}
-
-	ss.vmssCache, err = ss.newVMSSCache(ctx)
-	if err != nil {
-		return nil, err
-	}
-
-	ss.vmssVMCache, err = ss.newVMSSVirtualMachinesCache()
-	if err != nil {
+	if err := ss.RefreshCaches(ctx); err != nil {
 		return nil, err
 	}
 
@@ -426,7 +441,6 @@ func (ss *ScaleSet) GetInstanceIDByNodeName(name string) (string, error) {
 // azure:///subscriptions/subsid/resourceGroups/rg/providers/Microsoft.Compute/virtualMachineScaleSets/aks-agentpool-22126781-vmss/virtualMachines/1
 // /subscriptions/subsid/resourceGroups/rg/providers/Microsoft.Compute/virtualMachineScaleSets/aks-agentpool-22126781-vmss/virtualMachines/k8s-agentpool-36841236-vmss_1
 func (ss *ScaleSet) GetNodeNameByProviderID(providerID string) (types.NodeName, error) {
-
 	vmManagementType, err := ss.getVMManagementTypeByProviderID(providerID, azcache.CacheReadTypeUnsafe)
 	if err != nil {
 		klog.Errorf("Failed to check VM management type: %v", err)
@@ -2056,7 +2070,6 @@ func (ss *ScaleSet) EnsureBackendPoolDeleted(service *v1.Service, backendPoolIDs
 	}
 
 	return updated, nil
-
 }
 
 // GetNodeCIDRMaskByProviderID returns the node CIDR subnet mask by provider ID.
diff --git a/pkg/provider/azure_vmssflex.go b/pkg/provider/azure_vmssflex.go
index 72c5a9683..695d072cc 100644
--- a/pkg/provider/azure_vmssflex.go
+++ b/pkg/provider/azure_vmssflex.go
@@ -42,10 +42,8 @@ import (
 	vmutil "sigs.k8s.io/cloud-provider-azure/pkg/util/vm"
 )
 
-var (
-	// ErrorVmssIDIsEmpty indicates the vmss id is empty.
-	ErrorVmssIDIsEmpty = errors.New("VMSS ID is empty")
-)
+// ErrorVmssIDIsEmpty indicates the vmss id is empty.
+var ErrorVmssIDIsEmpty = errors.New("VMSS ID is empty")
 
 // FlexScaleSet implements VMSet interface for Azure Flexible VMSS.
 type FlexScaleSet struct {
@@ -60,6 +58,23 @@ type FlexScaleSet struct {
 	lockMap *LockMap
 }
 
+// RefreshCaches invalidates and renew all related caches.
+func (fs *FlexScaleSet) RefreshCaches(ctx context.Context) error {
+	logger := klog.Background().WithName("fs.RefreshCaches")
+	var err error
+	fs.vmssFlexCache, err = fs.newVmssFlexCache(ctx)
+	if err != nil {
+		logger.Error(err, "failed to create or refresh vmssFlexCache")
+		return err
+	}
+	fs.vmssFlexVMCache, err = fs.newVmssFlexVMCache(ctx)
+	if err != nil {
+		logger.Error(err, "failed to create or refresh vmssFlexVMCache")
+		return err
+	}
+	return nil
+}
+
 func newFlexScaleSet(ctx context.Context, az *Cloud) (VMSet, error) {
 	fs := &FlexScaleSet{
 		Cloud:                    az,
@@ -68,13 +83,7 @@ func newFlexScaleSet(ctx context.Context, az *Cloud) (VMSet, error) {
 		lockMap:                  newLockMap(),
 	}
 
-	var err error
-	fs.vmssFlexCache, err = fs.newVmssFlexCache(ctx)
-	if err != nil {
-		return nil, err
-	}
-	fs.vmssFlexVMCache, err = fs.newVmssFlexVMCache(ctx)
-	if err != nil {
+	if err := fs.RefreshCaches(ctx); err != nil {
 		return nil, err
 	}
 
@@ -98,7 +107,6 @@ func (fs *FlexScaleSet) getNodeVmssFlexName(nodeName string) (string, error) {
 		return "", err
 	}
 	return vmssFlexName, nil
-
 }
 
 // GetNodeVMSetName returns the availability set or vmss name by the node name.
@@ -196,7 +204,6 @@ func (fs *FlexScaleSet) GetInstanceIDByNodeName(name string) (string, error) {
 		return "", err
 	}
 	return convertedResourceID, nil
-
 }
 
 // GetInstanceTypeByNodeName gets the instance type by node name.
@@ -342,7 +349,6 @@ func (fs *FlexScaleSet) GetIPByNodeName(name string) (string, string, error) {
 	}
 
 	return privateIP, publicIP, nil
-
 }
 
 // GetPrivateIPsByNodeName returns a slice of all private ips assigned to node (ipv6 and ipv4)
@@ -397,7 +403,6 @@ func (fs *FlexScaleSet) getNodeInformationByIPConfigurationID(ipConfigurationID
 	}
 
 	vmssFlexName, err := fs.getNodeVmssFlexName(nodeName)
-
 	if err != nil {
 		klog.Errorf("Unable to get the vmss flex name by node name %s: %v", vmName, err)
 		return "", "", "", err
@@ -548,7 +553,6 @@ func (fs *FlexScaleSet) EnsureHostInPool(service *v1.Service, nodeName types.Nod
 	}
 
 	return nodeResourceGroup, vmssFlexName, name, nil, nil
-
 }
 
 func (fs *FlexScaleSet) ensureVMSSFlexInPool(_ *v1.Service, nodes []*v1.Node, backendPoolID string, vmSetNameOfLB string) error {
@@ -913,7 +917,6 @@ func (fs *FlexScaleSet) EnsureBackendPoolDeleted(service *v1.Service, backendPoo
 					continue
 				}
 			}
-
 		}
 	}
 
@@ -939,7 +942,6 @@ func (fs *FlexScaleSet) EnsureBackendPoolDeleted(service *v1.Service, backendPoo
 
 	isOperationSucceeded = true
 	return nicUpdated, nil
-
 }
 
 func (fs *FlexScaleSet) ensureBackendPoolDeletedFromNode(vmssFlexVMNameMap map[string]string, backendPoolIDs []string) (bool, error) {


Test patch:
diff --git a/e2e.Dockerfile b/e2e.Dockerfile
index 567135748..f53b9a4a5 100644
--- a/e2e.Dockerfile
+++ b/e2e.Dockerfile
@@ -14,7 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM golang:1.23-bullseye@sha256:ebaf58d74f200c27dce0181203a1347e41b5c159c9aa0517a95bec594aabbde4
+FROM golang:1.23-bullseye@sha256:48ac5022f9740543cac0eba41a4e37b721073a0103349e416678e0142a53b49a
 
 WORKDIR /go/src/sigs.k8s.io/cloud-provider-azure
 
diff --git a/pkg/provider/azure_loadbalancer_test.go b/pkg/provider/azure_loadbalancer_test.go
index 39e44c62a..f8a81d258 100644
--- a/pkg/provider/azure_loadbalancer_test.go
+++ b/pkg/provider/azure_loadbalancer_test.go
@@ -37,11 +37,14 @@ import (
 	"golang.org/x/text/cases"
 	"golang.org/x/text/language"
 
+	coordinationv1 "k8s.io/api/coordination/v1"
 	v1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/apimachinery/pkg/util/sets"
 	"k8s.io/client-go/kubernetes/fake"
+	k8stesting "k8s.io/client-go/testing"
 	"k8s.io/utils/pointer"
 	"k8s.io/utils/ptr"
 
@@ -795,6 +798,118 @@ func TestEnsureLoadBalancerDeleted(t *testing.T) {
 	}
 }
 
+func TestEnsureLoadBalancerLock(t *testing.T) {
+	ctrl := gomock.NewController(t)
+	defer ctrl.Finish()
+
+	az := GetTestCloud(ctrl)
+	kubeClient := fake.NewSimpleClientset()
+	kubeClient.PrependReactor(
+		"get", "leases",
+		func(_ k8stesting.Action) (handled bool, ret runtime.Object, err error) {
+			return true, nil, errors.New("get lease failed")
+		})
+	az.KubeClient = kubeClient
+	az.azureResourceLocker = NewAzureResourceLocker(
+		az, "holder", "aks-managed-resource-locker", "kube-system", 900,
+	)
+
+	svc := getTestService("service", v1.ProtocolTCP, nil, false, 80)
+	_, err := az.EnsureLoadBalancer(context.Background(), testClusterName, &svc, nil)
+	assert.Error(t, err)
+	assert.Contains(t, err.Error(), "get lease failed")
+	assert.Contains(t, err.Error(), "EnsureLoadBalancer failed due to fail to lock azure resources")
+
+	kubeClient = fake.NewSimpleClientset(&coordinationv1.Lease{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "aks-managed-resource-locker",
+			Namespace: "kube-system",
+		},
+		Spec: coordinationv1.LeaseSpec{
+			LeaseDurationSeconds: ptr.To[int32](900),
+		},
+	})
+	// update lease failed for the second time
+	count := 0
+	kubeClient.PrependReactor(
+		"update", "leases",
+		func(_ k8stesting.Action) (handled bool, ret runtime.Object, err error) {
+			count++
+			if count == 2 {
+				return true, nil, errors.New("update lease failed")
+			}
+			return false, nil, nil
+		})
+	az.KubeClient = kubeClient
+	az.azureResourceLocker = NewAzureResourceLocker(
+		az, "holder", "aks-managed-resource-locker", "kube-system", 900,
+	)
+	mockLBClient := az.LoadBalancerClient.(*mockloadbalancerclient.MockInterface)
+	mockLBClient.EXPECT().List(gomock.Any(), gomock.Any()).
+		Return(nil, retry.NewError(false, errors.New("list lb failed")))
+
+	_, err = az.EnsureLoadBalancer(context.Background(), testClusterName, &svc, nil)
+	assert.Error(t, err)
+	assert.Contains(t, err.Error(), "update lease failed")
+	assert.Contains(t, err.Error(), "list lb failed")
+}
+
+func TestEnsureLoadBalancerDeletedLock(t *testing.T) {
+	ctrl := gomock.NewController(t)
+	defer ctrl.Finish()
+
+	az := GetTestCloud(ctrl)
+	kubeClient := fake.NewSimpleClientset()
+	kubeClient.PrependReactor(
+		"get", "leases",
+		func(_ k8stesting.Action) (handled bool, ret runtime.Object, err error) {
+			return true, nil, errors.New("get lease failed")
+		})
+	az.KubeClient = kubeClient
+	az.azureResourceLocker = NewAzureResourceLocker(
+		az, "holder", "aks-managed-resource-locker", "kube-system", 900,
+	)
+
+	svc := getTestService("service", v1.ProtocolTCP, nil, false, 80)
+	err := az.EnsureLoadBalancerDeleted(context.Background(), testClusterName, &svc)
+	assert.Error(t, err)
+	assert.Contains(t, err.Error(), "get lease failed")
+	assert.Contains(t, err.Error(), "EnsureLoadBalancerDeleted failed due to fail to lock azure resources")
+
+	kubeClient = fake.NewSimpleClientset(&coordinationv1.Lease{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "aks-managed-resource-locker",
+			Namespace: "kube-system",
+		},
+		Spec: coordinationv1.LeaseSpec{
+			LeaseDurationSeconds: ptr.To[int32](900),
+		},
+	})
+	// update lease failed for the second time
+	count := 0
+	kubeClient.PrependReactor(
+		"update", "leases",
+		func(_ k8stesting.Action) (handled bool, ret runtime.Object, err error) {
+			count++
+			if count == 2 {
+				return true, nil, errors.New("update lease failed")
+			}
+			return false, nil, nil
+		})
+	az.KubeClient = kubeClient
+	az.azureResourceLocker = NewAzureResourceLocker(
+		az, "holder", "aks-managed-resource-locker", "kube-system", 900,
+	)
+	mockLBClient := az.LoadBalancerClient.(*mockloadbalancerclient.MockInterface)
+	mockLBClient.EXPECT().List(gomock.Any(), gomock.Any()).
+		Return(nil, retry.NewError(false, errors.New("list lb failed")))
+
+	err = az.EnsureLoadBalancerDeleted(context.Background(), testClusterName, &svc)
+	assert.Error(t, err)
+	assert.Contains(t, err.Error(), "update lease failed")
+	assert.Contains(t, err.Error(), "list lb failed")
+}
+
 func TestServiceOwnsPublicIP(t *testing.T) {
 	tests := []struct {
 		desc                    string
@@ -1403,7 +1518,6 @@ func TestConvertIPTagMapToSlice(t *testing.T) {
 				ipTagSlice := *c.expected
 				return pointer.StringDeref(ipTagSlice[i].IPTagType, "") < pointer.StringDeref(ipTagSlice[j].IPTagType, "")
 			})
-
 		}
 
 		assert.Equal(t, c.expected, actual, "TestCase[%d]: %s", i, c.desc)
@@ -1490,7 +1604,6 @@ func TestGetserviceIPTagRequestForPublicIP(t *testing.T) {
 				ipTagSlice := *c.expected.IPTags
 				return pointer.StringDeref(ipTagSlice[i].IPTagType, "") < pointer.StringDeref(ipTagSlice[j].IPTagType, "")
 			})
-
 		}
 
 		assert.Equal(t, actual, c.expected, "TestCase[%d]: %s", i, c.desc)
@@ -4689,7 +4802,6 @@ func TestReconcilePublicIPsCommon(t *testing.T) {
 					}
 
 					return network.PublicIPAddress{}, &retry.Error{HTTPStatusCode: http.StatusNotFound}
-
 				})
 				deleter := mockPIPsClient.EXPECT().Delete(gomock.Any(), "rg", *pip.Name).Return(nil).AnyTimes()
 				deleter.Do(func(_ context.Context, _ string, publicIPAddressName string) *retry.Error {
@@ -5200,6 +5312,7 @@ func TestEnsurePublicIPExistsCommon(t *testing.T) {
 		})
 	}
 }
+
 func TestEnsurePublicIPExistsWithExtendedLocation(t *testing.T) {
 	ctrl := gomock.NewController(t)
 	defer ctrl.Finish()
@@ -6212,7 +6325,8 @@ func TestReconcileZonesForFrontendIPConfigs(t *testing.T) {
 			description: "reconcileFrontendIPConfigs should use the nil zones of the existing frontend",
 			service: getTestServiceWithAnnotation("test", map[string]string{
 				consts.ServiceAnnotationLoadBalancerInternalSubnet: "subnet",
-				consts.ServiceAnnotationLoadBalancerInternal:       consts.TrueAnnotationValue}, true, 80),
+				consts.ServiceAnnotationLoadBalancerInternal:       consts.TrueAnnotationValue,
+			}, true, 80),
 			existingFrontendIPConfigs: []network.FrontendIPConfiguration{
 				{
 					Name: pointer.String("atest1"),
@@ -6229,7 +6343,8 @@ func TestReconcileZonesForFrontendIPConfigs(t *testing.T) {
 			description: "reconcileFrontendIPConfigs should use the non-nil zones of the existing frontend",
 			service: getTestServiceWithAnnotation("test", map[string]string{
 				consts.ServiceAnnotationLoadBalancerInternalSubnet: "subnet",
-				consts.ServiceAnnotationLoadBalancerInternal:       consts.TrueAnnotationValue}, true, 80),
+				consts.ServiceAnnotationLoadBalancerInternal:       consts.TrueAnnotationValue,
+			}, true, 80),
 			existingFrontendIPConfigs: []network.FrontendIPConfiguration{
 				{
 					Name: pointer.String("not-this-one"),
@@ -7197,7 +7312,8 @@ func TestGetEligibleLoadBalancers(t *testing.T) {
 									Operator: metav1.LabelSelectorOpIn,
 									Values:   []string{},
 								},
-							}},
+							},
+						},
 					},
 				},
 			},
@@ -7218,7 +7334,8 @@ func TestGetEligibleLoadBalancers(t *testing.T) {
 									Operator: metav1.LabelSelectorOpIn,
 									Values:   []string{},
 								},
-							}},
+							},
+						},
 					},
 				},
 			},
@@ -7670,7 +7787,6 @@ func TestReconcileMultipleStandardLoadBalancerConfigurations(t *testing.T) {
 			for lbConfigName, svcNames := range tc.expectedActiveServices {
 				assert.Equal(t, svcNames, activeServices[lbConfigName])
 			}
-
 		})
 	}
 }
diff --git a/pkg/provider/azure_lock_test.go b/pkg/provider/azure_lock_test.go
new file mode 100644
index 000000000..3c037f41c
--- /dev/null
+++ b/pkg/provider/azure_lock_test.go
@@ -0,0 +1,242 @@
+/*
+Copyright 2024 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package provider
+
+import (
+	"context"
+	"errors"
+	"testing"
+	"time"
+
+	"github.com/Azure/azure-sdk-for-go/sdk/azcore/to"
+	"github.com/stretchr/testify/assert"
+
+	"go.uber.org/mock/gomock"
+
+	v1 "k8s.io/api/coordination/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/client-go/kubernetes"
+	"k8s.io/client-go/kubernetes/fake"
+	client_go_testing "k8s.io/client-go/testing"
+)
+
+func newTestAzureResourceLocker(ctrl *gomock.Controller, kubeClient kubernetes.Interface) *AzureResourceLocker {
+	cloud := GetTestCloud(ctrl)
+	cloud.KubeClient = kubeClient
+	return NewAzureResourceLocker(cloud, "holder", "aks-managed-resource-locker", "kube-system", 900)
+}
+
+func TestLock(t *testing.T) {
+	ctrl := gomock.NewController(t)
+	defer ctrl.Finish()
+
+	for _, tc := range []struct {
+		desc        string
+		getErr      bool
+		createErr   bool
+		updateErr   bool
+		existed     bool
+		expired     bool
+		sameHolder  bool
+		expectedErr string
+	}{
+		{
+			desc:        "should return an error if failed to get the lease",
+			getErr:      true,
+			expectedErr: "failed to get leases",
+		},
+		{
+			desc:        "should return an error if failed to create the lease",
+			createErr:   true,
+			expectedErr: "failed to create lease",
+		},
+		{
+			desc:        "should return an error if failed to get the lease when acquiring it",
+			getErr:      true,
+			existed:     true,
+			expectedErr: "failed to get leases",
+		},
+		{
+			desc:        "should return an error when trying to acquire a lease which has not expired",
+			existed:     true,
+			expectedErr: "Lease has not expired yet",
+		},
+		{
+			desc:       "should not return an error if the lease has not expired but the previous holder is the same",
+			existed:    true,
+			sameHolder: true,
+		},
+		{
+			desc:        "should return an error if failed to update the lease",
+			existed:     true,
+			updateErr:   true,
+			expired:     true,
+			expectedErr: "failed to update lease",
+		},
+	} {
+		t.Run(tc.desc, func(t *testing.T) {
+			testLease := getTestLease()
+			kubeClient := fake.NewSimpleClientset()
+			if !tc.expired {
+				testLease.Spec.HolderIdentity = to.Ptr("prevHolder")
+				testLease.Spec.RenewTime = &metav1.MicroTime{Time: time.Now().Add(-100)}
+				if tc.sameHolder {
+					testLease.Spec.HolderIdentity = to.Ptr("holder")
+				}
+			}
+			if tc.existed {
+				kubeClient = fake.NewSimpleClientset(testLease)
+			}
+			getCount := 0
+			if tc.getErr {
+				kubeClient.PrependReactor(
+					"get", "leases",
+					func(_ client_go_testing.Action) (handled bool, ret runtime.Object, err error) {
+						getCount++
+						if tc.existed {
+							if getCount == 2 {
+								return true, nil, errors.New("failed to get leases")
+							}
+							return true, testLease, nil
+						}
+						return true, nil, errors.New("failed to get leases")
+					},
+				)
+			}
+			if tc.createErr {
+				kubeClient.PrependReactor(
+					"create", "leases",
+					func(_ client_go_testing.Action) (handled bool, ret runtime.Object, err error) {
+						return true, nil, errors.New("failed to create lease")
+					},
+				)
+			}
+			if tc.updateErr {
+				kubeClient.PrependReactor(
+					"update", "leases",
+					func(_ client_go_testing.Action) (handled bool, ret runtime.Object, err error) {
+						return true, nil, errors.New("failed to update lease")
+					},
+				)
+			}
+			if !tc.getErr && !tc.createErr {
+				kubeClient.PrependReactor(
+					"get", "leases",
+					func(_ client_go_testing.Action) (handled bool, ret runtime.Object, err error) {
+						return true, testLease, nil
+					},
+				)
+			}
+
+			locker := newTestAzureResourceLocker(ctrl, kubeClient)
+			err := locker.Lock(context.TODO())
+			if tc.expectedErr != "" {
+				assert.Error(t, err)
+				assert.Contains(t, err.Error(), tc.expectedErr)
+			} else {
+				assert.Nil(t, err)
+			}
+		})
+	}
+}
+
+func TestUnlock(t *testing.T) {
+	ctrl := gomock.NewController(t)
+	defer ctrl.Finish()
+
+	for _, tc := range []struct {
+		desc            string
+		getErr          bool
+		updateErr       bool
+		expired         bool
+		differentHolder bool
+		expectedErr     string
+	}{
+		{
+			desc:        "should return an error if failed to get the lease",
+			getErr:      true,
+			expectedErr: "failed to get lease",
+		},
+		{
+			desc:    "should do nothing if the lease has expired",
+			expired: true,
+		},
+		{
+			desc:            "should do nothing if the lease is holding by another holder",
+			differentHolder: true,
+		},
+		{
+			desc:        "should return an error if failed to update the lease",
+			updateErr:   true,
+			expired:     true,
+			expectedErr: "failed to update lease",
+		},
+	} {
+		t.Run(tc.desc, func(t *testing.T) {
+			testLease := getTestLease()
+			testLease.Spec.HolderIdentity = to.Ptr("holder")
+			testLease.Spec.RenewTime = &metav1.MicroTime{Time: time.Now().Add(-1000)}
+			if !tc.expired {
+				testLease.Spec.RenewTime = &metav1.MicroTime{Time: time.Now().Add(-100)}
+				if tc.differentHolder {
+					testLease.Spec.HolderIdentity = to.Ptr("prevHolder")
+				}
+			}
+
+			kubeClient := fake.NewSimpleClientset(testLease)
+			if tc.getErr {
+				kubeClient.PrependReactor(
+					"get", "leases",
+					func(_ client_go_testing.Action) (handled bool, ret runtime.Object, err error) {
+						return true, nil, errors.New("failed to get lease")
+					},
+				)
+			}
+			if tc.updateErr {
+				kubeClient.PrependReactor(
+					"update", "leases",
+					func(_ client_go_testing.Action) (handled bool, ret runtime.Object, err error) {
+						return true, nil, errors.New("failed to update lease")
+					},
+				)
+			}
+
+			locker := newTestAzureResourceLocker(ctrl, kubeClient)
+			err := locker.Unlock(context.TODO())
+			if tc.expectedErr != "" {
+				assert.Error(t, err)
+				assert.Contains(t, err.Error(), tc.expectedErr)
+			} else {
+				assert.Nil(t, err)
+			}
+		})
+	}
+}
+
+func getTestLease() *v1.Lease {
+	return &v1.Lease{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "aks-managed-resource-locker",
+			Namespace: "kube-system",
+		},
+		Spec: v1.LeaseSpec{
+			HolderIdentity:       to.Ptr(""),
+			LeaseDurationSeconds: to.Ptr[int32](900),
+		},
+	}
+}