Repository: kubernetes-sigs/cloud-provider-azure
Pull request #7337 (2024-10-16 17:30:36)

build(deps): bump github.com/prometheus/client_golang from 1.20.4 to 1.20.5
Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.20.4 to 1.20.5.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/prometheus/client_golang/releases">github.com/prometheus/client_golang's releases</a>.</em></p>
<blockquote>
<h2>v1.20.5 / 2024-10-15</h2>
<p>We decided to revert <a href="https://redirect.github.com/prometheus/client_golang/pull/1424">the <code>testutil</code> change</a> that made our util functions less error-prone, but created a lot of work for our downstream users. Apologies for the pain! This revert should not cause any major breaking change, even if you already did the work--unless you depend on the <a href="https://redirect.github.com/grafana/mimir/pull/9624#issuecomment-2413401565">exact error message</a>.</p>
<p>Going forward, we plan to reinforce our release testing strategy <a href="https://redirect.github.com/prometheus/client_golang/issues/1646">[1]</a>,<a href="https://redirect.github.com/prometheus/client_golang/issues/1648">[2]</a> and deliver an enhanced <a href="https://redirect.github.com/prometheus/client_golang/issues/1639"><code>testutil</code> package/module</a> with more flexible and safer APIs.</p>
<p>Thanks to <a href="https://github.com/dashpole"><code>@​dashpole</code></a> <a href="https://github.com/dgrisonnet"><code>@​dgrisonnet</code></a> <a href="https://github.com/kakkoyun"><code>@​kakkoyun</code></a> <a href="https://github.com/ArthurSens"><code>@​ArthurSens</code></a> <a href="https://github.com/vesari"><code>@​vesari</code></a> <a href="https://github.com/logicalhan"><code>@​logicalhan</code></a> <a href="https://github.com/krajorama"><code>@​krajorama</code></a> <a href="https://github.com/bwplotka"><code>@​bwplotka</code></a> who helped in this patch release! 🤗</p>
<h3>Changelog</h3>
<p>[BUGFIX] testutil: Reverted <a href="https://redirect.github.com/prometheus/client_golang/issues/1424">#1424</a>; functions using compareMetricFamilies are (again) only failing if filtered metricNames are in the expected input. <a href="https://redirect.github.com/prometheus/client_golang/issues/1645">#1645</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md">github.com/prometheus/client_golang's changelog</a>.</em></p>
<blockquote>
<h2>1.20.5 / 2024-10-15</h2>
<ul>
<li>[BUGFIX] testutil: Reverted <a href="https://redirect.github.com/prometheus/client_golang/issues/1424">#1424</a>; functions using compareMetricFamilies are (again) only failing if filtered metricNames are in the expected input.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/prometheus/client_golang/commit/48e12a185519fd76b4e514b597483781d9ba4093"><code>48e12a1</code></a> Merge pull request <a href="https://redirect.github.com/prometheus/client_golang/issues/1645">#1645</a> from prometheus/cut-1204-pr1424</li>
<li><a href="https://github.com/prometheus/client_golang/commit/504ad9bf5c6419449d2cacf8cf8855bfdcfcfc18"><code>504ad9b</code></a> Cut 1.20.5; update comments.</li>
<li><a href="https://github.com/prometheus/client_golang/commit/584a7ce3d935e4fdca7b893f5f741d59f3289140"><code>584a7ce</code></a> Revert &quot;testutil compareMetricFamilies: make less error-prone (<a href="https://redirect.github.com/prometheus/client_golang/issues/1424">#1424</a>)&quot;</li>
<li>See full diff in <a href="https://github.com/prometheus/client_golang/compare/v1.20.4...v1.20.5">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/prometheus/client_golang&package-manager=go_modules&previous-version=1.20.4&new-version=1.20.5)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>

k8s-ci-robot: Hi @dependabot[bot]. Thanks for your PR.

I'm waiting for a [kubernetes-sigs](https://github.com/orgs/kubernetes-sigs/people) member to verify that this patch is reasonable to test. If it is, they should reply with `/ok-to-test` on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should [join the org](https://git.k8s.io/community/community-membership.md#member) to skip this step.

Once the patch is verified, the new status will be reflected by the `ok-to-test` label.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>

k8s-ci-robot: @dependabot[bot]: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz-1-31 | 8464498dcaad6b8d0920916bea3f4c2581c4ac84 | [link](https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/7337/pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz-1-31/1846604683372662784) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-ip-lb-capz-1-31`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=7337). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Adependabot[bot]). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->

Patch:
diff --git a/Dockerfile b/Dockerfile
index 80e961757..5123d1b8e 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -14,7 +14,7 @@
 
 # syntax=docker/dockerfile:1
 
-FROM --platform=linux/amd64 golang:1.23-bullseye@sha256:ebaf58d74f200c27dce0181203a1347e41b5c159c9aa0517a95bec594aabbde4 AS builder
+FROM --platform=linux/amd64 golang:1.23-bullseye@sha256:48ac5022f9740543cac0eba41a4e37b721073a0103349e416678e0142a53b49a AS builder
 
 ARG ENABLE_GIT_COMMAND=true
 ARG ARCH=amd64
diff --git a/cloud-node-manager.Dockerfile b/cloud-node-manager.Dockerfile
index 67e390a07..061a03240 100644
--- a/cloud-node-manager.Dockerfile
+++ b/cloud-node-manager.Dockerfile
@@ -14,7 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM --platform=linux/amd64 golang:1.23-bullseye@sha256:ebaf58d74f200c27dce0181203a1347e41b5c159c9aa0517a95bec594aabbde4 AS builder
+FROM --platform=linux/amd64 golang:1.23-bullseye@sha256:48ac5022f9740543cac0eba41a4e37b721073a0103349e416678e0142a53b49a AS builder
 
 ARG ENABLE_GIT_COMMAND=true
 ARG ARCH=amd64
diff --git a/cmd/acr-credential-provider/main.go b/cmd/acr-credential-provider/main.go
index 93a78925b..9cb1840ec 100644
--- a/cmd/acr-credential-provider/main.go
+++ b/cmd/acr-credential-provider/main.go
@@ -33,6 +33,9 @@ import (
 
 func main() {
 	rand.Seed(time.Now().UnixNano())
+
+	var RegistryMirrorStr string
+
 	command := &cobra.Command{
 		Use:   "acr-credential-provider configFile",
 		Short: "Acr credential provider for Kubelet",
@@ -44,7 +47,7 @@ func main() {
 				os.Exit(1)
 			}
 
-			acrProvider, err := credentialprovider.NewAcrProviderFromConfig(args[0])
+			acrProvider, err := credentialprovider.NewAcrProviderFromConfig(args[0], RegistryMirrorStr)
 			if err != nil {
 				klog.Errorf("Failed to initialize ACR provider: %v", err)
 				os.Exit(1)
@@ -60,6 +63,10 @@ func main() {
 	logs.InitLogs()
 	defer logs.FlushLogs()
 
+	// Flags
+	command.Flags().StringVarP(&RegistryMirrorStr, "registry-mirror", "r", "",
+		"Mirror a source registry host to a target registry host, and image pull credential will be requested to the target registry host when the image is from source registry host")
+
 	if err := command.Execute(); err != nil {
 		os.Exit(1)
 	}
diff --git a/cmd/cloud-controller-manager/app/controllermanager.go b/cmd/cloud-controller-manager/app/controllermanager.go
index 77af51091..3f0bd2b84 100644
--- a/cmd/cloud-controller-manager/app/controllermanager.go
+++ b/cmd/cloud-controller-manager/app/controllermanager.go
@@ -348,7 +348,7 @@ func Run(ctx context.Context, c *cloudcontrollerconfig.CompletedConfig, h *contr
 	)
 
 	if c.ComponentConfig.KubeCloudShared.CloudProvider.CloudConfigFile != "" {
-		cloud, err = provider.NewCloudFromConfigFile(ctx, c.ComponentConfig.KubeCloudShared.CloudProvider.CloudConfigFile, true)
+		cloud, err = provider.NewCloudFromConfigFile(ctx, c.ClientBuilder, c.ComponentConfig.KubeCloudShared.CloudProvider.CloudConfigFile, true)
 		if err != nil {
 			klog.Fatalf("Cloud provider azure could not be initialized: %v", err)
 		}
diff --git a/examples/out-of-tree/credential-provider-config.yaml b/examples/out-of-tree/credential-provider-config.yaml
index f079b0539..f9f6e3b6b 100644
--- a/examples/out-of-tree/credential-provider-config.yaml
+++ b/examples/out-of-tree/credential-provider-config.yaml
@@ -9,5 +9,7 @@ providers:
   - "*.azurecr.cn"
   - "*.azurecr.de"
   - "*.azurecr.us"
+  - "mcr.microsoft.com"
   args:
   - /etc/kubernetes/azure.json
+  - --registry-mirror=mcr.microsoft.com:xxx.azurecr.io
diff --git a/go.mod b/go.mod
index d9f2e7479..532e9d9ed 100644
--- a/go.mod
+++ b/go.mod
@@ -24,7 +24,7 @@ require (
 	github.com/go-logr/logr v1.4.2
 	github.com/onsi/ginkgo/v2 v2.20.2
 	github.com/onsi/gomega v1.34.2
-	github.com/prometheus/client_golang v1.20.4
+	github.com/prometheus/client_golang v1.20.5
 	github.com/spf13/cobra v1.8.1
 	github.com/spf13/pflag v1.0.5
 	github.com/stretchr/testify v1.9.0
diff --git a/go.sum b/go.sum
index ddc3ae76b..f9c0ee74c 100644
--- a/go.sum
+++ b/go.sum
@@ -185,6 +185,7 @@ github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnr
 github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
 github.com/keybase/go-keychain v0.0.0-20231219164618-57a3676c3af6 h1:IsMZxCuZqKuao2vNdfD82fjjgPLfyHLpR41Z88viRWs=
 github.com/keybase/go-keychain v0.0.0-20231219164618-57a3676c3af6/go.mod h1:3VeWNIJaW+O5xpRQbPp0Ybqu1vJd/pm7s2F473HRrkw=
+github.com/julienschmidt/httprouter v1.3.0/go.mod h1:JR6WtHb+2LUe8TCKY3cZOxFyyO8IZAc4RVcycCCAKdM=
 github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
 github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
 github.com/klauspost/compress v1.17.9 h1:6KIumPrER1LHsvBVuDa0r5xaG0Es51mhhB9BQB2qeMA=
@@ -220,8 +221,8 @@ github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINE
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
 github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
-github.com/prometheus/client_golang v1.20.4 h1:Tgh3Yr67PaOv/uTqloMsCEdeuFTatm5zIq5+qNN23vI=
-github.com/prometheus/client_golang v1.20.4/go.mod h1:PIEt8X02hGcP8JWbeHyeZ53Y/jReSnHgO035n//V5WE=
+github.com/prometheus/client_golang v1.20.5 h1:cxppBPuYhUnsO6yo/aoRol4L7q7UFfdm+bR9r+8l63Y=
+github.com/prometheus/client_golang v1.20.5/go.mod h1:PIEt8X02hGcP8JWbeHyeZ53Y/jReSnHgO035n//V5WE=
 github.com/prometheus/client_model v0.6.1 h1:ZKSh/rekM+n3CeS952MLRAdFwIKqeY8b62p8ais2e9E=
 github.com/prometheus/client_model v0.6.1/go.mod h1:OrxVMOVHjw3lKMa8+x6HeMGkHMQyHDk9E3jmP2AmGiY=
 github.com/prometheus/common v0.60.0 h1:+V9PAREWNvJMAuJ1x1BaWl9dewMW4YrHZQbx0sJNllA=
diff --git a/pkg/azureclients/containerserviceclient/azure_containerserviceclient.go b/pkg/azureclients/containerserviceclient/azure_containerserviceclient.go
deleted file mode 100644
index 6edc4a9aa..000000000
--- a/pkg/azureclients/containerserviceclient/azure_containerserviceclient.go
+++ /dev/null
@@ -1,415 +0,0 @@
-/*
-Copyright 2020 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package containerserviceclient
-
-import (
-	"context"
-	"net/http"
-	"time"
-
-	"github.com/Azure/azure-sdk-for-go/services/containerservice/mgmt/2021-10-01/containerservice"
-	"github.com/Azure/go-autorest/autorest"
-	"github.com/Azure/go-autorest/autorest/azure"
-
-	"k8s.io/client-go/util/flowcontrol"
-	"k8s.io/klog/v2"
-	"k8s.io/utils/ptr"
-
-	azclients "sigs.k8s.io/cloud-provider-azure/pkg/azureclients"
-	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/armclient"
-	"sigs.k8s.io/cloud-provider-azure/pkg/metrics"
-	"sigs.k8s.io/cloud-provider-azure/pkg/retry"
-)
-
-var _ Interface = &Client{}
-
-const managedClusterResourceType = "Microsoft.ContainerService/managedClusters"
-
-// Client implements ContainerService client Interface.
-type Client struct {
-	armClient      armclient.Interface
-	subscriptionID string
-
-	// Rate limiting configures.
-	rateLimiterReader flowcontrol.RateLimiter
-	rateLimiterWriter flowcontrol.RateLimiter
-
-	// ARM throttling configures.
-	RetryAfterReader time.Time
-	RetryAfterWriter time.Time
-}
-
-// New creates a new ContainerServiceClient client with ratelimiting.
-func New(config *azclients.ClientConfig) *Client {
-	baseURI := config.ResourceManagerEndpoint
-	authorizer := config.Authorizer
-	armClient := armclient.New(authorizer, *config, baseURI, APIVersion)
-	rateLimiterReader, rateLimiterWriter := azclients.NewRateLimiter(config.RateLimitConfig)
-
-	if azclients.RateLimitEnabled(config.RateLimitConfig) {
-		klog.V(2).Infof("Azure ContainerServiceClient (read ops) using rate limit config: QPS=%g, bucket=%d",
-			config.RateLimitConfig.CloudProviderRateLimitQPS,
-			config.RateLimitConfig.CloudProviderRateLimitBucket)
-		klog.V(2).Infof("Azure ContainerServiceClient (write ops) using rate limit config: QPS=%g, bucket=%d",
-			config.RateLimitConfig.CloudProviderRateLimitQPSWrite,
-			config.RateLimitConfig.CloudProviderRateLimitBucketWrite)
-	}
-
-	client := &Client{
-		armClient:         armClient,
-		rateLimiterReader: rateLimiterReader,
-		rateLimiterWriter: rateLimiterWriter,
-		subscriptionID:    config.SubscriptionID,
-	}
-
-	return client
-}
-
-// Get gets a ManagedCluster.
-func (c *Client) Get(ctx context.Context, resourceGroupName string, managedClusterName string) (containerservice.ManagedCluster, *retry.Error) {
-	mc := metrics.NewMetricContext("managed_clusters", "get", resourceGroupName, c.subscriptionID, "")
-
-	// Report errors if the client is rate limited.
-	if !c.rateLimiterReader.TryAccept() {
-		mc.RateLimitedCount()
-		return containerservice.ManagedCluster{}, retry.GetRateLimitError(false, "GetManagedCluster")
-	}
-
-	// Report errors if the client is throttled.
-	if c.RetryAfterReader.After(time.Now()) {
-		mc.ThrottledCount()
-		rerr := retry.GetThrottlingError("GetManagedCluster", "client throttled", c.RetryAfterReader)
-		return containerservice.ManagedCluster{}, rerr
-	}
-
-	result, rerr := c.getManagedCluster(ctx, resourceGroupName, managedClusterName)
-	mc.Observe(rerr)
-	if rerr != nil {
-		if rerr.IsThrottled() {
-			// Update RetryAfterReader so that no more requests would be sent until RetryAfter expires.
-			c.RetryAfterReader = rerr.RetryAfter
-		}
-
-		return result, rerr
-	}
-
-	return result, nil
-}
-
-// getManagedCluster gets a ManagedCluster.
-func (c *Client) getManagedCluster(ctx context.Context, resourceGroupName string, managedClusterName string) (containerservice.ManagedCluster, *retry.Error) {
-	resourceID := armclient.GetResourceID(
-		c.subscriptionID,
-		resourceGroupName,
-		managedClusterResourceType,
-		managedClusterName,
-	)
-	result := containerservice.ManagedCluster{}
-
-	response, rerr := c.armClient.GetResource(ctx, resourceID)
-	defer c.armClient.CloseResponse(ctx, response)
-	if rerr != nil {
-		klog.V(5).Infof("Received error in %s: resourceID: %s, error: %s", "managedcluster.get.request", resourceID, rerr.Error())
-		return result, rerr
-	}
-
-	err := autorest.Respond(
-		response,
-		azure.WithErrorUnlessStatusCode(http.StatusOK),
-		autorest.ByUnmarshallingJSON(&result))
-	if err != nil {
-		klog.V(5).Infof("Received error in %s: resourceID: %s, error: %s", "managedcluster.get.respond", resourceID, err)
-		return result, retry.GetError(response, err)
-	}
-
-	result.Response = autorest.Response{Response: response}
-	return result, nil
-}
-
-// List gets a list of ManagedClusters in the resource group.
-func (c *Client) List(ctx context.Context, resourceGroupName string) ([]containerservice.ManagedCluster, *retry.Error) {
-	mc := metrics.NewMetricContext("managed_clusters", "list", resourceGroupName, c.subscriptionID, "")
-
-	// Report errors if the client is rate limited.
-	if !c.rateLimiterReader.TryAccept() {
-		mc.RateLimitedCount()
-		return nil, retry.GetRateLimitError(false, "ListManagedCluster")
-	}
-
-	// Report errors if the client is throttled.
-	if c.RetryAfterReader.After(time.Now()) {
-		mc.ThrottledCount()
-		rerr := retry.GetThrottlingError("ListManagedCluster", "client throttled", c.RetryAfterReader)
-		return nil, rerr
-	}
-
-	result, rerr := c.listManagedCluster(ctx, resourceGroupName)
-	mc.Observe(rerr)
-	if rerr != nil {
-		if rerr.IsThrottled() {
-			// Update RetryAfterReader so that no more requests would be sent until RetryAfter expires.
-			c.RetryAfterReader = rerr.RetryAfter
-		}
-
-		return result, rerr
-	}
-
-	return result, nil
-}
-
-// listManagedCluster gets a list of ManagedClusters in the resource group.
-func (c *Client) listManagedCluster(ctx context.Context, resourceGroupName string) ([]containerservice.ManagedCluster, *retry.Error) {
-	resourceID := armclient.GetResourceListID(c.subscriptionID, resourceGroupName, managedClusterResourceType)
-	result := make([]containerservice.ManagedCluster, 0)
-	page := &ManagedClusterResultPage{}
-	page.fn = c.listNextResults
-
-	resp, rerr := c.armClient.GetResource(ctx, resourceID)
-	defer c.armClient.CloseResponse(ctx, resp)
-	if rerr != nil {
-		klog.V(5).Infof("Received error in %s: resourceID: %s, error: %s", "managedcluster.list.request", resourceID, rerr.Error())
-		return result, rerr
-	}
-
-	var err error
-	page.mclr, err = c.listResponder(resp)
-	if err != nil {
-		klog.V(5).Infof("Received error in %s: resourceID: %s, error: %s", "managedcluster.list.respond", resourceID, err)
-		return result, retry.GetError(resp, err)
-	}
-
-	for {
-		result = append(result, page.Values()...)
-
-		// Abort the loop when there's no nextLink in the response.
-		if ptr.Deref(page.Response().NextLink, "") == "" {
-			break
-		}
-
-		if err = page.NextWithContext(ctx); err != nil {
-			klog.V(5).Infof("Received error in %s: resourceID: %s, error: %s", "managedcluster.list.next", resourceID, err)
-			return result, retry.GetError(page.Response().Response.Response, err)
-		}
-	}
-
-	return result, nil
-}
-
-func (c *Client) listResponder(resp *http.Response) (result containerservice.ManagedClusterListResult, err error) {
-	err = autorest.Respond(
-		resp,
-		autorest.ByIgnoring(),
-		azure.WithErrorUnlessStatusCode(http.StatusOK),
-		autorest.ByUnmarshallingJSON(&result))
-	result.Response = autorest.Response{Response: resp}
-	return
-}
-
-// managedClusterListResultPreparer prepares a request to retrieve the next set of results.
-// It returns nil if no more results exist.
-func (c *Client) managedClusterListResultPreparer(ctx context.Context, mclr containerservice.ManagedClusterListResult) (*http.Request, error) {
-	if mclr.NextLink == nil || len(ptr.Deref(mclr.NextLink, "")) < 1 {
-		return nil, nil
-	}
-
-	decorators := []autorest.PrepareDecorator{
-		autorest.WithBaseURL(ptr.Deref(mclr.NextLink, "")),
-	}
-	return c.armClient.PrepareGetRequest(ctx, decorators...)
-}
-
-// listNextResults retrieves the next set of results, if any.
-func (c *Client) listNextResults(ctx context.Context, lastResults containerservice.ManagedClusterListResult) (result containerservice.ManagedClusterListResult, err error) {
-	req, err := c.managedClusterListResultPreparer(ctx, lastResults)
-	if err != nil {
-		return result, autorest.NewErrorWithError(err, "managedclusterclient", "listNextResults", nil, "Failure preparing next results request")
-	}
-	if req == nil {
-		return
-	}
-
-	resp, rerr := c.armClient.Send(ctx, req)
-	defer c.armClient.CloseResponse(ctx, resp)
-	if rerr != nil {
-		result.Response = autorest.Response{Response: resp}
-		return result, autorest.NewErrorWithError(rerr.Error(), "managedclusterclient", "listNextResults", resp, "Failure sending next results request")
-	}
-
-	result, err = c.listResponder(resp)
-	if err != nil {
-		err = autorest.NewErrorWithError(err, "managedclusterclient", "listNextResults", resp, "Failure responding to next results request")
-	}
-
-	return
-}
-
-// ManagedClusterResultPage contains a page of ManagedCluster values.
-type ManagedClusterResultPage struct {
-	fn   func(context.Context, containerservice.ManagedClusterListResult) (containerservice.ManagedClusterListResult, error)
-	mclr containerservice.ManagedClusterListResult
-}
-
-// NextWithContext advances to the next page of values.  If there was an error making
-// the request the page does not advance and the error is returned.
-func (page *ManagedClusterResultPage) NextWithContext(ctx context.Context) (err error) {
-	next, err := page.fn(ctx, page.mclr)
-	if err != nil {
-		return err
-	}
-	page.mclr = next
-	return nil
-}
-
-// Next advances to the next page of values.  If there was an error making
-// the request the page does not advance and the error is returned.
-// Deprecated: Use NextWithContext() instead.
-func (page *ManagedClusterResultPage) Next() error {
-	return page.NextWithContext(context.Background())
-}
-
-// NotDone returns true if the page enumeration should be started or is not yet complete.
-func (page ManagedClusterResultPage) NotDone() bool {
-	return !page.mclr.IsEmpty()
-}
-
-// Response returns the raw server response from the last page request.
-func (page ManagedClusterResultPage) Response() containerservice.ManagedClusterListResult {
-	return page.mclr
-}
-
-// Values returns the slice of values for the current page or nil if there are no values.
-func (page ManagedClusterResultPage) Values() []containerservice.ManagedCluster {
-	if page.mclr.IsEmpty() {
-		return nil
-	}
-	return *page.mclr.Value
-}
-
-// CreateOrUpdate creates or updates a ManagedCluster.
-func (c *Client) CreateOrUpdate(ctx context.Context, resourceGroupName string, managedClusterName string, parameters containerservice.ManagedCluster, etag string) *retry.Error {
-	mc := metrics.NewMetricContext("managed_clusters", "create_or_update", resourceGroupName, c.subscriptionID, "")
-
-	// Report errors if the client is rate limited.
-	if !c.rateLimiterWriter.TryAccept() {
-		mc.RateLimitedCount()
-		return retry.GetRateLimitError(true, "CreateOrUpdateManagedCluster")
-	}
-
-	// Report errors if the client is throttled.
-	if c.RetryAfterWriter.After(time.Now()) {
-		mc.ThrottledCount()
-		rerr := retry.GetThrottlingError("CreateOrUpdateManagedCluster", "client throttled", c.RetryAfterWriter)
-		return rerr
-	}
-
-	rerr := c.createOrUpdateManagedCluster(ctx, resourceGroupName, managedClusterName, parameters, etag)
-	mc.Observe(rerr)
-	if rerr != nil {
-		if rerr.IsThrottled() {
-			// Update RetryAfterReader so that no more requests would be sent until RetryAfter expires.
-			c.RetryAfterWriter = rerr.RetryAfter
-		}
-
-		return rerr
-	}
-
-	return nil
-}
-
-// createOrUpdateManagedCluster creates or updates a ManagedCluster.
-func (c *Client) createOrUpdateManagedCluster(ctx context.Context, resourceGroupName string, managedClusterName string, parameters containerservice.ManagedCluster, etag string) *retry.Error {
-	resourceID := armclient.GetResourceID(
-		c.subscriptionID,
-		resourceGroupName,
-		managedClusterResourceType,
-		managedClusterName,
-	)
-	decorators := []autorest.PrepareDecorator{}
-	if etag != "" {
-		decorators = append(decorators, autorest.WithHeader("If-Match", autorest.String(etag)))
-	}
-
-	response, rerr := c.armClient.PutResource(ctx, resourceID, parameters, decorators...)
-	defer c.armClient.CloseResponse(ctx, response)
-	if rerr != nil {
-		klog.V(5).Infof("Received error in %s: resourceID: %s, error: %s", "managedCluster.put.request", resourceID, rerr.Error())
-		return rerr
-	}
-
-	if response != nil && response.StatusCode != http.StatusNoContent {
-		_, rerr = c.createOrUpdateResponder(response)
-		if rerr != nil {
-			klog.V(5).Infof("Received error in %s: resourceID: %s, error: %s", "managedCluster.put.respond", resourceID, rerr.Error())
-			return rerr
-		}
-	}
-
-	return nil
-}
-
-func (c *Client) createOrUpdateResponder(resp *http.Response) (*containerservice.ManagedCluster, *retry.Error) {
-	result := &containerservice.ManagedCluster{}
-	err := autorest.Respond(
-		resp,
-		azure.WithErrorUnlessStatusCode(http.StatusOK, http.StatusCreated),
-		autorest.ByUnmarshallingJSON(&result))
-	result.Response = autorest.Response{Response: resp}
-	return result, retry.GetError(resp, err)
-}
-
-// Delete deletes a ManagedCluster by name.
-func (c *Client) Delete(ctx context.Context, resourceGroupName string, managedClusterName string) *retry.Error {
-	mc := metrics.NewMetricContext("managed_clusters", "delete", resourceGroupName, c.subscriptionID, "")
-
-	// Report errors if the client is rate limited.
-	if !c.rateLimiterWriter.TryAccept() {
-		mc.RateLimitedCount()
-		return retry.GetRateLimitError(true, "DeleteManagedCluster")
-	}
-
-	// Report errors if the client is throttled.
-	if c.RetryAfterWriter.After(time.Now()) {
-		mc.ThrottledCount()
-		rerr := retry.GetThrottlingError("DeleteManagedCluster", "client throttled", c.RetryAfterWriter)
-		return rerr
-	}
-
-	rerr := c.deleteManagedCluster(ctx, resourceGroupName, managedClusterName)
-	mc.Observe(rerr)
-	if rerr != nil {
-		if rerr.IsThrottled() {
-			// Update RetryAfterReader so that no more requests would be sent until RetryAfter expires.
-			c.RetryAfterWriter = rerr.RetryAfter
-		}
-
-		return rerr
-	}
-
-	return nil
-}
-
-// deleteManagedCluster deletes a ManagedCluster by name.
-func (c *Client) deleteManagedCluster(ctx context.Context, resourceGroupName string, managedClusterName string) *retry.Error {
-	resourceID := armclient.GetResourceID(
-		c.subscriptionID,
-		resourceGroupName,
-		managedClusterResourceType,
-		managedClusterName,
-	)
-
-	return c.armClient.DeleteResource(ctx, resourceID)
-}
diff --git a/pkg/azureclients/containerserviceclient/doc.go b/pkg/azureclients/containerserviceclient/doc.go
deleted file mode 100644
index 9af55eb66..000000000
--- a/pkg/azureclients/containerserviceclient/doc.go
+++ /dev/null
@@ -1,18 +0,0 @@
-/*
-Copyright 2020 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-// Package containerserviceclient implements the client for azure container service.
-package containerserviceclient // import "sigs.k8s.io/cloud-provider-azure/pkg/azureclients/containerserviceclient"
diff --git a/pkg/azureclients/containerserviceclient/interface.go b/pkg/azureclients/containerserviceclient/interface.go
deleted file mode 100644
index 477b6b4fb..000000000
--- a/pkg/azureclients/containerserviceclient/interface.go
+++ /dev/null
@@ -1,39 +0,0 @@
-/*
-Copyright 2020 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package containerserviceclient
-
-import (
-	"context"
-
-	"github.com/Azure/azure-sdk-for-go/services/containerservice/mgmt/2021-10-01/containerservice"
-
-	"sigs.k8s.io/cloud-provider-azure/pkg/retry"
-)
-
-const (
-	// APIVersion is the API version for containerservice.
-	APIVersion = "2021-10-01"
-)
-
-// Interface is the client interface for ContainerService.
-// Don't forget to run "hack/update-mock-clients.sh" command to generate the mock client.
-type Interface interface {
-	CreateOrUpdate(ctx context.Context, resourceGroupName string, managedClusterName string, parameters containerservice.ManagedCluster, etag string) *retry.Error
-	Delete(ctx context.Context, resourceGroupName string, managedClusterName string) *retry.Error
-	Get(ctx context.Context, resourceGroupName string, managedClusterName string) (containerservice.ManagedCluster, *retry.Error)
-	List(ctx context.Context, resourceGroupName string) ([]containerservice.ManagedCluster, *retry.Error)
-}
diff --git a/pkg/azureclients/containerserviceclient/mockcontainerserviceclient/doc.go b/pkg/azureclients/containerserviceclient/mockcontainerserviceclient/doc.go
deleted file mode 100644
index b9fa86d42..000000000
--- a/pkg/azureclients/containerserviceclient/mockcontainerserviceclient/doc.go
+++ /dev/null
@@ -1,18 +0,0 @@
-/*
-Copyright 2020 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-// Package mockcontainerserviceclient implements the mock client for azure container service.
-package mockcontainerserviceclient // import "sigs.k8s.io/cloud-provider-azure/pkg/azureclients/containerserviceclient/mockcontainerserviceclient"
diff --git a/pkg/azureclients/containerserviceclient/mockcontainerserviceclient/interface.go b/pkg/azureclients/containerserviceclient/mockcontainerserviceclient/interface.go
deleted file mode 100644
index 51765fa85..000000000
--- a/pkg/azureclients/containerserviceclient/mockcontainerserviceclient/interface.go
+++ /dev/null
@@ -1,117 +0,0 @@
-// /*
-// Copyright The Kubernetes Authors.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-// */
-//
-
-// Code generated by MockGen. DO NOT EDIT.
-// Source: pkg/azureclients/containerserviceclient/interface.go
-//
-// Generated by this command:
-//
-//	mockgen -copyright_file=/home/runner/work/cloud-provider-azure/cloud-provider-azure/hack/boilerplate/boilerplate.generatego.txt -source=pkg/azureclients/containerserviceclient/interface.go -package=mockcontainerserviceclient Interface
-//
-
-// Package mockcontainerserviceclient is a generated GoMock package.
-package mockcontainerserviceclient
-
-import (
-	context "context"
-	reflect "reflect"
-
-	containerservice "github.com/Azure/azure-sdk-for-go/services/containerservice/mgmt/2021-10-01/containerservice"
-	gomock "go.uber.org/mock/gomock"
-	retry "sigs.k8s.io/cloud-provider-azure/pkg/retry"
-)
-
-// MockInterface is a mock of Interface interface.
-type MockInterface struct {
-	ctrl     *gomock.Controller
-	recorder *MockInterfaceMockRecorder
-}
-
-// MockInterfaceMockRecorder is the mock recorder for MockInterface.
-type MockInterfaceMockRecorder struct {
-	mock *MockInterface
-}
-
-// NewMockInterface creates a new mock instance.
-func NewMockInterface(ctrl *gomock.Controller) *MockInterface {
-	mock := &MockInterface{ctrl: ctrl}
-	mock.recorder = &MockInterfaceMockRecorder{mock}
-	return mock
-}
-
-// EXPECT returns an object that allows the caller to indicate expected use.
-func (m *MockInterface) EXPECT() *MockInterfaceMockRecorder {
-	return m.recorder
-}
-
-// CreateOrUpdate mocks base method.
-func (m *MockInterface) CreateOrUpdate(ctx context.Context, resourceGroupName, managedClusterName string, parameters containerservice.ManagedCluster, etag string) *retry.Error {
-	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "CreateOrUpdate", ctx, resourceGroupName, managedClusterName, parameters, etag)
-	ret0, _ := ret[0].(*retry.Error)
-	return ret0
-}
-
-// CreateOrUpdate indicates an expected call of CreateOrUpdate.
-func (mr *MockInterfaceMockRecorder) CreateOrUpdate(ctx, resourceGroupName, managedClusterName, parameters, etag any) *gomock.Call {
-	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CreateOrUpdate", reflect.TypeOf((*MockInterface)(nil).CreateOrUpdate), ctx, resourceGroupName, managedClusterName, parameters, etag)
-}
-
-// Delete mocks base method.
-func (m *MockInterface) Delete(ctx context.Context, resourceGroupName, managedClusterName string) *retry.Error {
-	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "Delete", ctx, resourceGroupName, managedClusterName)
-	ret0, _ := ret[0].(*retry.Error)
-	return ret0
-}
-
-// Delete indicates an expected call of Delete.
-func (mr *MockInterfaceMockRecorder) Delete(ctx, resourceGroupName, managedClusterName any) *gomock.Call {
-	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Delete", reflect.TypeOf((*MockInterface)(nil).Delete), ctx, resourceGroupName, managedClusterName)
-}
-
-// Get mocks base method.
-func (m *MockInterface) Get(ctx context.Context, resourceGroupName, managedClusterName string) (containerservice.ManagedCluster, *retry.Error) {
-	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "Get", ctx, resourceGroupName, managedClusterName)
-	ret0, _ := ret[0].(containerservice.ManagedCluster)
-	ret1, _ := ret[1].(*retry.Error)
-	return ret0, ret1
-}
-
-// Get indicates an expected call of Get.
-func (mr *MockInterfaceMockRecorder) Get(ctx, resourceGroupName, managedClusterName any) *gomock.Call {
-	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Get", reflect.TypeOf((*MockInterface)(nil).Get), ctx, resourceGroupName, managedClusterName)
-}
-
-// List mocks base method.
-func (m *MockInterface) List(ctx context.Context, resourceGroupName string) ([]containerservice.ManagedCluster, *retry.Error) {
-	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "List", ctx, resourceGroupName)
-	ret0, _ := ret[0].([]containerservice.ManagedCluster)
-	ret1, _ := ret[1].(*retry.Error)
-	return ret0, ret1
-}
-
-// List indicates an expected call of List.
-func (mr *MockInterfaceMockRecorder) List(ctx, resourceGroupName any) *gomock.Call {
-	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "List", reflect.TypeOf((*MockInterface)(nil).List), ctx, resourceGroupName)
-}
diff --git a/pkg/azureclients/routeclient/azure_routeclient.go b/pkg/azureclients/routeclient/azure_routeclient.go
deleted file mode 100644
index 1e63b53b7..000000000
--- a/pkg/azureclients/routeclient/azure_routeclient.go
+++ /dev/null
@@ -1,205 +0,0 @@
-/*
-Copyright 2020 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package routeclient
-
-import (
-	"context"
-	"net/http"
-	"strings"
-	"time"
-
-	"github.com/Azure/azure-sdk-for-go/services/network/mgmt/2022-07-01/network"
-	"github.com/Azure/go-autorest/autorest"
-	"github.com/Azure/go-autorest/autorest/azure"
-
-	"k8s.io/client-go/util/flowcontrol"
-	"k8s.io/klog/v2"
-
-	azclients "sigs.k8s.io/cloud-provider-azure/pkg/azureclients"
-	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/armclient"
-	"sigs.k8s.io/cloud-provider-azure/pkg/metrics"
-	"sigs.k8s.io/cloud-provider-azure/pkg/retry"
-)
-
-var _ Interface = &Client{}
-
-const routeTablesResourceType = "Microsoft.Network/routeTables"
-
-// Client implements Route client Interface.
-type Client struct {
-	armClient      armclient.Interface
-	subscriptionID string
-	cloudName      string
-
-	// Rate limiting configures.
-	rateLimiterReader flowcontrol.RateLimiter
-	rateLimiterWriter flowcontrol.RateLimiter
-
-	// ARM throttling configures.
-	RetryAfterReader time.Time
-	RetryAfterWriter time.Time
-}
-
-// New creates a new Route client with ratelimiting.
-func New(config *azclients.ClientConfig) *Client {
-	baseURI := config.ResourceManagerEndpoint
-	authorizer := config.Authorizer
-	apiVersion := APIVersion
-	if strings.EqualFold(config.CloudName, AzureStackCloudName) && !config.DisableAzureStackCloud {
-		apiVersion = AzureStackCloudAPIVersion
-	}
-	armClient := armclient.New(authorizer, *config, baseURI, apiVersion)
-	rateLimiterReader, rateLimiterWriter := azclients.NewRateLimiter(config.RateLimitConfig)
-
-	if azclients.RateLimitEnabled(config.RateLimitConfig) {
-		klog.V(2).Infof("Azure RoutesClient (read ops) using rate limit config: QPS=%g, bucket=%d",
-			config.RateLimitConfig.CloudProviderRateLimitQPS,
-			config.RateLimitConfig.CloudProviderRateLimitBucket)
-		klog.V(2).Infof("Azure RoutesClient (write ops) using rate limit config: QPS=%g, bucket=%d",
-			config.RateLimitConfig.CloudProviderRateLimitQPSWrite,
-			config.RateLimitConfig.CloudProviderRateLimitBucketWrite)
-	}
-
-	client := &Client{
-		armClient:         armClient,
-		rateLimiterReader: rateLimiterReader,
-		rateLimiterWriter: rateLimiterWriter,
-		subscriptionID:    config.SubscriptionID,
-		cloudName:         config.CloudName,
-	}
-
-	return client
-}
-
-// CreateOrUpdate creates or updates a Route.
-func (c *Client) CreateOrUpdate(ctx context.Context, resourceGroupName string, routeTableName string, routeName string, routeParameters network.Route, etag string) *retry.Error {
-	mc := metrics.NewMetricContext("routes", "create_or_update", resourceGroupName, c.subscriptionID, "")
-
-	// Report errors if the client is rate limited.
-	if !c.rateLimiterWriter.TryAccept() {
-		mc.RateLimitedCount()
-		return retry.GetRateLimitError(true, "RouteCreateOrUpdate")
-	}
-
-	// Report errors if the client is throttled.
-	if c.RetryAfterWriter.After(time.Now()) {
-		mc.ThrottledCount()
-		rerr := retry.GetThrottlingError("RouteCreateOrUpdate", "client throttled", c.RetryAfterWriter)
-		return rerr
-	}
-
-	rerr := c.createOrUpdateRoute(ctx, resourceGroupName, routeTableName, routeName, routeParameters, etag)
-	mc.Observe(rerr)
-	if rerr != nil {
-		if rerr.IsThrottled() {
-			// Update RetryAfterReader so that no more requests would be sent until RetryAfter expires.
-			c.RetryAfterWriter = rerr.RetryAfter
-		}
-
-		return rerr
-	}
-
-	return nil
-}
-
-// createOrUpdateRoute creates or updates a Route.
-func (c *Client) createOrUpdateRoute(ctx context.Context, resourceGroupName string, routeTableName string, routeName string, routeParameters network.Route, etag string) *retry.Error {
-	resourceID := armclient.GetChildResourceID(
-		c.subscriptionID,
-		resourceGroupName,
-		routeTablesResourceType,
-		routeTableName,
-		"routes",
-		routeName,
-	)
-	decorators := []autorest.PrepareDecorator{}
-	if etag != "" {
-		decorators = append(decorators, autorest.WithHeader("If-Match", autorest.String(etag)))
-	}
-
-	response, rerr := c.armClient.PutResource(ctx, resourceID, routeParameters, decorators...)
-	defer c.armClient.CloseResponse(ctx, response)
-	if rerr != nil {
-		klog.V(5).Infof("Received error in %s: resourceID: %s, error: %s", "route.put.request", resourceID, rerr.Error())
-		return rerr
-	}
-
-	if response != nil && response.StatusCode != http.StatusNoContent {
-		_, rerr = c.createOrUpdateResponder(response)
-		if rerr != nil {
-			klog.V(5).Infof("Received error in %s: resourceID: %s, error: %s", "route.put.respond", resourceID, rerr.Error())
-			return rerr
-		}
-	}
-
-	return nil
-}
-
-func (c *Client) createOrUpdateResponder(resp *http.Response) (*network.Route, *retry.Error) {
-	result := &network.Route{}
-	err := autorest.Respond(
-		resp,
-		azure.WithErrorUnlessStatusCode(http.StatusOK, http.StatusCreated),
-		autorest.ByUnmarshallingJSON(&result))
-	result.Response = autorest.Response{Response: resp}
-	return result, retry.GetError(resp, err)
-}
-
-// Delete deletes a Route by name.
-func (c *Client) Delete(ctx context.Context, resourceGroupName string, routeTableName string, routeName string) *retry.Error {
-	mc := metrics.NewMetricContext("routes", "delete", resourceGroupName, c.subscriptionID, "")
-
-	// Report errors if the client is rate limited.
-	if !c.rateLimiterWriter.TryAccept() {
-		mc.RateLimitedCount()
-		return retry.GetRateLimitError(true, "RouteDelete")
-	}
-
-	// Report errors if the client is throttled.
-	if c.RetryAfterWriter.After(time.Now()) {
-		mc.ThrottledCount()
-		rerr := retry.GetThrottlingError("RouteDelete", "client throttled", c.RetryAfterWriter)
-		return rerr
-	}
-
-	rerr := c.deleteRoute(ctx, resourceGroupName, routeTableName, routeName)
-	mc.Observe(rerr)
-	if rerr != nil {
-		if rerr.IsThrottled() {
-			// Update RetryAfterReader so that no more requests would be sent until RetryAfter expires.
-			c.RetryAfterWriter = rerr.RetryAfter
-		}
-
-		return rerr
-	}
-
-	return nil
-}
-
-// deleteRoute deletes a Route by name.
-func (c *Client) deleteRoute(ctx context.Context, resourceGroupName string, routeTableName string, routeName string) *retry.Error {
-	resourceID := armclient.GetChildResourceID(
-		c.subscriptionID,
-		resourceGroupName,
-		routeTablesResourceType,
-		routeTableName,
-		"routes",
-		routeName,
-	)
-
-	return c.armClient.DeleteResource(ctx, resourceID)
-}
diff --git a/pkg/azureclients/routeclient/doc.go b/pkg/azureclients/routeclient/doc.go
deleted file mode 100644
index 18e39cd10..000000000
--- a/pkg/azureclients/routeclient/doc.go
+++ /dev/null
@@ -1,18 +0,0 @@
-/*
-Copyright 2020 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-// Package routeclient implements the client for Route.
-package routeclient // import "sigs.k8s.io/cloud-provider-azure/pkg/azureclients/routeclient"
diff --git a/pkg/azureclients/routeclient/interface.go b/pkg/azureclients/routeclient/interface.go
deleted file mode 100644
index 163436ec1..000000000
--- a/pkg/azureclients/routeclient/interface.go
+++ /dev/null
@@ -1,45 +0,0 @@
-/*
-Copyright 2020 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package routeclient
-
-import (
-	"context"
-
-	"github.com/Azure/azure-sdk-for-go/services/network/mgmt/2022-07-01/network"
-
-	"sigs.k8s.io/cloud-provider-azure/pkg/retry"
-)
-
-const (
-	// APIVersion is the API version for network.
-	APIVersion = "2022-07-01"
-	// AzureStackCloudAPIVersion is the API version for Azure Stack
-	AzureStackCloudAPIVersion = "2018-11-01"
-	// AzureStackCloudName is the cloud name of Azure Stack
-	AzureStackCloudName = "AZURESTACKCLOUD"
-)
-
-// Interface is the client interface for Route.
-// Don't forget to run the following command to generate the mock client:
-// Don't forget to run "hack/update-mock-clients.sh" command to generate the mock client.
-type Interface interface {
-	// CreateOrUpdate creates or updates a Route.
-	CreateOrUpdate(ctx context.Context, resourceGroupName string, routeTableName string, routeName string, routeParameters network.Route, etag string) *retry.Error
-
-	// Delete deletes a Route by name.
-	Delete(ctx context.Context, resourceGroupName string, routeTableName string, routeName string) *retry.Error
-}
diff --git a/pkg/azureclients/routeclient/mockrouteclient/interface.go b/pkg/azureclients/routeclient/mockrouteclient/interface.go
deleted file mode 100644
index df9ef1351..000000000
--- a/pkg/azureclients/routeclient/mockrouteclient/interface.go
+++ /dev/null
@@ -1,87 +0,0 @@
-// /*
-// Copyright The Kubernetes Authors.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-// */
-//
-
-// Code generated by MockGen. DO NOT EDIT.
-// Source: pkg/azureclients/routeclient/interface.go
-//
-// Generated by this command:
-//
-//	mockgen -copyright_file=/home/runner/work/cloud-provider-azure/cloud-provider-azure/hack/boilerplate/boilerplate.generatego.txt -source=pkg/azureclients/routeclient/interface.go -package=mockrouteclient Interface
-//
-
-// Package mockrouteclient is a generated GoMock package.
-package mockrouteclient
-
-import (
-	context "context"
-	reflect "reflect"
-
-	network "github.com/Azure/azure-sdk-for-go/services/network/mgmt/2022-07-01/network"
-	gomock "go.uber.org/mock/gomock"
-	retry "sigs.k8s.io/cloud-provider-azure/pkg/retry"
-)
-
-// MockInterface is a mock of Interface interface.
-type MockInterface struct {
-	ctrl     *gomock.Controller
-	recorder *MockInterfaceMockRecorder
-}
-
-// MockInterfaceMockRecorder is the mock recorder for MockInterface.
-type MockInterfaceMockRecorder struct {
-	mock *MockInterface
-}
-
-// NewMockInterface creates a new mock instance.
-func NewMockInterface(ctrl *gomock.Controller) *MockInterface {
-	mock := &MockInterface{ctrl: ctrl}
-	mock.recorder = &MockInterfaceMockRecorder{mock}
-	return mock
-}
-
-// EXPECT returns an object that allows the caller to indicate expected use.
-func (m *MockInterface) EXPECT() *MockInterfaceMockRecorder {
-	return m.recorder
-}
-
-// CreateOrUpdate mocks base method.
-func (m *MockInterface) CreateOrUpdate(ctx context.Context, resourceGroupName, routeTableName, routeName string, routeParameters network.Route, etag string) *retry.Error {
-	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "CreateOrUpdate", ctx, resourceGroupName, routeTableName, routeName, routeParameters, etag)
-	ret0, _ := ret[0].(*retry.Error)
-	return ret0
-}
-
-// CreateOrUpdate indicates an expected call of CreateOrUpdate.
-func (mr *MockInterfaceMockRecorder) CreateOrUpdate(ctx, resourceGroupName, routeTableName, routeName, routeParameters, etag any) *gomock.Call {
-	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CreateOrUpdate", reflect.TypeOf((*MockInterface)(nil).CreateOrUpdate), ctx, resourceGroupName, routeTableName, routeName, routeParameters, etag)
-}
-
-// Delete mocks base method.
-func (m *MockInterface) Delete(ctx context.Context, resourceGroupName, routeTableName, routeName string) *retry.Error {
-	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "Delete", ctx, resourceGroupName, routeTableName, routeName)
-	ret0, _ := ret[0].(*retry.Error)
-	return ret0
-}
-
-// Delete indicates an expected call of Delete.
-func (mr *MockInterfaceMockRecorder) Delete(ctx, resourceGroupName, routeTableName, routeName any) *gomock.Call {
-	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Delete", reflect.TypeOf((*MockInterface)(nil).Delete), ctx, resourceGroupName, routeTableName, routeName)
-}
diff --git a/pkg/azureclients/snapshotclient/azure_snapshotclient.go b/pkg/azureclients/snapshotclient/azure_snapshotclient.go
deleted file mode 100644
index fcad9e6b7..000000000
--- a/pkg/azureclients/snapshotclient/azure_snapshotclient.go
+++ /dev/null
@@ -1,433 +0,0 @@
-/*
-Copyright 2020 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package snapshotclient
-
-import (
-	"context"
-	"net/http"
-	"strings"
-	"time"
-
-	"github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2022-08-01/compute"
-	"github.com/Azure/go-autorest/autorest"
-	"github.com/Azure/go-autorest/autorest/azure"
-
-	"k8s.io/client-go/util/flowcontrol"
-	"k8s.io/klog/v2"
-	"k8s.io/utils/ptr"
-
-	azclients "sigs.k8s.io/cloud-provider-azure/pkg/azureclients"
-	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/armclient"
-	"sigs.k8s.io/cloud-provider-azure/pkg/metrics"
-	"sigs.k8s.io/cloud-provider-azure/pkg/retry"
-)
-
-var _ Interface = &Client{}
-
-const snapshotsResourceType = "Microsoft.Compute/snapshots"
-
-// Client implements Snapshot client Interface.
-type Client struct {
-	armClient      armclient.Interface
-	subscriptionID string
-	cloudName      string
-
-	// Rate limiting configures.
-	rateLimiterReader flowcontrol.RateLimiter
-	rateLimiterWriter flowcontrol.RateLimiter
-
-	// ARM throttling configures.
-	RetryAfterReader time.Time
-	RetryAfterWriter time.Time
-}
-
-// New creates a new Snapshot client with ratelimiting.
-func New(config *azclients.ClientConfig) *Client {
-	baseURI := config.ResourceManagerEndpoint
-	authorizer := config.Authorizer
-	apiVersion := APIVersion
-	if strings.EqualFold(config.CloudName, AzureStackCloudName) && !config.DisableAzureStackCloud {
-		apiVersion = AzureStackCloudAPIVersion
-	}
-	armClient := armclient.New(authorizer, *config, baseURI, apiVersion)
-	rateLimiterReader, rateLimiterWriter := azclients.NewRateLimiter(config.RateLimitConfig)
-
-	if azclients.RateLimitEnabled(config.RateLimitConfig) {
-		klog.V(2).Infof("Azure SnapshotClient (read ops) using rate limit config: QPS=%g, bucket=%d",
-			config.RateLimitConfig.CloudProviderRateLimitQPS,
-			config.RateLimitConfig.CloudProviderRateLimitBucket)
-		klog.V(2).Infof("Azure SnapshotClient (write ops) using rate limit config: QPS=%g, bucket=%d",
-			config.RateLimitConfig.CloudProviderRateLimitQPSWrite,
-			config.RateLimitConfig.CloudProviderRateLimitBucketWrite)
-	}
-
-	client := &Client{
-		armClient:         armClient,
-		rateLimiterReader: rateLimiterReader,
-		rateLimiterWriter: rateLimiterWriter,
-		subscriptionID:    config.SubscriptionID,
-		cloudName:         config.CloudName,
-	}
-
-	return client
-}
-
-// Get gets a Snapshot.
-func (c *Client) Get(ctx context.Context, subsID, resourceGroupName, snapshotName string) (compute.Snapshot, *retry.Error) {
-	if subsID == "" {
-		subsID = c.subscriptionID
-	}
-	mc := metrics.NewMetricContext("snapshot", "get", resourceGroupName, subsID, "")
-
-	// Report errors if the client is rate limited.
-	if !c.rateLimiterReader.TryAccept() {
-		mc.RateLimitedCount()
-		return compute.Snapshot{}, retry.GetRateLimitError(false, "SnapshotGet")
-	}
-
-	// Report errors if the client is throttled.
-	if c.RetryAfterReader.After(time.Now()) {
-		mc.ThrottledCount()
-		rerr := retry.GetThrottlingError("SnapshotGet", "client throttled", c.RetryAfterReader)
-		return compute.Snapshot{}, rerr
-	}
-
-	result, rerr := c.getSnapshot(ctx, subsID, resourceGroupName, snapshotName)
-	mc.Observe(rerr)
-	if rerr != nil {
-		if rerr.IsThrottled() {
-			// Update RetryAfterReader so that no more requests would be sent until RetryAfter expires.
-			c.RetryAfterReader = rerr.RetryAfter
-		}
-
-		return result, rerr
-	}
-
-	return result, nil
-}
-
-// getSnapshot gets a Snapshot.
-func (c *Client) getSnapshot(ctx context.Context, subsID, resourceGroupName, snapshotName string) (compute.Snapshot, *retry.Error) {
-	resourceID := armclient.GetResourceID(
-		subsID,
-		resourceGroupName,
-		snapshotsResourceType,
-		snapshotName,
-	)
-	result := compute.Snapshot{}
-
-	response, rerr := c.armClient.GetResource(ctx, resourceID)
-	defer c.armClient.CloseResponse(ctx, response)
-	if rerr != nil {
-		klog.V(5).Infof("Received error in %s: resourceID: %s, error: %s", "snapshot.get.request", resourceID, rerr.Error())
-		return result, rerr
-	}
-
-	err := autorest.Respond(
-		response,
-		azure.WithErrorUnlessStatusCode(http.StatusOK),
-		autorest.ByUnmarshallingJSON(&result))
-	if err != nil {
-		klog.V(5).Infof("Received error in %s: resourceID: %s, error: %s", "snapshot.get.respond", resourceID, err)
-		return result, retry.GetError(response, err)
-	}
-
-	result.Response = autorest.Response{Response: response}
-	return result, nil
-}
-
-// Delete deletes a Snapshot by name.
-func (c *Client) Delete(ctx context.Context, subsID, resourceGroupName, snapshotName string) *retry.Error {
-	if subsID == "" {
-		subsID = c.subscriptionID
-	}
-	mc := metrics.NewMetricContext("snapshot", "delete", resourceGroupName, subsID, "")
-
-	// Report errors if the client is rate limited.
-	if !c.rateLimiterWriter.TryAccept() {
-		mc.RateLimitedCount()
-		return retry.GetRateLimitError(true, "SnapshotDelete")
-	}
-
-	// Report errors if the client is throttled.
-	if c.RetryAfterWriter.After(time.Now()) {
-		mc.ThrottledCount()
-		rerr := retry.GetThrottlingError("SnapshotDelete", "client throttled", c.RetryAfterWriter)
-		return rerr
-	}
-
-	rerr := c.deleteSnapshot(ctx, subsID, resourceGroupName, snapshotName)
-	mc.Observe(rerr)
-	if rerr != nil {
-		if rerr.IsThrottled() {
-			// Update RetryAfterReader so that no more requests would be sent until RetryAfter expires.
-			c.RetryAfterWriter = rerr.RetryAfter
-		}
-
-		return rerr
-	}
-
-	return nil
-}
-
-// deleteSnapshot deletes a PublicIPAddress by name.
-func (c *Client) deleteSnapshot(ctx context.Context, subsID, resourceGroupName, snapshotName string) *retry.Error {
-	resourceID := armclient.GetResourceID(
-		subsID,
-		resourceGroupName,
-		snapshotsResourceType,
-		snapshotName,
-	)
-
-	return c.armClient.DeleteResource(ctx, resourceID)
-}
-
-// CreateOrUpdate creates or updates a Snapshot.
-func (c *Client) CreateOrUpdate(ctx context.Context, subsID, resourceGroupName, snapshotName string, snapshot compute.Snapshot) *retry.Error {
-	if subsID == "" {
-		subsID = c.subscriptionID
-	}
-	mc := metrics.NewMetricContext("snapshot", "create_or_update", resourceGroupName, subsID, "")
-
-	// Report errors if the client is rate limited.
-	if !c.rateLimiterWriter.TryAccept() {
-		mc.RateLimitedCount()
-		return retry.GetRateLimitError(true, "SnapshotCreateOrUpdate")
-	}
-
-	// Report errors if the client is throttled.
-	if c.RetryAfterWriter.After(time.Now()) {
-		mc.ThrottledCount()
-		rerr := retry.GetThrottlingError("SnapshotCreateOrUpdate", "client throttled", c.RetryAfterWriter)
-		return rerr
-	}
-
-	rerr := c.createOrUpdateSnapshot(ctx, subsID, resourceGroupName, snapshotName, snapshot)
-	mc.Observe(rerr)
-	if rerr != nil {
-		if rerr.IsThrottled() {
-			// Update RetryAfterReader so that no more requests would be sent until RetryAfter expires.
-			c.RetryAfterWriter = rerr.RetryAfter
-		}
-
-		return rerr
-	}
-
-	return nil
-}
-
-// createOrUpdateSnapshot creates or updates a Snapshot.
-func (c *Client) createOrUpdateSnapshot(ctx context.Context, subsID, resourceGroupName, snapshotName string, snapshot compute.Snapshot) *retry.Error {
-	resourceID := armclient.GetResourceID(
-		subsID,
-		resourceGroupName,
-		snapshotsResourceType,
-		snapshotName,
-	)
-
-	response, rerr := c.armClient.PutResource(ctx, resourceID, snapshot)
-	defer c.armClient.CloseResponse(ctx, response)
-	if rerr != nil {
-		klog.V(5).Infof("Received error in %s: resourceID: %s, error: %s", "snapshot.put.request", resourceID, rerr.Error())
-		return rerr
-	}
-
-	if response != nil && response.StatusCode != http.StatusNoContent {
-		_, rerr = c.createOrUpdateResponder(response)
-		if rerr != nil {
-			klog.V(5).Infof("Received error in %s: resourceID: %s, error: %s", "snapshot.put.respond", resourceID, rerr.Error())
-			return rerr
-		}
-	}
-
-	return nil
-}
-
-func (c *Client) createOrUpdateResponder(resp *http.Response) (*compute.Snapshot, *retry.Error) {
-	result := &compute.Snapshot{}
-	err := autorest.Respond(
-		resp,
-		azure.WithErrorUnlessStatusCode(http.StatusOK, http.StatusCreated),
-		autorest.ByUnmarshallingJSON(&result))
-	result.Response = autorest.Response{Response: resp}
-	return result, retry.GetError(resp, err)
-}
-
-// ListByResourceGroup get a list snapshots by resourceGroup.
-func (c *Client) ListByResourceGroup(ctx context.Context, subsID, resourceGroupName string) ([]compute.Snapshot, *retry.Error) {
-	if subsID == "" {
-		subsID = c.subscriptionID
-	}
-	mc := metrics.NewMetricContext("snapshot", "list_by_resource_group", resourceGroupName, subsID, "")
-
-	// Report errors if the client is rate limited.
-	if !c.rateLimiterReader.TryAccept() {
-		mc.RateLimitedCount()
-		return nil, retry.GetRateLimitError(false, "SnapshotListByResourceGroup")
-	}
-
-	// Report errors if the client is throttled.
-	if c.RetryAfterReader.After(time.Now()) {
-		mc.ThrottledCount()
-		rerr := retry.GetThrottlingError("SnapshotListByResourceGroup", "client throttled", c.RetryAfterReader)
-		return nil, rerr
-	}
-
-	result, rerr := c.listSnapshotsByResourceGroup(ctx, subsID, resourceGroupName)
-	mc.Observe(rerr)
-	if rerr != nil {
-		if rerr.IsThrottled() {
-			// Update RetryAfterReader so that no more requests would be sent until RetryAfter expires.
-			c.RetryAfterReader = rerr.RetryAfter
-		}
-
-		return result, rerr
-	}
-
-	return result, nil
-}
-
-// listSnapshotsByResourceGroup gets a list of snapshots in the resource group.
-func (c *Client) listSnapshotsByResourceGroup(ctx context.Context, subsID, resourceGroupName string) ([]compute.Snapshot, *retry.Error) {
-	if subsID == "" {
-		subsID = c.subscriptionID
-	}
-	resourceID := armclient.GetResourceListID(subsID, resourceGroupName, snapshotsResourceType)
-	result := make([]compute.Snapshot, 0)
-	page := &SnapshotListPage{}
-	page.fn = c.listNextResults
-
-	resp, rerr := c.armClient.GetResource(ctx, resourceID)
-	defer c.armClient.CloseResponse(ctx, resp)
-	if rerr != nil {
-		klog.V(5).Infof("Received error in %s: resourceID: %s, error: %s", "snapshot.list.request", resourceID, rerr.Error())
-		return result, rerr
-	}
-
-	var err error
-	page.sl, err = c.listResponder(resp)
-	if err != nil {
-		klog.V(5).Infof("Received error in %s: resourceID: %s, error: %s", "snapshot.list.respond", resourceID, err)
-		return result, retry.GetError(resp, err)
-	}
-
-	for {
-		result = append(result, page.Values()...)
-
-		// Abort the loop when there's no nextLink in the response.
-		if ptr.Deref(page.Response().NextLink, "") == "" {
-			break
-		}
-
-		if err = page.NextWithContext(ctx); err != nil {
-			klog.V(5).Infof("Received error in %s: resourceID: %s, error: %s", "snapshot.list.next", resourceID, err)
-			return result, retry.GetError(page.Response().Response.Response, err)
-		}
-	}
-
-	return result, nil
-}
-
-func (c *Client) listResponder(resp *http.Response) (result compute.SnapshotList, err error) {
-	err = autorest.Respond(
-		resp,
-		autorest.ByIgnoring(),
-		azure.WithErrorUnlessStatusCode(http.StatusOK),
-		autorest.ByUnmarshallingJSON(&result))
-	result.Response = autorest.Response{Response: resp}
-	return
-}
-
-// SnapshotListResultPreparer prepares a request to retrieve the next set of results.
-// It returns nil if no more results exist.
-func (c *Client) SnapshotListResultPreparer(ctx context.Context, lr compute.SnapshotList) (*http.Request, error) {
-	if lr.NextLink == nil || len(ptr.Deref(lr.NextLink, "")) < 1 {
-		return nil, nil
-	}
-
-	decorators := []autorest.PrepareDecorator{
-		autorest.WithBaseURL(ptr.Deref(lr.NextLink, "")),
-	}
-	return c.armClient.PrepareGetRequest(ctx, decorators...)
-}
-
-// listNextResults retrieves the next set of results, if any.
-func (c *Client) listNextResults(ctx context.Context, lastResults compute.SnapshotList) (result compute.SnapshotList, err error) {
-	req, err := c.SnapshotListResultPreparer(ctx, lastResults)
-	if err != nil {
-		return result, autorest.NewErrorWithError(err, "snapshotclient", "listNextResults", nil, "Failure preparing next results request")
-	}
-	if req == nil {
-		return
-	}
-
-	resp, rerr := c.armClient.Send(ctx, req)
-	defer c.armClient.CloseResponse(ctx, resp)
-	if rerr != nil {
-		result.Response = autorest.Response{Response: resp}
-		return result, autorest.NewErrorWithError(rerr.Error(), "snapshotclient", "listNextResults", resp, "Failure sending next results request")
-	}
-
-	result, err = c.listResponder(resp)
-	if err != nil {
-		err = autorest.NewErrorWithError(err, "snapshotclient", "listNextResults", resp, "Failure responding to next results request")
-	}
-
-	return
-}
-
-// SnapshotListPage contains a page of Snapshot values.
-type SnapshotListPage struct {
-	fn func(context.Context, compute.SnapshotList) (compute.SnapshotList, error)
-	sl compute.SnapshotList
-}
-
-// NextWithContext advances to the next page of values.  If there was an error making
-// the request the page does not advance and the error is returned.
-func (page *SnapshotListPage) NextWithContext(ctx context.Context) (err error) {
-	next, err := page.fn(ctx, page.sl)
-	if err != nil {
-		return err
-	}
-	page.sl = next
-	return nil
-}
-
-// Next advances to the next page of values.  If there was an error making
-// the request the page does not advance and the error is returned.
-// Deprecated: Use NextWithContext() instead.
-func (page *SnapshotListPage) Next() error {
-	return page.NextWithContext(context.Background())
-}
-
-// NotDone returns true if the page enumeration should be started or is not yet complete.
-func (page SnapshotListPage) NotDone() bool {
-	return !page.sl.IsEmpty()
-}
-
-// Response returns the raw server response from the last page request.
-func (page SnapshotListPage) Response() compute.SnapshotList {
-	return page.sl
-}
-
-// Values returns the slice of values for the current page or nil if there are no values.
-func (page SnapshotListPage) Values() []compute.Snapshot {
-	if page.sl.IsEmpty() {
-		return nil
-	}
-	return *page.sl.Value
-}
diff --git a/pkg/azureclients/snapshotclient/doc.go b/pkg/azureclients/snapshotclient/doc.go
deleted file mode 100644
index 408846fd8..000000000
--- a/pkg/azureclients/snapshotclient/doc.go
+++ /dev/null
@@ -1,18 +0,0 @@
-/*
-Copyright 2020 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-// Package snapshotclient implements the client for Snapshots.
-package snapshotclient // import "sigs.k8s.io/cloud-provider-azure/pkg/azureclients/snapshotclient"
diff --git a/pkg/azureclients/snapshotclient/interface.go b/pkg/azureclients/snapshotclient/interface.go
deleted file mode 100644
index 1cf0ead02..000000000
--- a/pkg/azureclients/snapshotclient/interface.go
+++ /dev/null
@@ -1,50 +0,0 @@
-/*
-Copyright 2020 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package snapshotclient
-
-import (
-	"context"
-
-	"github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2022-08-01/compute"
-
-	"sigs.k8s.io/cloud-provider-azure/pkg/retry"
-)
-
-const (
-	// APIVersion is the API version for compute.
-	APIVersion = "2022-03-02"
-	// AzureStackCloudAPIVersion is the API version for Azure Stack
-	AzureStackCloudAPIVersion = "2019-03-01"
-	// AzureStackCloudName is the cloud name of Azure Stack
-	AzureStackCloudName = "AZURESTACKCLOUD"
-)
-
-// Interface is the client interface for Snapshots.
-// Don't forget to run "hack/update-mock-clients.sh" command to generate the mock client.
-type Interface interface {
-	// Get gets a Snapshot.
-	Get(ctx context.Context, subsID, resourceGroupName, snapshotName string) (compute.Snapshot, *retry.Error)
-
-	// Delete deletes a Snapshot by name.
-	Delete(ctx context.Context, subsID, resourceGroupName, snapshotName string) *retry.Error
-
-	// ListByResourceGroup get a list snapshots by resourceGroup.
-	ListByResourceGroup(ctx context.Context, subsID, resourceGroupName string) ([]compute.Snapshot, *retry.Error)
-
-	// CreateOrUpdate creates or updates a Snapshot.
-	CreateOrUpdate(ctx context.Context, subsID, resourceGroupName, snapshotName string, snapshot compute.Snapshot) *retry.Error
-}
diff --git a/pkg/azureclients/snapshotclient/mocksnapshotclient/doc.go b/pkg/azureclients/snapshotclient/mocksnapshotclient/doc.go
deleted file mode 100644
index c7c97a208..000000000
--- a/pkg/azureclients/snapshotclient/mocksnapshotclient/doc.go
+++ /dev/null
@@ -1,18 +0,0 @@
-/*
-Copyright 2020 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-// Package mocksnapshotclient implements the mock client for Snapshots.
-package mocksnapshotclient // import "sigs.k8s.io/cloud-provider-azure/pkg/azureclients/snapshotclient/mocksnapshotclient"
diff --git a/pkg/azureclients/snapshotclient/mocksnapshotclient/interface.go b/pkg/azureclients/snapshotclient/mocksnapshotclient/interface.go
deleted file mode 100644
index d6d5831ed..000000000
--- a/pkg/azureclients/snapshotclient/mocksnapshotclient/interface.go
+++ /dev/null
@@ -1,117 +0,0 @@
-// /*
-// Copyright The Kubernetes Authors.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-// */
-//
-
-// Code generated by MockGen. DO NOT EDIT.
-// Source: pkg/azureclients/snapshotclient/interface.go
-//
-// Generated by this command:
-//
-//	mockgen -copyright_file=/home/runner/work/cloud-provider-azure/cloud-provider-azure/hack/boilerplate/boilerplate.generatego.txt -source=pkg/azureclients/snapshotclient/interface.go -package=mocksnapshotclient Interface
-//
-
-// Package mocksnapshotclient is a generated GoMock package.
-package mocksnapshotclient
-
-import (
-	context "context"
-	reflect "reflect"
-
-	compute "github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2022-08-01/compute"
-	gomock "go.uber.org/mock/gomock"
-	retry "sigs.k8s.io/cloud-provider-azure/pkg/retry"
-)
-
-// MockInterface is a mock of Interface interface.
-type MockInterface struct {
-	ctrl     *gomock.Controller
-	recorder *MockInterfaceMockRecorder
-}
-
-// MockInterfaceMockRecorder is the mock recorder for MockInterface.
-type MockInterfaceMockRecorder struct {
-	mock *MockInterface
-}
-
-// NewMockInterface creates a new mock instance.
-func NewMockInterface(ctrl *gomock.Controller) *MockInterface {
-	mock := &MockInterface{ctrl: ctrl}
-	mock.recorder = &MockInterfaceMockRecorder{mock}
-	return mock
-}
-
-// EXPECT returns an object that allows the caller to indicate expected use.
-func (m *MockInterface) EXPECT() *MockInterfaceMockRecorder {
-	return m.recorder
-}
-
-// CreateOrUpdate mocks base method.
-func (m *MockInterface) CreateOrUpdate(ctx context.Context, subsID, resourceGroupName, snapshotName string, snapshot compute.Snapshot) *retry.Error {
-	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "CreateOrUpdate", ctx, subsID, resourceGroupName, snapshotName, snapshot)
-	ret0, _ := ret[0].(*retry.Error)
-	return ret0
-}
-
-// CreateOrUpdate indicates an expected call of CreateOrUpdate.
-func (mr *MockInterfaceMockRecorder) CreateOrUpdate(ctx, subsID, resourceGroupName, snapshotName, snapshot any) *gomock.Call {
-	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CreateOrUpdate", reflect.TypeOf((*MockInterface)(nil).CreateOrUpdate), ctx, subsID, resourceGroupName, snapshotName, snapshot)
-}
-
-// Delete mocks base method.
-func (m *MockInterface) Delete(ctx context.Context, subsID, resourceGroupName, snapshotName string) *retry.Error {
-	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "Delete", ctx, subsID, resourceGroupName, snapshotName)
-	ret0, _ := ret[0].(*retry.Error)
-	return ret0
-}
-
-// Delete indicates an expected call of Delete.
-func (mr *MockInterfaceMockRecorder) Delete(ctx, subsID, resourceGroupName, snapshotName any) *gomock.Call {
-	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Delete", reflect.TypeOf((*MockInterface)(nil).Delete), ctx, subsID, resourceGroupName, snapshotName)
-}
-
-// Get mocks base method.
-func (m *MockInterface) Get(ctx context.Context, subsID, resourceGroupName, snapshotName string) (compute.Snapshot, *retry.Error) {
-	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "Get", ctx, subsID, resourceGroupName, snapshotName)
-	ret0, _ := ret[0].(compute.Snapshot)
-	ret1, _ := ret[1].(*retry.Error)
-	return ret0, ret1
-}
-
-// Get indicates an expected call of Get.
-func (mr *MockInterfaceMockRecorder) Get(ctx, subsID, resourceGroupName, snapshotName any) *gomock.Call {
-	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Get", reflect.TypeOf((*MockInterface)(nil).Get), ctx, subsID, resourceGroupName, snapshotName)
-}
-
-// ListByResourceGroup mocks base method.
-func (m *MockInterface) ListByResourceGroup(ctx context.Context, subsID, resourceGroupName string) ([]compute.Snapshot, *retry.Error) {
-	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "ListByResourceGroup", ctx, subsID, resourceGroupName)
-	ret0, _ := ret[0].([]compute.Snapshot)
-	ret1, _ := ret[1].(*retry.Error)
-	return ret0, ret1
-}
-
-// ListByResourceGroup indicates an expected call of ListByResourceGroup.
-func (mr *MockInterfaceMockRecorder) ListByResourceGroup(ctx, subsID, resourceGroupName any) *gomock.Call {
-	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "ListByResourceGroup", reflect.TypeOf((*MockInterface)(nil).ListByResourceGroup), ctx, subsID, resourceGroupName)
-}
diff --git a/pkg/consts/consts.go b/pkg/consts/consts.go
index f4cce2104..9cf389f05 100644
--- a/pkg/consts/consts.go
+++ b/pkg/consts/consts.go
@@ -592,3 +592,16 @@ const (
 	VMPowerStateDeallocating = "deallocating"
 	VMPowerStateUnknown      = "unknown"
 )
+
+// Azure resource lock
+const (
+	AzureResourceLockHolderNameCloudControllerManager = "cloud-controller-manager"
+	AzureResourceLockLeaseName                        = "aks-managed-resource-locker"
+	AzureResourceLockLeaseNamespace                   = "kube-system"
+	AzureResourceLockLeaseDuration                    = int32(15 * 60)
+	AzureResourceLockPreviousHolderNameAnnotation     = "aks-managed-resource-locker-previous-holder"
+
+	AzureResourceLockFailedToLockErrorTemplate                = "%s failed due to fail to lock azure resources. This may because another component is trying to update azure resources, e.g., load balancers. This will be automatically retried by cloud provider exponentially: %w"
+	AzureResourceLockFailedToUnlockErrorTemplate              = "%s failed due to fail to unlock azure resources. This will be automatically retried by cloud provider exponentially, but can also be manually unlocked by removing the holder of the lease '%s/%s'. Before doing this, please be aware that this could lead to unexpected issues: %w"
+	AzureResourceLockFailedToReconcileWithUnlockErrorTemplate = "%s failed due to %s, and when unlocking azure resources another error happened: %w"
+)
diff --git a/pkg/credentialprovider/azure_credentials.go b/pkg/credentialprovider/azure_credentials.go
index ae0dc8477..2ffdceee7 100644
--- a/pkg/credentialprovider/azure_credentials.go
+++ b/pkg/credentialprovider/azure_credentials.go
@@ -59,9 +59,10 @@ type CredentialProvider interface {
 
 // acrProvider implements the credential provider interface for Azure Container Registry.
 type acrProvider struct {
-	config      *providerconfig.AzureAuthConfig
-	environment *azclient.Environment
-	credential  azcore.TokenCredential
+	config         *providerconfig.AzureAuthConfig
+	environment    *azclient.Environment
+	credential     azcore.TokenCredential
+	registryMirror map[string]string // Registry mirror relation: source registry -> target registry
 }
 
 func NewAcrProvider(config *providerconfig.AzureAuthConfig, environment *azclient.Environment, credential azcore.TokenCredential) CredentialProvider {
@@ -73,7 +74,7 @@ func NewAcrProvider(config *providerconfig.AzureAuthConfig, environment *azclien
 }
 
 // NewAcrProvider creates a new instance of the ACR provider.
-func NewAcrProviderFromConfig(configFile string) (CredentialProvider, error) {
+func NewAcrProviderFromConfig(configFile string, registryMirrorStr string) (CredentialProvider, error) {
 	if len(configFile) == 0 {
 		return nil, errors.New("no azure credential file is provided")
 	}
@@ -118,15 +119,16 @@ func NewAcrProviderFromConfig(configFile string) (CredentialProvider, error) {
 	}
 
 	return &acrProvider{
-		config:      config,
-		credential:  managedIdentityCredential,
-		environment: &envConfig,
+		config:         config,
+		credential:     managedIdentityCredential,
+		environment:    &envConfig,
+		registryMirror: parseRegistryMirror(registryMirrorStr),
 	}, nil
 }
 
 func (a *acrProvider) GetCredentials(ctx context.Context, image string, _ []string) (*v1.CredentialProviderResponse, error) {
-	loginServer := a.parseACRLoginServerFromImage(image)
-	if loginServer == "" {
+	targetloginServer, sourceloginServer := a.parseACRLoginServerFromImage(image)
+	if targetloginServer == "" {
 		klog.V(2).Infof("image(%s) is not from ACR, return empty authentication", image)
 		return &v1.CredentialProviderResponse{
 			CacheKeyType:  v1.RegistryPluginCacheKeyType,
@@ -148,16 +150,20 @@ func (a *acrProvider) GetCredentials(ctx context.Context, image string, _ []stri
 	}
 
 	if a.config.UseManagedIdentityExtension {
-		username, password, err := a.getFromACR(ctx, loginServer)
+		username, password, err := a.getFromACR(ctx, targetloginServer)
 		if err != nil {
-			klog.Errorf("error getting credentials from ACR for %s: %s", loginServer, err)
+			klog.Errorf("error getting credentials from ACR for %s: %s", targetloginServer, err)
 			return nil, err
 		}
 
-		response.Auth[loginServer] = v1.AuthConfig{
+		authConfig := v1.AuthConfig{
 			Username: username,
 			Password: password,
 		}
+		response.Auth[targetloginServer] = authConfig
+		if sourceloginServer != "" {
+			response.Auth[sourceloginServer] = authConfig
+		}
 	} else {
 		// Add our entry for each of the supported container registry URLs
 		for _, url := range containerRegistryUrls {
@@ -227,13 +233,16 @@ func (a *acrProvider) getFromACR(ctx context.Context, loginServer string) (strin
 	return dockerTokenLoginUsernameGUID, registryRefreshToken, nil
 }
 
-// parseACRLoginServerFromImage takes image as parameter and returns login server of it.
-// Parameter `image` is expected in following format: foo.azurecr.io/bar/imageName:version
+// parseACRLoginServerFromImage inputs an image URL and outputs login servers of target registry and source registry if --registry-mirror is set.
+// Input is expected in following format: foo.azurecr.io/bar/imageName:version
 // If the provided image is not an acr image, this function will return an empty string.
-func (a *acrProvider) parseACRLoginServerFromImage(image string) string {
-	match := acrRE.FindAllString(image, -1)
+func (a *acrProvider) parseACRLoginServerFromImage(image string) (string, string) {
+	targetImage, sourceRegistry := a.processImageWithRegistryMirror(image)
+
+	match := acrRE.FindAllString(targetImage, -1)
 	if len(match) == 1 {
-		return match[0]
+		targetRegistry := match[0]
+		return targetRegistry, sourceRegistry
 	}
 
 	// handle the custom cloud case
@@ -241,13 +250,42 @@ func (a *acrProvider) parseACRLoginServerFromImage(image string) string {
 		cloudAcrSuffix := a.environment.ContainerRegistryDNSSuffix
 		cloudAcrSuffixLength := len(cloudAcrSuffix)
 		if cloudAcrSuffixLength > 0 {
-			customAcrSuffixIndex := strings.Index(image, cloudAcrSuffix)
+			customAcrSuffixIndex := strings.Index(targetImage, cloudAcrSuffix)
 			if customAcrSuffixIndex != -1 {
 				endIndex := customAcrSuffixIndex + cloudAcrSuffixLength
-				return image[0:endIndex]
+				return targetImage[0:endIndex], sourceRegistry
 			}
 		}
 	}
 
-	return ""
+	return "", ""
+}
+
+// With acrProvider registry mirror, e.g. {"mcr.microsoft.com": "abc.azurecr.io"}
+// processImageWithRegistryMirror input format: "mcr.microsoft.com/bar/image:version"
+// output format: "abc.azurecr.io/bar/image:version", "mcr.microsoft.com"
+func (a *acrProvider) processImageWithRegistryMirror(image string) (string, string) {
+	for sourceRegistry, targetRegistry := range a.registryMirror {
+		if strings.HasPrefix(image, sourceRegistry) {
+			return strings.Replace(image, sourceRegistry, targetRegistry, 1), sourceRegistry
+		}
+	}
+	return image, ""
+}
+
+// parseRegistryMirror input format: "--registry-mirror=aaa:bbb,ccc:ddd"
+// output format: map[string]string{"aaa": "bbb", "ccc": "ddd"}
+func parseRegistryMirror(registryMirrorStr string) map[string]string {
+	registryMirror := map[string]string{}
+
+	registryMirrorStr = strings.ReplaceAll(registryMirrorStr, " ", "")
+	for _, mapping := range strings.Split(registryMirrorStr, ",") {
+		parts := strings.Split(mapping, ":")
+		if len(parts) != 2 {
+			klog.Errorf("Invalid registry mirror format: %s", mapping)
+			continue
+		}
+		registryMirror[parts[0]] = parts[1]
+	}
+	return registryMirror
 }
diff --git a/pkg/node/node.go b/pkg/node/node.go
index 5e6f4c628..4b705d1f0 100644
--- a/pkg/node/node.go
+++ b/pkg/node/node.go
@@ -34,7 +34,7 @@ type IMDSNodeProvider struct {
 
 // NewIMDSNodeProvider creates a new IMDSNodeProvider.
 func NewIMDSNodeProvider(ctx context.Context) *IMDSNodeProvider {
-	az, err := azureprovider.NewCloud(ctx, &azureprovider.Config{
+	az, err := azureprovider.NewCloud(ctx, nil, &azureprovider.Config{
 		UseInstanceMetadata: true,
 		VMType:              "vmss",
 	}, false)
diff --git a/pkg/node/nodearm.go b/pkg/node/nodearm.go
index 16da82fc0..00bf6d55c 100644
--- a/pkg/node/nodearm.go
+++ b/pkg/node/nodearm.go
@@ -37,7 +37,7 @@ type ARMNodeProvider struct {
 func NewARMNodeProvider(ctx context.Context, cloudConfigFilePath string) *ARMNodeProvider {
 	var err error
 	var az cloudprovider.Interface
-	az, err = azureprovider.NewCloudFromConfigFile(ctx, cloudConfigFilePath, false)
+	az, err = azureprovider.NewCloudFromConfigFile(ctx, nil, cloudConfigFilePath, false)
 	if err != nil {
 		klog.Fatalf("Failed to initialize Azure cloud provider: %v", err)
 	}
diff --git a/pkg/provider/azure.go b/pkg/provider/azure.go
index 4eb6adcc4..7d85c2a29 100644
--- a/pkg/provider/azure.go
+++ b/pkg/provider/azure.go
@@ -53,7 +53,6 @@ import (
 	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/configloader"
 	azclients "sigs.k8s.io/cloud-provider-azure/pkg/azureclients"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/blobclient"
-	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/containerserviceclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/deploymentclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/diskclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/fileclient"
@@ -63,9 +62,7 @@ import (
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/privateendpointclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/privatelinkserviceclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/publicipclient"
-	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/routeclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/routetableclient"
-	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/snapshotclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/storageaccountclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/subnetclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/vmasclient"
@@ -80,6 +77,7 @@ import (
 	azcache "sigs.k8s.io/cloud-provider-azure/pkg/cache"
 	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
 	ratelimitconfig "sigs.k8s.io/cloud-provider-azure/pkg/provider/config"
+	"sigs.k8s.io/cloud-provider-azure/pkg/provider/securitygroup"
 	"sigs.k8s.io/cloud-provider-azure/pkg/retry"
 	utilsets "sigs.k8s.io/cloud-provider-azure/pkg/util/sets"
 	"sigs.k8s.io/cloud-provider-azure/pkg/util/taints"
@@ -358,7 +356,6 @@ type Cloud struct {
 	Config
 	Environment azure.Environment
 
-	RoutesClient                    routeclient.Interface
 	SubnetsClient                   subnetclient.Interface
 	InterfacesClient                interfaceclient.Interface
 	RouteTablesClient               routetableclient.Interface
@@ -367,7 +364,6 @@ type Cloud struct {
 	VirtualMachinesClient           vmclient.Interface
 	StorageAccountClient            storageaccountclient.Interface
 	DisksClient                     diskclient.Interface
-	SnapshotsClient                 snapshotclient.Interface
 	FileClient                      fileclient.Interface
 	BlobClient                      blobclient.Interface
 	VirtualMachineScaleSetsClient   vmssclient.Interface
@@ -378,7 +374,6 @@ type Cloud struct {
 	privateendpointclient           privateendpointclient.Interface
 	privatednszonegroupclient       privatednszonegroupclient.Interface
 	PrivateLinkServiceClient        privatelinkserviceclient.Interface
-	containerServiceClient          containerserviceclient.Interface
 	deploymentClient                deploymentclient.Interface
 	ComputeClientFactory            azclient.ClientFactory
 	NetworkClientFactory            azclient.ClientFactory
@@ -423,10 +418,10 @@ type Cloud struct {
 	routeUpdater       batchProcessor
 	backendPoolUpdater batchProcessor
 
-	vmCache  azcache.Resource
-	lbCache  azcache.Resource
-	nsgCache azcache.Resource
-	rtCache  azcache.Resource
+	vmCache azcache.Resource
+	lbCache azcache.Resource
+	nsgRepo securitygroup.Repository
+	rtCache azcache.Resource
 	// public ip cache
 	// key: [resourceGroupName]
 	// Value: sync.Map of [pipName]*PublicIPAddress
@@ -453,10 +448,12 @@ type Cloud struct {
 	multipleStandardLoadBalancersActiveNodesLock    sync.Mutex
 	localServiceNameToServiceInfoMap                sync.Map
 	endpointSlicesCache                             sync.Map
+
+	azureResourceLocker *AzureResourceLocker
 }
 
 // NewCloud returns a Cloud with initialized clients
-func NewCloud(ctx context.Context, config *Config, callFromCCM bool) (cloudprovider.Interface, error) {
+func NewCloud(ctx context.Context, clientBuilder cloudprovider.ControllerClientBuilder, config *Config, callFromCCM bool) (cloudprovider.Interface, error) {
 	az := &Cloud{
 		nodeNames:                  utilsets.NewString(),
 		nodeZones:                  map[string]*utilsets.IgnoreCaseSet{},
@@ -477,10 +474,22 @@ func NewCloud(ctx context.Context, config *Config, callFromCCM bool) (cloudprovi
 	if az.lockMap == nil {
 		az.lockMap = newLockMap()
 	}
+
+	if clientBuilder != nil {
+		az.KubeClient = clientBuilder.ClientOrDie("azure-cloud-provider")
+	}
+	az.azureResourceLocker = NewAzureResourceLocker(
+		az,
+		consts.AzureResourceLockHolderNameCloudControllerManager,
+		consts.AzureResourceLockLeaseName,
+		consts.AzureResourceLockLeaseNamespace,
+		consts.AzureResourceLockLeaseDuration,
+	)
+
 	return az, nil
 }
 
-func NewCloudFromConfigFile(ctx context.Context, configFilePath string, calFromCCM bool) (cloudprovider.Interface, error) {
+func NewCloudFromConfigFile(ctx context.Context, clientBuilder cloudprovider.ControllerClientBuilder, configFilePath string, calFromCCM bool) (cloudprovider.Interface, error) {
 	var (
 		cloud cloudprovider.Interface
 		err   error
@@ -501,8 +510,7 @@ func NewCloudFromConfigFile(ctx context.Context, configFilePath string, calFromC
 			klog.Fatalf("Failed to parse Azure cloud provider config: %v", err)
 		}
 	}
-	cloud, err = NewCloud(ctx, configValue, calFromCCM && configFilePath != "")
-
+	cloud, err = NewCloud(ctx, clientBuilder, configValue, calFromCCM && configFilePath != "")
 	if err != nil {
 		return nil, fmt.Errorf("could not init cloud provider azure: %w", err)
 	}
@@ -525,7 +533,7 @@ func NewCloudFromSecret(ctx context.Context, clientBuilder cloudprovider.Control
 	if err != nil {
 		return nil, fmt.Errorf("NewCloudFromSecret: failed to get config from secret %s/%s: %w", secretNamespace, secretName, err)
 	}
-	az, err := NewCloud(ctx, config, true)
+	az, err := NewCloud(ctx, clientBuilder, config, true)
 	if err != nil {
 		return nil, fmt.Errorf("NewCloudFromSecret: failed to initialize cloud from secret %s/%s: %w", secretNamespace, secretName, err)
 	}
@@ -727,8 +735,16 @@ func (az *Cloud) InitializeCloudFromConfig(ctx context.Context, config *Config,
 		if err != nil {
 			return err
 		}
-	}
 
+		networkClientFactory := az.NetworkClientFactory
+		if networkClientFactory == nil {
+			networkClientFactory = az.ComputeClientFactory
+		}
+		az.nsgRepo, err = securitygroup.NewSecurityGroupRepo(az.SecurityGroupResourceGroup, az.SecurityGroupName, az.NsgCacheTTLInSeconds, az.DisableAPICallCache, networkClientFactory.GetSecurityGroupClient())
+		if err != nil {
+			return err
+		}
+	}
 	err = az.initCaches()
 	if err != nil {
 		return err
@@ -841,11 +857,6 @@ func (az *Cloud) initCaches() (err error) {
 		return err
 	}
 
-	az.nsgCache, err = az.newNSGCache()
-	if err != nil {
-		return err
-	}
-
 	az.rtCache, err = az.newRouteTableCache()
 	if err != nil {
 		return err
@@ -954,13 +965,13 @@ func (az *Cloud) setCloudProviderBackoffDefaults(config *Config) wait.Backoff {
 func (az *Cloud) configAzureClients(
 	servicePrincipalToken *adal.ServicePrincipalToken,
 	multiTenantOAuthTokenProvider adal.MultitenantOAuthTokenProvider,
-	networkResourceServicePrincipalToken adal.OAuthTokenProvider) {
+	networkResourceServicePrincipalToken adal.OAuthTokenProvider,
+) {
 	azClientConfig := az.getAzureClientConfig(servicePrincipalToken)
 
 	// Prepare AzureClientConfig for all azure clients
 	interfaceClientConfig := azClientConfig.WithRateLimiter(az.Config.InterfaceRateLimit)
 	vmSizeClientConfig := azClientConfig.WithRateLimiter(az.Config.VirtualMachineSizeRateLimit)
-	snapshotClientConfig := azClientConfig.WithRateLimiter(az.Config.SnapshotRateLimit)
 	storageAccountClientConfig := azClientConfig.WithRateLimiter(az.Config.StorageAccountRateLimit)
 	diskClientConfig := azClientConfig.WithRateLimiter(az.Config.DiskRateLimit)
 	vmClientConfig := azClientConfig.WithRateLimiter(az.Config.VirtualMachineRateLimit)
@@ -974,7 +985,6 @@ func (az *Cloud) configAzureClients(
 	routeTableClientConfig := azClientConfig.WithRateLimiter(az.Config.RouteTableRateLimit)
 	loadBalancerClientConfig := azClientConfig.WithRateLimiter(az.Config.LoadBalancerRateLimit)
 	publicIPClientConfig := azClientConfig.WithRateLimiter(az.Config.PublicIPAddressRateLimit)
-	containerServiceConfig := azClientConfig.WithRateLimiter(az.Config.ContainerServiceRateLimit)
 	deploymentConfig := azClientConfig.WithRateLimiter(az.Config.DeploymentRateLimit)
 	privateDNSZoenGroupConfig := azClientConfig.WithRateLimiter(az.Config.PrivateDNSZoneGroupRateLimit)
 	privateEndpointConfig := azClientConfig.WithRateLimiter(az.Config.PrivateEndpointRateLimit)
@@ -1016,13 +1026,11 @@ func (az *Cloud) configAzureClients(
 	// Initialize all azure clients based on client config
 	az.InterfacesClient = interfaceclient.New(interfaceClientConfig)
 	az.VirtualMachineSizesClient = vmsizeclient.New(vmSizeClientConfig)
-	az.SnapshotsClient = snapshotclient.New(snapshotClientConfig)
 	az.StorageAccountClient = storageaccountclient.New(storageAccountClientConfig)
 	az.DisksClient = diskclient.New(diskClientConfig)
 	az.VirtualMachinesClient = vmclient.New(vmClientConfig)
 	az.VirtualMachineScaleSetsClient = vmssclient.New(vmssClientConfig)
 	az.VirtualMachineScaleSetVMsClient = vmssvmclient.New(vmssVMClientConfig)
-	az.RoutesClient = routeclient.New(routeClientConfig)
 	az.SubnetsClient = subnetclient.New(subnetClientConfig)
 	az.RouteTablesClient = routetableclient.New(routeTableClientConfig)
 	az.LoadBalancerClient = loadbalancerclient.New(loadBalancerClientConfig)
@@ -1033,7 +1041,6 @@ func (az *Cloud) configAzureClients(
 	az.privateendpointclient = privateendpointclient.New(privateEndpointConfig)
 	az.privatednszonegroupclient = privatednszonegroupclient.New(privateDNSZoenGroupConfig)
 	az.PrivateLinkServiceClient = privatelinkserviceclient.New(privateLinkServiceConfig)
-	az.containerServiceClient = containerserviceclient.New(containerServiceConfig)
 	az.deploymentClient = deploymentclient.New(deploymentConfig)
 
 	if az.ZoneClient == nil {
@@ -1526,5 +1533,4 @@ func (az *Cloud) GetNodeVMSet(ctx context.Context, nodeName types.NodeName, crt
 
 	// 5. Node is managed by vmss
 	return ss, nil
-
 }
diff --git a/pkg/provider/azure_fakes.go b/pkg/provider/azure_fakes.go
index b86c4fcae..127b6713f 100644
--- a/pkg/provider/azure_fakes.go
+++ b/pkg/provider/azure_fakes.go
@@ -36,9 +36,7 @@ import (
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/loadbalancerclient/mockloadbalancerclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/privatelinkserviceclient/mockprivatelinkserviceclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/publicipclient/mockpublicipclient"
-	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/routeclient/mockrouteclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/routetableclient/mockroutetableclient"
-	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/snapshotclient/mocksnapshotclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/subnetclient/mocksubnetclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/vmclient/mockvmclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/vmssclient/mockvmssclient"
@@ -46,6 +44,7 @@ import (
 	azcache "sigs.k8s.io/cloud-provider-azure/pkg/cache"
 	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
 	"sigs.k8s.io/cloud-provider-azure/pkg/provider/config"
+	"sigs.k8s.io/cloud-provider-azure/pkg/provider/securitygroup"
 	utilsets "sigs.k8s.io/cloud-provider-azure/pkg/util/sets"
 )
 
@@ -112,11 +111,9 @@ func GetTestCloud(ctrl *gomock.Controller) (az *Cloud) {
 		lockMap:                  newLockMap(),
 	}
 	az.DisksClient = mockdiskclient.NewMockInterface(ctrl)
-	az.SnapshotsClient = mocksnapshotclient.NewMockInterface(ctrl)
 	az.InterfacesClient = mockinterfaceclient.NewMockInterface(ctrl)
 	az.LoadBalancerClient = mockloadbalancerclient.NewMockInterface(ctrl)
 	az.PublicIPAddressesClient = mockpublicipclient.NewMockInterface(ctrl)
-	az.RoutesClient = mockrouteclient.NewMockInterface(ctrl)
 	az.RouteTablesClient = mockroutetableclient.NewMockInterface(ctrl)
 	az.SubnetsClient = mocksubnetclient.NewMockInterface(ctrl)
 	az.VirtualMachineScaleSetsClient = mockvmssclient.NewMockInterface(ctrl)
@@ -138,7 +135,7 @@ func GetTestCloud(ctrl *gomock.Controller) (az *Cloud) {
 	az.VMSet, _ = newAvailabilitySet(az)
 	az.vmCache, _ = az.newVMCache()
 	az.lbCache, _ = az.newLBCache()
-	az.nsgCache, _ = az.newNSGCache()
+	az.nsgRepo, _ = securitygroup.NewSecurityGroupRepo(az.SecurityGroupResourceGroup, az.SecurityGroupName, az.NsgCacheTTLInSeconds, az.Config.DisableAPICallCache, securtyGrouptrack2Client)
 	az.rtCache, _ = az.newRouteTableCache()
 	az.pipCache, _ = az.newPIPCache()
 	az.plsCache, _ = az.newPLSCache()
diff --git a/pkg/provider/azure_loadbalancer.go b/pkg/provider/azure_loadbalancer.go
index 931f06b24..e05cceaf8 100644
--- a/pkg/provider/azure_loadbalancer.go
+++ b/pkg/provider/azure_loadbalancer.go
@@ -46,10 +46,10 @@ import (
 	"sigs.k8s.io/cloud-provider-azure/pkg/log"
 	"sigs.k8s.io/cloud-provider-azure/pkg/metrics"
 	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer"
-	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer/iputil"
 	"sigs.k8s.io/cloud-provider-azure/pkg/retry"
 	"sigs.k8s.io/cloud-provider-azure/pkg/trace"
 	"sigs.k8s.io/cloud-provider-azure/pkg/trace/attributes"
+	"sigs.k8s.io/cloud-provider-azure/pkg/util/iputil"
 	utilsets "sigs.k8s.io/cloud-provider-azure/pkg/util/sets"
 )
 
@@ -202,13 +202,12 @@ func (az *Cloud) reconcileService(ctx context.Context, clusterName string, servi
 // load balancer is not ready yet (e.g., it is still being provisioned) and
 // polling at a fixed rate is preferred over backing off exponentially in
 // order to minimize latency.
-func (az *Cloud) EnsureLoadBalancer(ctx context.Context, clusterName string, service *v1.Service, nodes []*v1.Node) (*v1.LoadBalancerStatus, error) {
+func (az *Cloud) EnsureLoadBalancer(ctx context.Context, clusterName string, service *v1.Service, nodes []*v1.Node) (lbStatus *v1.LoadBalancerStatus, err error) {
 	// When a client updates the internal load balancer annotation,
 	// the service may be switched from an internal LB to a public one, or vice versa.
 	// Here we'll firstly ensure service do not lie in the opposite LB.
 	const Operation = "EnsureLoadBalancer"
 
-	var err error
 	ctx, span := trace.BeginReconcile(ctx, trace.DefaultTracer(), Operation, attributes.FeatureOfService(service)...)
 	defer func() { span.Observe(ctx, err) }()
 
@@ -223,6 +222,41 @@ func (az *Cloud) EnsureLoadBalancer(ctx context.Context, clusterName string, ser
 		isOperationSucceeded = false
 	)
 
+	if az.azureResourceLocker != nil {
+		err = az.azureResourceLocker.Lock(ctx)
+		if err != nil {
+			logger.Error(err, "failed to lock azure resources")
+			return nil, fmt.Errorf(
+				consts.AzureResourceLockFailedToLockErrorTemplate,
+				"EnsureLoadBalancer",
+				err,
+			)
+		}
+
+		defer func() {
+			unlockErr := az.azureResourceLocker.Unlock(ctx)
+			if unlockErr != nil {
+				unlockErr = fmt.Errorf(
+					consts.AzureResourceLockFailedToUnlockErrorTemplate,
+					"EnsureLoadBalancer",
+					consts.AzureResourceLockLeaseNamespace,
+					consts.AzureResourceLockLeaseName,
+					unlockErr,
+				)
+			}
+			if err == nil {
+				err = unlockErr
+			} else if unlockErr != nil {
+				err = fmt.Errorf(
+					consts.AzureResourceLockFailedToReconcileWithUnlockErrorTemplate,
+					"EnsureLoadBalancer",
+					err,
+					unlockErr,
+				)
+			}
+		}()
+	}
+
 	logger.V(5).Info("Starting", "service-spec", log.ValueAsMap(service))
 
 	defer func() {
@@ -234,7 +268,7 @@ func (az *Cloud) EnsureLoadBalancer(ctx context.Context, clusterName string, ser
 		}
 	}()
 
-	lbStatus, err := az.reconcileService(ctx, clusterName, service, nodes)
+	lbStatus, err = az.reconcileService(ctx, clusterName, service, nodes)
 	if err != nil {
 		return nil, err
 	}
@@ -315,6 +349,41 @@ func (az *Cloud) UpdateLoadBalancer(ctx context.Context, clusterName string, ser
 		return nil
 	}
 
+	if az.azureResourceLocker != nil {
+		err = az.azureResourceLocker.Lock(ctx)
+		if err != nil {
+			logger.Error(err, "failed to lock azure resources")
+			return fmt.Errorf(
+				consts.AzureResourceLockFailedToLockErrorTemplate,
+				"UpdateLoadBalancer",
+				err,
+			)
+		}
+
+		defer func() {
+			unlockErr := az.azureResourceLocker.Unlock(ctx)
+			if unlockErr != nil {
+				unlockErr = fmt.Errorf(
+					consts.AzureResourceLockFailedToUnlockErrorTemplate,
+					"UpdateLoadBalancer",
+					consts.AzureResourceLockLeaseNamespace,
+					consts.AzureResourceLockLeaseName,
+					unlockErr,
+				)
+			}
+			if err == nil {
+				err = unlockErr
+			} else if unlockErr != nil {
+				err = fmt.Errorf(
+					consts.AzureResourceLockFailedToReconcileWithUnlockErrorTemplate,
+					"UpdateLoadBalancer",
+					err,
+					unlockErr,
+				)
+			}
+		}()
+	}
+
 	_, err = az.reconcileService(ctx, clusterName, service, nodes)
 	if err != nil {
 		return err
@@ -332,10 +401,9 @@ func (az *Cloud) UpdateLoadBalancer(ctx context.Context, clusterName string, ser
 // doesn't exist even if some part of it is still laying around.
 // Implementations must treat the *v1.Service parameter as read-only and not modify it.
 // Parameter 'clusterName' is the name of the cluster as presented to kube-controller-manager
-func (az *Cloud) EnsureLoadBalancerDeleted(ctx context.Context, clusterName string, service *v1.Service) error {
+func (az *Cloud) EnsureLoadBalancerDeleted(ctx context.Context, clusterName string, service *v1.Service) (err error) {
 	const Operation = "EnsureLoadBalancerDeleted"
 
-	var err error
 	ctx, span := trace.BeginReconcile(ctx, trace.DefaultTracer(), Operation, attributes.FeatureOfService(service)...)
 	defer func() { span.Observe(ctx, err) }()
 
@@ -350,6 +418,40 @@ func (az *Cloud) EnsureLoadBalancerDeleted(ctx context.Context, clusterName stri
 		isOperationSucceeded = false
 	)
 	ctx = log.NewContext(ctx, logger)
+	if az.azureResourceLocker != nil {
+		err = az.azureResourceLocker.Lock(ctx)
+		if err != nil {
+			logger.Error(err, "failed to lock azure resources")
+			return fmt.Errorf(
+				consts.AzureResourceLockFailedToLockErrorTemplate,
+				"EnsureLoadBalancerDeleted",
+				err,
+			)
+		}
+
+		defer func() {
+			unlockErr := az.azureResourceLocker.Unlock(ctx)
+			if unlockErr != nil {
+				unlockErr = fmt.Errorf(
+					consts.AzureResourceLockFailedToUnlockErrorTemplate,
+					"EnsureLoadBalancerDeleted",
+					consts.AzureResourceLockLeaseNamespace,
+					consts.AzureResourceLockLeaseName,
+					unlockErr,
+				)
+			}
+			if err == nil {
+				err = unlockErr
+			} else if unlockErr != nil {
+				err = fmt.Errorf(
+					consts.AzureResourceLockFailedToReconcileWithUnlockErrorTemplate,
+					"EnsureLoadBalancerDeleted",
+					err,
+					unlockErr,
+				)
+			}
+		}()
+	}
 
 	logger.V(5).Info("Starting", "service-spec", log.ValueAsMap(service))
 	defer func() {
@@ -2211,8 +2313,7 @@ func (az *Cloud) recordExistingNodesOnLoadBalancers(clusterName string, lbs *[]n
 					if strings.EqualFold(trimSuffixIgnoreCase(
 						lbName, consts.InternalLoadBalancerNameSuffix,
 					), multiSLBConfig.Name) {
-						az.MultipleStandardLoadBalancerConfigurations[i].ActiveNodes =
-							utilsets.SafeInsert(multiSLBConfig.ActiveNodes, nodeNames...)
+						az.MultipleStandardLoadBalancerConfigurations[i].ActiveNodes = utilsets.SafeInsert(multiSLBConfig.ActiveNodes, nodeNames...)
 					}
 				}
 			}
@@ -2339,7 +2440,8 @@ func (az *Cloud) reconcileFrontendIPConfigs(
 	lb *network.LoadBalancer,
 	status *v1.LoadBalancerStatus,
 	wantLb bool,
-	lbFrontendIPConfigNames map[bool]string) ([]*network.FrontendIPConfiguration, []network.FrontendIPConfiguration, bool, error) {
+	lbFrontendIPConfigNames map[bool]string,
+) ([]*network.FrontendIPConfiguration, []network.FrontendIPConfiguration, bool, error) {
 	var err error
 	lbName := *lb.Name
 	serviceName := getServiceName(service)
@@ -2546,7 +2648,8 @@ func (az *Cloud) getFrontendZones(
 	fipConfig *network.FrontendIPConfiguration,
 	previousZone *[]string,
 	isFipChanged bool,
-	serviceName, lbFrontendIPConfigName string) error {
+	serviceName, lbFrontendIPConfigName string,
+) error {
 	if !isFipChanged { // fetch zone information from API for new frontends
 		// only add zone information for new internal frontend IP configurations for standard load balancer not deployed to an edge zone.
 		location := az.Location
@@ -2678,8 +2781,8 @@ func (az *Cloud) getExpectedLBRules(
 	lbFrontendIPConfigID string,
 	lbBackendPoolID string,
 	lbName string,
-	isIPv6 bool) ([]network.Probe, []network.LoadBalancingRule, error) {
-
+	isIPv6 bool,
+) ([]network.Probe, []network.LoadBalancingRule, error) {
 	var expectedRules []network.LoadBalancingRule
 	var expectedProbes []network.Probe
 
@@ -2728,7 +2831,7 @@ func (az *Cloud) getExpectedLBRules(
 		if err != nil {
 			return nil, nil, fmt.Errorf("error generate lb rule for ha mod loadbalancer. err: %w", err)
 		}
-		//Here we need to find one health probe rule for the HA lb rule.
+		// Here we need to find one health probe rule for the HA lb rule.
 		if nodeEndpointHealthprobe == nil {
 			// use user customized health probe rule if any
 			for _, port := range service.Spec.Ports {
@@ -2736,7 +2839,7 @@ func (az *Cloud) getExpectedLBRules(
 				if err != nil {
 					klog.V(2).ErrorS(err, "error occurred when buildHealthProbeRulesForPort", "service", service.Name, "namespace", service.Namespace,
 						"rule-name", lbRuleName, "port", port.Port)
-					//ignore error because we only need one correct rule
+					// ignore error because we only need one correct rule
 				}
 				if portprobe != nil {
 					props.Probe = &network.SubResource{
@@ -2833,7 +2936,8 @@ func (az *Cloud) getExpectedLBRules(
 func (az *Cloud) getExpectedLoadBalancingRulePropertiesForPort(
 	service *v1.Service,
 	lbFrontendIPConfigID string,
-	lbBackendPoolID string, servicePort v1.ServicePort, transportProto network.TransportProtocol) (*network.LoadBalancingRulePropertiesFormat, error) {
+	lbBackendPoolID string, servicePort v1.ServicePort, transportProto network.TransportProtocol,
+) (*network.LoadBalancingRulePropertiesFormat, error) {
 	var err error
 
 	loadDistribution := network.LoadDistributionDefault
@@ -2889,7 +2993,8 @@ func (az *Cloud) getExpectedLoadBalancingRulePropertiesForPort(
 func (az *Cloud) getExpectedHAModeLoadBalancingRuleProperties(
 	service *v1.Service,
 	lbFrontendIPConfigID string,
-	lbBackendPoolID string) (*network.LoadBalancingRulePropertiesFormat, error) {
+	lbBackendPoolID string,
+) (*network.LoadBalancingRulePropertiesFormat, error) {
 	props, err := az.getExpectedLoadBalancingRulePropertiesForPort(service, lbFrontendIPConfigID, lbBackendPoolID, v1.ServicePort{}, network.TransportProtocolAll)
 	if err != nil {
 		return nil, fmt.Errorf("error generate lb rule for ha mod loadbalancer. err: %w", err)
@@ -2924,7 +3029,7 @@ func (az *Cloud) reconcileSecurityGroup(
 
 	var accessControl *loadbalancer.AccessControl
 	{
-		sg, err := az.getSecurityGroup(ctx, azcache.CacheReadTypeDefault)
+		sg, err := az.nsgRepo.GetSecurityGroup(ctx)
 		if err != nil {
 			return nil, err
 		}
@@ -3017,13 +3122,12 @@ func (az *Cloud) reconcileSecurityGroup(
 	if updated {
 		logger.V(2).Info("Preparing to update security group")
 		logger.V(5).Info("CreateOrUpdateSecurityGroup begin")
-		err := az.CreateOrUpdateSecurityGroup(rv)
+		err := az.nsgRepo.CreateOrUpdateSecurityGroup(ctx, rv)
 		if err != nil {
 			logger.Error(err, "Failed to update security group")
 			return nil, err
 		}
 		logger.V(5).Info("CreateOrUpdateSecurityGroup end")
-		_ = az.nsgCache.Delete(ptr.Deref(rv.Name, ""))
 	}
 	return rv, nil
 }
diff --git a/pkg/provider/azure_loadbalancer_accesscontrol.go b/pkg/provider/azure_loadbalancer_accesscontrol.go
index 60394aa77..e2e48a799 100644
--- a/pkg/provider/azure_loadbalancer_accesscontrol.go
+++ b/pkg/provider/azure_loadbalancer_accesscontrol.go
@@ -28,7 +28,7 @@ import (
 	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
 	"sigs.k8s.io/cloud-provider-azure/pkg/log"
 	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer"
-	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer/fnutil"
+	fnutil "sigs.k8s.io/cloud-provider-azure/pkg/util/collectionutil"
 )
 
 func filterServicesByIngressIPs(services []*v1.Service, ips []netip.Addr) []*v1.Service {
diff --git a/pkg/provider/azure_lock.go b/pkg/provider/azure_lock.go
new file mode 100644
index 000000000..d58a5103f
--- /dev/null
+++ b/pkg/provider/azure_lock.go
@@ -0,0 +1,244 @@
+/*
+Copyright 2024 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package provider
+
+import (
+	"context"
+	"errors"
+	"strings"
+	"time"
+
+	v1 "k8s.io/api/coordination/v1"
+	k8sapierrors "k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/client-go/kubernetes"
+	"k8s.io/klog/v2"
+	"k8s.io/utils/ptr"
+
+	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
+)
+
+type AzureResourceLocker struct {
+	*Cloud
+	holder               string
+	leaseName            string
+	leaseNamespace       string
+	leaseDurationSeconds int32
+}
+
+func NewAzureResourceLocker(
+	cloud *Cloud,
+	holder, leaseName, leaseNamespace string,
+	leaseDurationSeconds int32,
+) *AzureResourceLocker {
+	return &AzureResourceLocker{
+		Cloud:                cloud,
+		holder:               holder,
+		leaseName:            leaseName,
+		leaseNamespace:       leaseNamespace,
+		leaseDurationSeconds: leaseDurationSeconds,
+	}
+}
+
+// Lock creates a lease if it does not exist and acquires the lease.
+// If the lease has not expired yet and is held by another holder, it will return an error.
+func (l *AzureResourceLocker) Lock(ctx context.Context) error {
+	if err := createLeaseIfNotExists(
+		ctx, l.leaseNamespace, l.leaseName, l.leaseDurationSeconds, l.KubeClient,
+	); err != nil {
+		return err
+	}
+	if err := l.acquireLease(
+		ctx, l.KubeClient, l.holder, l.leaseNamespace, l.leaseName); err != nil {
+		return err
+	}
+
+	return nil
+}
+
+// Unlock releases the lease if needed.
+func (l *AzureResourceLocker) Unlock(ctx context.Context) error {
+	if err := releaseLease(
+		ctx, l.KubeClient, l.leaseNamespace, l.leaseName, l.holder,
+	); err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func createLeaseIfNotExists(
+	ctx context.Context,
+	leaseNamespace, leaseName string,
+	leaseDurationSeconds int32,
+	clientset kubernetes.Interface,
+) error {
+	logger := klog.Background().WithName("createLeaseIfNotExists").
+		WithValues("leaseNamespace", leaseNamespace, "leaseName", leaseName,
+			"leaseDurationSeconds", leaseDurationSeconds).V(4)
+
+	lease := &v1.Lease{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      leaseName,
+			Namespace: leaseNamespace,
+		},
+		Spec: v1.LeaseSpec{
+			LeaseDurationSeconds: ptr.To(leaseDurationSeconds),
+		},
+	}
+
+	_, err := clientset.CoordinationV1().Leases(leaseNamespace).Get(ctx, leaseName, metav1.GetOptions{})
+	if err != nil {
+		if !k8sapierrors.IsNotFound(err) {
+			logger.Error(err, "failed to get lease")
+			return err
+		}
+		// Lease does not exist, create it
+		_, err = clientset.CoordinationV1().Leases(leaseNamespace).Create(ctx, lease, metav1.CreateOptions{})
+		if err != nil {
+			logger.Error(err, "failed to create lease")
+			return err
+		}
+		logger.Info("Successfully created lease.")
+	} else {
+		logger.Info("Lease already exists.")
+	}
+
+	return nil
+}
+
+func (l *AzureResourceLocker) acquireLease(
+	ctx context.Context,
+	clientset kubernetes.Interface,
+	holder, leaseNamespace, leaseName string,
+) error {
+	logger := klog.Background().WithName("acquireLease").
+		WithValues("holder", holder, "leaseNamespace", leaseNamespace, "leaseName", leaseName).V(4)
+
+	lease, err := clientset.CoordinationV1().Leases(leaseNamespace).Get(ctx, leaseName, metav1.GetOptions{})
+	if err != nil {
+		logger.Error(err, "failed to get lease")
+		return err
+	}
+
+	if !hasExpired(lease) {
+		// If the lease has not expired, but the previous holder is the same,
+		// it means the same previous operation failed and the releaseLease
+		// also failed. In this case we should continue acquiring the lease.
+		// This should be a rare case, otherwise we should not acquire the lease.
+		prevHolder := ptr.Deref(lease.Spec.HolderIdentity, "")
+		if !strings.EqualFold(prevHolder, holder) {
+			errMsg := "Lease has not expired yet, another component such as aks rp may be processing another request. This would be automatically recovered after the lease expires."
+			err := errors.New(errMsg)
+			logger.Error(err, "failed to acquire lease")
+			return err
+		}
+	}
+
+	lease.Spec.HolderIdentity = ptr.To(holder)
+	now := &metav1.MicroTime{Time: time.Now()}
+	lease.Spec.AcquireTime = now
+	lease.Spec.RenewTime = now
+	_, err = clientset.CoordinationV1().
+		Leases(leaseNamespace).Update(ctx, lease, metav1.UpdateOptions{})
+	if err != nil {
+		logger.Error(err, "failed to acquire lease")
+		return err
+	}
+
+	// invalidate caches if the previous holder was not the cloud controller manager
+	if lease.Annotations != nil &&
+		!strings.EqualFold(
+			lease.Annotations[consts.AzureResourceLockPreviousHolderNameAnnotation],
+			consts.AzureResourceLockHolderNameCloudControllerManager,
+		) {
+		l.lbCache, err = l.newLBCache()
+		if err != nil {
+			return err
+		}
+		if err := l.VMSet.RefreshCaches(); err != nil {
+			return err
+		}
+	}
+
+	logger.Info("Successfully acquired lease.")
+	return nil
+}
+
+func hasExpired(lease *v1.Lease) bool {
+	if ptr.Deref(lease.Spec.HolderIdentity, "") == "" {
+		return true // No holder, hence expired
+	}
+
+	// Calculate the expiration time
+	leaseDuration := time.Duration(ptr.Deref(lease.Spec.LeaseDurationSeconds, 0)) * time.Second
+	var expirationTime time.Time
+	if lease.Spec.RenewTime != nil {
+		expirationTime = lease.Spec.RenewTime.Add(leaseDuration)
+	}
+
+	return time.Now().After(expirationTime)
+}
+
+func releaseLease(
+	ctx context.Context,
+	clientset kubernetes.Interface,
+	leaseNamespace, leaseName, holder string,
+) error {
+	logger := klog.Background().WithName("releaseLease").
+		WithValues("leaseNamespace", leaseNamespace, "leaseName", leaseName, "holder", holder).V(4)
+
+	lease, err := clientset.CoordinationV1().
+		Leases(leaseNamespace).Get(ctx, leaseName, metav1.GetOptions{})
+	if err != nil {
+		if k8sapierrors.IsNotFound(err) {
+			logger.Info("lease does not exist, no need to release it.")
+			return nil
+		}
+		logger.Error(err, "failed to get lease")
+		return err
+	}
+
+	prevHolder := ptr.Deref(lease.Spec.HolderIdentity, "")
+	if !strings.EqualFold(prevHolder, holder) {
+		logger.Info(
+			"%s is holding the lease instead of %s, no need to release it.",
+			prevHolder, holder,
+		)
+		return nil
+	}
+
+	if hasExpired(lease) {
+		logger.Info("lease has already expired, no need to release it.")
+		return nil
+	}
+
+	lease.Spec.HolderIdentity = ptr.To("")
+	if lease.Annotations == nil {
+		lease.Annotations = make(map[string]string)
+	}
+	lease.Annotations[consts.AzureResourceLockPreviousHolderNameAnnotation] = consts.AzureResourceLockHolderNameCloudControllerManager
+	_, err = clientset.CoordinationV1().
+		Leases(leaseNamespace).Update(ctx, lease, metav1.UpdateOptions{})
+	if err != nil {
+		logger.Error(err, "failed to release lease")
+		return err
+	}
+
+	logger.Info("Successfully released lease.")
+	return nil
+}
diff --git a/pkg/provider/azure_mock_vmsets.go b/pkg/provider/azure_mock_vmsets.go
index de3198a43..5476b3b6b 100644
--- a/pkg/provider/azure_mock_vmsets.go
+++ b/pkg/provider/azure_mock_vmsets.go
@@ -16,11 +16,11 @@
 //
 
 // Code generated by MockGen. DO NOT EDIT.
-// Source: azure_vmsets.go
+// Source: /Users/niqi/go/src/sigs.k8s.io/cloud-provider-azure/pkg/provider/azure_vmsets.go
 //
 // Generated by this command:
 //
-//	mockgen -package provider -source azure_vmsets.go -self_package sigs.k8s.io/cloud-provider-azure/pkg/provider -copyright_file ../../hack/boilerplate/boilerplate.generatego.txt
+//	mockgen -destination=/Users/niqi/go/src/sigs.k8s.io/cloud-provider-azure/pkg/provider/azure_mock_vmsets.go -source=/Users/niqi/go/src/sigs.k8s.io/cloud-provider-azure/pkg/provider/azure_vmsets.go -package=provider VMSet
 //
 
 // Package provider is a generated GoMock package.
@@ -30,14 +30,13 @@ import (
 	context "context"
 	reflect "reflect"
 
-	v6 "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v6"
+	armcompute "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v6"
 	compute "github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2022-08-01/compute"
 	network "github.com/Azure/azure-sdk-for-go/services/network/mgmt/2022-07-01/network"
 	gomock "go.uber.org/mock/gomock"
 	v1 "k8s.io/api/core/v1"
 	types "k8s.io/apimachinery/pkg/types"
-	cloud_provider "k8s.io/cloud-provider"
-
+	cloudprovider "k8s.io/cloud-provider"
 	cache "sigs.k8s.io/cloud-provider-azure/pkg/cache"
 )
 
@@ -183,10 +182,10 @@ func (mr *MockVMSetMockRecorder) GetAgentPoolVMSetNames(ctx, nodes any) *gomock.
 }
 
 // GetDataDisks mocks base method.
-func (m *MockVMSet) GetDataDisks(ctx context.Context, nodeName types.NodeName, crt cache.AzureCacheReadType) ([]*v6.DataDisk, *string, error) {
+func (m *MockVMSet) GetDataDisks(ctx context.Context, nodeName types.NodeName, crt cache.AzureCacheReadType) ([]*armcompute.DataDisk, *string, error) {
 	m.ctrl.T.Helper()
 	ret := m.ctrl.Call(m, "GetDataDisks", ctx, nodeName, crt)
-	ret0, _ := ret[0].([]*v6.DataDisk)
+	ret0, _ := ret[0].([]*armcompute.DataDisk)
 	ret1, _ := ret[1].(*string)
 	ret2, _ := ret[2].(error)
 	return ret0, ret1, ret2
@@ -396,10 +395,10 @@ func (mr *MockVMSetMockRecorder) GetVMSetNames(ctx, service, nodes any) *gomock.
 }
 
 // GetZoneByNodeName mocks base method.
-func (m *MockVMSet) GetZoneByNodeName(ctx context.Context, name string) (cloud_provider.Zone, error) {
+func (m *MockVMSet) GetZoneByNodeName(ctx context.Context, name string) (cloudprovider.Zone, error) {
 	m.ctrl.T.Helper()
 	ret := m.ctrl.Call(m, "GetZoneByNodeName", ctx, name)
-	ret0, _ := ret[0].(cloud_provider.Zone)
+	ret0, _ := ret[0].(cloudprovider.Zone)
 	ret1, _ := ret[1].(error)
 	return ret0, ret1
 }
@@ -410,6 +409,20 @@ func (mr *MockVMSetMockRecorder) GetZoneByNodeName(ctx, name any) *gomock.Call {
 	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetZoneByNodeName", reflect.TypeOf((*MockVMSet)(nil).GetZoneByNodeName), ctx, name)
 }
 
+// RefreshCaches mocks base method.
+func (m *MockVMSet) RefreshCaches() error {
+	m.ctrl.T.Helper()
+	ret := m.ctrl.Call(m, "RefreshCaches")
+	ret0, _ := ret[0].(error)
+	return ret0
+}
+
+// RefreshCaches indicates an expected call of RefreshCaches.
+func (mr *MockVMSetMockRecorder) RefreshCaches() *gomock.Call {
+	mr.mock.ctrl.T.Helper()
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "RefreshCaches", reflect.TypeOf((*MockVMSet)(nil).RefreshCaches))
+}
+
 // UpdateVM mocks base method.
 func (m *MockVMSet) UpdateVM(ctx context.Context, nodeName types.NodeName) error {
 	m.ctrl.T.Helper()
diff --git a/pkg/provider/azure_standard.go b/pkg/provider/azure_standard.go
index 78c4f757d..fdfbd9933 100644
--- a/pkg/provider/azure_standard.go
+++ b/pkg/provider/azure_standard.go
@@ -189,7 +189,6 @@ func getProtocolsFromKubernetesProtocol(protocol v1.Protocol) (*network.Transpor
 	default:
 		return &transportProto, securityProto, &probeProto, fmt.Errorf("only TCP, UDP and SCTP are supported for Azure LoadBalancers")
 	}
-
 }
 
 // This returns the full identifier of the primary NIC for the given VM.
@@ -446,15 +445,24 @@ func (as *availabilitySet) newVMASCache() (azcache.Resource, error) {
 	return azcache.NewTimedCache(time.Duration(as.Config.AvailabilitySetsCacheTTLInSeconds)*time.Second, getter, as.Cloud.Config.DisableAPICallCache)
 }
 
+// RefreshCaches invalidates and renew all related caches.
+func (as *availabilitySet) RefreshCaches() error {
+	var err error
+	as.vmasCache, err = as.newVMASCache()
+	if err != nil {
+		klog.Errorf("as.RefreshCaches: failed to create or refresh VMAS cache: %s", err)
+		return err
+	}
+	return nil
+}
+
 // newStandardSet creates a new availabilitySet.
 func newAvailabilitySet(az *Cloud) (VMSet, error) {
 	as := &availabilitySet{
 		Cloud: az,
 	}
 
-	var err error
-	as.vmasCache, err = as.newVMASCache()
-	if err != nil {
+	if err := as.RefreshCaches(); err != nil {
 		return nil, err
 	}
 
@@ -1010,7 +1018,6 @@ func (as *availabilitySet) EnsureBackendPoolDeleted(ctx context.Context, service
 				}
 			}
 		}
-
 	}
 	nicUpdaters := make([]func() error, 0)
 	allErrs := make([]error, 0)
diff --git a/pkg/provider/azure_vmsets.go b/pkg/provider/azure_vmsets.go
index 52861cd71..5b7c62acd 100644
--- a/pkg/provider/azure_vmsets.go
+++ b/pkg/provider/azure_vmsets.go
@@ -72,7 +72,7 @@ type VMSet interface {
 	EnsureHostInPool(ctx context.Context, service *v1.Service, nodeName types.NodeName, backendPoolID string, vmSetName string) (string, string, string, *compute.VirtualMachineScaleSetVM, error)
 	// EnsureBackendPoolDeleted ensures the loadBalancer backendAddressPools deleted from the specified nodes.
 	EnsureBackendPoolDeleted(ctx context.Context, service *v1.Service, backendPoolIDs []string, vmSetName string, backendAddressPools *[]network.BackendAddressPool, deleteFromVMSet bool) (bool, error)
-	//EnsureBackendPoolDeletedFromVMSets ensures the loadBalancer backendAddressPools deleted from the specified VMSS/VMAS
+	// EnsureBackendPoolDeletedFromVMSets ensures the loadBalancer backendAddressPools deleted from the specified VMSS/VMAS
 	EnsureBackendPoolDeletedFromVMSets(ctx context.Context, vmSetNamesMap map[string]bool, backendPoolIDs []string) error
 
 	// AttachDisk attaches a disk to vm
@@ -107,6 +107,9 @@ type VMSet interface {
 
 	// DeleteCacheForNode removes the node entry from cache.
 	DeleteCacheForNode(ctx context.Context, nodeName string) error
+
+	// RefreshCaches invalidates and renew all related caches.
+	RefreshCaches() error
 }
 
 // AttachDiskOptions attach disk options
diff --git a/pkg/provider/azure_vmss.go b/pkg/provider/azure_vmss.go
index 167c2fb04..10be64201 100644
--- a/pkg/provider/azure_vmss.go
+++ b/pkg/provider/azure_vmss.go
@@ -105,6 +105,34 @@ type ScaleSet struct {
 	lockMap *LockMap
 }
 
+// RefreshCaches invalidates and renew all related caches.
+func (ss *ScaleSet) RefreshCaches() error {
+	logger := klog.Background().WithName("ss.RefreshCaches")
+
+	var err error
+	ss.vmssCache, err = ss.newVMSSCache()
+	if err != nil {
+		logger.Error(err, "failed to create or refresh vmss cache")
+		return err
+	}
+
+	if !ss.DisableAvailabilitySetNodes || ss.EnableVmssFlexNodes {
+		ss.nonVmssUniformNodesCache, err = ss.newNonVmssUniformNodesCache()
+		if err != nil {
+			logger.Error(err, "failed to create or refresh nonVmssUniformNodes cache")
+			return err
+		}
+	}
+
+	ss.vmssVMCache, err = ss.newVMSSVirtualMachinesCache()
+	if err != nil {
+		logger.Error(err, "failed to create or refresh vmssVM cache")
+		return err
+	}
+
+	return nil
+}
+
 // newScaleSet creates a new ScaleSet.
 func newScaleSet(az *Cloud) (VMSet, error) {
 	if az.Config.VmssVirtualMachinesCacheTTLInSeconds == 0 {
@@ -128,20 +156,7 @@ func newScaleSet(az *Cloud) (VMSet, error) {
 		lockMap:         newLockMap(),
 	}
 
-	if !ss.DisableAvailabilitySetNodes || ss.EnableVmssFlexNodes {
-		ss.nonVmssUniformNodesCache, err = ss.newNonVmssUniformNodesCache()
-		if err != nil {
-			return nil, err
-		}
-	}
-
-	ss.vmssCache, err = ss.newVMSSCache()
-	if err != nil {
-		return nil, err
-	}
-
-	ss.vmssVMCache, err = ss.newVMSSVirtualMachinesCache()
-	if err != nil {
+	if err := ss.RefreshCaches(); err != nil {
 		return nil, err
 	}
 
@@ -425,7 +440,6 @@ func (ss *ScaleSet) GetInstanceIDByNodeName(ctx context.Context, name string) (s
 // azure:///subscriptions/subsid/resourceGroups/rg/providers/Microsoft.Compute/virtualMachineScaleSets/aks-agentpool-22126781-vmss/virtualMachines/1
 // /subscriptions/subsid/resourceGroups/rg/providers/Microsoft.Compute/virtualMachineScaleSets/aks-agentpool-22126781-vmss/virtualMachines/k8s-agentpool-36841236-vmss_1
 func (ss *ScaleSet) GetNodeNameByProviderID(ctx context.Context, providerID string) (types.NodeName, error) {
-
 	vmManagementType, err := ss.getVMManagementTypeByProviderID(ctx, providerID, azcache.CacheReadTypeUnsafe)
 	if err != nil {
 		klog.Errorf("Failed to check VM management type: %v", err)
@@ -1904,7 +1918,6 @@ func (ss *ScaleSet) ensureBackendPoolDeleted(ctx context.Context, service *v1.Se
 		meta := meta
 		update := update
 		hostUpdates = append(hostUpdates, func() error {
-
 			logFields := []interface{}{
 				"operation", "EnsureBackendPoolDeleted UpdateVMSSVMs",
 				"vmssName", meta.vmssName,
@@ -2053,7 +2066,6 @@ func (ss *ScaleSet) EnsureBackendPoolDeleted(ctx context.Context, service *v1.Se
 	}
 
 	return updated, nil
-
 }
 
 // GetNodeCIDRMaskByProviderID returns the node CIDR subnet mask by provider ID.
diff --git a/pkg/provider/azure_vmssflex.go b/pkg/provider/azure_vmssflex.go
index 6dea67625..d2b779768 100644
--- a/pkg/provider/azure_vmssflex.go
+++ b/pkg/provider/azure_vmssflex.go
@@ -41,10 +41,8 @@ import (
 	vmutil "sigs.k8s.io/cloud-provider-azure/pkg/util/vm"
 )
 
-var (
-	// ErrorVmssIDIsEmpty indicates the vmss id is empty.
-	ErrorVmssIDIsEmpty = errors.New("VMSS ID is empty")
-)
+// ErrorVmssIDIsEmpty indicates the vmss id is empty.
+var ErrorVmssIDIsEmpty = errors.New("VMSS ID is empty")
 
 // FlexScaleSet implements VMSet interface for Azure Flexible VMSS.
 type FlexScaleSet struct {
@@ -59,6 +57,23 @@ type FlexScaleSet struct {
 	lockMap *LockMap
 }
 
+// RefreshCaches invalidates and renew all related caches.
+func (fs *FlexScaleSet) RefreshCaches() error {
+	logger := klog.Background().WithName("fs.RefreshCaches")
+	var err error
+	fs.vmssFlexCache, err = fs.newVmssFlexCache()
+	if err != nil {
+		logger.Error(err, "failed to create or refresh vmssFlexCache")
+		return err
+	}
+	fs.vmssFlexVMCache, err = fs.newVmssFlexVMCache()
+	if err != nil {
+		logger.Error(err, "failed to create or refresh vmssFlexVMCache")
+		return err
+	}
+	return nil
+}
+
 func newFlexScaleSet(az *Cloud) (VMSet, error) {
 	fs := &FlexScaleSet{
 		Cloud:                    az,
@@ -67,13 +82,7 @@ func newFlexScaleSet(az *Cloud) (VMSet, error) {
 		lockMap:                  newLockMap(),
 	}
 
-	var err error
-	fs.vmssFlexCache, err = fs.newVmssFlexCache()
-	if err != nil {
-		return nil, err
-	}
-	fs.vmssFlexVMCache, err = fs.newVmssFlexVMCache()
-	if err != nil {
+	if err := fs.RefreshCaches(); err != nil {
 		return nil, err
 	}
 
@@ -97,7 +106,6 @@ func (fs *FlexScaleSet) getNodeVmssFlexName(ctx context.Context, nodeName string
 		return "", err
 	}
 	return vmssFlexName, nil
-
 }
 
 // GetNodeVMSetName returns the availability set or vmss name by the node name.
@@ -195,7 +203,6 @@ func (fs *FlexScaleSet) GetInstanceIDByNodeName(ctx context.Context, name string
 		return "", err
 	}
 	return convertedResourceID, nil
-
 }
 
 // GetInstanceTypeByNodeName gets the instance type by node name.
@@ -339,7 +346,6 @@ func (fs *FlexScaleSet) GetIPByNodeName(ctx context.Context, name string) (strin
 	}
 
 	return privateIP, publicIP, nil
-
 }
 
 // GetPrivateIPsByNodeName returns a slice of all private ips assigned to node (ipv6 and ipv4)
@@ -394,7 +400,6 @@ func (fs *FlexScaleSet) getNodeInformationByIPConfigurationID(ctx context.Contex
 	}
 
 	vmssFlexName, err := fs.getNodeVmssFlexName(ctx, nodeName)
-
 	if err != nil {
 		klog.Errorf("Unable to get the vmss flex name by node name %s: %v", vmName, err)
 		return "", "", "", err
@@ -545,7 +550,6 @@ func (fs *FlexScaleSet) EnsureHostInPool(ctx context.Context, service *v1.Servic
 	}
 
 	return nodeResourceGroup, vmssFlexName, name, nil, nil
-
 }
 
 func (fs *FlexScaleSet) ensureVMSSFlexInPool(ctx context.Context, _ *v1.Service, nodes []*v1.Node, backendPoolID string, vmSetNameOfLB string) error {
@@ -910,7 +914,6 @@ func (fs *FlexScaleSet) EnsureBackendPoolDeleted(ctx context.Context, service *v
 					continue
 				}
 			}
-
 		}
 	}
 
@@ -936,7 +939,6 @@ func (fs *FlexScaleSet) EnsureBackendPoolDeleted(ctx context.Context, service *v
 
 	isOperationSucceeded = true
 	return nicUpdated, nil
-
 }
 
 func (fs *FlexScaleSet) ensureBackendPoolDeletedFromNode(ctx context.Context, vmssFlexVMNameMap map[string]string, backendPoolIDs []string) (bool, error) {
diff --git a/pkg/provider/azure_wrap.go b/pkg/provider/azure_wrap.go
index 1e270b998..6a819329a 100644
--- a/pkg/provider/azure_wrap.go
+++ b/pkg/provider/azure_wrap.go
@@ -17,14 +17,11 @@ limitations under the License.
 package provider
 
 import (
-	"errors"
 	"fmt"
 	"net/http"
 	"regexp"
 	"strings"
 
-	"github.com/Azure/azure-sdk-for-go/sdk/azcore"
-
 	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
 	"sigs.k8s.io/cloud-provider-azure/pkg/retry"
 )
@@ -32,7 +29,6 @@ import (
 var (
 	vmCacheTTLDefaultInSeconds           = 60
 	loadBalancerCacheTTLDefaultInSeconds = 120
-	nsgCacheTTLDefaultInSeconds          = 120
 	routeTableCacheTTLDefaultInSeconds   = 120
 	publicIPCacheTTLDefaultInSeconds     = 120
 	plsCacheTTLDefaultInSeconds          = 120
@@ -56,20 +52,6 @@ func checkResourceExistsFromError(err *retry.Error) (bool, *retry.Error) {
 	return false, err
 }
 
-func checkResourceExistsFromAzcoreError(err error) (bool, error) {
-	if err == nil {
-		return true, nil
-	}
-	var respError *azcore.ResponseError
-	if errors.As(err, &respError) && respError != nil {
-		if respError.StatusCode == http.StatusNotFound {
-			return false, nil
-		}
-	}
-
-	return false, err
-}
-
 func (az *Cloud) useStandardLoadBalancer() bool {
 	return strings.EqualFold(az.LoadBalancerSku, consts.LoadBalancerSkuStandard)
 }
diff --git a/pkg/provider/loadbalancer/accesscontrol.go b/pkg/provider/loadbalancer/accesscontrol.go
index 7ada58579..71c1ad72f 100644
--- a/pkg/provider/loadbalancer/accesscontrol.go
+++ b/pkg/provider/loadbalancer/accesscontrol.go
@@ -27,9 +27,9 @@ import (
 	"k8s.io/utils/ptr"
 
 	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
-	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer/fnutil"
-	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer/iputil"
-	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer/securitygroup"
+	"sigs.k8s.io/cloud-provider-azure/pkg/provider/securitygroup"
+	fnutil "sigs.k8s.io/cloud-provider-azure/pkg/util/collectionutil"
+	"sigs.k8s.io/cloud-provider-azure/pkg/util/iputil"
 )
 
 var (
diff --git a/pkg/provider/loadbalancer/configuration.go b/pkg/provider/loadbalancer/configuration.go
index 95d2dc6bf..d1f3d4683 100644
--- a/pkg/provider/loadbalancer/configuration.go
+++ b/pkg/provider/loadbalancer/configuration.go
@@ -25,7 +25,7 @@ import (
 	v1 "k8s.io/api/core/v1"
 
 	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
-	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer/iputil"
+	"sigs.k8s.io/cloud-provider-azure/pkg/util/iputil"
 )
 
 // IsInternal returns true if the given service is internal load balancer.
diff --git a/pkg/provider/azure_securitygroup_repo.go b/pkg/provider/securitygroup/azure_securitygroup_repo.go
similarity index 55%
rename from pkg/provider/azure_securitygroup_repo.go
rename to pkg/provider/securitygroup/azure_securitygroup_repo.go
index c33c3e012..0c602b8ee 100644
--- a/pkg/provider/azure_securitygroup_repo.go
+++ b/pkg/provider/securitygroup/azure_securitygroup_repo.go
@@ -14,7 +14,7 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
-package provider
+package securitygroup
 
 import (
 	"context"
@@ -31,20 +31,66 @@ import (
 	"k8s.io/klog/v2"
 	"k8s.io/utils/ptr"
 
+	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/securitygroupclient"
 	azcache "sigs.k8s.io/cloud-provider-azure/pkg/cache"
 	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
+	"sigs.k8s.io/cloud-provider-azure/pkg/util/errutils"
 )
 
-// CreateOrUpdateSecurityGroup invokes az.SecurityGroupsClient.CreateOrUpdate with exponential backoff retry
-func (az *Cloud) CreateOrUpdateSecurityGroup(sg *armnetwork.SecurityGroup) error {
-	ctx, cancel := getContextWithCancel()
-	defer cancel()
-	clientFactory := az.NetworkClientFactory
-	if clientFactory == nil {
-		clientFactory = az.ComputeClientFactory
+const (
+	nsgCacheTTLDefaultInSeconds = 120
+)
+
+type Repository interface {
+	GetSecurityGroup(ctx context.Context) (*armnetwork.SecurityGroup, error)
+	CreateOrUpdateSecurityGroup(ctx context.Context, sg *armnetwork.SecurityGroup) error
+}
+
+type securityGroupRepo struct {
+	securityGroupResourceGroup string
+	securityGroupName          string
+	nsgCacheTTLInSeconds       int
+	securigyGroupClient        securitygroupclient.Interface
+	nsgCache                   azcache.Resource
+}
+
+func NewSecurityGroupRepo(securityGroupResourceGroup string, securityGroupName string, nsgCacheTTLInSeconds int, disableAPICallCache bool, securityGroupClient securitygroupclient.Interface) (Repository, error) {
+	getter := func(ctx context.Context, key string) (interface{}, error) {
+		nsg, err := securityGroupClient.Get(ctx, securityGroupResourceGroup, key)
+		exists, rerr := errutils.CheckResourceExistsFromAzcoreError(err)
+		if rerr != nil {
+			return nil, err
+		}
+
+		if !exists {
+			klog.V(2).Infof("Security group %q not found", key)
+			return nil, nil
+		}
+
+		return nsg, nil
+	}
+
+	if nsgCacheTTLInSeconds == 0 {
+		nsgCacheTTLInSeconds = nsgCacheTTLDefaultInSeconds
+	}
+	cache, err := azcache.NewTimedCache(time.Duration(nsgCacheTTLInSeconds)*time.Second, getter, disableAPICallCache)
+	if err != nil {
+		klog.Errorf("Failed to create cache for security group %q: %v", securityGroupName, err)
+		return nil, err
 	}
-	sgClient := clientFactory.GetSecurityGroupClient()
-	_, rerr := sgClient.CreateOrUpdate(ctx, az.SecurityGroupResourceGroup, *sg.Name, *sg)
+
+	return &securityGroupRepo{
+		securityGroupResourceGroup: securityGroupResourceGroup,
+		securityGroupName:          securityGroupName,
+		nsgCacheTTLInSeconds:       nsgCacheTTLDefaultInSeconds,
+		securigyGroupClient:        securityGroupClient,
+		nsgCache:                   cache,
+	}, nil
+}
+
+// CreateOrUpdateSecurityGroup invokes az.SecurityGroupsClient.CreateOrUpdate with exponential backoff retry
+func (az *securityGroupRepo) CreateOrUpdateSecurityGroup(ctx context.Context, sg *armnetwork.SecurityGroup) error {
+	_, rerr := az.securigyGroupClient.CreateOrUpdate(ctx, az.securityGroupResourceGroup, *sg.Name, *sg)
 	klog.V(10).Infof("SecurityGroupsClient.CreateOrUpdate(%s): end", *sg.Name)
 	if rerr == nil {
 		// Invalidate the cache right after updating
@@ -71,47 +117,19 @@ func (az *Cloud) CreateOrUpdateSecurityGroup(sg *armnetwork.SecurityGroup) error
 	return rerr
 }
 
-func (az *Cloud) newNSGCache() (azcache.Resource, error) {
-	getter := func(ctx context.Context, key string) (interface{}, error) {
-		clientFactory := az.NetworkClientFactory
-		if clientFactory == nil {
-			clientFactory = az.ComputeClientFactory
-		}
-		sgClient := clientFactory.GetSecurityGroupClient()
-
-		nsg, err := sgClient.Get(ctx, az.SecurityGroupResourceGroup, key)
-		exists, rerr := checkResourceExistsFromAzcoreError(err)
-		if rerr != nil {
-			return nil, err
-		}
-
-		if !exists {
-			klog.V(2).Infof("Security group %q not found", key)
-			return nil, nil
-		}
-
-		return nsg, nil
-	}
-
-	if az.NsgCacheTTLInSeconds == 0 {
-		az.NsgCacheTTLInSeconds = nsgCacheTTLDefaultInSeconds
-	}
-	return azcache.NewTimedCache(time.Duration(az.NsgCacheTTLInSeconds)*time.Second, getter, az.Config.DisableAPICallCache)
-}
-
-func (az *Cloud) getSecurityGroup(ctx context.Context, crt azcache.AzureCacheReadType) (*armnetwork.SecurityGroup, error) {
+func (az *securityGroupRepo) GetSecurityGroup(ctx context.Context) (*armnetwork.SecurityGroup, error) {
 	nsg := &armnetwork.SecurityGroup{}
-	if az.SecurityGroupName == "" {
+	if az.securityGroupName == "" {
 		return nsg, fmt.Errorf("securityGroupName is not configured")
 	}
 
-	securityGroup, err := az.nsgCache.GetWithDeepCopy(ctx, az.SecurityGroupName, crt)
+	securityGroup, err := az.nsgCache.GetWithDeepCopy(ctx, az.securityGroupName, azcache.CacheReadTypeDefault)
 	if err != nil {
 		return nsg, err
 	}
 
 	if securityGroup == nil {
-		return nsg, fmt.Errorf("nsg %q not found", az.SecurityGroupName)
+		return nsg, fmt.Errorf("nsg %q not found", az.securityGroupName)
 	}
 
 	return securityGroup.(*armnetwork.SecurityGroup), nil
diff --git a/pkg/provider/securitygroup/azure_securitygroup_repo_mock.go b/pkg/provider/securitygroup/azure_securitygroup_repo_mock.go
new file mode 100644
index 000000000..017bc064b
--- /dev/null
+++ b/pkg/provider/securitygroup/azure_securitygroup_repo_mock.go
@@ -0,0 +1,87 @@
+// /*
+// Copyright The Kubernetes Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+// */
+//
+
+// Code generated by MockGen. DO NOT EDIT.
+// Source: azure_securitygroup_repo.go
+//
+// Generated by this command:
+//
+//	mockgen -package securitygroup -source azure_securitygroup_repo.go -self_package sigs.k8s.io/cloud-provider-azure/pkg/provider/securitygroup -copyright_file ../../../hack/boilerplate/boilerplate.generatego.txt
+//
+
+// Package securitygroup is a generated GoMock package.
+package securitygroup
+
+import (
+	context "context"
+	reflect "reflect"
+
+	armnetwork "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork/v6"
+	gomock "go.uber.org/mock/gomock"
+)
+
+// MockRepository is a mock of Repository interface.
+type MockRepository struct {
+	ctrl     *gomock.Controller
+	recorder *MockRepositoryMockRecorder
+}
+
+// MockRepositoryMockRecorder is the mock recorder for MockRepository.
+type MockRepositoryMockRecorder struct {
+	mock *MockRepository
+}
+
+// NewMockRepository creates a new mock instance.
+func NewMockRepository(ctrl *gomock.Controller) *MockRepository {
+	mock := &MockRepository{ctrl: ctrl}
+	mock.recorder = &MockRepositoryMockRecorder{mock}
+	return mock
+}
+
+// EXPECT returns an object that allows the caller to indicate expected use.
+func (m *MockRepository) EXPECT() *MockRepositoryMockRecorder {
+	return m.recorder
+}
+
+// CreateOrUpdateSecurityGroup mocks base method.
+func (m *MockRepository) CreateOrUpdateSecurityGroup(ctx context.Context, sg *armnetwork.SecurityGroup) error {
+	m.ctrl.T.Helper()
+	ret := m.ctrl.Call(m, "CreateOrUpdateSecurityGroup", ctx, sg)
+	ret0, _ := ret[0].(error)
+	return ret0
+}
+
+// CreateOrUpdateSecurityGroup indicates an expected call of CreateOrUpdateSecurityGroup.
+func (mr *MockRepositoryMockRecorder) CreateOrUpdateSecurityGroup(ctx, sg any) *gomock.Call {
+	mr.mock.ctrl.T.Helper()
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CreateOrUpdateSecurityGroup", reflect.TypeOf((*MockRepository)(nil).CreateOrUpdateSecurityGroup), ctx, sg)
+}
+
+// GetSecurityGroup mocks base method.
+func (m *MockRepository) GetSecurityGroup(ctx context.Context) (*armnetwork.SecurityGroup, error) {
+	m.ctrl.T.Helper()
+	ret := m.ctrl.Call(m, "GetSecurityGroup", ctx)
+	ret0, _ := ret[0].(*armnetwork.SecurityGroup)
+	ret1, _ := ret[1].(error)
+	return ret0, ret1
+}
+
+// GetSecurityGroup indicates an expected call of GetSecurityGroup.
+func (mr *MockRepositoryMockRecorder) GetSecurityGroup(ctx any) *gomock.Call {
+	mr.mock.ctrl.T.Helper()
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetSecurityGroup", reflect.TypeOf((*MockRepository)(nil).GetSecurityGroup), ctx)
+}
diff --git a/pkg/provider/loadbalancer/securitygroup/securitygroup.go b/pkg/provider/securitygroup/securitygroup.go
similarity index 99%
rename from pkg/provider/loadbalancer/securitygroup/securitygroup.go
rename to pkg/provider/securitygroup/securitygroup.go
index 8fb6213fc..7ed4be70f 100644
--- a/pkg/provider/loadbalancer/securitygroup/securitygroup.go
+++ b/pkg/provider/securitygroup/securitygroup.go
@@ -30,8 +30,8 @@ import (
 	"k8s.io/utils/ptr"
 
 	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
-	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer/fnutil"
-	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer/iputil"
+	fnutil "sigs.k8s.io/cloud-provider-azure/pkg/util/collectionutil"
+	"sigs.k8s.io/cloud-provider-azure/pkg/util/iputil"
 )
 
 const (
diff --git a/pkg/provider/loadbalancer/securitygroup/securityrule.go b/pkg/provider/securitygroup/securityrule.go
similarity index 97%
rename from pkg/provider/loadbalancer/securitygroup/securityrule.go
rename to pkg/provider/securitygroup/securityrule.go
index 1cd843d44..481c26f13 100644
--- a/pkg/provider/loadbalancer/securitygroup/securityrule.go
+++ b/pkg/provider/securitygroup/securityrule.go
@@ -27,8 +27,8 @@ import (
 	"github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork/v6"
 	v1 "k8s.io/api/core/v1"
 
-	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer/fnutil"
-	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer/iputil"
+	fnutil "sigs.k8s.io/cloud-provider-azure/pkg/util/collectionutil"
+	"sigs.k8s.io/cloud-provider-azure/pkg/util/iputil"
 )
 
 // GenerateAllowSecurityRuleName returns the AllowInbound rule name based on the given rule properties.
diff --git a/pkg/provider/loadbalancer/fnutil/map.go b/pkg/util/collectionutil/map.go
similarity index 100%
rename from pkg/provider/loadbalancer/fnutil/map.go
rename to pkg/util/collectionutil/map.go
diff --git a/pkg/provider/loadbalancer/fnutil/slice.go b/pkg/util/collectionutil/slice.go
similarity index 100%
rename from pkg/provider/loadbalancer/fnutil/slice.go
rename to pkg/util/collectionutil/slice.go
diff --git a/pkg/azureclients/routeclient/mockrouteclient/doc.go b/pkg/util/errutils/err.go
similarity index 55%
rename from pkg/azureclients/routeclient/mockrouteclient/doc.go
rename to pkg/util/errutils/err.go
index 5c78c86ae..477b816e4 100644
--- a/pkg/azureclients/routeclient/mockrouteclient/doc.go
+++ b/pkg/util/errutils/err.go
@@ -1,5 +1,5 @@
 /*
-Copyright 2020 The Kubernetes Authors.
+Copyright 2022 The Kubernetes Authors.
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -14,5 +14,24 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
-// Package mockrouteclient implements the mock client for Route.
-package mockrouteclient // import "sigs.k8s.io/cloud-provider-azure/pkg/azureclients/routeclient/mockrouteclient"
+package errutils
+
+import (
+	"errors"
+	"net/http"
+
+	"github.com/Azure/azure-sdk-for-go/sdk/azcore"
+)
+
+func CheckResourceExistsFromAzcoreError(err error) (bool, error) {
+	if err == nil {
+		return true, nil
+	}
+	var respError *azcore.ResponseError
+	if errors.As(err, &respError) && respError != nil {
+		if respError.StatusCode == http.StatusNotFound {
+			return false, nil
+		}
+	}
+	return false, err
+}
diff --git a/pkg/provider/loadbalancer/iputil/addr.go b/pkg/util/iputil/addr.go
similarity index 100%
rename from pkg/provider/loadbalancer/iputil/addr.go
rename to pkg/util/iputil/addr.go
diff --git a/pkg/provider/loadbalancer/iputil/addr_test.go b/pkg/util/iputil/addr_test.go
similarity index 100%
rename from pkg/provider/loadbalancer/iputil/addr_test.go
rename to pkg/util/iputil/addr_test.go
diff --git a/pkg/provider/loadbalancer/iputil/family.go b/pkg/util/iputil/family.go
similarity index 94%
rename from pkg/provider/loadbalancer/iputil/family.go
rename to pkg/util/iputil/family.go
index 9ac42b92f..65a8ce220 100644
--- a/pkg/provider/loadbalancer/iputil/family.go
+++ b/pkg/util/iputil/family.go
@@ -19,7 +19,7 @@ package iputil
 import (
 	"net/netip"
 
-	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer/fnutil"
+	fnutil "sigs.k8s.io/cloud-provider-azure/pkg/util/collectionutil"
 )
 
 type Family string
diff --git a/vendor/modules.txt b/vendor/modules.txt
index 838db5ce4..bb7628e90 100644
--- a/vendor/modules.txt
+++ b/vendor/modules.txt
@@ -422,7 +422,7 @@ github.com/pkg/errors
 # github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2
 ## explicit
 github.com/pmezard/go-difflib/difflib
-# github.com/prometheus/client_golang v1.20.4
+# github.com/prometheus/client_golang v1.20.5
 ## explicit; go 1.20
 github.com/prometheus/client_golang/internal/github.com/golang/gddo/httputil
 github.com/prometheus/client_golang/internal/github.com/golang/gddo/httputil/header


Test patch:
diff --git a/e2e.Dockerfile b/e2e.Dockerfile
index 567135748..f53b9a4a5 100644
--- a/e2e.Dockerfile
+++ b/e2e.Dockerfile
@@ -14,7 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM golang:1.23-bullseye@sha256:ebaf58d74f200c27dce0181203a1347e41b5c159c9aa0517a95bec594aabbde4
+FROM golang:1.23-bullseye@sha256:48ac5022f9740543cac0eba41a4e37b721073a0103349e416678e0142a53b49a
 
 WORKDIR /go/src/sigs.k8s.io/cloud-provider-azure
 
diff --git a/internal/testutil/fixture/azure_securitygroup.go b/internal/testutil/fixture/azure_securitygroup.go
index 721e381aa..a979fa364 100644
--- a/internal/testutil/fixture/azure_securitygroup.go
+++ b/internal/testutil/fixture/azure_securitygroup.go
@@ -27,9 +27,9 @@ import (
 
 	"sigs.k8s.io/cloud-provider-azure/internal/testutil"
 	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
-	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer/fnutil"
-	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer/iputil"
-	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer/securitygroup"
+	"sigs.k8s.io/cloud-provider-azure/pkg/provider/securitygroup"
+	fnutil "sigs.k8s.io/cloud-provider-azure/pkg/util/collectionutil"
+	"sigs.k8s.io/cloud-provider-azure/pkg/util/iputil"
 )
 
 // NoiseSecurityRules returns 3 non cloud-provider-specific security rules.
diff --git a/internal/testutil/fixture/fixture.go b/internal/testutil/fixture/fixture.go
index 3278676ad..92aeba488 100644
--- a/internal/testutil/fixture/fixture.go
+++ b/internal/testutil/fixture/fixture.go
@@ -24,7 +24,7 @@ import (
 	"math/big"
 	"net/netip"
 
-	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer/fnutil"
+	fnutil "sigs.k8s.io/cloud-provider-azure/pkg/util/collectionutil"
 )
 
 type Fixture struct{}
diff --git a/pkg/azureclients/containerserviceclient/azure_containerserviceclient_test.go b/pkg/azureclients/containerserviceclient/azure_containerserviceclient_test.go
deleted file mode 100644
index 615815eea..000000000
--- a/pkg/azureclients/containerserviceclient/azure_containerserviceclient_test.go
+++ /dev/null
@@ -1,615 +0,0 @@
-/*
-Copyright 2020 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package containerserviceclient
-
-import (
-	"bytes"
-	"context"
-	"encoding/json"
-	"fmt"
-	"io"
-	"net/http"
-	"testing"
-	"time"
-
-	"github.com/Azure/azure-sdk-for-go/services/containerservice/mgmt/2021-10-01/containerservice"
-	"github.com/Azure/go-autorest/autorest"
-	"github.com/stretchr/testify/assert"
-
-	"go.uber.org/mock/gomock"
-
-	"k8s.io/client-go/util/flowcontrol"
-	"k8s.io/utils/ptr"
-
-	azclients "sigs.k8s.io/cloud-provider-azure/pkg/azureclients"
-	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/armclient"
-	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/armclient/mockarmclient"
-	"sigs.k8s.io/cloud-provider-azure/pkg/retry"
-)
-
-const (
-	testResourceID     = "/subscriptions/subscriptionID/resourceGroups/rg/providers/Microsoft.ContainerService/managedClusters/cluster"
-	testResourcePrefix = "/subscriptions/subscriptionID/resourceGroups/rg/providers/Microsoft.ContainerService/managedClusters"
-)
-
-// 2065-01-24 05:20:00 +0000 UTC
-func getFutureTime() time.Time {
-	return time.Unix(3000000000, 0)
-}
-
-func getTestManagedClusterClient(armClient armclient.Interface) *Client {
-	rateLimiterReader, rateLimiterWriter := azclients.NewRateLimiter(&azclients.RateLimitConfig{})
-	return &Client{
-		armClient:         armClient,
-		subscriptionID:    "subscriptionID",
-		rateLimiterReader: rateLimiterReader,
-		rateLimiterWriter: rateLimiterWriter,
-	}
-}
-
-func getTestManagedClusterClientWithNeverRateLimiter(armClient armclient.Interface) *Client {
-	rateLimiterReader := flowcontrol.NewFakeNeverRateLimiter()
-	rateLimiterWriter := flowcontrol.NewFakeNeverRateLimiter()
-	return &Client{
-		armClient:         armClient,
-		subscriptionID:    "subscriptionID",
-		rateLimiterReader: rateLimiterReader,
-		rateLimiterWriter: rateLimiterWriter,
-	}
-}
-
-func getTestManagedClusterClientWithRetryAfterReader(armClient armclient.Interface) *Client {
-	rateLimiterReader := flowcontrol.NewFakeAlwaysRateLimiter()
-	rateLimiterWriter := flowcontrol.NewFakeAlwaysRateLimiter()
-	return &Client{
-		armClient:         armClient,
-		subscriptionID:    "subscriptionID",
-		rateLimiterReader: rateLimiterReader,
-		rateLimiterWriter: rateLimiterWriter,
-		RetryAfterReader:  getFutureTime(),
-		RetryAfterWriter:  getFutureTime(),
-	}
-}
-
-func getTestManagedCluster(name string) containerservice.ManagedCluster {
-	return containerservice.ManagedCluster{
-		ID:       ptr.To(fmt.Sprintf("/subscriptions/subscriptionID/resourceGroups/rg/providers/Microsoft.ContainerService/managedClusters/%s", name)),
-		Name:     ptr.To(name),
-		Location: ptr.To("eastus"),
-	}
-}
-
-func TestNew(t *testing.T) {
-	config := &azclients.ClientConfig{
-		SubscriptionID:          "sub",
-		ResourceManagerEndpoint: "endpoint",
-		Location:                "eastus",
-		RateLimitConfig: &azclients.RateLimitConfig{
-			CloudProviderRateLimit:            true,
-			CloudProviderRateLimitQPS:         0.5,
-			CloudProviderRateLimitBucket:      1,
-			CloudProviderRateLimitQPSWrite:    0.5,
-			CloudProviderRateLimitBucketWrite: 1,
-		},
-		Backoff: &retry.Backoff{Steps: 1},
-	}
-
-	mcClient := New(config)
-	assert.Equal(t, "sub", mcClient.subscriptionID)
-	assert.NotEmpty(t, mcClient.rateLimiterReader)
-	assert.NotEmpty(t, mcClient.rateLimiterWriter)
-}
-
-func TestGet(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	response := &http.Response{
-		StatusCode: http.StatusOK,
-		Body:       io.NopCloser(bytes.NewReader([]byte("{}"))),
-	}
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	armClient.EXPECT().GetResource(gomock.Any(), testResourceID).Return(response, nil).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-
-	expected := containerservice.ManagedCluster{}
-	expected.Response = autorest.Response{Response: response}
-	mcClient := getTestManagedClusterClient(armClient)
-	result, rerr := mcClient.Get(context.TODO(), "rg", "cluster")
-	assert.Equal(t, expected, result)
-	assert.Nil(t, rerr)
-}
-
-func TestGetNeverRateLimiter(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	mcGetErr := retry.GetRateLimitError(false, "GetManagedCluster")
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-
-	mcClient := getTestManagedClusterClientWithNeverRateLimiter(armClient)
-	expected := containerservice.ManagedCluster{}
-	result, rerr := mcClient.Get(context.TODO(), "rg", "cluster")
-	assert.Equal(t, expected, result)
-	assert.Equal(t, mcGetErr, rerr)
-}
-
-func TestGetRetryAfterReader(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	mcGetErr := retry.GetThrottlingError("GetManagedCluster", "client throttled", getFutureTime())
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-
-	mcClient := getTestManagedClusterClientWithRetryAfterReader(armClient)
-	expected := containerservice.ManagedCluster{}
-	result, rerr := mcClient.Get(context.TODO(), "rg", "cluster")
-	assert.Equal(t, expected, result)
-	assert.Equal(t, mcGetErr, rerr)
-}
-
-func TestGetThrottle(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	response := &http.Response{
-		StatusCode: http.StatusTooManyRequests,
-		Body:       io.NopCloser(bytes.NewReader([]byte("{}"))),
-	}
-	throttleErr := &retry.Error{
-		HTTPStatusCode: http.StatusTooManyRequests,
-		RawError:       fmt.Errorf("error"),
-		Retriable:      true,
-		RetryAfter:     time.Unix(100, 0),
-	}
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	armClient.EXPECT().GetResource(gomock.Any(), testResourceID).Return(response, throttleErr).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-
-	mcClient := getTestManagedClusterClient(armClient)
-	result, rerr := mcClient.Get(context.TODO(), "rg", "cluster")
-	assert.Empty(t, result)
-	assert.Equal(t, throttleErr, rerr)
-}
-
-func TestGetNotFound(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	response := &http.Response{
-		StatusCode: http.StatusNotFound,
-		Body:       io.NopCloser(bytes.NewReader([]byte("{}"))),
-	}
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	armClient.EXPECT().GetResource(gomock.Any(), testResourceID).Return(response, nil).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-
-	mcClient := getTestManagedClusterClient(armClient)
-	expected := containerservice.ManagedCluster{Response: autorest.Response{}}
-	result, rerr := mcClient.Get(context.TODO(), "rg", "cluster")
-	assert.Equal(t, expected, result)
-	assert.NotNil(t, rerr)
-	assert.Equal(t, http.StatusNotFound, rerr.HTTPStatusCode)
-}
-
-func TestGetInternalError(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	response := &http.Response{
-		StatusCode: http.StatusInternalServerError,
-		Body:       io.NopCloser(bytes.NewReader([]byte("{}"))),
-	}
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	armClient.EXPECT().GetResource(gomock.Any(), testResourceID).Return(response, nil).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-
-	mcClient := getTestManagedClusterClient(armClient)
-	expected := containerservice.ManagedCluster{Response: autorest.Response{}}
-	result, rerr := mcClient.Get(context.TODO(), "rg", "cluster")
-	assert.Equal(t, expected, result)
-	assert.NotNil(t, rerr)
-	assert.Equal(t, http.StatusInternalServerError, rerr.HTTPStatusCode)
-}
-
-func TestList(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	mcList := []containerservice.ManagedCluster{getTestManagedCluster("cluster"), getTestManagedCluster("cluster1"), getTestManagedCluster("cluster2")}
-	responseBody, err := json.Marshal(containerservice.ManagedClusterListResult{Value: &mcList})
-	assert.NoError(t, err)
-	armClient.EXPECT().GetResource(gomock.Any(), testResourcePrefix).Return(
-		&http.Response{
-			StatusCode: http.StatusOK,
-			Body:       io.NopCloser(bytes.NewReader(responseBody)),
-		}, nil).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-
-	mcClient := getTestManagedClusterClient(armClient)
-	result, rerr := mcClient.List(context.TODO(), "rg")
-	assert.Nil(t, rerr)
-	assert.Equal(t, 3, len(result))
-}
-
-func TestListNextResultsMultiPages(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	tests := []struct {
-		prepareErr error
-		sendErr    *retry.Error
-		statusCode int
-	}{
-		{
-			prepareErr: nil,
-			sendErr:    nil,
-		},
-		{
-			prepareErr: fmt.Errorf("error"),
-		},
-		{
-			sendErr: &retry.Error{RawError: fmt.Errorf("error")},
-		},
-	}
-
-	lastResult := containerservice.ManagedClusterListResult{
-		NextLink: ptr.To("next"),
-	}
-
-	for _, test := range tests {
-		armClient := mockarmclient.NewMockInterface(ctrl)
-		req := &http.Request{
-			Method: "GET",
-		}
-		armClient.EXPECT().PrepareGetRequest(gomock.Any(), gomock.Any()).Return(req, test.prepareErr)
-		if test.prepareErr == nil {
-			armClient.EXPECT().Send(gomock.Any(), req).Return(&http.Response{
-				StatusCode: http.StatusOK,
-				Body:       io.NopCloser(bytes.NewReader([]byte(`{"foo":"bar"}`))),
-			}, test.sendErr)
-			armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any())
-		}
-
-		mcClient := getTestManagedClusterClient(armClient)
-		result, err := mcClient.listNextResults(context.TODO(), lastResult)
-		if test.prepareErr != nil || test.sendErr != nil {
-			assert.Error(t, err)
-		} else {
-			assert.NoError(t, err)
-		}
-		if test.prepareErr != nil {
-			assert.Empty(t, result)
-		} else {
-			assert.NotEmpty(t, result)
-		}
-	}
-}
-
-func TestListNextResultsMultiPagesWithListResponderError(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	test := struct {
-		prepareErr error
-		sendErr    *retry.Error
-	}{
-		prepareErr: nil,
-		sendErr:    nil,
-	}
-
-	lastResult := containerservice.ManagedClusterListResult{
-		NextLink: ptr.To("next"),
-	}
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	req := &http.Request{
-		Method: "GET",
-	}
-	armClient.EXPECT().PrepareGetRequest(gomock.Any(), gomock.Any()).Return(req, test.prepareErr)
-	if test.prepareErr == nil {
-		armClient.EXPECT().Send(gomock.Any(), req).Return(&http.Response{
-			StatusCode: http.StatusNotFound,
-			Body:       io.NopCloser(bytes.NewReader([]byte(`{"foo":"bar"}`))),
-		}, test.sendErr)
-		armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any())
-	}
-
-	response := &http.Response{
-		StatusCode: http.StatusNotFound,
-		Body:       io.NopCloser(bytes.NewBuffer([]byte(`{"foo":"bar"}`))),
-	}
-	expected := containerservice.ManagedClusterListResult{}
-	expected.Response = autorest.Response{Response: response}
-	mcClient := getTestManagedClusterClient(armClient)
-	result, err := mcClient.listNextResults(context.TODO(), lastResult)
-	assert.Error(t, err)
-	assert.Equal(t, expected, result)
-}
-
-func TestListWithListResponderError(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	mcList := []containerservice.ManagedCluster{getTestManagedCluster("cluster"), getTestManagedCluster("cluster1"), getTestManagedCluster("cluster2")}
-	responseBody, err := json.Marshal(containerservice.ManagedClusterListResult{Value: &mcList})
-	assert.NoError(t, err)
-	armClient.EXPECT().GetResource(gomock.Any(), testResourcePrefix).Return(
-		&http.Response{
-			StatusCode: http.StatusNotFound,
-			Body:       io.NopCloser(bytes.NewReader(responseBody)),
-		}, nil).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-	mcClient := getTestManagedClusterClient(armClient)
-	result, rerr := mcClient.List(context.TODO(), "rg")
-	assert.NotNil(t, rerr)
-	assert.Equal(t, 0, len(result))
-}
-
-func TestListWithNextPage(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	mcList := []containerservice.ManagedCluster{getTestManagedCluster("cluster"), getTestManagedCluster("cluster1"), getTestManagedCluster("cluster2")}
-	responseBody, err := json.Marshal(containerservice.ManagedClusterListResult{Value: &mcList, NextLink: ptr.To("nextLink")})
-	assert.NoError(t, err)
-	_, err = json.Marshal(containerservice.ManagedClusterListResult{Value: &mcList})
-	assert.NoError(t, err)
-	armClient.EXPECT().GetResource(gomock.Any(), testResourcePrefix).Return(
-		&http.Response{
-			StatusCode: http.StatusOK,
-			Body:       io.NopCloser(bytes.NewReader(responseBody)),
-		}, nil).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-	mcClient := getTestManagedClusterClient(armClient)
-	result, rerr := mcClient.List(context.TODO(), "rg")
-	assert.Nil(t, rerr)
-	assert.Equal(t, 3, len(result))
-}
-
-func TestListNeverRateLimiter(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	mcListErr := &retry.Error{
-		RawError:  fmt.Errorf("azure cloud provider rate limited(%s) for operation %q", "read", "ListManagedCluster"),
-		Retriable: true,
-	}
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	mcClient := getTestManagedClusterClientWithNeverRateLimiter(armClient)
-	result, rerr := mcClient.List(context.TODO(), "rg")
-	assert.Equal(t, 0, len(result))
-	assert.NotNil(t, rerr)
-	assert.Equal(t, mcListErr, rerr)
-}
-
-func TestListRetryAfterReader(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	mcListErr := &retry.Error{
-		RawError:   fmt.Errorf("azure cloud provider throttled for operation %s with reason %q", "ListManagedCluster", "client throttled"),
-		Retriable:  true,
-		RetryAfter: getFutureTime(),
-	}
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	mcClient := getTestManagedClusterClientWithRetryAfterReader(armClient)
-	result, rerr := mcClient.List(context.TODO(), "rg")
-	assert.Equal(t, 0, len(result))
-	assert.NotNil(t, rerr)
-	assert.Equal(t, mcListErr, rerr)
-}
-
-func TestListThrottle(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	response := &http.Response{
-		StatusCode: http.StatusTooManyRequests,
-		Body:       io.NopCloser(bytes.NewReader([]byte("{}"))),
-	}
-	throttleErr := &retry.Error{
-		HTTPStatusCode: http.StatusTooManyRequests,
-		RawError:       fmt.Errorf("error"),
-		Retriable:      true,
-		RetryAfter:     time.Unix(100, 0),
-	}
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	armClient.EXPECT().GetResource(gomock.Any(), testResourcePrefix).Return(response, throttleErr).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-
-	mcClient := getTestManagedClusterClient(armClient)
-	result, rerr := mcClient.List(context.TODO(), "rg")
-	assert.Empty(t, result)
-	assert.NotNil(t, rerr)
-	assert.Equal(t, throttleErr, rerr)
-}
-
-func TestCreateOrUpdate(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	mc := getTestManagedCluster("cluster")
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	response := &http.Response{
-		StatusCode: http.StatusOK,
-		Body:       io.NopCloser(bytes.NewReader([]byte(""))),
-	}
-	armClient.EXPECT().PutResource(gomock.Any(), ptr.Deref(mc.ID, ""), mc, gomock.Any()).Return(response, nil).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-
-	mcClient := getTestManagedClusterClient(armClient)
-	rerr := mcClient.CreateOrUpdate(context.TODO(), "rg", "cluster", mc, "*")
-	assert.Nil(t, rerr)
-}
-
-func TestCreateOrUpdateWithCreateOrUpdateResponderError(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-	mc := getTestManagedCluster("cluster")
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	response := &http.Response{
-		StatusCode: http.StatusNotFound,
-		Body:       io.NopCloser(bytes.NewReader([]byte(""))),
-	}
-	armClient.EXPECT().PutResource(gomock.Any(), ptr.Deref(mc.ID, ""), mc, gomock.Any()).Return(response, nil).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-
-	mcClient := getTestManagedClusterClient(armClient)
-	rerr := mcClient.CreateOrUpdate(context.TODO(), "rg", "cluster", mc, "")
-	assert.NotNil(t, rerr)
-}
-
-func TestCreateOrUpdateNeverRateLimiter(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	mcCreateOrUpdateErr := &retry.Error{
-		RawError:  fmt.Errorf("azure cloud provider rate limited(%s) for operation %q", "write", "CreateOrUpdateManagedCluster"),
-		Retriable: true,
-	}
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-
-	mcClient := getTestManagedClusterClientWithNeverRateLimiter(armClient)
-	mc := getTestManagedCluster("cluster")
-	rerr := mcClient.CreateOrUpdate(context.TODO(), "rg", "cluster", mc, "")
-	assert.NotNil(t, rerr)
-	assert.Equal(t, mcCreateOrUpdateErr, rerr)
-}
-
-func TestCreateOrUpdateRetryAfterReader(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	mcCreateOrUpdateErr := &retry.Error{
-		RawError:   fmt.Errorf("azure cloud provider throttled for operation %s with reason %q", "CreateOrUpdateManagedCluster", "client throttled"),
-		Retriable:  true,
-		RetryAfter: getFutureTime(),
-	}
-
-	mc := getTestManagedCluster("cluster")
-	armClient := mockarmclient.NewMockInterface(ctrl)
-
-	mcClient := getTestManagedClusterClientWithRetryAfterReader(armClient)
-	rerr := mcClient.CreateOrUpdate(context.TODO(), "rg", "cluster", mc, "")
-	assert.NotNil(t, rerr)
-	assert.Equal(t, mcCreateOrUpdateErr, rerr)
-}
-
-func TestCreateOrUpdateThrottle(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	response := &http.Response{
-		StatusCode: http.StatusTooManyRequests,
-		Body:       io.NopCloser(bytes.NewReader([]byte("{}"))),
-	}
-	throttleErr := &retry.Error{
-		HTTPStatusCode: http.StatusTooManyRequests,
-		RawError:       fmt.Errorf("error"),
-		Retriable:      true,
-		RetryAfter:     time.Unix(100, 0),
-	}
-
-	mc := getTestManagedCluster("cluster")
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	armClient.EXPECT().PutResource(gomock.Any(), ptr.Deref(mc.ID, ""), mc, gomock.Any()).Return(response, throttleErr).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-
-	mcClient := getTestManagedClusterClient(armClient)
-	rerr := mcClient.CreateOrUpdate(context.TODO(), "rg", "cluster", mc, "")
-	assert.NotNil(t, rerr)
-	assert.Equal(t, throttleErr, rerr)
-}
-
-func TestDelete(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	mc := getTestManagedCluster("cluster")
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	armClient.EXPECT().DeleteResource(gomock.Any(), ptr.Deref(mc.ID, "")).Return(nil).Times(1)
-
-	mcClient := getTestManagedClusterClient(armClient)
-	rerr := mcClient.Delete(context.TODO(), "rg", "cluster")
-	assert.Nil(t, rerr)
-}
-
-func TestDeleteNeverRateLimiter(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	mcDeleteErr := &retry.Error{
-		RawError:  fmt.Errorf("azure cloud provider rate limited(%s) for operation %q", "write", "DeleteManagedCluster"),
-		Retriable: true,
-	}
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-
-	mcClient := getTestManagedClusterClientWithNeverRateLimiter(armClient)
-	rerr := mcClient.Delete(context.TODO(), "rg", "cluster")
-	assert.NotNil(t, rerr)
-	assert.Equal(t, mcDeleteErr, rerr)
-}
-
-func TestDeleteRetryAfterReader(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	mcDeleteErr := &retry.Error{
-		RawError:   fmt.Errorf("azure cloud provider throttled for operation %s with reason %q", "DeleteManagedCluster", "client throttled"),
-		Retriable:  true,
-		RetryAfter: getFutureTime(),
-	}
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-
-	mcClient := getTestManagedClusterClientWithRetryAfterReader(armClient)
-	rerr := mcClient.Delete(context.TODO(), "rg", "cluster")
-	assert.NotNil(t, rerr)
-	assert.Equal(t, mcDeleteErr, rerr)
-}
-
-func TestDeleteThrottle(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	throttleErr := &retry.Error{
-		HTTPStatusCode: http.StatusTooManyRequests,
-		RawError:       fmt.Errorf("error"),
-		Retriable:      true,
-		RetryAfter:     time.Unix(100, 0),
-	}
-
-	mc := getTestManagedCluster("cluster")
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	armClient.EXPECT().DeleteResource(gomock.Any(), ptr.Deref(mc.ID, "")).Return(throttleErr).Times(1)
-
-	mcClient := getTestManagedClusterClient(armClient)
-	rerr := mcClient.Delete(context.TODO(), "rg", "cluster")
-	assert.NotNil(t, rerr)
-	assert.Equal(t, throttleErr, rerr)
-}
diff --git a/pkg/azureclients/routeclient/azure_routeclient_test.go b/pkg/azureclients/routeclient/azure_routeclient_test.go
deleted file mode 100644
index 17214d3dd..000000000
--- a/pkg/azureclients/routeclient/azure_routeclient_test.go
+++ /dev/null
@@ -1,285 +0,0 @@
-/*
-Copyright 2020 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package routeclient
-
-import (
-	"bytes"
-	"context"
-	"fmt"
-	"io"
-	"net/http"
-	"testing"
-	"time"
-
-	"github.com/Azure/azure-sdk-for-go/services/network/mgmt/2022-07-01/network"
-	"github.com/stretchr/testify/assert"
-
-	"go.uber.org/mock/gomock"
-
-	"k8s.io/client-go/util/flowcontrol"
-	"k8s.io/utils/ptr"
-
-	azclients "sigs.k8s.io/cloud-provider-azure/pkg/azureclients"
-	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/armclient"
-	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/armclient/mockarmclient"
-	"sigs.k8s.io/cloud-provider-azure/pkg/retry"
-)
-
-// 2065-01-24 05:20:00 +0000 UTC
-func getFutureTime() time.Time {
-	return time.Unix(3000000000, 0)
-}
-
-func TestNew(t *testing.T) {
-	config := &azclients.ClientConfig{
-		SubscriptionID:          "sub",
-		ResourceManagerEndpoint: "endpoint",
-		Location:                "eastus",
-		RateLimitConfig: &azclients.RateLimitConfig{
-			CloudProviderRateLimit:            true,
-			CloudProviderRateLimitQPS:         0.5,
-			CloudProviderRateLimitBucket:      1,
-			CloudProviderRateLimitQPSWrite:    0.5,
-			CloudProviderRateLimitBucketWrite: 1,
-		},
-		Backoff: &retry.Backoff{Steps: 1},
-	}
-
-	routeClient := New(config)
-	assert.Equal(t, "sub", routeClient.subscriptionID)
-	assert.NotEmpty(t, routeClient.rateLimiterReader)
-	assert.NotEmpty(t, routeClient.rateLimiterWriter)
-}
-
-func TestNewAzureStack(t *testing.T) {
-	config := &azclients.ClientConfig{
-		CloudName:               "AZURESTACKCLOUD",
-		SubscriptionID:          "sub",
-		ResourceManagerEndpoint: "endpoint",
-		Location:                "eastus",
-		RateLimitConfig: &azclients.RateLimitConfig{
-			CloudProviderRateLimit:            true,
-			CloudProviderRateLimitQPS:         0.5,
-			CloudProviderRateLimitBucket:      1,
-			CloudProviderRateLimitQPSWrite:    0.5,
-			CloudProviderRateLimitBucketWrite: 1,
-		},
-		Backoff: &retry.Backoff{Steps: 1},
-	}
-
-	routeClient := New(config)
-	assert.Equal(t, "AZURESTACKCLOUD", routeClient.cloudName)
-	assert.Equal(t, "sub", routeClient.subscriptionID)
-}
-
-func TestCreateOrUpdate(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	r := getTestRoute("r1")
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	response := &http.Response{
-		StatusCode: http.StatusOK,
-		Body:       io.NopCloser(bytes.NewReader([]byte(""))),
-	}
-	armClient.EXPECT().PutResource(gomock.Any(), ptr.Deref(r.ID, ""), r, gomock.Any()).Return(response, nil).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-
-	routeClient := getTestRouteClient(armClient)
-	rerr := routeClient.CreateOrUpdate(context.TODO(), "rg", "rt", "r1", r, "*")
-	assert.Nil(t, rerr)
-}
-
-func TestCreateOrUpdateWithNeverRateLimiter(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	rcCreateOrUpdateErr := retry.GetRateLimitError(true, "RouteCreateOrUpdate")
-
-	r := getTestRoute("r1")
-	armClient := mockarmclient.NewMockInterface(ctrl)
-
-	routeClient := getTestRouteClientWithNeverRateLimiter(armClient)
-	rerr := routeClient.CreateOrUpdate(context.TODO(), "rg", "rt", "r1", r, "")
-	assert.Equal(t, rcCreateOrUpdateErr, rerr)
-}
-
-func TestCreateOrUpdateRetryAfterReader(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	rcCreateOrUpdateErr := retry.GetThrottlingError("RouteCreateOrUpdate", "client throttled", getFutureTime())
-
-	r := getTestRoute("r1")
-	armClient := mockarmclient.NewMockInterface(ctrl)
-
-	routeClient := getTestRouteClientWithRetryAfterReader(armClient)
-	rerr := routeClient.CreateOrUpdate(context.TODO(), "rg", "rt", "r1", r, "")
-	assert.NotNil(t, rerr)
-	assert.Equal(t, rcCreateOrUpdateErr, rerr)
-}
-
-func TestCreateOrUpdateThrottle(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	response := &http.Response{
-		StatusCode: http.StatusTooManyRequests,
-		Body:       io.NopCloser(bytes.NewReader([]byte("{}"))),
-	}
-	throttleErr := &retry.Error{
-		HTTPStatusCode: http.StatusTooManyRequests,
-		RawError:       fmt.Errorf("error"),
-		Retriable:      true,
-		RetryAfter:     time.Unix(100, 0),
-	}
-
-	r := getTestRoute("r1")
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	armClient.EXPECT().PutResource(gomock.Any(), ptr.Deref(r.ID, ""), r, gomock.Any()).Return(response, throttleErr).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-
-	routeClient := getTestRouteClient(armClient)
-	rerr := routeClient.CreateOrUpdate(context.TODO(), "rg", "rt", "r1", r, "")
-	assert.Equal(t, throttleErr, rerr)
-}
-
-func TestCreateOrUpdateWithCreateOrUpdateResponderError(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	r := getTestRoute("r1")
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	response := &http.Response{
-		StatusCode: http.StatusNotFound,
-		Body:       io.NopCloser(bytes.NewReader([]byte(""))),
-	}
-
-	armClient.EXPECT().PutResource(gomock.Any(), ptr.Deref(r.ID, ""), r, gomock.Any()).Return(response, nil).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-
-	routeClient := getTestRouteClient(armClient)
-	rerr := routeClient.CreateOrUpdate(context.TODO(), "rg", "rt", "r1", r, "")
-	assert.NotNil(t, rerr)
-}
-
-func TestDelete(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	r := getTestRoute("r1")
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	armClient.EXPECT().DeleteResource(gomock.Any(), ptr.Deref(r.ID, "")).Return(nil).Times(1)
-
-	routeClient := getTestRouteClient(armClient)
-	rerr := routeClient.Delete(context.TODO(), "rg", "rt", "r1")
-	assert.Nil(t, rerr)
-}
-
-func TestDeleteWithNeverRateLimiter(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	rcDeleteErr := &retry.Error{
-		RawError:  fmt.Errorf("azure cloud provider rate limited(%s) for operation %q", "write", "RouteDelete"),
-		Retriable: true,
-	}
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-
-	routeClient := getTestRouteClientWithNeverRateLimiter(armClient)
-	rerr := routeClient.Delete(context.TODO(), "rg", "rt", "r1")
-	assert.Equal(t, rcDeleteErr, rerr)
-}
-
-func TestDeleteRetryAfterReader(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	rcDeleteErr := &retry.Error{
-		RawError:   fmt.Errorf("azure cloud provider throttled for operation %s with reason %q", "RouteDelete", "client throttled"),
-		Retriable:  true,
-		RetryAfter: getFutureTime(),
-	}
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-
-	routeClient := getTestRouteClientWithRetryAfterReader(armClient)
-	rerr := routeClient.Delete(context.TODO(), "rg", "rt", "r1")
-	assert.Equal(t, rcDeleteErr, rerr)
-}
-
-func TestDeleteThrottle(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	throttleErr := &retry.Error{
-		HTTPStatusCode: http.StatusTooManyRequests,
-		RawError:       fmt.Errorf("error"),
-		Retriable:      true,
-		RetryAfter:     time.Unix(100, 0),
-	}
-
-	r := getTestRoute("r1")
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	armClient.EXPECT().DeleteResource(gomock.Any(), ptr.Deref(r.ID, "")).Return(throttleErr).Times(1)
-
-	routeClient := getTestRouteClient(armClient)
-	rerr := routeClient.Delete(context.TODO(), "rg", "rt", "r1")
-	assert.Equal(t, throttleErr, rerr)
-}
-
-func getTestRoute(name string) network.Route {
-	return network.Route{
-		ID:   ptr.To(fmt.Sprintf("/subscriptions/subscriptionID/resourceGroups/rg/providers/Microsoft.Network/routeTables/rt/routes/%s", name)),
-		Name: ptr.To(name),
-	}
-}
-
-func getTestRouteClient(armClient armclient.Interface) *Client {
-	rateLimiterReader, rateLimiterWriter := azclients.NewRateLimiter(&azclients.RateLimitConfig{})
-	return &Client{
-		armClient:         armClient,
-		subscriptionID:    "subscriptionID",
-		rateLimiterReader: rateLimiterReader,
-		rateLimiterWriter: rateLimiterWriter,
-	}
-}
-
-func getTestRouteClientWithNeverRateLimiter(armClient armclient.Interface) *Client {
-	rateLimiterReader := flowcontrol.NewFakeNeverRateLimiter()
-	rateLimiterWriter := flowcontrol.NewFakeNeverRateLimiter()
-	return &Client{
-		armClient:         armClient,
-		subscriptionID:    "subscriptionID",
-		rateLimiterReader: rateLimiterReader,
-		rateLimiterWriter: rateLimiterWriter,
-	}
-}
-
-func getTestRouteClientWithRetryAfterReader(armClient armclient.Interface) *Client {
-	rateLimiterReader := flowcontrol.NewFakeAlwaysRateLimiter()
-	rateLimiterWriter := flowcontrol.NewFakeAlwaysRateLimiter()
-	return &Client{
-		armClient:         armClient,
-		subscriptionID:    "subscriptionID",
-		rateLimiterReader: rateLimiterReader,
-		rateLimiterWriter: rateLimiterWriter,
-		RetryAfterReader:  getFutureTime(),
-		RetryAfterWriter:  getFutureTime(),
-	}
-}
diff --git a/pkg/azureclients/snapshotclient/azure_snapshotclient_test.go b/pkg/azureclients/snapshotclient/azure_snapshotclient_test.go
deleted file mode 100644
index a1ff2c6ad..000000000
--- a/pkg/azureclients/snapshotclient/azure_snapshotclient_test.go
+++ /dev/null
@@ -1,672 +0,0 @@
-/*
-Copyright 2020 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package snapshotclient
-
-import (
-	"bytes"
-	"context"
-	"encoding/json"
-	"fmt"
-	"io"
-	"net/http"
-	"testing"
-	"time"
-
-	"github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2022-08-01/compute"
-	"github.com/Azure/go-autorest/autorest"
-	"github.com/stretchr/testify/assert"
-
-	"go.uber.org/mock/gomock"
-
-	"k8s.io/client-go/util/flowcontrol"
-	"k8s.io/utils/ptr"
-
-	azclients "sigs.k8s.io/cloud-provider-azure/pkg/azureclients"
-	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/armclient"
-	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/armclient/mockarmclient"
-	"sigs.k8s.io/cloud-provider-azure/pkg/retry"
-)
-
-const (
-	testResourceID     = "/subscriptions/subscriptionID/resourceGroups/rg/providers/Microsoft.Compute/snapshots/sn1"
-	testResourcePrefix = "/subscriptions/subscriptionID/resourceGroups/rg/providers/Microsoft.Compute/snapshots"
-)
-
-func TestNew(t *testing.T) {
-	config := &azclients.ClientConfig{
-		SubscriptionID:          "sub",
-		ResourceManagerEndpoint: "endpoint",
-		Location:                "eastus",
-		RateLimitConfig: &azclients.RateLimitConfig{
-			CloudProviderRateLimit:            true,
-			CloudProviderRateLimitQPS:         0.5,
-			CloudProviderRateLimitBucket:      1,
-			CloudProviderRateLimitQPSWrite:    0.5,
-			CloudProviderRateLimitBucketWrite: 1,
-		},
-		Backoff: &retry.Backoff{Steps: 1},
-	}
-
-	snClient := New(config)
-	assert.Equal(t, "sub", snClient.subscriptionID)
-	assert.NotEmpty(t, snClient.rateLimiterReader)
-	assert.NotEmpty(t, snClient.rateLimiterWriter)
-}
-
-func TestNewAzureStack(t *testing.T) {
-	config := &azclients.ClientConfig{
-		CloudName:               "AZURESTACKCLOUD",
-		SubscriptionID:          "sub",
-		ResourceManagerEndpoint: "endpoint",
-		Location:                "eastus",
-		RateLimitConfig: &azclients.RateLimitConfig{
-			CloudProviderRateLimit:            true,
-			CloudProviderRateLimitQPS:         0.5,
-			CloudProviderRateLimitBucket:      1,
-			CloudProviderRateLimitQPSWrite:    0.5,
-			CloudProviderRateLimitBucketWrite: 1,
-		},
-		Backoff: &retry.Backoff{Steps: 1},
-	}
-
-	snClient := New(config)
-	assert.Equal(t, "AZURESTACKCLOUD", snClient.cloudName)
-	assert.Equal(t, "sub", snClient.subscriptionID)
-}
-
-func TestGet(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	response := &http.Response{
-		StatusCode: http.StatusOK,
-		Body:       io.NopCloser(bytes.NewReader([]byte("{}"))),
-	}
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	armClient.EXPECT().GetResource(gomock.Any(), testResourceID).Return(response, nil).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-
-	expected := compute.Snapshot{Response: autorest.Response{Response: response}}
-	snClient := getTestSnapshotClient(armClient)
-	result, rerr := snClient.Get(context.TODO(), "", "rg", "sn1")
-	assert.Equal(t, expected, result)
-	assert.Nil(t, rerr)
-}
-
-func TestGetNeverRateLimiter(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	snGetErr := &retry.Error{
-		RawError:  fmt.Errorf("azure cloud provider rate limited(%s) for operation %q", "read", "SnapshotGet"),
-		Retriable: true,
-	}
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	snClient := getTestSnapshotClientWithNeverRateLimiter(armClient)
-	expected := compute.Snapshot{}
-	result, rerr := snClient.Get(context.TODO(), "", "rg", "sn1")
-	assert.Equal(t, expected, result)
-	assert.Equal(t, snGetErr, rerr)
-}
-
-func TestGetRetryAfterReader(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	snGetErr := &retry.Error{
-		RawError:   fmt.Errorf("azure cloud provider throttled for operation %s with reason %q", "SnapshotGet", "client throttled"),
-		Retriable:  true,
-		RetryAfter: getFutureTime(),
-	}
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	snClient := getTestSnapshotClientWithRetryAfterReader(armClient)
-	expected := compute.Snapshot{}
-	result, rerr := snClient.Get(context.TODO(), "", "rg", "sn1")
-	assert.Equal(t, expected, result)
-	assert.Equal(t, snGetErr, rerr)
-}
-
-func TestGetNotFound(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	response := &http.Response{
-		StatusCode: http.StatusNotFound,
-		Body:       io.NopCloser(bytes.NewReader([]byte("{}"))),
-	}
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	armClient.EXPECT().GetResource(gomock.Any(), testResourceID).Return(response, nil).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-
-	snClient := getTestSnapshotClient(armClient)
-	expected := compute.Snapshot{Response: autorest.Response{}}
-	result, rerr := snClient.Get(context.TODO(), "", "rg", "sn1")
-	assert.Equal(t, expected, result)
-	assert.NotNil(t, rerr)
-	assert.Equal(t, http.StatusNotFound, rerr.HTTPStatusCode)
-}
-
-func TestGetThrottle(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	response := &http.Response{
-		StatusCode: http.StatusTooManyRequests,
-		Body:       io.NopCloser(bytes.NewReader([]byte("{}"))),
-	}
-	throttleErr := &retry.Error{
-		HTTPStatusCode: http.StatusTooManyRequests,
-		RawError:       fmt.Errorf("error"),
-		Retriable:      true,
-		RetryAfter:     time.Unix(100, 0),
-	}
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	armClient.EXPECT().GetResource(gomock.Any(), testResourceID).Return(response, throttleErr).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-
-	snClient := getTestSnapshotClient(armClient)
-	result, rerr := snClient.Get(context.TODO(), "", "rg", "sn1")
-	assert.Empty(t, result)
-	assert.Equal(t, throttleErr, rerr)
-}
-
-func TestGetInternalError(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	response := &http.Response{
-		StatusCode: http.StatusInternalServerError,
-		Body:       io.NopCloser(bytes.NewReader([]byte("{}"))),
-	}
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	armClient.EXPECT().GetResource(gomock.Any(), testResourceID).Return(response, nil).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-
-	snClient := getTestSnapshotClient(armClient)
-	expected := compute.Snapshot{Response: autorest.Response{}}
-	result, rerr := snClient.Get(context.TODO(), "", "rg", "sn1")
-	assert.Equal(t, expected, result)
-	assert.NotNil(t, rerr)
-	assert.Equal(t, http.StatusInternalServerError, rerr.HTTPStatusCode)
-}
-
-func TestListByResourceGroup(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	snList := []compute.Snapshot{getTestSnapshot("sn1"), getTestSnapshot("sn2"), getTestSnapshot("sn3")}
-	responseBody, err := json.Marshal(compute.SnapshotList{Value: &snList})
-	assert.NoError(t, err)
-	armClient.EXPECT().GetResource(gomock.Any(), testResourcePrefix).Return(
-		&http.Response{
-			StatusCode: http.StatusOK,
-			Body:       io.NopCloser(bytes.NewReader(responseBody)),
-		}, nil).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-
-	snClient := getTestSnapshotClient(armClient)
-	result, rerr := snClient.ListByResourceGroup(context.TODO(), "", "rg")
-	assert.Nil(t, rerr)
-	assert.Equal(t, 3, len(result))
-}
-
-func TestListByResourceGroupNotFound(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	response := &http.Response{
-		StatusCode: http.StatusNotFound,
-		Body:       io.NopCloser(bytes.NewReader([]byte("{}"))),
-	}
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	armClient.EXPECT().GetResource(gomock.Any(), testResourcePrefix).Return(response, nil).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-
-	snClient := getTestSnapshotClient(armClient)
-	expected := []compute.Snapshot{}
-	result, rerr := snClient.ListByResourceGroup(context.TODO(), "", "rg")
-	assert.Equal(t, expected, result)
-	assert.NotNil(t, rerr)
-	assert.Equal(t, http.StatusNotFound, rerr.HTTPStatusCode)
-}
-
-func TestListByResourceGroupInternalError(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	response := &http.Response{
-		StatusCode: http.StatusInternalServerError,
-		Body:       io.NopCloser(bytes.NewReader([]byte("{}"))),
-	}
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	armClient.EXPECT().GetResource(gomock.Any(), testResourcePrefix).Return(response, nil).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-
-	snClient := getTestSnapshotClient(armClient)
-	expected := []compute.Snapshot{}
-	result, rerr := snClient.ListByResourceGroup(context.TODO(), "", "rg")
-	assert.Equal(t, expected, result)
-	assert.NotNil(t, rerr)
-	assert.Equal(t, http.StatusInternalServerError, rerr.HTTPStatusCode)
-}
-
-func TestListByResourceGroupThrottle(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	response := &http.Response{
-		StatusCode: http.StatusTooManyRequests,
-		Body:       io.NopCloser(bytes.NewReader([]byte("{}"))),
-	}
-	throttleErr := &retry.Error{
-		HTTPStatusCode: http.StatusTooManyRequests,
-		RawError:       fmt.Errorf("error"),
-		Retriable:      true,
-		RetryAfter:     time.Unix(100, 0),
-	}
-	armClient.EXPECT().GetResource(gomock.Any(), testResourcePrefix).Return(response, throttleErr).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-	snClient := getTestSnapshotClient(armClient)
-	result, rerr := snClient.ListByResourceGroup(context.TODO(), "", "rg")
-	assert.Empty(t, result)
-	assert.NotNil(t, rerr)
-	assert.Equal(t, throttleErr, rerr)
-}
-
-func TestListByResourceGroupWithListResponderError(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	snList := []compute.Snapshot{getTestSnapshot("sn1"), getTestSnapshot("sn2"), getTestSnapshot("sn3")}
-	responseBody, err := json.Marshal(compute.SnapshotList{Value: &snList})
-	assert.NoError(t, err)
-	armClient.EXPECT().GetResource(gomock.Any(), testResourcePrefix).Return(
-		&http.Response{
-			StatusCode: http.StatusNotFound,
-			Body:       io.NopCloser(bytes.NewReader(responseBody)),
-		}, nil).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-	snClient := getTestSnapshotClient(armClient)
-	result, rerr := snClient.ListByResourceGroup(context.TODO(), "", "rg")
-	assert.NotNil(t, rerr)
-	assert.Equal(t, 0, len(result))
-}
-
-func TestListWithNextPage(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	snList := []compute.Snapshot{getTestSnapshot("sn1"), getTestSnapshot("sn2"), getTestSnapshot("sn3")}
-	partialResponse, err := json.Marshal(compute.SnapshotList{Value: &snList, NextLink: ptr.To("nextLink")})
-	assert.NoError(t, err)
-	pagedResponse, err := json.Marshal(compute.SnapshotList{Value: &snList})
-	assert.NoError(t, err)
-	armClient.EXPECT().PrepareGetRequest(gomock.Any(), gomock.Any()).Return(&http.Request{}, nil)
-	armClient.EXPECT().Send(gomock.Any(), gomock.Any()).Return(
-		&http.Response{
-			StatusCode: http.StatusOK,
-			Body:       io.NopCloser(bytes.NewReader(pagedResponse)),
-		}, nil)
-	armClient.EXPECT().GetResource(gomock.Any(), testResourcePrefix).Return(
-		&http.Response{
-			StatusCode: http.StatusOK,
-			Body:       io.NopCloser(bytes.NewReader(partialResponse)),
-		}, nil).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(2)
-	snClient := getTestSnapshotClient(armClient)
-	result, rerr := snClient.ListByResourceGroup(context.TODO(), "", "rg")
-	assert.Nil(t, rerr)
-	assert.Equal(t, 6, len(result))
-}
-
-func TestListByResourceGroupNeverRateLimiter(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	snListErr := &retry.Error{
-		RawError:  fmt.Errorf("azure cloud provider rate limited(%s) for operation %q", "read", "SnapshotListByResourceGroup"),
-		Retriable: true,
-	}
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	snClient := getTestSnapshotClientWithNeverRateLimiter(armClient)
-	result, rerr := snClient.ListByResourceGroup(context.TODO(), "", "rg")
-	assert.Equal(t, 0, len(result))
-	assert.NotNil(t, rerr)
-	assert.Equal(t, snListErr, rerr)
-}
-
-func TestListByResourceGroupRetryAfterReader(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	snListErr := &retry.Error{
-		RawError:   fmt.Errorf("azure cloud provider throttled for operation %s with reason %q", "SnapshotListByResourceGroup", "client throttled"),
-		Retriable:  true,
-		RetryAfter: getFutureTime(),
-	}
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	snClient := getTestSnapshotClientWithRetryAfterReader(armClient)
-	result, rerr := snClient.ListByResourceGroup(context.TODO(), "", "rg")
-	assert.Equal(t, 0, len(result))
-	assert.NotNil(t, rerr)
-	assert.Equal(t, snListErr, rerr)
-}
-
-func TestListNextResultsMultiPages(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	tests := []struct {
-		prepareErr error
-		sendErr    *retry.Error
-	}{
-		{
-			prepareErr: nil,
-			sendErr:    nil,
-		},
-		{
-			prepareErr: fmt.Errorf("error"),
-		},
-		{
-			sendErr: &retry.Error{RawError: fmt.Errorf("error")},
-		},
-	}
-
-	lastResult := compute.SnapshotList{
-		NextLink: ptr.To("next"),
-	}
-
-	for _, test := range tests {
-		armClient := mockarmclient.NewMockInterface(ctrl)
-		req := &http.Request{
-			Method: "GET",
-		}
-		armClient.EXPECT().PrepareGetRequest(gomock.Any(), gomock.Any()).Return(req, test.prepareErr)
-		if test.prepareErr == nil {
-			armClient.EXPECT().Send(gomock.Any(), req).Return(&http.Response{
-				StatusCode: http.StatusOK,
-				Body:       io.NopCloser(bytes.NewReader([]byte(`{"foo":"bar"}`))),
-			}, test.sendErr)
-			armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any())
-		}
-
-		snClient := getTestSnapshotClient(armClient)
-		result, err := snClient.listNextResults(context.TODO(), lastResult)
-		if test.prepareErr != nil || test.sendErr != nil {
-			assert.Error(t, err)
-		} else {
-			assert.NoError(t, err)
-		}
-		if test.prepareErr != nil {
-			assert.Empty(t, result)
-		} else {
-			assert.NotEmpty(t, result)
-		}
-	}
-}
-
-func TestListNextResultsMultiPagesWithListResponderError(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	test := struct {
-		prepareErr error
-		sendErr    *retry.Error
-	}{
-		prepareErr: nil,
-		sendErr:    nil,
-	}
-
-	lastResult := compute.SnapshotList{
-		NextLink: ptr.To("next"),
-	}
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	req := &http.Request{
-		Method: "GET",
-	}
-	armClient.EXPECT().PrepareGetRequest(gomock.Any(), gomock.Any()).Return(req, test.prepareErr)
-	if test.prepareErr == nil {
-		armClient.EXPECT().Send(gomock.Any(), req).Return(&http.Response{
-			StatusCode: http.StatusNotFound,
-			Body:       io.NopCloser(bytes.NewReader([]byte(`{"foo":"bar"}`))),
-		}, test.sendErr)
-		armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any())
-	}
-
-	response := &http.Response{
-		StatusCode: http.StatusNotFound,
-		Body:       io.NopCloser(bytes.NewBuffer([]byte(`{"foo":"bar"}`))),
-	}
-	expected := compute.SnapshotList{}
-	expected.Response = autorest.Response{Response: response}
-	snClient := getTestSnapshotClient(armClient)
-	result, err := snClient.listNextResults(context.TODO(), lastResult)
-	assert.Error(t, err)
-	assert.Equal(t, expected, result)
-}
-
-func TestCreateOrUpdate(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	sn := getTestSnapshot("sn1")
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	response := &http.Response{
-		StatusCode: http.StatusOK,
-		Body:       io.NopCloser(bytes.NewReader([]byte(""))),
-	}
-	armClient.EXPECT().PutResource(gomock.Any(), ptr.Deref(sn.ID, ""), sn).Return(response, nil).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-
-	snClient := getTestSnapshotClient(armClient)
-	rerr := snClient.CreateOrUpdate(context.TODO(), "", "rg", "sn1", sn)
-	assert.Nil(t, rerr)
-}
-
-func TestCreateOrUpdateWithCreateOrUpdateResponderError(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-	sn := getTestSnapshot("sn1")
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	response := &http.Response{
-		StatusCode: http.StatusNotFound,
-		Body:       io.NopCloser(bytes.NewReader([]byte(""))),
-	}
-	armClient.EXPECT().PutResource(gomock.Any(), ptr.Deref(sn.ID, ""), sn).Return(response, nil).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-
-	snClient := getTestSnapshotClient(armClient)
-	rerr := snClient.CreateOrUpdate(context.TODO(), "", "rg", "sn1", sn)
-	assert.NotNil(t, rerr)
-}
-
-func TestCreateOrUpdateNeverRateLimiter(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	snCreateOrUpdateErr := retry.GetRateLimitError(true, "SnapshotCreateOrUpdate")
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	snClient := getTestSnapshotClientWithNeverRateLimiter(armClient)
-	sn := getTestSnapshot("sn1")
-	rerr := snClient.CreateOrUpdate(context.TODO(), "", "rg", "sn1", sn)
-	assert.NotNil(t, rerr)
-	assert.Equal(t, snCreateOrUpdateErr, rerr)
-}
-
-func TestCreateOrUpdateRetryAfterReader(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	snCreateOrUpdateErr := retry.GetThrottlingError("SnapshotCreateOrUpdate", "client throttled", getFutureTime())
-
-	sn := getTestSnapshot("sn1")
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	snClient := getTestSnapshotClientWithRetryAfterReader(armClient)
-	rerr := snClient.CreateOrUpdate(context.TODO(), "", "rg", "sn1", sn)
-	assert.NotNil(t, rerr)
-	assert.Equal(t, snCreateOrUpdateErr, rerr)
-}
-
-func TestCreateOrUpdateThrottle(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	response := &http.Response{
-		StatusCode: http.StatusTooManyRequests,
-		Body:       io.NopCloser(bytes.NewReader([]byte("{}"))),
-	}
-	throttleErr := &retry.Error{
-		HTTPStatusCode: http.StatusTooManyRequests,
-		RawError:       fmt.Errorf("error"),
-		Retriable:      true,
-		RetryAfter:     time.Unix(100, 0),
-	}
-
-	sn := getTestSnapshot("sn1")
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	armClient.EXPECT().PutResource(gomock.Any(), ptr.Deref(sn.ID, ""), sn).Return(response, throttleErr).Times(1)
-	armClient.EXPECT().CloseResponse(gomock.Any(), gomock.Any()).Times(1)
-
-	snClient := getTestSnapshotClient(armClient)
-	rerr := snClient.CreateOrUpdate(context.TODO(), "", "rg", "sn1", sn)
-	assert.NotNil(t, rerr)
-	assert.Equal(t, throttleErr, rerr)
-}
-
-func TestDelete(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	r := getTestSnapshot("sn1")
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	armClient.EXPECT().DeleteResource(gomock.Any(), ptr.Deref(r.ID, "")).Return(nil).Times(1)
-
-	rtClient := getTestSnapshotClient(armClient)
-	rerr := rtClient.Delete(context.TODO(), "", "rg", "sn1")
-	assert.Nil(t, rerr)
-}
-
-func TestDeleteNeverRateLimiter(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	snDeleteErr := &retry.Error{
-		RawError:  fmt.Errorf("azure cloud provider rate limited(%s) for operation %q", "write", "SnapshotDelete"),
-		Retriable: true,
-	}
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	snClient := getTestSnapshotClientWithNeverRateLimiter(armClient)
-	rerr := snClient.Delete(context.TODO(), "", "rg", "sn1")
-	assert.NotNil(t, rerr)
-	assert.Equal(t, snDeleteErr, rerr)
-}
-
-func TestDeleteRetryAfterReader(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	snDeleteErr := &retry.Error{
-		RawError:   fmt.Errorf("azure cloud provider throttled for operation %s with reason %q", "SnapshotDelete", "client throttled"),
-		Retriable:  true,
-		RetryAfter: getFutureTime(),
-	}
-
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	snClient := getTestSnapshotClientWithRetryAfterReader(armClient)
-	rerr := snClient.Delete(context.TODO(), "", "rg", "sn1")
-	assert.NotNil(t, rerr)
-	assert.Equal(t, snDeleteErr, rerr)
-}
-
-func TestDeleteThrottle(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	throttleErr := &retry.Error{
-		HTTPStatusCode: http.StatusTooManyRequests,
-		RawError:       fmt.Errorf("error"),
-		Retriable:      true,
-		RetryAfter:     time.Unix(100, 0),
-	}
-
-	sn := getTestSnapshot("sn1")
-	armClient := mockarmclient.NewMockInterface(ctrl)
-	armClient.EXPECT().DeleteResource(gomock.Any(), ptr.Deref(sn.ID, "")).Return(throttleErr).Times(1)
-
-	snClient := getTestSnapshotClient(armClient)
-	rerr := snClient.Delete(context.TODO(), "", "rg", "sn1")
-	assert.NotNil(t, rerr)
-	assert.Equal(t, throttleErr, rerr)
-}
-
-func getTestSnapshot(name string) compute.Snapshot {
-	return compute.Snapshot{
-		ID:       ptr.To(fmt.Sprintf("/subscriptions/subscriptionID/resourceGroups/rg/providers/Microsoft.Compute/snapshots/%s", name)),
-		Name:     ptr.To(name),
-		Location: ptr.To("eastus"),
-	}
-}
-
-func getTestSnapshotClient(armClient armclient.Interface) *Client {
-	rateLimiterReader, rateLimiterWriter := azclients.NewRateLimiter(&azclients.RateLimitConfig{})
-	return &Client{
-		armClient:         armClient,
-		subscriptionID:    "subscriptionID",
-		rateLimiterReader: rateLimiterReader,
-		rateLimiterWriter: rateLimiterWriter,
-	}
-}
-
-func getTestSnapshotClientWithNeverRateLimiter(armClient armclient.Interface) *Client {
-	rateLimiterReader := flowcontrol.NewFakeNeverRateLimiter()
-	rateLimiterWriter := flowcontrol.NewFakeNeverRateLimiter()
-	return &Client{
-		armClient:         armClient,
-		subscriptionID:    "subscriptionID",
-		rateLimiterReader: rateLimiterReader,
-		rateLimiterWriter: rateLimiterWriter,
-	}
-}
-
-func getTestSnapshotClientWithRetryAfterReader(armClient armclient.Interface) *Client {
-	rateLimiterReader := flowcontrol.NewFakeAlwaysRateLimiter()
-	rateLimiterWriter := flowcontrol.NewFakeAlwaysRateLimiter()
-	return &Client{
-		armClient:         armClient,
-		subscriptionID:    "subscriptionID",
-		rateLimiterReader: rateLimiterReader,
-		rateLimiterWriter: rateLimiterWriter,
-		RetryAfterReader:  getFutureTime(),
-		RetryAfterWriter:  getFutureTime(),
-	}
-}
-
-func getFutureTime() time.Time {
-	return time.Unix(3000000000, 0)
-}
diff --git a/pkg/credentialprovider/azure_credentials_test.go b/pkg/credentialprovider/azure_credentials_test.go
index 5097e5ff3..2dd8d6918 100644
--- a/pkg/credentialprovider/azure_credentials_test.go
+++ b/pkg/credentialprovider/azure_credentials_test.go
@@ -21,6 +21,7 @@ import (
 	"net/http"
 	"net/http/httptest"
 	"os"
+	"reflect"
 	"testing"
 
 	"github.com/stretchr/testify/assert"
@@ -137,7 +138,7 @@ func TestGetCredentialsConfig(t *testing.T) {
 		if err != nil {
 			t.Fatalf("Unexpected error when closing temp file: %v", err)
 		}
-		provider, err := NewAcrProviderFromConfig(configFile.Name())
+		provider, err := NewAcrProviderFromConfig(configFile.Name(), "")
 		if err != nil && !test.expectError {
 			t.Fatalf("Unexpected error when creating new acr provider: %v", err)
 		}
@@ -163,6 +164,53 @@ func TestGetCredentialsConfig(t *testing.T) {
 	}
 }
 
+func TestProcessImageWithMirrorMapping(t *testing.T) {
+	configStr := `
+	{
+	    "aadClientId": "foo",
+	    "aadClientSecret": "bar"
+	}`
+
+	configFile, err := os.CreateTemp(".", "config.json")
+	assert.Nilf(t, err, "Unexpected error when creating temp file")
+	defer os.Remove(configFile.Name())
+	_, err = configFile.WriteString(configStr)
+	assert.Nilf(t, err, "Unexpected error when writing to temp file")
+	assert.Nilf(t, configFile.Close(), "Unexpected error when closing temp file")
+
+	provider, err := NewAcrProviderFromConfig(configFile.Name(), "mcr.microsoft.com:abc.azurecr.io")
+	assert.Nilf(t, err, "Unexpected error when creating new acr provider")
+	acrProvider := provider.(*acrProvider)
+
+	testcases := []struct {
+		description               string
+		image                     string
+		expectedLoginServer       string
+		expectedLoginServerMirror string
+	}{
+		{
+			description:               "image in registry mirror map",
+			image:                     "mcr.microsoft.com/bar/image:version",
+			expectedLoginServer:       "abc.azurecr.io",
+			expectedLoginServerMirror: "mcr.microsoft.com",
+		},
+		{
+			description:               "image not in registry mirror map",
+			image:                     "foo.azurecr.io/bar/image:version",
+			expectedLoginServer:       "foo.azurecr.io",
+			expectedLoginServerMirror: "",
+		},
+	}
+
+	for _, test := range testcases {
+		t.Run(test.description, func(t *testing.T) {
+			targetloginServer, sourceloginServer := acrProvider.parseACRLoginServerFromImage(test.image)
+			assert.Equal(t, targetloginServer, test.expectedLoginServer)
+			assert.Equal(t, sourceloginServer, test.expectedLoginServerMirror)
+		})
+	}
+}
+
 func TestParseACRLoginServerFromImage(t *testing.T) {
 
 	providerInterface := NewAcrProvider(&config.AzureAuthConfig{
@@ -215,8 +263,48 @@ func TestParseACRLoginServerFromImage(t *testing.T) {
 		},
 	}
 	for _, test := range tests {
-		if loginServer := provider.parseACRLoginServerFromImage(test.image); loginServer != test.expected {
-			t.Errorf("function parseACRLoginServerFromImage returns \"%s\" for image %s, expected \"%s\"", loginServer, test.image, test.expected)
-		}
+		t.Run(test.image, func(t *testing.T) {
+			targetloginServer, _ := provider.parseACRLoginServerFromImage(test.image)
+			assert.Equal(t, targetloginServer, test.expected)
+		})
+	}
+}
+
+func TestProcessMirrorMapping(t *testing.T) {
+	testcases := []struct {
+		description      string
+		mirrorMappingStr string
+		expected         map[string]string
+	}{
+		{
+			"multiple",
+			"aaa:bbb,ccc:ddd",
+			map[string]string{
+				"aaa": "bbb",
+				"ccc": "ddd",
+			},
+		},
+		{
+			"multiple with some spaces",
+			"aaa: bbb, ccc:ddd",
+			map[string]string{
+				"aaa": "bbb",
+				"ccc": "ddd",
+			},
+		},
+		{
+			"single",
+			"aaa:bbb",
+			map[string]string{
+				"aaa": "bbb",
+			},
+		},
+	}
+
+	for _, tc := range testcases {
+		t.Run(tc.description, func(t *testing.T) {
+			result := parseRegistryMirror(tc.mirrorMappingStr)
+			assert.True(t, reflect.DeepEqual(result, tc.expected))
+		})
 	}
 }
diff --git a/pkg/provider/azure_loadbalancer_accesscontrol_test.go b/pkg/provider/azure_loadbalancer_accesscontrol_test.go
index 3f6cdfed2..cd853b50e 100644
--- a/pkg/provider/azure_loadbalancer_accesscontrol_test.go
+++ b/pkg/provider/azure_loadbalancer_accesscontrol_test.go
@@ -42,9 +42,9 @@ import (
 	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
 	"sigs.k8s.io/cloud-provider-azure/pkg/log"
 	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer"
-	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer/iputil"
-	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer/securitygroup"
+	"sigs.k8s.io/cloud-provider-azure/pkg/provider/securitygroup"
 	"sigs.k8s.io/cloud-provider-azure/pkg/retry"
+	"sigs.k8s.io/cloud-provider-azure/pkg/util/iputil"
 )
 
 func TestCloud_reconcileSecurityGroup(t *testing.T) {
diff --git a/pkg/provider/azure_loadbalancer_test.go b/pkg/provider/azure_loadbalancer_test.go
index d8e0bf7ab..204cb9734 100644
--- a/pkg/provider/azure_loadbalancer_test.go
+++ b/pkg/provider/azure_loadbalancer_test.go
@@ -37,11 +37,14 @@ import (
 	"golang.org/x/text/cases"
 	"golang.org/x/text/language"
 
+	coordinationv1 "k8s.io/api/coordination/v1"
 	v1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/apimachinery/pkg/util/sets"
 	"k8s.io/client-go/kubernetes/fake"
+	k8stesting "k8s.io/client-go/testing"
 	"k8s.io/utils/ptr"
 
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/loadbalancerclient/mockloadbalancerclient"
@@ -794,6 +797,118 @@ func TestEnsureLoadBalancerDeleted(t *testing.T) {
 	}
 }
 
+func TestEnsureLoadBalancerLock(t *testing.T) {
+	ctrl := gomock.NewController(t)
+	defer ctrl.Finish()
+
+	az := GetTestCloud(ctrl)
+	kubeClient := fake.NewSimpleClientset()
+	kubeClient.PrependReactor(
+		"get", "leases",
+		func(_ k8stesting.Action) (handled bool, ret runtime.Object, err error) {
+			return true, nil, errors.New("get lease failed")
+		})
+	az.KubeClient = kubeClient
+	az.azureResourceLocker = NewAzureResourceLocker(
+		az, "holder", "aks-managed-resource-locker", "kube-system", 900,
+	)
+
+	svc := getTestService("service", v1.ProtocolTCP, nil, false, 80)
+	_, err := az.EnsureLoadBalancer(context.Background(), testClusterName, &svc, nil)
+	assert.Error(t, err)
+	assert.Contains(t, err.Error(), "get lease failed")
+	assert.Contains(t, err.Error(), "EnsureLoadBalancer failed due to fail to lock azure resources")
+
+	kubeClient = fake.NewSimpleClientset(&coordinationv1.Lease{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "aks-managed-resource-locker",
+			Namespace: "kube-system",
+		},
+		Spec: coordinationv1.LeaseSpec{
+			LeaseDurationSeconds: ptr.To[int32](900),
+		},
+	})
+	// update lease failed for the second time
+	count := 0
+	kubeClient.PrependReactor(
+		"update", "leases",
+		func(_ k8stesting.Action) (handled bool, ret runtime.Object, err error) {
+			count++
+			if count == 2 {
+				return true, nil, errors.New("update lease failed")
+			}
+			return false, nil, nil
+		})
+	az.KubeClient = kubeClient
+	az.azureResourceLocker = NewAzureResourceLocker(
+		az, "holder", "aks-managed-resource-locker", "kube-system", 900,
+	)
+	mockLBClient := az.LoadBalancerClient.(*mockloadbalancerclient.MockInterface)
+	mockLBClient.EXPECT().List(gomock.Any(), gomock.Any()).
+		Return(nil, retry.NewError(false, errors.New("list lb failed")))
+
+	_, err = az.EnsureLoadBalancer(context.Background(), testClusterName, &svc, nil)
+	assert.Error(t, err)
+	assert.Contains(t, err.Error(), "update lease failed")
+	assert.Contains(t, err.Error(), "list lb failed")
+}
+
+func TestEnsureLoadBalancerDeletedLock(t *testing.T) {
+	ctrl := gomock.NewController(t)
+	defer ctrl.Finish()
+
+	az := GetTestCloud(ctrl)
+	kubeClient := fake.NewSimpleClientset()
+	kubeClient.PrependReactor(
+		"get", "leases",
+		func(_ k8stesting.Action) (handled bool, ret runtime.Object, err error) {
+			return true, nil, errors.New("get lease failed")
+		})
+	az.KubeClient = kubeClient
+	az.azureResourceLocker = NewAzureResourceLocker(
+		az, "holder", "aks-managed-resource-locker", "kube-system", 900,
+	)
+
+	svc := getTestService("service", v1.ProtocolTCP, nil, false, 80)
+	err := az.EnsureLoadBalancerDeleted(context.Background(), testClusterName, &svc)
+	assert.Error(t, err)
+	assert.Contains(t, err.Error(), "get lease failed")
+	assert.Contains(t, err.Error(), "EnsureLoadBalancerDeleted failed due to fail to lock azure resources")
+
+	kubeClient = fake.NewSimpleClientset(&coordinationv1.Lease{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "aks-managed-resource-locker",
+			Namespace: "kube-system",
+		},
+		Spec: coordinationv1.LeaseSpec{
+			LeaseDurationSeconds: ptr.To[int32](900),
+		},
+	})
+	// update lease failed for the second time
+	count := 0
+	kubeClient.PrependReactor(
+		"update", "leases",
+		func(_ k8stesting.Action) (handled bool, ret runtime.Object, err error) {
+			count++
+			if count == 2 {
+				return true, nil, errors.New("update lease failed")
+			}
+			return false, nil, nil
+		})
+	az.KubeClient = kubeClient
+	az.azureResourceLocker = NewAzureResourceLocker(
+		az, "holder", "aks-managed-resource-locker", "kube-system", 900,
+	)
+	mockLBClient := az.LoadBalancerClient.(*mockloadbalancerclient.MockInterface)
+	mockLBClient.EXPECT().List(gomock.Any(), gomock.Any()).
+		Return(nil, retry.NewError(false, errors.New("list lb failed")))
+
+	err = az.EnsureLoadBalancerDeleted(context.Background(), testClusterName, &svc)
+	assert.Error(t, err)
+	assert.Contains(t, err.Error(), "update lease failed")
+	assert.Contains(t, err.Error(), "list lb failed")
+}
+
 func TestServiceOwnsPublicIP(t *testing.T) {
 	tests := []struct {
 		desc                    string
@@ -1402,7 +1517,6 @@ func TestConvertIPTagMapToSlice(t *testing.T) {
 				ipTagSlice := *c.expected
 				return ptr.Deref(ipTagSlice[i].IPTagType, "") < ptr.Deref(ipTagSlice[j].IPTagType, "")
 			})
-
 		}
 
 		assert.Equal(t, c.expected, actual, "TestCase[%d]: %s", i, c.desc)
@@ -1489,7 +1603,6 @@ func TestGetserviceIPTagRequestForPublicIP(t *testing.T) {
 				ipTagSlice := *c.expected.IPTags
 				return ptr.Deref(ipTagSlice[i].IPTagType, "") < ptr.Deref(ipTagSlice[j].IPTagType, "")
 			})
-
 		}
 
 		assert.Equal(t, actual, c.expected, "TestCase[%d]: %s", i, c.desc)
@@ -3543,7 +3656,7 @@ func TestReconcileLoadBalancerCommon(t *testing.T) {
 		},
 	}
 
-	//change to false to test that reconciliation will fix it (despite the fact that disable-tcp-reset was removed in 1.20)
+	// change to false to test that reconciliation will fix it (despite the fact that disable-tcp-reset was removed in 1.20)
 	(*slb5.LoadBalancerPropertiesFormat.LoadBalancingRules)[0].EnableTCPReset = ptr.To(false)
 	(*slb5.LoadBalancerPropertiesFormat.LoadBalancingRules)[1].EnableTCPReset = ptr.To(false)
 
@@ -4688,7 +4801,6 @@ func TestReconcilePublicIPsCommon(t *testing.T) {
 					}
 
 					return network.PublicIPAddress{}, &retry.Error{HTTPStatusCode: http.StatusNotFound}
-
 				})
 				deleter := mockPIPsClient.EXPECT().Delete(gomock.Any(), "rg", *pip.Name).Return(nil).AnyTimes()
 				deleter.Do(func(_ context.Context, _ string, publicIPAddressName string) *retry.Error {
@@ -4972,7 +5084,6 @@ func TestEnsurePublicIPExistsCommon(t *testing.T) {
 			foundDNSLabelAnnotation: true,
 			isIPv6:                  false,
 			existingPIPs: []network.PublicIPAddress{{
-
 				Name: ptr.To("pip1"),
 				PublicIPAddressPropertiesFormat: &network.PublicIPAddressPropertiesFormat{
 					DNSSettings: &network.PublicIPAddressDNSSettings{
@@ -5199,6 +5310,7 @@ func TestEnsurePublicIPExistsCommon(t *testing.T) {
 		})
 	}
 }
+
 func TestEnsurePublicIPExistsWithExtendedLocation(t *testing.T) {
 	ctrl := gomock.NewController(t)
 	defer ctrl.Finish()
@@ -6211,7 +6323,8 @@ func TestReconcileZonesForFrontendIPConfigs(t *testing.T) {
 			description: "reconcileFrontendIPConfigs should use the nil zones of the existing frontend",
 			service: getTestServiceWithAnnotation("test", map[string]string{
 				consts.ServiceAnnotationLoadBalancerInternalSubnet: "subnet",
-				consts.ServiceAnnotationLoadBalancerInternal:       consts.TrueAnnotationValue}, true, 80),
+				consts.ServiceAnnotationLoadBalancerInternal:       consts.TrueAnnotationValue,
+			}, true, 80),
 			existingFrontendIPConfigs: []network.FrontendIPConfiguration{
 				{
 					Name: ptr.To("atest1"),
@@ -6228,7 +6341,8 @@ func TestReconcileZonesForFrontendIPConfigs(t *testing.T) {
 			description: "reconcileFrontendIPConfigs should use the non-nil zones of the existing frontend",
 			service: getTestServiceWithAnnotation("test", map[string]string{
 				consts.ServiceAnnotationLoadBalancerInternalSubnet: "subnet",
-				consts.ServiceAnnotationLoadBalancerInternal:       consts.TrueAnnotationValue}, true, 80),
+				consts.ServiceAnnotationLoadBalancerInternal:       consts.TrueAnnotationValue,
+			}, true, 80),
 			existingFrontendIPConfigs: []network.FrontendIPConfiguration{
 				{
 					Name: ptr.To("not-this-one"),
@@ -7197,7 +7311,8 @@ func TestGetEligibleLoadBalancers(t *testing.T) {
 									Operator: metav1.LabelSelectorOpIn,
 									Values:   []string{},
 								},
-							}},
+							},
+						},
 					},
 				},
 			},
@@ -7218,7 +7333,8 @@ func TestGetEligibleLoadBalancers(t *testing.T) {
 									Operator: metav1.LabelSelectorOpIn,
 									Values:   []string{},
 								},
-							}},
+							},
+						},
 					},
 				},
 			},
@@ -7670,7 +7786,6 @@ func TestReconcileMultipleStandardLoadBalancerConfigurations(t *testing.T) {
 			for lbConfigName, svcNames := range tc.expectedActiveServices {
 				assert.Equal(t, svcNames, activeServices[lbConfigName])
 			}
-
 		})
 	}
 }
diff --git a/pkg/provider/azure_lock_test.go b/pkg/provider/azure_lock_test.go
new file mode 100644
index 000000000..3c037f41c
--- /dev/null
+++ b/pkg/provider/azure_lock_test.go
@@ -0,0 +1,242 @@
+/*
+Copyright 2024 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package provider
+
+import (
+	"context"
+	"errors"
+	"testing"
+	"time"
+
+	"github.com/Azure/azure-sdk-for-go/sdk/azcore/to"
+	"github.com/stretchr/testify/assert"
+
+	"go.uber.org/mock/gomock"
+
+	v1 "k8s.io/api/coordination/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/client-go/kubernetes"
+	"k8s.io/client-go/kubernetes/fake"
+	client_go_testing "k8s.io/client-go/testing"
+)
+
+func newTestAzureResourceLocker(ctrl *gomock.Controller, kubeClient kubernetes.Interface) *AzureResourceLocker {
+	cloud := GetTestCloud(ctrl)
+	cloud.KubeClient = kubeClient
+	return NewAzureResourceLocker(cloud, "holder", "aks-managed-resource-locker", "kube-system", 900)
+}
+
+func TestLock(t *testing.T) {
+	ctrl := gomock.NewController(t)
+	defer ctrl.Finish()
+
+	for _, tc := range []struct {
+		desc        string
+		getErr      bool
+		createErr   bool
+		updateErr   bool
+		existed     bool
+		expired     bool
+		sameHolder  bool
+		expectedErr string
+	}{
+		{
+			desc:        "should return an error if failed to get the lease",
+			getErr:      true,
+			expectedErr: "failed to get leases",
+		},
+		{
+			desc:        "should return an error if failed to create the lease",
+			createErr:   true,
+			expectedErr: "failed to create lease",
+		},
+		{
+			desc:        "should return an error if failed to get the lease when acquiring it",
+			getErr:      true,
+			existed:     true,
+			expectedErr: "failed to get leases",
+		},
+		{
+			desc:        "should return an error when trying to acquire a lease which has not expired",
+			existed:     true,
+			expectedErr: "Lease has not expired yet",
+		},
+		{
+			desc:       "should not return an error if the lease has not expired but the previous holder is the same",
+			existed:    true,
+			sameHolder: true,
+		},
+		{
+			desc:        "should return an error if failed to update the lease",
+			existed:     true,
+			updateErr:   true,
+			expired:     true,
+			expectedErr: "failed to update lease",
+		},
+	} {
+		t.Run(tc.desc, func(t *testing.T) {
+			testLease := getTestLease()
+			kubeClient := fake.NewSimpleClientset()
+			if !tc.expired {
+				testLease.Spec.HolderIdentity = to.Ptr("prevHolder")
+				testLease.Spec.RenewTime = &metav1.MicroTime{Time: time.Now().Add(-100)}
+				if tc.sameHolder {
+					testLease.Spec.HolderIdentity = to.Ptr("holder")
+				}
+			}
+			if tc.existed {
+				kubeClient = fake.NewSimpleClientset(testLease)
+			}
+			getCount := 0
+			if tc.getErr {
+				kubeClient.PrependReactor(
+					"get", "leases",
+					func(_ client_go_testing.Action) (handled bool, ret runtime.Object, err error) {
+						getCount++
+						if tc.existed {
+							if getCount == 2 {
+								return true, nil, errors.New("failed to get leases")
+							}
+							return true, testLease, nil
+						}
+						return true, nil, errors.New("failed to get leases")
+					},
+				)
+			}
+			if tc.createErr {
+				kubeClient.PrependReactor(
+					"create", "leases",
+					func(_ client_go_testing.Action) (handled bool, ret runtime.Object, err error) {
+						return true, nil, errors.New("failed to create lease")
+					},
+				)
+			}
+			if tc.updateErr {
+				kubeClient.PrependReactor(
+					"update", "leases",
+					func(_ client_go_testing.Action) (handled bool, ret runtime.Object, err error) {
+						return true, nil, errors.New("failed to update lease")
+					},
+				)
+			}
+			if !tc.getErr && !tc.createErr {
+				kubeClient.PrependReactor(
+					"get", "leases",
+					func(_ client_go_testing.Action) (handled bool, ret runtime.Object, err error) {
+						return true, testLease, nil
+					},
+				)
+			}
+
+			locker := newTestAzureResourceLocker(ctrl, kubeClient)
+			err := locker.Lock(context.TODO())
+			if tc.expectedErr != "" {
+				assert.Error(t, err)
+				assert.Contains(t, err.Error(), tc.expectedErr)
+			} else {
+				assert.Nil(t, err)
+			}
+		})
+	}
+}
+
+func TestUnlock(t *testing.T) {
+	ctrl := gomock.NewController(t)
+	defer ctrl.Finish()
+
+	for _, tc := range []struct {
+		desc            string
+		getErr          bool
+		updateErr       bool
+		expired         bool
+		differentHolder bool
+		expectedErr     string
+	}{
+		{
+			desc:        "should return an error if failed to get the lease",
+			getErr:      true,
+			expectedErr: "failed to get lease",
+		},
+		{
+			desc:    "should do nothing if the lease has expired",
+			expired: true,
+		},
+		{
+			desc:            "should do nothing if the lease is holding by another holder",
+			differentHolder: true,
+		},
+		{
+			desc:        "should return an error if failed to update the lease",
+			updateErr:   true,
+			expired:     true,
+			expectedErr: "failed to update lease",
+		},
+	} {
+		t.Run(tc.desc, func(t *testing.T) {
+			testLease := getTestLease()
+			testLease.Spec.HolderIdentity = to.Ptr("holder")
+			testLease.Spec.RenewTime = &metav1.MicroTime{Time: time.Now().Add(-1000)}
+			if !tc.expired {
+				testLease.Spec.RenewTime = &metav1.MicroTime{Time: time.Now().Add(-100)}
+				if tc.differentHolder {
+					testLease.Spec.HolderIdentity = to.Ptr("prevHolder")
+				}
+			}
+
+			kubeClient := fake.NewSimpleClientset(testLease)
+			if tc.getErr {
+				kubeClient.PrependReactor(
+					"get", "leases",
+					func(_ client_go_testing.Action) (handled bool, ret runtime.Object, err error) {
+						return true, nil, errors.New("failed to get lease")
+					},
+				)
+			}
+			if tc.updateErr {
+				kubeClient.PrependReactor(
+					"update", "leases",
+					func(_ client_go_testing.Action) (handled bool, ret runtime.Object, err error) {
+						return true, nil, errors.New("failed to update lease")
+					},
+				)
+			}
+
+			locker := newTestAzureResourceLocker(ctrl, kubeClient)
+			err := locker.Unlock(context.TODO())
+			if tc.expectedErr != "" {
+				assert.Error(t, err)
+				assert.Contains(t, err.Error(), tc.expectedErr)
+			} else {
+				assert.Nil(t, err)
+			}
+		})
+	}
+}
+
+func getTestLease() *v1.Lease {
+	return &v1.Lease{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "aks-managed-resource-locker",
+			Namespace: "kube-system",
+		},
+		Spec: v1.LeaseSpec{
+			HolderIdentity:       to.Ptr(""),
+			LeaseDurationSeconds: to.Ptr[int32](900),
+		},
+	}
+}
diff --git a/pkg/provider/loadbalancer/accesscontrol_test.go b/pkg/provider/loadbalancer/accesscontrol_test.go
index 17284b795..e1eb2eb42 100644
--- a/pkg/provider/loadbalancer/accesscontrol_test.go
+++ b/pkg/provider/loadbalancer/accesscontrol_test.go
@@ -30,9 +30,9 @@ import (
 	"sigs.k8s.io/cloud-provider-azure/internal/testutil"
 	"sigs.k8s.io/cloud-provider-azure/internal/testutil/fixture"
 	"sigs.k8s.io/cloud-provider-azure/pkg/log"
-	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer/fnutil"
-	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer/iputil"
-	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer/securitygroup"
+	"sigs.k8s.io/cloud-provider-azure/pkg/provider/securitygroup"
+	fnutil "sigs.k8s.io/cloud-provider-azure/pkg/util/collectionutil"
+	"sigs.k8s.io/cloud-provider-azure/pkg/util/iputil"
 )
 
 func TestAccessControl_IsAllowFromInternet(t *testing.T) {
diff --git a/pkg/provider/azure_securitygroup_repo_test.go b/pkg/provider/securitygroup/azure_securitygroup_repo_test.go
similarity index 66%
rename from pkg/provider/azure_securitygroup_repo_test.go
rename to pkg/provider/securitygroup/azure_securitygroup_repo_test.go
index 4d6be443c..7bfd7ae11 100644
--- a/pkg/provider/azure_securitygroup_repo_test.go
+++ b/pkg/provider/securitygroup/azure_securitygroup_repo_test.go
@@ -14,7 +14,7 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
-package provider
+package securitygroup
 
 import (
 	"context"
@@ -31,7 +31,6 @@ import (
 
 	"k8s.io/utils/ptr"
 
-	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/mock_azclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/securitygroupclient/mock_securitygroupclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/cache"
 	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
@@ -40,24 +39,23 @@ import (
 func TestCreateOrUpdateSecurityGroupCanceled(t *testing.T) {
 	ctrl := gomock.NewController(t)
 	defer ctrl.Finish()
+	mockSGClient := mock_securitygroupclient.NewMockInterface(ctrl)
+	az, err := NewSecurityGroupRepo("rg", "sg", 120, false, mockSGClient)
+	assert.NoError(t, err)
+	az.(*securityGroupRepo).nsgCache.Set("sg", "test")
 
-	az := GetTestCloud(ctrl)
-	az.nsgCache.Set("sg", "test")
-	clientFactory := az.NetworkClientFactory.(*mock_azclient.MockClientFactory)
-	mockSGClient := clientFactory.GetSecurityGroupClient().(*mock_securitygroupclient.MockInterface)
-
-	mockSGClient.EXPECT().CreateOrUpdate(gomock.Any(), az.ResourceGroup, gomock.Any(), gomock.Any()).Return(nil, &azcore.ResponseError{
+	mockSGClient.EXPECT().CreateOrUpdate(gomock.Any(), "rg", "sg", gomock.Any()).Return(nil, &azcore.ResponseError{
 		RawResponse: &http.Response{
 			Body: io.NopCloser(strings.NewReader(consts.OperationCanceledErrorMessage)),
 		},
 	})
-	mockSGClient.EXPECT().Get(gomock.Any(), az.ResourceGroup, "sg").Return(&armnetwork.SecurityGroup{}, nil)
+	mockSGClient.EXPECT().Get(gomock.Any(), "rg", "sg").Return(&armnetwork.SecurityGroup{}, nil)
 
-	err := az.CreateOrUpdateSecurityGroup(&armnetwork.SecurityGroup{Name: ptr.To("sg")})
+	err = az.CreateOrUpdateSecurityGroup(context.TODO(), &armnetwork.SecurityGroup{Name: ptr.To("sg")})
 	assert.Contains(t, err.Error(), "canceledandsupersededduetoanotheroperation")
 
 	// security group should be removed from cache if the operation is canceled
-	shouldBeEmpty, err := az.nsgCache.GetWithDeepCopy(context.TODO(), "sg", cache.CacheReadTypeDefault)
+	shouldBeEmpty, err := az.(*securityGroupRepo).nsgCache.GetWithDeepCopy(context.TODO(), "sg", cache.CacheReadTypeDefault)
 	assert.NoError(t, err)
 	assert.Empty(t, shouldBeEmpty)
 }
diff --git a/pkg/provider/loadbalancer/securitygroup/securitygroup_test.go b/pkg/provider/securitygroup/securitygroup_test.go
similarity index 99%
rename from pkg/provider/loadbalancer/securitygroup/securitygroup_test.go
rename to pkg/provider/securitygroup/securitygroup_test.go
index dd7c3e5cb..90b5dfbc4 100644
--- a/pkg/provider/loadbalancer/securitygroup/securitygroup_test.go
+++ b/pkg/provider/securitygroup/securitygroup_test.go
@@ -29,9 +29,9 @@ import (
 	"sigs.k8s.io/cloud-provider-azure/internal/testutil"
 	"sigs.k8s.io/cloud-provider-azure/internal/testutil/fixture"
 	"sigs.k8s.io/cloud-provider-azure/pkg/log"
-	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer/fnutil"
-	"sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer/iputil"
-	. "sigs.k8s.io/cloud-provider-azure/pkg/provider/loadbalancer/securitygroup" //nolint:revive
+	. "sigs.k8s.io/cloud-provider-azure/pkg/provider/securitygroup" //nolint:revive
+	fnutil "sigs.k8s.io/cloud-provider-azure/pkg/util/collectionutil"
+	"sigs.k8s.io/cloud-provider-azure/pkg/util/iputil"
 )
 
 func ExpectNewSecurityGroupHelper(t *testing.T, sg *armnetwork.SecurityGroup) *RuleHelper {
diff --git a/pkg/provider/loadbalancer/iputil/family_test.go b/pkg/util/iputil/family_test.go
similarity index 100%
rename from pkg/provider/loadbalancer/iputil/family_test.go
rename to pkg/util/iputil/family_test.go
diff --git a/pkg/provider/loadbalancer/iputil/prefix.go b/pkg/util/iputil/prefix.go
similarity index 100%
rename from pkg/provider/loadbalancer/iputil/prefix.go
rename to pkg/util/iputil/prefix.go
diff --git a/pkg/provider/loadbalancer/iputil/prefix_test.go b/pkg/util/iputil/prefix_test.go
similarity index 100%
rename from pkg/provider/loadbalancer/iputil/prefix_test.go
rename to pkg/util/iputil/prefix_test.go
diff --git a/pkg/provider/loadbalancer/iputil/prefix_tree.go b/pkg/util/iputil/prefix_tree.go
similarity index 100%
rename from pkg/provider/loadbalancer/iputil/prefix_tree.go
rename to pkg/util/iputil/prefix_tree.go
diff --git a/pkg/provider/loadbalancer/iputil/prefix_tree_test.go b/pkg/util/iputil/prefix_tree_test.go
similarity index 100%
rename from pkg/provider/loadbalancer/iputil/prefix_tree_test.go
rename to pkg/util/iputil/prefix_tree_test.go
diff --git a/vendor/github.com/prometheus/client_golang/prometheus/testutil/testutil.go b/vendor/github.com/prometheus/client_golang/prometheus/testutil/testutil.go
index e0ac34666..6f1200180 100644
--- a/vendor/github.com/prometheus/client_golang/prometheus/testutil/testutil.go
+++ b/vendor/github.com/prometheus/client_golang/prometheus/testutil/testutil.go
@@ -158,6 +158,9 @@ func GatherAndCount(g prometheus.Gatherer, metricNames ...string) (int, error) {
 // ScrapeAndCompare calls a remote exporter's endpoint which is expected to return some metrics in
 // plain text format. Then it compares it with the results that the `expected` would return.
 // If the `metricNames` is not empty it would filter the comparison only to the given metric names.
+//
+// NOTE: Be mindful of accidental discrepancies between expected and metricNames; metricNames filter
+// both expected and scraped metrics. See https://github.com/prometheus/client_golang/issues/1351.
 func ScrapeAndCompare(url string, expected io.Reader, metricNames ...string) error {
 	resp, err := http.Get(url)
 	if err != nil {
@@ -185,6 +188,9 @@ func ScrapeAndCompare(url string, expected io.Reader, metricNames ...string) err
 
 // CollectAndCompare collects the metrics identified by `metricNames` and compares them in the Prometheus text
 // exposition format to the data read from expected.
+//
+// NOTE: Be mindful of accidental discrepancies between expected and metricNames; metricNames filter
+// both expected and collected metrics. See https://github.com/prometheus/client_golang/issues/1351.
 func CollectAndCompare(c prometheus.Collector, expected io.Reader, metricNames ...string) error {
 	reg := prometheus.NewPedanticRegistry()
 	if err := reg.Register(c); err != nil {
@@ -197,6 +203,9 @@ func CollectAndCompare(c prometheus.Collector, expected io.Reader, metricNames .
 // it to an expected output read from the provided Reader in the Prometheus text
 // exposition format. If any metricNames are provided, only metrics with those
 // names are compared.
+//
+// NOTE: Be mindful of accidental discrepancies between expected and metricNames; metricNames filter
+// both expected and gathered metrics. See https://github.com/prometheus/client_golang/issues/1351.
 func GatherAndCompare(g prometheus.Gatherer, expected io.Reader, metricNames ...string) error {
 	return TransactionalGatherAndCompare(prometheus.ToTransactionalGatherer(g), expected, metricNames...)
 }
@@ -205,6 +214,9 @@ func GatherAndCompare(g prometheus.Gatherer, expected io.Reader, metricNames ...
 // it to an expected output read from the provided Reader in the Prometheus text
 // exposition format. If any metricNames are provided, only metrics with those
 // names are compared.
+//
+// NOTE: Be mindful of accidental discrepancies between expected and metricNames; metricNames filter
+// both expected and gathered metrics. See https://github.com/prometheus/client_golang/issues/1351.
 func TransactionalGatherAndCompare(g prometheus.TransactionalGatherer, expected io.Reader, metricNames ...string) error {
 	got, done, err := g.Gather()
 	defer done()
@@ -277,15 +289,6 @@ func compareMetricFamilies(got, expected []*dto.MetricFamily, metricNames ...str
 	if metricNames != nil {
 		got = filterMetrics(got, metricNames)
 		expected = filterMetrics(expected, metricNames)
-		if len(metricNames) > len(got) {
-			var missingMetricNames []string
-			for _, name := range metricNames {
-				if ok := hasMetricByName(got, name); !ok {
-					missingMetricNames = append(missingMetricNames, name)
-				}
-			}
-			return fmt.Errorf("expected metric name(s) not found: %v", missingMetricNames)
-		}
 	}
 
 	return compare(got, expected)
@@ -327,12 +330,3 @@ func filterMetrics(metrics []*dto.MetricFamily, names []string) []*dto.MetricFam
 	}
 	return filtered
 }
-
-func hasMetricByName(metrics []*dto.MetricFamily, name string) bool {
-	for _, mf := range metrics {
-		if mf.GetName() == name {
-			return true
-		}
-	}
-	return false
-}