Repository: kubernetes-sigs/cloud-provider-azure
Pull request #6965 (2024-09-05 06:20:47)

fix: Do not move nodes that have already been attached to load balanc…


<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide/first-contribution.md#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?

<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature
/kind design

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->

/kind bug

#### What this PR does / why we need it:

fix: Do not move nodes that have already been attached to load balancers after restarting

When using multislb, the node distribution could be changed after restarting the ccm. This is because the node distribution cache would be lost after restarting. This PR restores the node distribution each time the ccm is restarted.

#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes #

#### Special notes for your reviewer:

This only affects cluster services.

#### Does this PR introduce a user-facing change?
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://git.k8s.io/community/contributors/guide/release-notes.md
-->
```release-note
fix: Do not move nodes that have already been attached to load balancers after restarting

When using multislb, the node distribution could be changed after restarting the ccm. This is because the node distribution cache would be lost after restarting. This PR restores the node distribution each time the ccm is restarted.
```

#### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

<!--
This section can be blank if this pull request does not require a release note.

When adding links which point to resources within git repositories, like
KEPs or supporting documentation, please reference a specific commit and avoid
linking directly to the master branch. This ensures that links reference a
specific point in time, rather than a document that may change over time.

See here for guidance on getting permanent links to files: https://help.github.com/en/articles/getting-permanent-links-to-files

Please use the following format for linking documentation:
- [KEP]: <link>
- [Usage]: <link>
- [Other doc]: <link>
-->
```docs

```


k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/6965#" title="Author self-approved">nilo19</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/master/OWNERS)~~ [nilo19]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
k8s-ci-robot: @nilo19: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-unit | 5a603a8fe21ff1166cbbd017a7cb3bb5b6753b5e | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/6965/pull-cloud-provider-azure-unit/1831583461685596160) | true | `/test pull-cloud-provider-azure-unit`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=6965). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Anilo19). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
coveralls: 
[![Coverage Status](https://coveralls.io/builds/69644481/badge)](https://coveralls.io/builds/69644481)

coverage: 77.245% (+0.01%) from 77.233%
when pulling **01f3d9f47691120152d33a85d6270d701e87b1b9 on nilo19:fix/multi-slb-node**
into **516ad1896dd52dfe61fd6ef0d1debcc20af7bcdb on kubernetes-sigs:master**.

k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/6965#pullrequestreview-2291362913" title="Approved">feiskyer</a>*, *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/6965#" title="Author self-approved">nilo19</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/master/OWNERS)~~ [feiskyer,nilo19]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->

Patch:
diff --git a/.github/workflows/trivy.yaml b/.github/workflows/trivy.yaml
index a2468e58e..1b17cf4a4 100644
--- a/.github/workflows/trivy.yaml
+++ b/.github/workflows/trivy.yaml
@@ -7,6 +7,7 @@ on:
     paths:
       - '.github/workflows/trivy.yaml'
       - 'pkg/**.go'
+      - '!pkg/azureclients/v2/**.go'
       - '!kubetest2-aks/**.go'
       - 'cmd/**.go'
       - 'go.*'
@@ -17,19 +18,20 @@ on:
     paths:
       - '.github/workflows/trivy.yaml'
       - 'pkg/**.go'
+      - '!pkg/azureclients/v2/**.go'
       - '!kubetest2-aks/**.go'
       - 'cmd/**.go'
       - 'go.*'
       - '!vendor/**'
       - 'health-probe-proxy/**'
   schedule:
-    - cron: '0 */24 * * *'
+    - cron: '35 12 * * 4'
 permissions:
   contents: read
 jobs:
   build:
     permissions:
-      contents: write # for sbom
+      contents: read # for actions/checkout to fetch code
       security-events: write # for github/codeql-action/upload-sarif to upload SARIF results
     name: Build
     runs-on: ubuntu-latest
@@ -102,7 +104,8 @@ jobs:
         uses: aquasecurity/trivy-action@6e7b7d1fd3e4fef0c5fa8cce1229c54b2c9bd0d8 # master
         with:
           scan-type: 'fs'
+          ignore-unfixed: true
           format: 'github'
           output: 'dependency-results.sbom.json'
-          scan-ref: '.'
+          image-ref: '.'
           github-pat: ${{ secrets.GITHUB_TOKEN }}
diff --git a/Dockerfile b/Dockerfile
index ad1b0514f..6e494e55b 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -14,7 +14,7 @@
 
 # syntax=docker/dockerfile:1
 
-FROM --platform=linux/amd64 golang:1.23-bullseye@sha256:45b43371f21ec51276118e6806a22cbb0bca087ddd54c491fdc7149be01035d5 AS builder
+FROM --platform=linux/amd64 golang:1.23-bullseye@sha256:ecef8303ced05b7cd1addf3c8ea98974f9231d4c5a0c230d23b37bb623714a23 AS builder
 
 ARG ENABLE_GIT_COMMAND=true
 ARG ARCH=amd64
diff --git a/cloud-node-manager.Dockerfile b/cloud-node-manager.Dockerfile
index 0df48a71c..c4441d61c 100644
--- a/cloud-node-manager.Dockerfile
+++ b/cloud-node-manager.Dockerfile
@@ -14,7 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM --platform=linux/amd64 golang:1.23-bullseye@sha256:45b43371f21ec51276118e6806a22cbb0bca087ddd54c491fdc7149be01035d5 AS builder
+FROM --platform=linux/amd64 golang:1.23-bullseye@sha256:ecef8303ced05b7cd1addf3c8ea98974f9231d4c5a0c230d23b37bb623714a23 AS builder
 
 ARG ENABLE_GIT_COMMAND=true
 ARG ARCH=amd64
diff --git a/go.mod b/go.mod
index 00392ff41..633adf457 100644
--- a/go.mod
+++ b/go.mod
@@ -24,7 +24,7 @@ require (
 	github.com/go-logr/logr v1.4.2
 	github.com/onsi/ginkgo/v2 v2.20.2
 	github.com/onsi/gomega v1.34.2
-	github.com/prometheus/client_golang v1.20.3
+	github.com/prometheus/client_golang v1.20.2
 	github.com/spf13/cobra v1.8.1
 	github.com/spf13/pflag v1.0.5
 	github.com/stretchr/testify v1.9.0
diff --git a/go.sum b/go.sum
index b57234313..20eb9768b 100644
--- a/go.sum
+++ b/go.sum
@@ -206,8 +206,8 @@ github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINE
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
 github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
-github.com/prometheus/client_golang v1.20.3 h1:oPksm4K8B+Vt35tUhw6GbSNSgVlVSBH0qELP/7u83l4=
-github.com/prometheus/client_golang v1.20.3/go.mod h1:PIEt8X02hGcP8JWbeHyeZ53Y/jReSnHgO035n//V5WE=
+github.com/prometheus/client_golang v1.20.2 h1:5ctymQzZlyOON1666svgwn3s6IKWgfbjsejTMiXIyjg=
+github.com/prometheus/client_golang v1.20.2/go.mod h1:PIEt8X02hGcP8JWbeHyeZ53Y/jReSnHgO035n//V5WE=
 github.com/prometheus/client_model v0.6.1 h1:ZKSh/rekM+n3CeS952MLRAdFwIKqeY8b62p8ais2e9E=
 github.com/prometheus/client_model v0.6.1/go.mod h1:OrxVMOVHjw3lKMa8+x6HeMGkHMQyHDk9E3jmP2AmGiY=
 github.com/prometheus/common v0.55.0 h1:KEi6DK7lXW/m7Ig5i47x0vRzuBsHuvJdi5ee6Y3G1dc=
diff --git a/pkg/azclient/Makefile b/pkg/azclient/Makefile
index d3faf00a0..e0efaecec 100644
--- a/pkg/azclient/Makefile
+++ b/pkg/azclient/Makefile
@@ -103,8 +103,6 @@ generatecode: build ## Generate client
 	$(TYPESCAFFOLD) --package github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/keyvault/armkeyvault --package-alias armkeyvault --resource Vault --subresource Secret --client-name SecretsClient --verbs get,createorupdate,delete,list
 	$(TYPESCAFFOLD) --package github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/keyvault/armkeyvault --package-alias armkeyvault --resource Vault --client-name VaultsClient --verbs get,createorupdate,delete,list
 	$(TYPESCAFFOLD) --package github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources --package-alias armresources --resource Provider --client-name ProvidersClient 
-	$(TYPESCAFFOLD) --package github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2 --package-alias armauthorization --resource RoleAssignment --client-name RoleAssignmentsClient
-	$(TYPESCAFFOLD) --package github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2 --package-alias armauthorization --resource RoleDefinition --client-name RoleDefinitionsClient
 
 
 .PHONY: generateimpl
diff --git a/pkg/azclient/client-gen/generator/codegenerator.go b/pkg/azclient/client-gen/generator/codegenerator.go
index d2606c0f1..bb6d5ccfd 100644
--- a/pkg/azclient/client-gen/generator/codegenerator.go
+++ b/pkg/azclient/client-gen/generator/codegenerator.go
@@ -109,11 +109,10 @@ func (g Generator) Generate(ctx *genall.GenerationContext) error {
 					root.AddError(err)
 					return
 				}
-				if !markerConf.OutOfSubscriptionScope {
-					if err := factoryGenerator.RegisterClient(ctx, root, typeInfo.Name, markerConf, headerText); err != nil {
-						root.AddError(err)
-						return
-					}
+
+				if err := factoryGenerator.RegisterClient(ctx, root, typeInfo.Name, markerConf, headerText); err != nil {
+					root.AddError(err)
+					return
 				}
 			}
 		})
@@ -130,7 +129,7 @@ func (g Generator) Generate(ctx *genall.GenerationContext) error {
 	fmt.Println("Run go test ")
 
 	//nolint:gosec // G204 ignore this!
-	cmd = exec.Command("go", "test", "./...", "--timeout=30m")
+	cmd = exec.Command("go", "test", "./...")
 	cmd.Stdout = os.Stdout
 	cmd.Stderr = os.Stderr
 	return cmd.Run()
diff --git a/pkg/azclient/client-gen/generator/gencode_action.go b/pkg/azclient/client-gen/generator/gencode_action.go
index e44e59d08..251c07b43 100644
--- a/pkg/azclient/client-gen/generator/gencode_action.go
+++ b/pkg/azclient/client-gen/generator/gencode_action.go
@@ -85,9 +85,6 @@ func generateClient(ctx *genall.GenerationContext, root *loader.Package, _ strin
 	importList["github.com/Azure/azure-sdk-for-go/sdk/azcore"] = make(map[string]struct{})
 	importList["github.com/Azure/azure-sdk-for-go/sdk/azcore/arm"] = make(map[string]struct{})
 	importList["github.com/Azure/azure-sdk-for-go/sdk/azcore/tracing"] = make(map[string]struct{})
-	if markerConf.OutOfSubscriptionScope && len(markerConf.Verbs) > 0 {
-		importList["go.opentelemetry.io/otel/attribute"] = make(map[string]struct{})
-	}
 	if err := WriteToFile(ctx, root, "zz_generated_client.go", headerText, importList, &outContent); err != nil {
 		return err
 	}
diff --git a/pkg/azclient/client-gen/generator/template.go b/pkg/azclient/client-gen/generator/template.go
index 3d9dd54c8..bd9c354c4 100644
--- a/pkg/azclient/client-gen/generator/template.go
+++ b/pkg/azclient/client-gen/generator/template.go
@@ -19,22 +19,21 @@ package generator
 import "html/template"
 
 type ClientGenConfig struct {
-	Verbs                  []string `marker:",optional"`
-	Resource               string
-	SubResource            string `marker:"subResource,optional"`
-	PackageName            string
-	PackageAlias           string
-	ClientName             string
-	OutOfSubscriptionScope bool   `marker:"outOfSubscriptionScope,optional"`
-	Expand                 bool   `marker:"expand,optional"`
-	RateLimitKey           string `marker:"rateLimitKey,optional"`
-	CrossSubFactory        bool   `marker:"crossSubFactory,optional"`
+	Verbs           []string `marker:",optional"`
+	Resource        string
+	SubResource     string `marker:"subResource,optional"`
+	PackageName     string
+	PackageAlias    string
+	ClientName      string
+	Expand          bool   `marker:"expand,optional"`
+	RateLimitKey    string `marker:"rateLimitKey,optional"`
+	CrossSubFactory bool   `marker:"crossSubFactory,optional"`
 }
 
 var ClientTemplate = template.Must(template.New("object-scaffolding-client-struct").Parse(`
 type Client struct{
 	*{{.PackageAlias}}.{{.ClientName}}
-	{{if not .OutOfSubscriptionScope -}}subscriptionID string {{- end}}
+	subscriptionID string
 	tracer tracing.Tracer
 }
 `))
@@ -46,13 +45,13 @@ func New(subscriptionID string, credential azcore.TokenCredential, options *arm.
 	}
 	tr := options.TracingProvider.NewTracer(utils.ModuleName, utils.ModuleVersion)
 
-	client, err := {{.PackageAlias}}.New{{.ClientName}}({{if not .OutOfSubscriptionScope}}subscriptionID,{{end}} credential, options)
+	client, err := {{.PackageAlias}}.New{{.ClientName}}(subscriptionID, credential, options)
 	if err != nil {
 		return nil, err
 	}
 	return &Client{ 
 		{{.ClientName}}: client, 
-		{{if not .OutOfSubscriptionScope}}subscriptionID: subscriptionID,{{end}}
+		subscriptionID:  subscriptionID,
 		tracer:          tr, 
 	}, nil
 }
@@ -66,11 +65,7 @@ var CreateOrUpdateFuncTemplate = template.Must(template.New("object-scaffolding-
 const CreateOrUpdateOperationName = "{{.ClientName}}.Create"
 // CreateOrUpdate creates or updates a {{$resource}}.
 func (client *Client) CreateOrUpdate(ctx context.Context, resourceGroupName string, resourceName string,{{with .SubResource}}parentResourceName string, {{end}} resource {{.PackageAlias}}.{{$resource}}) (result *{{.PackageAlias}}.{{$resource}}, err error) {
-	{{if .OutOfSubscriptionScope -}}
-	metricsCtx := metrics.BeginARMRequestWithAttributes(attribute.String("resource", "{{ $resource }}"), attribute.String("method", "create_or_update"))
-	{{else -}}
 	metricsCtx := metrics.BeginARMRequest(client.subscriptionID, resourceGroupName, "{{ $resource }}", "create_or_update")
-	{{end -}}
 	defer func() { metricsCtx.Observe(ctx, err) }()
 	ctx, endSpan := runtime.StartSpan(ctx, CreateOrUpdateOperationName, client.tracer, nil)
 	defer endSpan(err)
@@ -116,16 +111,12 @@ var ListFuncTemplate = template.Must(template.New("object-scaffolding-list-func"
 {{- end }}
 const ListOperationName = "{{.ClientName}}.List"
 // List gets a list of {{$resource}} in the resource group.
-func (client *Client) List(ctx context.Context,{{if .OutOfSubscriptionScope}} scopeName{{else}} resourceGroupName{{end}} string{{with .SubResource}}, parentResourceName string{{end}}) (result []*{{.PackageAlias}}.{{$resource}}, err error) {
-	{{if .OutOfSubscriptionScope -}}
-	metricsCtx := metrics.BeginARMRequestWithAttributes(attribute.String("resource", "{{ $resource }}"), attribute.String("method", "list"))
-	{{else -}}
-	metricsCtx := metrics.BeginARMRequest({{if .OutOfSubscriptionScope}}scopeName{{else}}client.subscriptionID, resourceGroupName,{{end}} "{{ $resource }}", "list")
-	{{end -}}
+func (client *Client) List(ctx context.Context, resourceGroupName string{{with .SubResource}}, parentResourceName string{{end}}) (result []*{{.PackageAlias}}.{{$resource}}, err error) {
+	metricsCtx := metrics.BeginARMRequest(client.subscriptionID, resourceGroupName, "{{ $resource }}", "list")
 	defer func() { metricsCtx.Observe(ctx, err) }()
 	ctx, endSpan := runtime.StartSpan(ctx, ListOperationName, client.tracer, nil)
 	defer endSpan(err)
-	pager := client.{{.ClientName}}.NewListPager({{if .OutOfSubscriptionScope}}scopeName{{else}}resourceGroupName{{end}},{{with .SubResource}} parentResourceName,{{end}} nil)
+	pager := client.{{.ClientName}}.NewListPager(resourceGroupName,{{with .SubResource}}parentResourceName,{{end}} nil)
 	for pager.More() {
 		nextResult, err := pager.NextPage(ctx)
 		if err != nil {
@@ -162,15 +153,15 @@ var GetFuncTemplate = template.Must(template.New("object-scaffolding-get-func").
 const GetOperationName = "{{.ClientName}}.Get"
 // Get gets the {{$resource}}
 func (client *Client) Get(ctx context.Context, resourceGroupName string, {{with .SubResource}}parentResourceName string,{{end}} resourceName string{{if .Expand}}, expand *string{{end}}) (result *{{.PackageAlias}}.{{$resource}}, err error) {
-	{{ if .Expand -}}var ops *{{.PackageAlias}}.{{.ClientName}}GetOptions
+	{{ if .Expand}}var ops *{{.PackageAlias}}.{{.ClientName}}GetOptions
 	if expand != nil {
 		ops = &{{.PackageAlias}}.{{.ClientName}}GetOptions{ Expand: expand }
-	}{{ end }}
+	}{{- end}}
 	metricsCtx := metrics.BeginARMRequest(client.subscriptionID, resourceGroupName, "{{ $resource }}", "get")
 	defer func() { metricsCtx.Observe(ctx, err) }()
 	ctx, endSpan := runtime.StartSpan(ctx, GetOperationName, client.tracer, nil)
 	defer endSpan(err)
-	resp, err := client.{{.ClientName}}.Get(ctx, resourceGroupName,{{with .SubResource}}parentResourceName,{{end}} resourceName,{{if .Expand}} ops{{else}} nil{{end}})
+	resp, err := client.{{.ClientName}}.Get(ctx, resourceGroupName,{{with .SubResource}}parentResourceName,{{end}} resourceName,{{if .Expand}}ops{{else}}nil{{end}} )
 	if err != nil {
 		return nil, err
 	}
diff --git a/pkg/azclient/factory.go b/pkg/azclient/factory.go
index 697f9d24d..f51b85dc8 100644
--- a/pkg/azclient/factory.go
+++ b/pkg/azclient/factory.go
@@ -37,7 +37,6 @@ import (
 	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/publicipprefixclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/registryclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/resourcegroupclient"
-	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/roleassignmentclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/routetableclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/secretclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/securitygroupclient"
@@ -76,7 +75,6 @@ type ClientFactory interface {
 	GetPublicIPPrefixClient() publicipprefixclient.Interface
 	GetRegistryClient() registryclient.Interface
 	GetResourceGroupClient() resourcegroupclient.Interface
-	GetRoleAssignmentClient() roleassignmentclient.Interface
 	GetRouteTableClient() routetableclient.Interface
 	GetSecretClient() secretclient.Interface
 	GetSecurityGroupClient() securitygroupclient.Interface
diff --git a/pkg/azclient/factory_gen.go b/pkg/azclient/factory_gen.go
index 6a3df53a3..1c2f36ee1 100644
--- a/pkg/azclient/factory_gen.go
+++ b/pkg/azclient/factory_gen.go
@@ -45,7 +45,6 @@ import (
 	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/publicipprefixclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/registryclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/resourcegroupclient"
-	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/roleassignmentclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/routetableclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/secretclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/securitygroupclient"
@@ -84,7 +83,6 @@ type ClientFactoryImpl struct {
 	publicipprefixclientInterface           publicipprefixclient.Interface
 	registryclientInterface                 registryclient.Interface
 	resourcegroupclientInterface            resourcegroupclient.Interface
-	roleassignmentclientInterface           roleassignmentclient.Interface
 	routetableclientInterface               routetableclient.Interface
 	secretclientInterface                   secretclient.Interface
 	securitygroupclientInterface            securitygroupclient.Interface
@@ -230,12 +228,6 @@ func NewClientFactory(config *ClientFactoryConfig, armConfig *ARMClientConfig, c
 		return nil, err
 	}
 
-	//initialize roleassignmentclient
-	factory.roleassignmentclientInterface, err = factory.createRoleAssignmentClient(config.SubscriptionID)
-	if err != nil {
-		return nil, err
-	}
-
 	//initialize routetableclient
 	factory.routetableclientInterface, err = factory.createRouteTableClient(config.SubscriptionID)
 	if err != nil {
@@ -805,25 +797,6 @@ func (factory *ClientFactoryImpl) GetResourceGroupClient() resourcegroupclient.I
 	return factory.resourcegroupclientInterface
 }
 
-func (factory *ClientFactoryImpl) createRoleAssignmentClient(subscription string) (roleassignmentclient.Interface, error) {
-	//initialize roleassignmentclient
-	options, err := GetDefaultResourceClientOption(factory.armConfig, factory.facotryConfig)
-	if err != nil {
-		return nil, err
-	}
-
-	for _, optionMutFn := range factory.clientOptionsMutFn {
-		if optionMutFn != nil {
-			optionMutFn(options)
-		}
-	}
-	return roleassignmentclient.New(subscription, factory.cred, options)
-}
-
-func (factory *ClientFactoryImpl) GetRoleAssignmentClient() roleassignmentclient.Interface {
-	return factory.roleassignmentclientInterface
-}
-
 func (factory *ClientFactoryImpl) createRouteTableClient(subscription string) (routetableclient.Interface, error) {
 	//initialize routetableclient
 	options, err := GetDefaultResourceClientOption(factory.armConfig, factory.facotryConfig)
diff --git a/pkg/azclient/go.mod b/pkg/azclient/go.mod
index 2a1439ecc..eb9f733f8 100644
--- a/pkg/azclient/go.mod
+++ b/pkg/azclient/go.mod
@@ -7,7 +7,6 @@ toolchain go1.23.0
 require (
 	github.com/Azure/azure-sdk-for-go/sdk/azcore v1.14.0
 	github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0
-	github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2 v2.2.0
 	github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v5 v5.7.0
 	github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerregistry/armcontainerregistry v1.2.0
 	github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v4 v4.8.0
diff --git a/pkg/azclient/go.sum b/pkg/azclient/go.sum
index 53109c6e6..21ed6fb8c 100644
--- a/pkg/azclient/go.sum
+++ b/pkg/azclient/go.sum
@@ -4,8 +4,6 @@ github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0 h1:tfLQ34V6F7tVSwoTf/4lH
 github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0/go.mod h1:9kIvujWAA58nmPmWB1m23fyWic1kYZMxD9CxaWn4Qpg=
 github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0 h1:ywEEhmNahHBihViHepv3xPBn1663uRv2t2q/ESv9seY=
 github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0/go.mod h1:iZDifYGJTIgIIkYRNWPENUnqx6bJ2xnSDFI2tjwZNuY=
-github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2 v2.2.0 h1:Hp+EScFOu9HeCbeW8WU2yQPJd4gGwhMgKxWe+G6jNzw=
-github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2 v2.2.0/go.mod h1:/pz8dyNQe+Ey3yBp/XuYz7oqX8YDNWVpPB0hH3XWfbc=
 github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v5 v5.7.0 h1:LkHbJbgF3YyvC53aqYGR+wWQDn2Rdp9AQdGndf9QvY4=
 github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v5 v5.7.0/go.mod h1:QyiQdW4f4/BIfB8ZutZ2s+28RAgfa/pT+zS++ZHyM1I=
 github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerregistry/armcontainerregistry v1.2.0 h1:DWlwvVV5r/Wy1561nZ3wrpI1/vDIBRY/Wd1HWaRBZWA=
diff --git a/pkg/azclient/metrics/metrics.go b/pkg/azclient/metrics/metrics.go
index f6dadcb16..ee25b950e 100644
--- a/pkg/azclient/metrics/metrics.go
+++ b/pkg/azclient/metrics/metrics.go
@@ -47,19 +47,14 @@ type ARMContext struct {
 
 // BeginARMRequest creates a new ARMContext for an ARM request.
 func BeginARMRequest(subscriptionID, resourceGroup, resource, method string) *ARMContext {
-	return BeginARMRequestWithAttributes(
-		attribute.String("subscription_id", subscriptionID),
-		attribute.String("resource_group", strings.ToLower(resourceGroup)),
-		attribute.String("resource", resource),
-		attribute.String("method", method),
-	)
-}
-
-// BeginARMRequest creates a new ARMContext for an ARM request.
-func BeginARMRequestWithAttributes(attributes ...attribute.KeyValue) *ARMContext {
 	return &ARMContext{
-		startedAt:  time.Now(),
-		attributes: attributes,
+		startedAt: time.Now(),
+		attributes: []attribute.KeyValue{
+			attribute.String("subscription_id", subscriptionID),
+			attribute.String("resource_group", strings.ToLower(resourceGroup)),
+			attribute.String("resource", resource),
+			attribute.String("method", method),
+		},
 	}
 }
 
diff --git a/pkg/azclient/mock_azclient/interface.go b/pkg/azclient/mock_azclient/interface.go
index 7d7f0cac8..8e0445440 100644
--- a/pkg/azclient/mock_azclient/interface.go
+++ b/pkg/azclient/mock_azclient/interface.go
@@ -49,7 +49,6 @@ import (
 	publicipprefixclient "sigs.k8s.io/cloud-provider-azure/pkg/azclient/publicipprefixclient"
 	registryclient "sigs.k8s.io/cloud-provider-azure/pkg/azclient/registryclient"
 	resourcegroupclient "sigs.k8s.io/cloud-provider-azure/pkg/azclient/resourcegroupclient"
-	roleassignmentclient "sigs.k8s.io/cloud-provider-azure/pkg/azclient/roleassignmentclient"
 	routetableclient "sigs.k8s.io/cloud-provider-azure/pkg/azclient/routetableclient"
 	secretclient "sigs.k8s.io/cloud-provider-azure/pkg/azclient/secretclient"
 	securitygroupclient "sigs.k8s.io/cloud-provider-azure/pkg/azclient/securitygroupclient"
@@ -413,20 +412,6 @@ func (mr *MockClientFactoryMockRecorder) GetResourceGroupClient() *gomock.Call {
 	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetResourceGroupClient", reflect.TypeOf((*MockClientFactory)(nil).GetResourceGroupClient))
 }
 
-// GetRoleAssignmentClient mocks base method.
-func (m *MockClientFactory) GetRoleAssignmentClient() roleassignmentclient.Interface {
-	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "GetRoleAssignmentClient")
-	ret0, _ := ret[0].(roleassignmentclient.Interface)
-	return ret0
-}
-
-// GetRoleAssignmentClient indicates an expected call of GetRoleAssignmentClient.
-func (mr *MockClientFactoryMockRecorder) GetRoleAssignmentClient() *gomock.Call {
-	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetRoleAssignmentClient", reflect.TypeOf((*MockClientFactory)(nil).GetRoleAssignmentClient))
-}
-
 // GetRouteTableClient mocks base method.
 func (m *MockClientFactory) GetRouteTableClient() routetableclient.Interface {
 	m.ctrl.T.Helper()
diff --git a/pkg/azclient/roleassignmentclient/custom.go b/pkg/azclient/roleassignmentclient/custom.go
deleted file mode 100644
index 3506235ae..000000000
--- a/pkg/azclient/roleassignmentclient/custom.go
+++ /dev/null
@@ -1,72 +0,0 @@
-/*
-Copyright 2024 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-// +azure:enableclientgen:=true
-package roleassignmentclient
-
-import (
-	"context"
-
-	"github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime"
-	armauthorization "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2"
-
-	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/metrics"
-)
-
-// Get gets a role assignment.
-const GetOperationName = "RoleAssignmentsClient.Get"
-
-func (client *Client) Get(ctx context.Context, scope string, roleAssignmentName string, options *armauthorization.RoleAssignmentsClientGetOptions) (result *armauthorization.RoleAssignment, err error) {
-	metricsCtx := metrics.BeginARMRequest(client.subscriptionID, scope, "RoleAssignment", "get")
-	defer func() { metricsCtx.Observe(ctx, err) }()
-	ctx, endSpan := runtime.StartSpan(ctx, GetOperationName, client.tracer, nil)
-	defer endSpan(err)
-	resp, err := client.RoleAssignmentsClient.Get(ctx, scope, roleAssignmentName, options)
-	if err != nil {
-		return nil, err
-	}
-	return &resp.RoleAssignment, nil
-}
-
-const DeleteOperationName = "RoleAssignmentsClient.Delete"
-
-// Delete deletes a Subnet by name.
-func (client *Client) Delete(ctx context.Context, scope string, roleAssignmentName string, options *armauthorization.RoleAssignmentsClientDeleteOptions) (result *armauthorization.RoleAssignment, err error) {
-	metricsCtx := metrics.BeginARMRequest(client.subscriptionID, scope, "RoleAssignment", "delete")
-	defer func() { metricsCtx.Observe(ctx, err) }()
-	ctx, endSpan := runtime.StartSpan(ctx, DeleteOperationName, client.tracer, nil)
-	defer endSpan(err)
-	resp, err := client.RoleAssignmentsClient.Delete(ctx, scope, roleAssignmentName, options)
-	if err != nil {
-		return nil, err
-	}
-	return &resp.RoleAssignment, nil
-}
-
-const CreateOrUpdateOperationName = "RoleAssignmentsClient.Create"
-
-// CreateOrUpdate creates or updates a Subnet.
-func (client *Client) Create(ctx context.Context, scope string, roleAssignmentName string, parameters armauthorization.RoleAssignmentCreateParameters) (result *armauthorization.RoleAssignment, err error) {
-	metricsCtx := metrics.BeginARMRequest(client.subscriptionID, scope, "RoleAssignment", "create")
-	defer func() { metricsCtx.Observe(ctx, err) }()
-	ctx, endSpan := runtime.StartSpan(ctx, CreateOrUpdateOperationName, client.tracer, nil)
-	defer endSpan(err)
-	resp, err := client.RoleAssignmentsClient.Create(ctx, scope, roleAssignmentName, parameters, nil)
-	if err != nil {
-		return nil, err
-	}
-	return &resp.RoleAssignment, nil
-}
diff --git a/pkg/azclient/roleassignmentclient/interface.go b/pkg/azclient/roleassignmentclient/interface.go
deleted file mode 100644
index 9f77b123e..000000000
--- a/pkg/azclient/roleassignmentclient/interface.go
+++ /dev/null
@@ -1,31 +0,0 @@
-/*
-Copyright 2024 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-// +azure:enableclientgen:=true
-package roleassignmentclient
-
-import (
-	"context"
-
-	armauthorization "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2"
-)
-
-// +azure:client:verbs=,resource=RoleAssignment,packageName=github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2,packageAlias=armauthorization,clientName=RoleAssignmentsClient,expand=false
-type Interface interface {
-	Get(ctx context.Context, scope string, roleAssignmentName string, options *armauthorization.RoleAssignmentsClientGetOptions) (*armauthorization.RoleAssignment, error)
-	Create(ctx context.Context, scope string, roleAssignmentName string, parameters armauthorization.RoleAssignmentCreateParameters) (*armauthorization.RoleAssignment, error)
-	Delete(ctx context.Context, scope string, roleAssignmentName string, options *armauthorization.RoleAssignmentsClientDeleteOptions) (*armauthorization.RoleAssignment, error)
-}
diff --git a/pkg/azclient/roleassignmentclient/mock_roleassignmentclient/interface.go b/pkg/azclient/roleassignmentclient/mock_roleassignmentclient/interface.go
deleted file mode 100644
index 41b8ea124..000000000
--- a/pkg/azclient/roleassignmentclient/mock_roleassignmentclient/interface.go
+++ /dev/null
@@ -1,102 +0,0 @@
-// /*
-// Copyright The Kubernetes Authors.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-// */
-
-// Code generated by MockGen. DO NOT EDIT.
-// Source: roleassignmentclient/interface.go
-//
-// Generated by this command:
-//
-//	mockgen -package mock_roleassignmentclient -source roleassignmentclient/interface.go
-//
-
-// Package mock_roleassignmentclient is a generated GoMock package.
-package mock_roleassignmentclient
-
-import (
-	context "context"
-	reflect "reflect"
-
-	armauthorization "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2"
-	gomock "go.uber.org/mock/gomock"
-)
-
-// MockInterface is a mock of Interface interface.
-type MockInterface struct {
-	ctrl     *gomock.Controller
-	recorder *MockInterfaceMockRecorder
-}
-
-// MockInterfaceMockRecorder is the mock recorder for MockInterface.
-type MockInterfaceMockRecorder struct {
-	mock *MockInterface
-}
-
-// NewMockInterface creates a new mock instance.
-func NewMockInterface(ctrl *gomock.Controller) *MockInterface {
-	mock := &MockInterface{ctrl: ctrl}
-	mock.recorder = &MockInterfaceMockRecorder{mock}
-	return mock
-}
-
-// EXPECT returns an object that allows the caller to indicate expected use.
-func (m *MockInterface) EXPECT() *MockInterfaceMockRecorder {
-	return m.recorder
-}
-
-// Create mocks base method.
-func (m *MockInterface) Create(ctx context.Context, scope, roleAssignmentName string, parameters armauthorization.RoleAssignmentCreateParameters) (*armauthorization.RoleAssignment, error) {
-	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "Create", ctx, scope, roleAssignmentName, parameters)
-	ret0, _ := ret[0].(*armauthorization.RoleAssignment)
-	ret1, _ := ret[1].(error)
-	return ret0, ret1
-}
-
-// Create indicates an expected call of Create.
-func (mr *MockInterfaceMockRecorder) Create(ctx, scope, roleAssignmentName, parameters any) *gomock.Call {
-	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Create", reflect.TypeOf((*MockInterface)(nil).Create), ctx, scope, roleAssignmentName, parameters)
-}
-
-// Delete mocks base method.
-func (m *MockInterface) Delete(ctx context.Context, scope, roleAssignmentName string, options *armauthorization.RoleAssignmentsClientDeleteOptions) (*armauthorization.RoleAssignment, error) {
-	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "Delete", ctx, scope, roleAssignmentName, options)
-	ret0, _ := ret[0].(*armauthorization.RoleAssignment)
-	ret1, _ := ret[1].(error)
-	return ret0, ret1
-}
-
-// Delete indicates an expected call of Delete.
-func (mr *MockInterfaceMockRecorder) Delete(ctx, scope, roleAssignmentName, options any) *gomock.Call {
-	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Delete", reflect.TypeOf((*MockInterface)(nil).Delete), ctx, scope, roleAssignmentName, options)
-}
-
-// Get mocks base method.
-func (m *MockInterface) Get(ctx context.Context, scope, roleAssignmentName string, options *armauthorization.RoleAssignmentsClientGetOptions) (*armauthorization.RoleAssignment, error) {
-	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "Get", ctx, scope, roleAssignmentName, options)
-	ret0, _ := ret[0].(*armauthorization.RoleAssignment)
-	ret1, _ := ret[1].(error)
-	return ret0, ret1
-}
-
-// Get indicates an expected call of Get.
-func (mr *MockInterfaceMockRecorder) Get(ctx, scope, roleAssignmentName, options any) *gomock.Call {
-	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Get", reflect.TypeOf((*MockInterface)(nil).Get), ctx, scope, roleAssignmentName, options)
-}
diff --git a/pkg/azclient/roleassignmentclient/zz_generated_client.go b/pkg/azclient/roleassignmentclient/zz_generated_client.go
deleted file mode 100644
index 3ea3b2f13..000000000
--- a/pkg/azclient/roleassignmentclient/zz_generated_client.go
+++ /dev/null
@@ -1,50 +0,0 @@
-// /*
-// Copyright The Kubernetes Authors.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-// */
-
-// Code generated by client-gen. DO NOT EDIT.
-package roleassignmentclient
-
-import (
-	"github.com/Azure/azure-sdk-for-go/sdk/azcore"
-	"github.com/Azure/azure-sdk-for-go/sdk/azcore/arm"
-	"github.com/Azure/azure-sdk-for-go/sdk/azcore/tracing"
-	armauthorization "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2"
-
-	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/utils"
-)
-
-type Client struct {
-	*armauthorization.RoleAssignmentsClient
-	subscriptionID string
-	tracer         tracing.Tracer
-}
-
-func New(subscriptionID string, credential azcore.TokenCredential, options *arm.ClientOptions) (Interface, error) {
-	if options == nil {
-		options = utils.GetDefaultOption()
-	}
-	tr := options.TracingProvider.NewTracer(utils.ModuleName, utils.ModuleVersion)
-
-	client, err := armauthorization.NewRoleAssignmentsClient(subscriptionID, credential, options)
-	if err != nil {
-		return nil, err
-	}
-	return &Client{
-		RoleAssignmentsClient: client,
-		subscriptionID:        subscriptionID,
-		tracer:                tr,
-	}, nil
-}
diff --git a/pkg/azclient/roledefinitionclient/custom.go b/pkg/azclient/roledefinitionclient/custom.go
deleted file mode 100644
index 8ab58f54d..000000000
--- a/pkg/azclient/roledefinitionclient/custom.go
+++ /dev/null
@@ -1,47 +0,0 @@
-/*
-Copyright 2024 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-// +azure:enableclientgen:=true
-package roledefinitionclient
-
-import (
-	"context"
-
-	"github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime"
-	armauthorization "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2"
-	"go.opentelemetry.io/otel/attribute"
-
-	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/metrics"
-)
-
-const ListOperationName = "RoleDefinitionsClient.List"
-
-// List gets a list of RoleDefinition in the resource group.
-func (client *Client) List(ctx context.Context, scopeName string, option *armauthorization.RoleDefinitionsClientListOptions) (result []*armauthorization.RoleDefinition, err error) {
-	metricsCtx := metrics.BeginARMRequestWithAttributes(attribute.String("resource", "RoleDefinition"), attribute.String("method", "list"))
-	defer func() { metricsCtx.Observe(ctx, err) }()
-	ctx, endSpan := runtime.StartSpan(ctx, ListOperationName, client.tracer, nil)
-	defer endSpan(err)
-	pager := client.RoleDefinitionsClient.NewListPager(scopeName, option)
-	for pager.More() {
-		nextResult, err := pager.NextPage(ctx)
-		if err != nil {
-			return nil, err
-		}
-		result = append(result, nextResult.Value...)
-	}
-	return result, nil
-}
diff --git a/pkg/azclient/roledefinitionclient/interface.go b/pkg/azclient/roledefinitionclient/interface.go
deleted file mode 100644
index 8e850c6c6..000000000
--- a/pkg/azclient/roledefinitionclient/interface.go
+++ /dev/null
@@ -1,29 +0,0 @@
-/*
-Copyright 2024 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-// +azure:enableclientgen:=true
-package roledefinitionclient
-
-import (
-	"context"
-
-	armauthorization "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2"
-)
-
-// +azure:client:verbs=,resource=RoleDefinition,packageName=github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2,packageAlias=armauthorization,clientName=RoleDefinitionsClient,expand=false,outOfSubscriptionScope=true
-type Interface interface {
-	List(ctx context.Context, scope string, option *armauthorization.RoleDefinitionsClientListOptions) (result []*armauthorization.RoleDefinition, rerr error)
-}
diff --git a/pkg/azclient/roledefinitionclient/mock_roledefinitionclient/interface.go b/pkg/azclient/roledefinitionclient/mock_roledefinitionclient/interface.go
deleted file mode 100644
index e92198459..000000000
--- a/pkg/azclient/roledefinitionclient/mock_roledefinitionclient/interface.go
+++ /dev/null
@@ -1,72 +0,0 @@
-// /*
-// Copyright The Kubernetes Authors.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-// */
-
-// Code generated by MockGen. DO NOT EDIT.
-// Source: roledefinitionclient/interface.go
-//
-// Generated by this command:
-//
-//	mockgen -package mock_roledefinitionclient -source roledefinitionclient/interface.go
-//
-
-// Package mock_roledefinitionclient is a generated GoMock package.
-package mock_roledefinitionclient
-
-import (
-	context "context"
-	reflect "reflect"
-
-	armauthorization "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2"
-	gomock "go.uber.org/mock/gomock"
-)
-
-// MockInterface is a mock of Interface interface.
-type MockInterface struct {
-	ctrl     *gomock.Controller
-	recorder *MockInterfaceMockRecorder
-}
-
-// MockInterfaceMockRecorder is the mock recorder for MockInterface.
-type MockInterfaceMockRecorder struct {
-	mock *MockInterface
-}
-
-// NewMockInterface creates a new mock instance.
-func NewMockInterface(ctrl *gomock.Controller) *MockInterface {
-	mock := &MockInterface{ctrl: ctrl}
-	mock.recorder = &MockInterfaceMockRecorder{mock}
-	return mock
-}
-
-// EXPECT returns an object that allows the caller to indicate expected use.
-func (m *MockInterface) EXPECT() *MockInterfaceMockRecorder {
-	return m.recorder
-}
-
-// List mocks base method.
-func (m *MockInterface) List(ctx context.Context, scope string, option *armauthorization.RoleDefinitionsClientListOptions) ([]*armauthorization.RoleDefinition, error) {
-	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "List", ctx, scope, option)
-	ret0, _ := ret[0].([]*armauthorization.RoleDefinition)
-	ret1, _ := ret[1].(error)
-	return ret0, ret1
-}
-
-// List indicates an expected call of List.
-func (mr *MockInterfaceMockRecorder) List(ctx, scope, option any) *gomock.Call {
-	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "List", reflect.TypeOf((*MockInterface)(nil).List), ctx, scope, option)
-}
diff --git a/pkg/azclient/roledefinitionclient/zz_generated_client.go b/pkg/azclient/roledefinitionclient/zz_generated_client.go
deleted file mode 100644
index 8d94af4a5..000000000
--- a/pkg/azclient/roledefinitionclient/zz_generated_client.go
+++ /dev/null
@@ -1,50 +0,0 @@
-// /*
-// Copyright The Kubernetes Authors.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-// */
-
-// Code generated by client-gen. DO NOT EDIT.
-package roledefinitionclient
-
-import (
-	"github.com/Azure/azure-sdk-for-go/sdk/azcore"
-	"github.com/Azure/azure-sdk-for-go/sdk/azcore/arm"
-	"github.com/Azure/azure-sdk-for-go/sdk/azcore/tracing"
-	armauthorization "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2"
-
-	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/utils"
-)
-
-type Client struct {
-	*armauthorization.RoleDefinitionsClient
-
-	tracer tracing.Tracer
-}
-
-func New(subscriptionID string, credential azcore.TokenCredential, options *arm.ClientOptions) (Interface, error) {
-	if options == nil {
-		options = utils.GetDefaultOption()
-	}
-	tr := options.TracingProvider.NewTracer(utils.ModuleName, utils.ModuleVersion)
-
-	client, err := armauthorization.NewRoleDefinitionsClient(credential, options)
-	if err != nil {
-		return nil, err
-	}
-	return &Client{
-		RoleDefinitionsClient: client,
-
-		tracer: tr,
-	}, nil
-}
diff --git a/pkg/azclient/trace/go.mod b/pkg/azclient/trace/go.mod
index c793afd5e..351b79c38 100644
--- a/pkg/azclient/trace/go.mod
+++ b/pkg/azclient/trace/go.mod
@@ -6,7 +6,7 @@ require (
 	github.com/Azure/azure-sdk-for-go/sdk/azcore v1.14.0
 	go.opentelemetry.io/otel v1.29.0
 	go.opentelemetry.io/otel/trace v1.29.0
-	sigs.k8s.io/cloud-provider-azure/pkg/azclient v0.0.49
+	sigs.k8s.io/cloud-provider-azure/pkg/azclient v0.0.46
 )
 
 require (
@@ -15,5 +15,5 @@ require (
 	github.com/go-logr/stdr v1.2.2 // indirect
 	go.opentelemetry.io/otel/metric v1.29.0 // indirect
 	golang.org/x/net v0.28.0 // indirect
-	golang.org/x/text v0.18.0 // indirect
+	golang.org/x/text v0.17.0 // indirect
 )
diff --git a/pkg/azclient/trace/go.sum b/pkg/azclient/trace/go.sum
index ac23f59ef..34d1e2045 100644
--- a/pkg/azclient/trace/go.sum
+++ b/pkg/azclient/trace/go.sum
@@ -31,13 +31,13 @@ go.opentelemetry.io/otel/trace v1.29.0 h1:J/8ZNK4XgR7a21DZUAsbF8pZ5Jcw1VhACmnYt3
 go.opentelemetry.io/otel/trace v1.29.0/go.mod h1:eHl3w0sp3paPkYstJOmAimxhiFXPg+MMTlEh3nsQgWQ=
 golang.org/x/net v0.28.0 h1:a9JDOJc5GMUJ0+UDqmLT86WiEy7iWyIhz8gz8E4e5hE=
 golang.org/x/net v0.28.0/go.mod h1:yqtgsTWOOnlGLG9GFRrK3++bGOUEkNBoHZc8MEDWPNg=
-golang.org/x/sys v0.25.0 h1:r+8e+loiHxRqhXVl6ML1nO3l1+oFoWbnlu2Ehimmi34=
-golang.org/x/sys v0.25.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
-golang.org/x/text v0.18.0 h1:XvMDiNzPAl0jr17s6W9lcaIhGUfUORdGCNsuLmPG224=
-golang.org/x/text v0.18.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY=
+golang.org/x/sys v0.24.0 h1:Twjiwq9dn6R1fQcyiK+wQyHWfaz/BJB+YIpzU/Cv3Xg=
+golang.org/x/sys v0.24.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
+golang.org/x/text v0.17.0 h1:XtiM5bkSOt+ewxlOE/aE/AKEHibwj/6gvWMl9Rsh0Qc=
+golang.org/x/text v0.17.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY=
 golang.org/x/tools v0.24.0 h1:J1shsA93PJUEVaUSaay7UXAyE8aimq3GW0pjlolpa24=
 golang.org/x/tools v0.24.0/go.mod h1:YhNqVBIfWHdzvTLs0d8LCuMhkKUgSUKldakyV7W/WDQ=
 gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
 gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
-sigs.k8s.io/cloud-provider-azure/pkg/azclient v0.0.49 h1:3AsgRzy3mz/AOsofFsuzQFs2paHACDl2J0Ibd6dh+5Q=
-sigs.k8s.io/cloud-provider-azure/pkg/azclient v0.0.49/go.mod h1:1M90A+akyTabHVnveSKlvIO/Kk9kEr1LjRx+08twKVU=
+sigs.k8s.io/cloud-provider-azure/pkg/azclient v0.0.46 h1:4n8pSFxd9zre5WVO3JZ/v9URyK8b1L0B9zUk3EUlE2g=
+sigs.k8s.io/cloud-provider-azure/pkg/azclient v0.0.46/go.mod h1:MDKJvFWPruKRfuG19tM1n8ZHUgy7eTbgdTIpBrlqn9w=
diff --git a/pkg/provider/azure_loadbalancer.go b/pkg/provider/azure_loadbalancer.go
index 4a4528beb..90400b4ca 100644
--- a/pkg/provider/azure_loadbalancer.go
+++ b/pkg/provider/azure_loadbalancer.go
@@ -1650,7 +1650,7 @@ func (az *Cloud) reconcileMultipleStandardLoadBalancerConfigurations(
 		}
 	}
 
-	return az.reconcileMultipleStandardLoadBalancerBackendNodes("", lbs, service, nodes)
+	return az.reconcileMultipleStandardLoadBalancerBackendNodes(clusterName, "", lbs, service, nodes, true)
 }
 
 // reconcileLoadBalancer ensures load balancer exists and the frontend ip config is setup.
@@ -1852,7 +1852,7 @@ func (az *Cloud) reconcileLoadBalancer(ctx context.Context, clusterName string,
 		}()
 
 		if az.useMultipleStandardLoadBalancers() {
-			err := az.reconcileMultipleStandardLoadBalancerBackendNodes(lbName, existingLBs, service, nodes)
+			err := az.reconcileMultipleStandardLoadBalancerBackendNodes(clusterName, lbName, existingLBs, service, nodes, false)
 			if err != nil {
 				return nil, err
 			}
@@ -2151,11 +2151,27 @@ func isLBInList(lbs *[]network.LoadBalancer, lbConfigName string) bool {
 // 2. We only check nodes that are not matched by primary vmSet before we ensure
 // hosts in pool. So the number API calls is under control.
 func (az *Cloud) reconcileMultipleStandardLoadBalancerBackendNodes(
+	clusterName string,
 	lbName string,
 	lbs *[]network.LoadBalancer,
 	service *v1.Service,
 	nodes []*v1.Node,
+	init bool,
 ) error {
+	logger := klog.Background().WithName("reconcileMultipleStandardLoadBalancerBackendNodes").
+		WithValues(
+			"clusterName", clusterName,
+			"lbName", lbName,
+			"service", service.Name,
+			"init", init,
+		)
+	if init {
+		if err := az.recordExistingNodesOnLoadBalancers(clusterName, lbs); err != nil {
+			logger.Error(err, "failed to record existing nodes on load balancers")
+			return err
+		}
+	}
+
 	// Remove the nodes from the load balancer configurations if they are not in the node list.
 	nodeNameToLBConfigIDXMap := az.removeDeletedNodesFromLoadBalancerConfigurations(nodes)
 
@@ -2172,6 +2188,38 @@ func (az *Cloud) reconcileMultipleStandardLoadBalancerBackendNodes(
 	return nil
 }
 
+// recordExistingNodesOnLoadBalancers restores the node distribution
+// across multiple load balancers each time the cloud provider restarts.
+func (az *Cloud) recordExistingNodesOnLoadBalancers(clusterName string, lbs *[]network.LoadBalancer) error {
+	bi, ok := az.LoadBalancerBackendPool.(*backendPoolTypeNodeIP)
+	if !ok {
+		return errors.New("must use backend pool type nodeIP")
+	}
+	bpNames := getBackendPoolNames(clusterName)
+	for _, lb := range *lbs {
+		if lb.LoadBalancerPropertiesFormat == nil ||
+			lb.LoadBalancerPropertiesFormat.BackendAddressPools == nil {
+			continue
+		}
+		lbName := ptr.Deref(lb.Name, "")
+		for _, backendPool := range *lb.LoadBalancerPropertiesFormat.BackendAddressPools {
+			backendPool := backendPool
+			if found, _ := isLBBackendPoolsExisting(bpNames, backendPool.Name); found {
+				nodeNames := bi.getBackendPoolNodeNames(&backendPool)
+				for i, multiSLBConfig := range az.MultipleStandardLoadBalancerConfigurations {
+					if strings.EqualFold(trimSuffixIgnoreCase(
+						lbName, consts.InternalLoadBalancerNameSuffix,
+					), multiSLBConfig.Name) {
+						az.MultipleStandardLoadBalancerConfigurations[i].ActiveNodes =
+							utilsets.SafeInsert(multiSLBConfig.ActiveNodes, nodeNames...)
+					}
+				}
+			}
+		}
+	}
+	return nil
+}
+
 func (az *Cloud) reconcileMultipleStandardLoadBalancerConfigurationStatus(wantLb bool, svcName, lbName string) {
 	lbName = trimSuffixIgnoreCase(lbName, consts.InternalLoadBalancerNameSuffix)
 	for i := range az.MultipleStandardLoadBalancerConfigurations {
diff --git a/pkg/provider/azure_loadbalancer_backendpool.go b/pkg/provider/azure_loadbalancer_backendpool.go
index 2962f47d2..2cf104761 100644
--- a/pkg/provider/azure_loadbalancer_backendpool.go
+++ b/pkg/provider/azure_loadbalancer_backendpool.go
@@ -774,6 +774,20 @@ func (bi *backendPoolTypeNodeIP) GetBackendPrivateIPs(clusterName string, servic
 	return backendPrivateIPv4s.UnsortedList(), backendPrivateIPv6s.UnsortedList()
 }
 
+// getBackendPoolNameForService returns all node names in the backend pool.
+func (bi *backendPoolTypeNodeIP) getBackendPoolNodeNames(bp *network.BackendAddressPool) []string {
+	nodeNames := utilsets.NewString()
+	if bp.BackendAddressPoolPropertiesFormat != nil && bp.LoadBalancerBackendAddresses != nil {
+		for _, backendAddress := range *bp.LoadBalancerBackendAddresses {
+			if backendAddress.LoadBalancerBackendAddressPropertiesFormat != nil {
+				ip := ptr.Deref(backendAddress.IPAddress, "")
+				nodeNames.Insert(bi.nodePrivateIPToNodeNameMap[ip])
+			}
+		}
+	}
+	return nodeNames.UnsortedList()
+}
+
 func newBackendPool(lb *network.LoadBalancer, isBackendPoolPreConfigured bool, preConfiguredBackendPoolLoadBalancerTypes, serviceName, lbBackendPoolName string) bool {
 	if isBackendPoolPreConfigured {
 		klog.V(2).Infof("newBackendPool for service (%s)(true): lb backendpool - PreConfiguredBackendPoolLoadBalancerTypes %s has been set but can not find corresponding backend pool %q, ignoring it",
diff --git a/vendor/github.com/prometheus/client_golang/prometheus/histogram.go b/vendor/github.com/prometheus/client_golang/prometheus/histogram.go
index 8a4f49a4c..8d35f2d8a 100644
--- a/vendor/github.com/prometheus/client_golang/prometheus/histogram.go
+++ b/vendor/github.com/prometheus/client_golang/prometheus/histogram.go
@@ -1658,10 +1658,7 @@ func addAndResetCounts(hot, cold *histogramCounts) {
 type nativeExemplars struct {
 	sync.Mutex
 
-	// Time-to-live for exemplars, it is set to -1 if exemplars are disabled, that is NativeHistogramMaxExemplars is below 0.
-	// The ttl is used on insertion to remove an exemplar that is older than ttl, if present.
-	ttl time.Duration
-
+	ttl       time.Duration
 	exemplars []*dto.Exemplar
 }
 
@@ -1676,7 +1673,6 @@ func makeNativeExemplars(ttl time.Duration, maxCount int) nativeExemplars {
 
 	if maxCount < 0 {
 		maxCount = 0
-		ttl = -1
 	}
 
 	return nativeExemplars{
@@ -1686,18 +1682,20 @@ func makeNativeExemplars(ttl time.Duration, maxCount int) nativeExemplars {
 }
 
 func (n *nativeExemplars) addExemplar(e *dto.Exemplar) {
-	if n.ttl == -1 {
+	if cap(n.exemplars) == 0 {
 		return
 	}
 
 	n.Lock()
 	defer n.Unlock()
 
+	// The index where to insert the new exemplar.
+	var nIdx int = -1
+
 	// When the number of exemplars has not yet exceeded or
 	// is equal to cap(n.exemplars), then
 	// insert the new exemplar directly.
 	if len(n.exemplars) < cap(n.exemplars) {
-		var nIdx int
 		for nIdx = 0; nIdx < len(n.exemplars); nIdx++ {
 			if *e.Value < *n.exemplars[nIdx].Value {
 				break
@@ -1707,46 +1705,17 @@ func (n *nativeExemplars) addExemplar(e *dto.Exemplar) {
 		return
 	}
 
-	if len(n.exemplars) == 1 {
-		// When the number of exemplars is 1, then
-		// replace the existing exemplar with the new exemplar.
-		n.exemplars[0] = e
-		return
-	}
-	// From this point on, the number of exemplars is greater than 1.
-
 	// When the number of exemplars exceeds the limit, remove one exemplar.
 	var (
-		ot    = time.Time{} // Oldest timestamp seen. Initial value doesn't matter as we replace it due to otIdx == -1 in the loop.
-		otIdx = -1          // Index of the exemplar with the oldest timestamp.
-
-		md = -1.0 // Logarithm of the delta of the closest pair of exemplars.
-
-		// The insertion point of the new exemplar in the exemplars slice after insertion.
-		// This is calculated purely based on the order of the exemplars by value.
-		// nIdx == len(n.exemplars) means the new exemplar is to be inserted after the end.
-		nIdx = -1
-
-		// rIdx is ultimately the index for the exemplar that we are replacing with the new exemplar.
-		// The aim is to keep a good spread of exemplars by value and not let them bunch up too much.
-		// It is calculated in 3 steps:
-		//   1. First we set rIdx to the index of the older exemplar within the closest pair by value.
-		//      That is the following will be true (on log scale):
-		//      either the exemplar pair on index (rIdx-1, rIdx) or (rIdx, rIdx+1) will have
-		//      the closest values to each other from all pairs.
-		//      For example, suppose the values are distributed like this:
-		//        |-----------x-------------x----------------x----x-----|
-		//                                                   ^--rIdx as this is older.
-		//      Or like this:
-		//        |-----------x-------------x----------------x----x-----|
-		//                                                        ^--rIdx as this is older.
-		//   2. If there is an exemplar that expired, then we simple reset rIdx to that index.
-		//   3. We check if by inserting the new exemplar we would create a closer pair at
-		//      (nIdx-1, nIdx) or (nIdx, nIdx+1) and set rIdx to nIdx-1 or nIdx accordingly to
-		//      keep the spread of exemplars by value; otherwise we keep rIdx as it is.
-		rIdx = -1
-		cLog float64 // Logarithm of the current exemplar.
-		pLog float64 // Logarithm of the previous exemplar.
+		rIdx int // The index where to remove the old exemplar.
+
+		ot    = time.Now() // Oldest timestamp seen.
+		otIdx = -1         // Index of the exemplar with the oldest timestamp.
+
+		md    = -1.0  // Logarithm of the delta of the closest pair of exemplars.
+		mdIdx = -1    // Index of the older exemplar within the closest pair.
+		cLog  float64 // Logarithm of the current exemplar.
+		pLog  float64 // Logarithm of the previous exemplar.
 	)
 
 	for i, exemplar := range n.exemplars {
@@ -1757,7 +1726,7 @@ func (n *nativeExemplars) addExemplar(e *dto.Exemplar) {
 		}
 
 		// Find the index at which to insert new the exemplar.
-		if nIdx == -1 && *e.Value <= *exemplar.Value {
+		if *e.Value <= *exemplar.Value && nIdx == -1 {
 			nIdx = i
 		}
 
@@ -1769,13 +1738,11 @@ func (n *nativeExemplars) addExemplar(e *dto.Exemplar) {
 		}
 		diff := math.Abs(cLog - pLog)
 		if md == -1 || diff < md {
-			// The closest exemplar pair is at index: i-1, i.
-			// Choose the exemplar with the older timestamp for replacement.
 			md = diff
 			if n.exemplars[i].Timestamp.AsTime().Before(n.exemplars[i-1].Timestamp.AsTime()) {
-				rIdx = i
+				mdIdx = i
 			} else {
-				rIdx = i - 1
+				mdIdx = i - 1
 			}
 		}
 
@@ -1786,12 +1753,8 @@ func (n *nativeExemplars) addExemplar(e *dto.Exemplar) {
 	if nIdx == -1 {
 		nIdx = len(n.exemplars)
 	}
-	// Here, we have the following relationships:
-	// n.exemplars[nIdx-1].Value < e.Value (if nIdx > 0)
-	// e.Value <= n.exemplars[nIdx].Value (if nIdx < len(n.exemplars))
 
 	if otIdx != -1 && e.Timestamp.AsTime().Sub(ot) > n.ttl {
-		// If the oldest exemplar has expired, then replace it with the new exemplar.
 		rIdx = otIdx
 	} else {
 		// In the previous for loop, when calculating the closest pair of exemplars,
@@ -1801,26 +1764,23 @@ func (n *nativeExemplars) addExemplar(e *dto.Exemplar) {
 		if nIdx > 0 {
 			diff := math.Abs(elog - math.Log(n.exemplars[nIdx-1].GetValue()))
 			if diff < md {
-				// The value we are about to insert is closer to the previous exemplar at the insertion point than what we calculated before in rIdx.
-				//                                            v--rIdx
-				// |-----------x-n-----------x----------------x----x-----|
-				//     nIdx-1--^ ^--new exemplar value
-				// Do not make the spread worse, replace nIdx-1 and not rIdx.
 				md = diff
-				rIdx = nIdx - 1
+				mdIdx = nIdx
+				if n.exemplars[nIdx-1].Timestamp.AsTime().Before(e.Timestamp.AsTime()) {
+					mdIdx = nIdx - 1
+				}
 			}
 		}
 		if nIdx < len(n.exemplars) {
 			diff := math.Abs(math.Log(n.exemplars[nIdx].GetValue()) - elog)
 			if diff < md {
-				// The value we are about to insert is closer to the next exemplar at the insertion point than what we calculated before in rIdx.
-				//                                            v--rIdx
-				// |-----------x-----------n-x----------------x----x-----|
-				//     new exemplar value--^ ^--nIdx
-				// Do not make the spread worse, replace nIdx-1 and not rIdx.
-				rIdx = nIdx
+				mdIdx = nIdx
+				if n.exemplars[nIdx].Timestamp.AsTime().Before(e.Timestamp.AsTime()) {
+					mdIdx = nIdx
+				}
 			}
 		}
+		rIdx = mdIdx
 	}
 
 	// Adjust the slice according to rIdx and nIdx.
diff --git a/vendor/modules.txt b/vendor/modules.txt
index e52517e24..5a553fe69 100644
--- a/vendor/modules.txt
+++ b/vendor/modules.txt
@@ -417,7 +417,7 @@ github.com/pkg/errors
 # github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2
 ## explicit
 github.com/pmezard/go-difflib/difflib
-# github.com/prometheus/client_golang v1.20.3
+# github.com/prometheus/client_golang v1.20.2
 ## explicit; go 1.20
 github.com/prometheus/client_golang/internal/github.com/golang/gddo/httputil
 github.com/prometheus/client_golang/internal/github.com/golang/gddo/httputil/header


Test patch:
diff --git a/e2e.Dockerfile b/e2e.Dockerfile
index fc95e2804..ba38f1251 100644
--- a/e2e.Dockerfile
+++ b/e2e.Dockerfile
@@ -14,7 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM golang:1.23-bullseye@sha256:45b43371f21ec51276118e6806a22cbb0bca087ddd54c491fdc7149be01035d5
+FROM golang:1.23-bullseye@sha256:ecef8303ced05b7cd1addf3c8ea98974f9231d4c5a0c230d23b37bb623714a23
 
 WORKDIR /go/src/sigs.k8s.io/cloud-provider-azure
 
diff --git a/pkg/azclient/factory_test.go b/pkg/azclient/factory_test.go
index 2406237ef..3c062d8a4 100644
--- a/pkg/azclient/factory_test.go
+++ b/pkg/azclient/factory_test.go
@@ -157,13 +157,6 @@ var _ = ginkgo.Describe("Factory", func() {
 			client := factory.GetResourceGroupClient()
 			gomega.Expect(client).NotTo(gomega.BeNil())
 		})
-		ginkgo.It("should create factory instance without painc - RoleAssignment", func() {
-			factory, err := NewClientFactory(nil, nil, nil)
-			gomega.Expect(err).NotTo(gomega.HaveOccurred())
-			gomega.Expect(factory).NotTo(gomega.BeNil())
-			client := factory.GetRoleAssignmentClient()
-			gomega.Expect(client).NotTo(gomega.BeNil())
-		})
 		ginkgo.It("should create factory instance without painc - RouteTable", func() {
 			factory, err := NewClientFactory(nil, nil, nil)
 			gomega.Expect(err).NotTo(gomega.HaveOccurred())
diff --git a/pkg/azclient/roleassignmentclient/custom_test.go b/pkg/azclient/roleassignmentclient/custom_test.go
deleted file mode 100644
index a9154551b..000000000
--- a/pkg/azclient/roleassignmentclient/custom_test.go
+++ /dev/null
@@ -1,33 +0,0 @@
-// /*
-// Copyright The Kubernetes Authors.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-// */
-
-// Code generated by client-gen. DO NOT EDIT.
-package roleassignmentclient
-
-import (
-	"context"
-)
-
-func init() {
-	additionalTestCases = func() {
-	}
-
-	beforeAllFunc = func(ctx context.Context) {
-	}
-
-	afterAllFunc = func(ctx context.Context) {
-	}
-}
diff --git a/pkg/azclient/roleassignmentclient/roleassignmentclient_suite_test.go b/pkg/azclient/roleassignmentclient/roleassignmentclient_suite_test.go
deleted file mode 100644
index 0fbb23c1e..000000000
--- a/pkg/azclient/roleassignmentclient/roleassignmentclient_suite_test.go
+++ /dev/null
@@ -1,87 +0,0 @@
-// /*
-// Copyright The Kubernetes Authors.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-// */
-
-// Code generated by client-gen. DO NOT EDIT.
-package roleassignmentclient
-
-import (
-	"context"
-	"testing"
-
-	"github.com/Azure/azure-sdk-for-go/sdk/azcore"
-	"github.com/Azure/azure-sdk-for-go/sdk/azcore/arm"
-	"github.com/Azure/azure-sdk-for-go/sdk/azcore/to"
-	"github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources"
-	"github.com/onsi/ginkgo/v2"
-	"github.com/onsi/gomega"
-
-	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/recording"
-	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/utils"
-)
-
-func TestClient(t *testing.T) {
-	gomega.RegisterFailHandler(ginkgo.Fail)
-	ginkgo.RunSpecs(t, "Client Suite")
-}
-
-var resourceGroupName = "aks-cit-RoleAssignment"
-var resourceName = "testResource"
-
-var subscriptionID string
-var location = "eastus"
-var resourceGroupClient *armresources.ResourceGroupsClient
-var err error
-var recorder *recording.Recorder
-var realClient Interface
-
-var _ = ginkgo.BeforeSuite(func(ctx context.Context) {
-	recorder, err = recording.NewRecorder("testdata/RoleAssignment")
-	gomega.Expect(err).ToNot(gomega.HaveOccurred())
-	subscriptionID = recorder.SubscriptionID()
-	gomega.Expect(err).NotTo(gomega.HaveOccurred())
-	cred := recorder.TokenCredential()
-	resourceGroupClient, err = armresources.NewResourceGroupsClient(subscriptionID, cred, &arm.ClientOptions{
-		ClientOptions: azcore.ClientOptions{
-			Transport:       recorder.HTTPClient(),
-			TracingProvider: utils.TracingProvider,
-		},
-	})
-	gomega.Expect(err).NotTo(gomega.HaveOccurred())
-	realClient, err = New(subscriptionID, recorder.TokenCredential(), &arm.ClientOptions{
-		ClientOptions: azcore.ClientOptions{
-			Transport:       recorder.HTTPClient(),
-			TracingProvider: utils.TracingProvider,
-		},
-	})
-	gomega.Expect(err).NotTo(gomega.HaveOccurred())
-	_, err = resourceGroupClient.CreateOrUpdate(
-		ctx,
-		resourceGroupName,
-		armresources.ResourceGroup{
-			Location: to.Ptr(location),
-		},
-		nil)
-	gomega.Expect(err).NotTo(gomega.HaveOccurred())
-})
-
-var _ = ginkgo.AfterSuite(func(ctx context.Context) {
-	poller, err := resourceGroupClient.BeginDelete(ctx, resourceGroupName, nil)
-	gomega.Expect(err).NotTo(gomega.HaveOccurred())
-	_, err = poller.PollUntilDone(ctx, nil)
-	gomega.Expect(err).ToNot(gomega.HaveOccurred())
-	err = recorder.Stop()
-	gomega.Expect(err).ToNot(gomega.HaveOccurred())
-})
diff --git a/pkg/azclient/roleassignmentclient/roleassignmentclient_test.go b/pkg/azclient/roleassignmentclient/roleassignmentclient_test.go
deleted file mode 100644
index 7e36fef3a..000000000
--- a/pkg/azclient/roleassignmentclient/roleassignmentclient_test.go
+++ /dev/null
@@ -1,43 +0,0 @@
-// /*
-// Copyright The Kubernetes Authors.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-// */
-
-// Code generated by client-gen. DO NOT EDIT.
-package roleassignmentclient
-
-import (
-	"context"
-
-	"github.com/onsi/ginkgo/v2"
-)
-
-var beforeAllFunc func(context.Context)
-var afterAllFunc func(context.Context)
-var additionalTestCases func()
-
-var _ = ginkgo.Describe("RoleAssignmentsClient", ginkgo.Ordered, func() {
-
-	if beforeAllFunc != nil {
-		ginkgo.BeforeAll(beforeAllFunc)
-	}
-
-	if additionalTestCases != nil {
-		additionalTestCases()
-	}
-
-	if afterAllFunc != nil {
-		ginkgo.AfterAll(afterAllFunc)
-	}
-})
diff --git a/pkg/azclient/roleassignmentclient/testdata/RoleAssignment.yaml b/pkg/azclient/roleassignmentclient/testdata/RoleAssignment.yaml
deleted file mode 100644
index 42c8223ba..000000000
--- a/pkg/azclient/roleassignmentclient/testdata/RoleAssignment.yaml
+++ /dev/null
@@ -1,144 +0,0 @@
----
-version: 2
-interactions:
-    - id: 0
-      request:
-        proto: HTTP/1.1
-        proto_major: 1
-        proto_minor: 1
-        content_length: 21
-        transfer_encoding: []
-        trailer: {}
-        host: management.azure.com
-        remote_addr: ""
-        request_uri: ""
-        body: '{"location":"eastus"}'
-        form: {}
-        headers:
-            Accept:
-                - application/json
-            Content-Length:
-                - "21"
-            Content-Type:
-                - application/json
-        url: https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/aks-cit-RoleAssignment?api-version=2021-04-01
-        method: PUT
-      response:
-        proto: HTTP/2.0
-        proto_major: 2
-        proto_minor: 0
-        transfer_encoding: []
-        trailer: {}
-        content_length: 241
-        uncompressed: false
-        body: '{"id":"/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/aks-cit-RoleAssignment","name":"aks-cit-RoleAssignment","type":"Microsoft.Resources/resourceGroups","location":"eastus","properties":{"provisioningState":"Succeeded"}}'
-        headers:
-            Cache-Control:
-                - no-cache
-            Content-Length:
-                - "241"
-            Content-Type:
-                - application/json; charset=utf-8
-            Expires:
-                - "-1"
-            Pragma:
-                - no-cache
-            Strict-Transport-Security:
-                - max-age=31536000; includeSubDomains
-            X-Cache:
-                - CONFIG_NOCACHE
-            X-Content-Type-Options:
-                - nosniff
-        status: 201 Created
-        code: 201
-        duration: 200ms
-    - id: 1
-      request:
-        proto: HTTP/1.1
-        proto_major: 1
-        proto_minor: 1
-        content_length: 0
-        transfer_encoding: []
-        trailer: {}
-        host: management.azure.com
-        remote_addr: ""
-        request_uri: ""
-        body: ""
-        form: {}
-        headers:
-            Accept:
-                - application/json
-        url: https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/aks-cit-RoleAssignment?api-version=2021-04-01
-        method: DELETE
-      response:
-        proto: HTTP/2.0
-        proto_major: 2
-        proto_minor: 0
-        transfer_encoding: []
-        trailer: {}
-        content_length: 0
-        uncompressed: false
-        body: ""
-        headers:
-            Cache-Control:
-                - no-cache
-            Content-Length:
-                - "0"
-            Expires:
-                - "-1"
-            Location:
-                - https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1BS1M6MkRDSVQ6MkRST0xFQVNTSUdOTUVOVC1FQVNUVVMiLCJqb2JMb2NhdGlvbiI6ImVhc3R1cyJ9?api-version=2021-04-01&c=c&h=h&s=s&t=t
-            Pragma:
-                - no-cache
-            Strict-Transport-Security:
-                - max-age=31536000; includeSubDomains
-            X-Cache:
-                - CONFIG_NOCACHE
-            X-Content-Type-Options:
-                - nosniff
-        status: 202 Accepted
-        code: 202
-        duration: 200ms
-    - id: 2
-      request:
-        proto: HTTP/1.1
-        proto_major: 1
-        proto_minor: 1
-        content_length: 0
-        transfer_encoding: []
-        trailer: {}
-        host: management.azure.com
-        remote_addr: ""
-        request_uri: ""
-        body: ""
-        form: {}
-        headers: {}
-        url: https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1BS1M6MkRDSVQ6MkRST0xFQVNTSUdOTUVOVC1FQVNUVVMiLCJqb2JMb2NhdGlvbiI6ImVhc3R1cyJ9?api-version=2021-04-01&c=c&h=h&s=s&t=t
-        method: GET
-      response:
-        proto: HTTP/2.0
-        proto_major: 2
-        proto_minor: 0
-        transfer_encoding: []
-        trailer: {}
-        content_length: 0
-        uncompressed: false
-        body: ""
-        headers:
-            Cache-Control:
-                - no-cache
-            Content-Length:
-                - "0"
-            Expires:
-                - "-1"
-            Pragma:
-                - no-cache
-            Strict-Transport-Security:
-                - max-age=31536000; includeSubDomains
-            X-Cache:
-                - CONFIG_NOCACHE
-            X-Content-Type-Options:
-                - nosniff
-        status: 200 OK
-        code: 200
-        duration: 200ms
diff --git a/pkg/azclient/roledefinitionclient/custom_test.go b/pkg/azclient/roledefinitionclient/custom_test.go
deleted file mode 100644
index 9c87e8d3b..000000000
--- a/pkg/azclient/roledefinitionclient/custom_test.go
+++ /dev/null
@@ -1,56 +0,0 @@
-// /*
-// Copyright The Kubernetes Authors.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-// */
-
-// Code generated by client-gen. DO NOT EDIT.
-package roledefinitionclient
-
-import (
-	"context"
-
-	"github.com/Azure/azure-sdk-for-go/sdk/azcore/to"
-	"github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v2"
-	"github.com/onsi/ginkgo/v2"
-	"github.com/onsi/gomega"
-)
-
-func init() {
-	additionalTestCases = func() {
-		ginkgo.When("list requests are raised", func() {
-			ginkgo.It("should not return error", func(ctx context.Context) {
-				resourceList, err := realClient.List(ctx, "/subscriptions/"+subscriptionID+"/resourceGroups/"+resourceGroupName, &armauthorization.RoleDefinitionsClientListOptions{
-					Filter: to.Ptr("roleName eq 'Contributor'"),
-				})
-				gomega.Expect(err).NotTo(gomega.HaveOccurred())
-				gomega.Expect(resourceList).NotTo(gomega.BeNil())
-				gomega.Expect(len(resourceList)).To(gomega.Equal(1))
-				gomega.Expect(*resourceList[0].Properties.RoleName).To(gomega.Equal("Contributor"))
-			})
-		})
-		ginkgo.When("invalid list requests are raised", func() {
-			ginkgo.It("should return error", func(ctx context.Context) {
-				resourceList, err := realClient.List(ctx, "/subscriptions/"+subscriptionID+"/resourceGroups/"+resourceGroupName+"notfound", nil)
-				gomega.Expect(err).To(gomega.HaveOccurred())
-				gomega.Expect(resourceList).To(gomega.BeNil())
-			})
-		})
-	}
-
-	beforeAllFunc = func(ctx context.Context) {
-
-	}
-	afterAllFunc = func(ctx context.Context) {
-	}
-}
diff --git a/pkg/azclient/roledefinitionclient/roledefinitionclient_suite_test.go b/pkg/azclient/roledefinitionclient/roledefinitionclient_suite_test.go
deleted file mode 100644
index f42cd7245..000000000
--- a/pkg/azclient/roledefinitionclient/roledefinitionclient_suite_test.go
+++ /dev/null
@@ -1,87 +0,0 @@
-// /*
-// Copyright The Kubernetes Authors.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-// */
-
-// Code generated by client-gen. DO NOT EDIT.
-package roledefinitionclient
-
-import (
-	"context"
-	"testing"
-
-	"github.com/Azure/azure-sdk-for-go/sdk/azcore"
-	"github.com/Azure/azure-sdk-for-go/sdk/azcore/arm"
-	"github.com/Azure/azure-sdk-for-go/sdk/azcore/to"
-	"github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources"
-	"github.com/onsi/ginkgo/v2"
-	"github.com/onsi/gomega"
-
-	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/recording"
-	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/utils"
-)
-
-func TestClient(t *testing.T) {
-	gomega.RegisterFailHandler(ginkgo.Fail)
-	ginkgo.RunSpecs(t, "Client Suite")
-}
-
-var resourceGroupName = "aks-cit-RoleDefinition"
-var resourceName = "testResource"
-
-var subscriptionID string
-var location = "eastus"
-var resourceGroupClient *armresources.ResourceGroupsClient
-var err error
-var recorder *recording.Recorder
-var realClient Interface
-
-var _ = ginkgo.BeforeSuite(func(ctx context.Context) {
-	recorder, err = recording.NewRecorder("testdata/RoleDefinition")
-	gomega.Expect(err).ToNot(gomega.HaveOccurred())
-	subscriptionID = recorder.SubscriptionID()
-	gomega.Expect(err).NotTo(gomega.HaveOccurred())
-	cred := recorder.TokenCredential()
-	resourceGroupClient, err = armresources.NewResourceGroupsClient(subscriptionID, cred, &arm.ClientOptions{
-		ClientOptions: azcore.ClientOptions{
-			Transport:       recorder.HTTPClient(),
-			TracingProvider: utils.TracingProvider,
-		},
-	})
-	gomega.Expect(err).NotTo(gomega.HaveOccurred())
-	realClient, err = New(subscriptionID, recorder.TokenCredential(), &arm.ClientOptions{
-		ClientOptions: azcore.ClientOptions{
-			Transport:       recorder.HTTPClient(),
-			TracingProvider: utils.TracingProvider,
-		},
-	})
-	gomega.Expect(err).NotTo(gomega.HaveOccurred())
-	_, err = resourceGroupClient.CreateOrUpdate(
-		ctx,
-		resourceGroupName,
-		armresources.ResourceGroup{
-			Location: to.Ptr(location),
-		},
-		nil)
-	gomega.Expect(err).NotTo(gomega.HaveOccurred())
-})
-
-var _ = ginkgo.AfterSuite(func(ctx context.Context) {
-	poller, err := resourceGroupClient.BeginDelete(ctx, resourceGroupName, nil)
-	gomega.Expect(err).NotTo(gomega.HaveOccurred())
-	_, err = poller.PollUntilDone(ctx, nil)
-	gomega.Expect(err).ToNot(gomega.HaveOccurred())
-	err = recorder.Stop()
-	gomega.Expect(err).ToNot(gomega.HaveOccurred())
-})
diff --git a/pkg/azclient/roledefinitionclient/roledefinitionclient_test.go b/pkg/azclient/roledefinitionclient/roledefinitionclient_test.go
deleted file mode 100644
index 31636a7a0..000000000
--- a/pkg/azclient/roledefinitionclient/roledefinitionclient_test.go
+++ /dev/null
@@ -1,43 +0,0 @@
-// /*
-// Copyright The Kubernetes Authors.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-// */
-
-// Code generated by client-gen. DO NOT EDIT.
-package roledefinitionclient
-
-import (
-	"context"
-
-	"github.com/onsi/ginkgo/v2"
-)
-
-var beforeAllFunc func(context.Context)
-var afterAllFunc func(context.Context)
-var additionalTestCases func()
-
-var _ = ginkgo.Describe("RoleDefinitionsClient", ginkgo.Ordered, func() {
-
-	if beforeAllFunc != nil {
-		ginkgo.BeforeAll(beforeAllFunc)
-	}
-
-	if additionalTestCases != nil {
-		additionalTestCases()
-	}
-
-	if afterAllFunc != nil {
-		ginkgo.AfterAll(afterAllFunc)
-	}
-})
diff --git a/pkg/azclient/roledefinitionclient/testdata/RoleDefinition.yaml b/pkg/azclient/roledefinitionclient/testdata/RoleDefinition.yaml
deleted file mode 100644
index e5a63697a..000000000
--- a/pkg/azclient/roledefinitionclient/testdata/RoleDefinition.yaml
+++ /dev/null
@@ -1,191 +0,0 @@
----
-version: 2
-interactions:
-    - id: 0
-      request:
-        proto: HTTP/1.1
-        proto_major: 1
-        proto_minor: 1
-        content_length: 21
-        transfer_encoding: []
-        trailer: {}
-        host: management.azure.com
-        remote_addr: ""
-        request_uri: ""
-        body: '{"location":"eastus"}'
-        form: {}
-        headers:
-            Accept:
-                - application/json
-            Content-Length:
-                - "21"
-            Content-Type:
-                - application/json
-        url: https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/aks-cit-RoleDefinition?api-version=2021-04-01
-        method: PUT
-      response:
-        proto: HTTP/2.0
-        proto_major: 2
-        proto_minor: 0
-        transfer_encoding: []
-        trailer: {}
-        content_length: 241
-        uncompressed: false
-        body: '{"id":"/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/aks-cit-RoleDefinition","name":"aks-cit-RoleDefinition","type":"Microsoft.Resources/resourceGroups","location":"eastus","properties":{"provisioningState":"Succeeded"}}'
-        headers:
-            Cache-Control:
-                - no-cache
-            Content-Length:
-                - "241"
-            Content-Type:
-                - application/json; charset=utf-8
-            Expires:
-                - "-1"
-            Pragma:
-                - no-cache
-            Strict-Transport-Security:
-                - max-age=31536000; includeSubDomains
-            X-Cache:
-                - CONFIG_NOCACHE
-            X-Content-Type-Options:
-                - nosniff
-        status: 201 Created
-        code: 201
-        duration: 200ms
-    - id: 1
-      request:
-        proto: HTTP/1.1
-        proto_major: 1
-        proto_minor: 1
-        content_length: 0
-        transfer_encoding: []
-        trailer: {}
-        host: management.azure.com
-        remote_addr: ""
-        request_uri: ""
-        body: ""
-        form: {}
-        headers:
-            Accept:
-                - application/json
-        url: https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/aks-cit-RoleDefinition/providers/Microsoft.Authorization/roleDefinitions?%24filter=roleName+eq+%27Contributor%27&api-version=2022-04-01
-        method: GET
-      response:
-        proto: HTTP/2.0
-        proto_major: 2
-        proto_minor: 0
-        transfer_encoding: []
-        trailer: {}
-        content_length: 1119
-        uncompressed: false
-        body: '{"value":[{"properties":{"roleName":"Contributor","type":"BuiltInRole","description":"Grants full access to manage all resources, but does not allow you to assign roles in Azure RBAC, manage assignments in Azure Blueprints, or share image galleries.","assignableScopes":["/"],"permissions":[{"actions":["*"],"notActions":["Microsoft.Authorization/*/Delete","Microsoft.Authorization/*/Write","Microsoft.Authorization/elevateAccess/Action","Microsoft.Blueprint/blueprintAssignments/write","Microsoft.Blueprint/blueprintAssignments/delete","Microsoft.Compute/galleries/share/action","Microsoft.Purview/consents/write","Microsoft.Purview/consents/delete","Microsoft.Resources/deploymentStacks/manageDenySetting/action"],"dataActions":[],"notDataActions":[]}],"createdOn":"2001-02-03T04:05:06Z","updatedOn":"2001-02-03T04:05:06Z","createdBy":null,"updatedBy":null},"id":"/subscriptions/00000000-0000-0000-0000-000000000000/providers/Microsoft.Authorization/roleDefinitions/00000000-0000-0000-0000-000000000000","type":"Microsoft.Authorization/roleDefinitions","name":"00000000-0000-0000-0000-000000000000"}]}'
-        headers:
-            Cache-Control:
-                - no-cache
-            Content-Length:
-                - "1119"
-            Content-Type:
-                - application/json; charset=utf-8
-            Expires:
-                - "-1"
-            Pragma:
-                - no-cache
-            Strict-Transport-Security:
-                - max-age=31536000; includeSubDomains
-            X-Cache:
-                - CONFIG_NOCACHE
-            X-Content-Type-Options:
-                - nosniff
-        status: 200 OK
-        code: 200
-        duration: 200ms
-    - id: 2
-      request:
-        proto: HTTP/1.1
-        proto_major: 1
-        proto_minor: 1
-        content_length: 0
-        transfer_encoding: []
-        trailer: {}
-        host: management.azure.com
-        remote_addr: ""
-        request_uri: ""
-        body: ""
-        form: {}
-        headers:
-            Accept:
-                - application/json
-        url: https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/aks-cit-RoleDefinition?api-version=2021-04-01
-        method: DELETE
-      response:
-        proto: HTTP/2.0
-        proto_major: 2
-        proto_minor: 0
-        transfer_encoding: []
-        trailer: {}
-        content_length: 0
-        uncompressed: false
-        body: ""
-        headers:
-            Cache-Control:
-                - no-cache
-            Content-Length:
-                - "0"
-            Expires:
-                - "-1"
-            Location:
-                - https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1BS1M6MkRDSVQ6MkRST0xFREVGSU5JVElPTi1FQVNUVVMiLCJqb2JMb2NhdGlvbiI6ImVhc3R1cyJ9?api-version=2021-04-01&c=c&h=h&s=s&t=t
-            Pragma:
-                - no-cache
-            Strict-Transport-Security:
-                - max-age=31536000; includeSubDomains
-            X-Cache:
-                - CONFIG_NOCACHE
-            X-Content-Type-Options:
-                - nosniff
-        status: 202 Accepted
-        code: 202
-        duration: 200ms
-    - id: 3
-      request:
-        proto: HTTP/1.1
-        proto_major: 1
-        proto_minor: 1
-        content_length: 0
-        transfer_encoding: []
-        trailer: {}
-        host: management.azure.com
-        remote_addr: ""
-        request_uri: ""
-        body: ""
-        form: {}
-        headers: {}
-        url: https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/operationresults/eyJqb2JJZCI6IlJFU09VUkNFR1JPVVBERUxFVElPTkpPQi1BS1M6MkRDSVQ6MkRST0xFREVGSU5JVElPTi1FQVNUVVMiLCJqb2JMb2NhdGlvbiI6ImVhc3R1cyJ9?api-version=2021-04-01&c=c&h=h&s=s&t=t
-        method: GET
-      response:
-        proto: HTTP/2.0
-        proto_major: 2
-        proto_minor: 0
-        transfer_encoding: []
-        trailer: {}
-        content_length: 0
-        uncompressed: false
-        body: ""
-        headers:
-            Cache-Control:
-                - no-cache
-            Content-Length:
-                - "0"
-            Expires:
-                - "-1"
-            Pragma:
-                - no-cache
-            Strict-Transport-Security:
-                - max-age=31536000; includeSubDomains
-            X-Cache:
-                - CONFIG_NOCACHE
-            X-Content-Type-Options:
-                - nosniff
-        status: 200 OK
-        code: 200
-        duration: 200ms
diff --git a/pkg/provider/azure_loadbalancer_test.go b/pkg/provider/azure_loadbalancer_test.go
index 3a116a426..0fb4d8a6f 100644
--- a/pkg/provider/azure_loadbalancer_test.go
+++ b/pkg/provider/azure_loadbalancer_test.go
@@ -7599,6 +7599,7 @@ func TestReconcileMultipleStandardLoadBalancerConfigurations(t *testing.T) {
 			}
 
 			if tc.useMultipleLB {
+				az.LoadBalancerBackendPool = newBackendPoolTypeNodeIP(az)
 				az.MultipleStandardLoadBalancerConfigurations = []MultipleStandardLoadBalancerConfiguration{
 					{Name: "lb1"},
 					{Name: "lb2"},
@@ -8076,6 +8077,7 @@ func TestReconcileMultipleStandardLoadBalancerNodes(t *testing.T) {
 	for _, tc := range []struct {
 		description          string
 		lbName               string
+		init                 bool
 		existingLBConfigs    []MultipleStandardLoadBalancerConfiguration
 		existingNodes        []*v1.Node
 		existingLBs          []network.LoadBalancer
@@ -8424,6 +8426,89 @@ func TestReconcileMultipleStandardLoadBalancerNodes(t *testing.T) {
 				"lb4": nil,
 			},
 		},
+		{
+			description: "should record current node distributions after restarting the controller",
+			init:        true,
+			existingLBConfigs: []MultipleStandardLoadBalancerConfiguration{
+				{
+					Name: "lb1",
+					MultipleStandardLoadBalancerConfigurationSpec: MultipleStandardLoadBalancerConfigurationSpec{
+						PrimaryVMSet: "vmss-1",
+					},
+				},
+				{
+					Name: "lb2",
+					MultipleStandardLoadBalancerConfigurationSpec: MultipleStandardLoadBalancerConfigurationSpec{
+						PrimaryVMSet: "vmss-2",
+					},
+				},
+			},
+			existingNodes: []*v1.Node{
+				getTestNodeWithMetadata("node1", "vmss-1", map[string]string{"k1": "v1"}, "10.1.0.1"),
+				getTestNodeWithMetadata("node2", "vmss-2", map[string]string{"k3": "v3"}, "10.1.0.2"),
+				getTestNodeWithMetadata("node3", "vmss-3", map[string]string{"k2": "v2"}, "10.1.0.3"),
+				getTestNodeWithMetadata("node5", "vmss-5", map[string]string{"k2": "v2"}, "10.1.0.5"),
+				getTestNodeWithMetadata("node6", "vmss-6", map[string]string{"k3": "v3"}, "10.1.0.6"),
+			},
+			existingLBs: []network.LoadBalancer{
+				{
+					Name: ptr.To("lb1-internal"),
+					LoadBalancerPropertiesFormat: &network.LoadBalancerPropertiesFormat{
+						BackendAddressPools: &[]network.BackendAddressPool{
+							{
+								Name: ptr.To("kubernetes"),
+								BackendAddressPoolPropertiesFormat: &network.BackendAddressPoolPropertiesFormat{
+									LoadBalancerBackendAddresses: &[]network.LoadBalancerBackendAddress{
+										{
+											Name: ptr.To("node2"),
+											LoadBalancerBackendAddressPropertiesFormat: &network.LoadBalancerBackendAddressPropertiesFormat{
+												IPAddress: ptr.To("10.1.0.2"),
+											},
+										},
+									},
+								},
+							},
+						},
+					},
+				},
+				{
+					Name: ptr.To("lb2"),
+					LoadBalancerPropertiesFormat: &network.LoadBalancerPropertiesFormat{
+						BackendAddressPools: &[]network.BackendAddressPool{
+							{
+								Name: ptr.To("kubernetes"),
+								BackendAddressPoolPropertiesFormat: &network.BackendAddressPoolPropertiesFormat{
+									LoadBalancerBackendAddresses: &[]network.LoadBalancerBackendAddress{
+										{
+											Name: ptr.To("node3"),
+											LoadBalancerBackendAddressPropertiesFormat: &network.LoadBalancerBackendAddressPropertiesFormat{
+												IPAddress: ptr.To("10.1.0.3"),
+											},
+										},
+										{
+											Name: ptr.To("node4"),
+											LoadBalancerBackendAddressPropertiesFormat: &network.LoadBalancerBackendAddressPropertiesFormat{
+												IPAddress: ptr.To("10.1.0.4"),
+											},
+										},
+										{
+											Name: ptr.To("node5"),
+											LoadBalancerBackendAddressPropertiesFormat: &network.LoadBalancerBackendAddressPropertiesFormat{
+												IPAddress: ptr.To("10.1.0.5"),
+											},
+										},
+									},
+								},
+							},
+						},
+					},
+				},
+			},
+			expectedLBToNodesMap: map[string]*utilsets.IgnoreCaseSet{
+				"lb1": utilsets.NewString("node1", "node6"),
+				"lb2": utilsets.NewString("node2", "node3", "node5"),
+			},
+		},
 	} {
 		tc := tc
 		t.Run(tc.description, func(t *testing.T) {
@@ -8431,9 +8516,18 @@ func TestReconcileMultipleStandardLoadBalancerNodes(t *testing.T) {
 			ss, _ := NewTestScaleSet(ctrl)
 			ss.DisableAvailabilitySetNodes = true
 			az.VMSet = ss
+			az.nodePrivateIPToNodeNameMap = map[string]string{
+				"10.1.0.1": "node1",
+				"10.1.0.2": "node2",
+				"10.1.0.3": "node3",
+				"10.1.0.4": "node4",
+				"10.1.0.5": "node5",
+				"10.1.0.6": "node6",
+			}
+			az.LoadBalancerBackendPool = newBackendPoolTypeNodeIP(az)
 			az.MultipleStandardLoadBalancerConfigurations = tc.existingLBConfigs
 			svc := getTestService("test", v1.ProtocolTCP, nil, false)
-			_ = az.reconcileMultipleStandardLoadBalancerBackendNodes(tc.lbName, &tc.existingLBs, &svc, tc.existingNodes)
+			_ = az.reconcileMultipleStandardLoadBalancerBackendNodes("kubernetes", tc.lbName, &tc.existingLBs, &svc, tc.existingNodes, tc.init)
 
 			expectedLBToNodesMap := make(map[string]*utilsets.IgnoreCaseSet)
 			for _, multiSLBConfig := range az.MultipleStandardLoadBalancerConfigurations {