Repository: kubernetes-sigs/cloud-provider-azure
Pull request #7564 (2024-11-13 02:09:35)

[release-1.30]migrate subnet to track2 sdk
<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide/first-contribution.md#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?
/kind feature

<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature
/kind design

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->

#### What this PR does / why we need it:
[release-1.30]migrate subnet to track2 sdk
#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes #

#### Special notes for your reviewer:

#### Does this PR introduce a user-facing change?
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://git.k8s.io/community/contributors/guide/release-notes.md
-->
```release-note
NONE
```

#### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

<!--
This section can be blank if this pull request does not require a release note.

When adding links which point to resources within git repositories, like
KEPs or supporting documentation, please reference a specific commit and avoid
linking directly to the master branch. This ensures that links reference a
specific point in time, rather than a document that may change over time.

See here for guidance on getting permanent links to files: https://help.github.com/en/articles/getting-permanent-links-to-files

Please use the following format for linking documentation:
- [KEP]: <link>
- [Usage]: <link>
- [Other doc]: <link>
-->
```docs

```


k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/7564#" title="Author self-approved">MartinForReal</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/release-1.30/OWNERS)~~ [MartinForReal]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/7564#pullrequestreview-2431599621" title="LGTM">andyzhangx</a>*, *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/7564#" title="Author self-approved">MartinForReal</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/release-1.30/OWNERS)~~ [MartinForReal,andyzhangx]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
k8s-ci-robot: @MartinForReal: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-capz-1-30 | 7e081c04c678d2746a999e1413b371f323fb2507 | [link](https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/7564/pull-cloud-provider-azure-e2e-ccm-capz-1-30/1856519760045412352) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz-1-30`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=7564). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3AMartinForReal). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @MartinForReal: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-e2e-ccm-capz-1-30 | 7e081c04c678d2746a999e1413b371f323fb2507 | [link](https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/7564/pull-cloud-provider-azure-e2e-ccm-capz-1-30/1856519760045412352) | true | `/test pull-cloud-provider-azure-e2e-ccm-capz-1-30`
pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-30 | 7e081c04c678d2746a999e1413b371f323fb2507 | [link](https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/7564/pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-30/1856519759932166144) | true | `/test pull-cloud-provider-azure-e2e-ccm-vmss-capz-1-30`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=7564). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3AMartinForReal). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
MartinForReal: /retest

Patch:
diff --git a/pkg/provider/azure.go b/pkg/provider/azure.go
index 7cb575398..21249eeea 100644
--- a/pkg/provider/azure.go
+++ b/pkg/provider/azure.go
@@ -71,6 +71,7 @@ import (
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/vmssclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/vmssvmclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/provider/privatelinkservice"
+	"sigs.k8s.io/cloud-provider-azure/pkg/provider/subnet"
 	"sigs.k8s.io/cloud-provider-azure/pkg/provider/zone"
 
 	"sigs.k8s.io/yaml"
@@ -418,12 +419,13 @@ type Cloud struct {
 	routeUpdater       batchProcessor
 	backendPoolUpdater batchProcessor
 
-	vmCache  azcache.Resource
-	lbCache  azcache.Resource
-	nsgRepo  securitygroup.Repository
-	zoneRepo zone.Repository
-	plsRepo  privatelinkservice.Repository
-	rtCache  azcache.Resource
+	vmCache    azcache.Resource
+	lbCache    azcache.Resource
+	nsgRepo    securitygroup.Repository
+	zoneRepo   zone.Repository
+	plsRepo    privatelinkservice.Repository
+	subnetRepo subnet.Repository
+	rtCache    azcache.Resource
 	// public ip cache
 	// key: [resourceGroupName]
 	// Value: sync.Map of [pipName]*PublicIPAddress
@@ -754,6 +756,10 @@ func (az *Cloud) InitializeCloudFromConfig(ctx context.Context, config *Config,
 		if err != nil {
 			return err
 		}
+		az.subnetRepo, err = subnet.NewRepo(networkClientFactory.GetSubnetClient())
+		if err != nil {
+			return err
+		}
 	}
 	err = az.initCaches()
 	if err != nil {
diff --git a/pkg/provider/azure_fakes.go b/pkg/provider/azure_fakes.go
index 07249d8d3..030128a85 100644
--- a/pkg/provider/azure_fakes.go
+++ b/pkg/provider/azure_fakes.go
@@ -30,6 +30,7 @@ import (
 	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/mock_azclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/privatezoneclient/mock_privatezoneclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/securitygroupclient/mock_securitygroupclient"
+	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/subnetclient/mock_subnetclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/virtualnetworklinkclient/mock_virtualnetworklinkclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/diskclient/mockdiskclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/interfaceclient/mockinterfaceclient"
@@ -45,6 +46,7 @@ import (
 	"sigs.k8s.io/cloud-provider-azure/pkg/provider/config"
 	"sigs.k8s.io/cloud-provider-azure/pkg/provider/privatelinkservice"
 	"sigs.k8s.io/cloud-provider-azure/pkg/provider/securitygroup"
+	"sigs.k8s.io/cloud-provider-azure/pkg/provider/subnet"
 	utilsets "sigs.k8s.io/cloud-provider-azure/pkg/util/sets"
 )
 
@@ -128,6 +130,8 @@ func GetTestCloud(ctrl *gomock.Controller) (az *Cloud) {
 	clientFactory.EXPECT().GetPrivateZoneClient().Return(mockPrivateDNSClient).AnyTimes()
 	virtualNetworkLinkClient := mock_virtualnetworklinkclient.NewMockInterface(ctrl)
 	clientFactory.EXPECT().GetVirtualNetworkLinkClient().Return(virtualNetworkLinkClient).AnyTimes()
+	subnetTrack2Client := mock_subnetclient.NewMockInterface(ctrl)
+	clientFactory.EXPECT().GetSubnetClient().Return(subnetTrack2Client).AnyTimes()
 	az.AuthProvider = &azclient.AuthProvider{
 		ComputeCredential: mock_azclient.NewMockTokenCredential(ctrl),
 	}
@@ -135,6 +139,7 @@ func GetTestCloud(ctrl *gomock.Controller) (az *Cloud) {
 	az.vmCache, _ = az.newVMCache()
 	az.lbCache, _ = az.newLBCache()
 	az.nsgRepo, _ = securitygroup.NewSecurityGroupRepo(az.SecurityGroupResourceGroup, az.SecurityGroupName, az.NsgCacheTTLInSeconds, az.Config.DisableAPICallCache, securtyGrouptrack2Client)
+	az.subnetRepo = subnet.NewMockRepository(ctrl)
 	az.rtCache, _ = az.newRouteTableCache()
 	az.pipCache, _ = az.newPIPCache()
 	az.LoadBalancerBackendPool = NewMockBackendPool(ctrl)
diff --git a/pkg/provider/azure_privatelinkservice.go b/pkg/provider/azure_privatelinkservice.go
index 0ec458832..684a2e474 100644
--- a/pkg/provider/azure_privatelinkservice.go
+++ b/pkg/provider/azure_privatelinkservice.go
@@ -18,10 +18,13 @@ package provider
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"net"
+	"net/http"
 	"strings"
 
+	"github.com/Azure/azure-sdk-for-go/sdk/azcore"
 	"github.com/Azure/azure-sdk-for-go/sdk/azcore/to"
 	"github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork/v6"
 	"github.com/Azure/azure-sdk-for-go/services/network/mgmt/2022-07-01/network"
@@ -144,14 +147,14 @@ func (az *Cloud) reconcilePrivateLinkService(
 			return err
 		}
 
-		dirtyPLS, err := az.getExpectedPrivateLinkService(existingPLS, &plsName, &clusterName, service, fipConfig)
+		dirtyPLS, err := az.getExpectedPrivateLinkService(ctx, existingPLS, &plsName, &clusterName, service, fipConfig)
 		if err != nil {
 			return err
 		}
 
 		if dirtyPLS {
 			klog.V(2).Infof("reconcilePrivateLinkService for service(%s): pls(%s) - updating", serviceName, plsName)
-			err := az.disablePLSNetworkPolicy(service)
+			err := az.disablePLSNetworkPolicy(ctx, service)
 			if err != nil {
 				klog.Errorf("reconcilePrivateLinkService for service(%s) disable PLS network policy failed for pls(%s): %v", serviceName, plsName, err.Error())
 				return err
@@ -196,28 +199,39 @@ func (az *Cloud) getPLSResourceGroup(service *v1.Service) string {
 	return az.PrivateLinkServiceResourceGroup
 }
 
-func (az *Cloud) disablePLSNetworkPolicy(service *v1.Service) error {
+func (az *Cloud) disablePLSNetworkPolicy(ctx context.Context, service *v1.Service) error {
 	serviceName := getServiceName(service)
 	subnetName := getPLSSubnetName(service)
 	if subnetName == nil {
 		subnetName = &az.SubnetName
 	}
 
-	subnet, existsSubnet, err := az.getSubnet("", az.VnetName, *subnetName)
+	rg := az.VnetResourceGroup
+	if rg == "" {
+		rg = az.ResourceGroup
+	}
+
+	subnet, err := az.subnetRepo.Get(ctx, rg, az.VnetName, *subnetName)
 	if err != nil {
+		var respErr *azcore.ResponseError
+		if errors.As(err, &respErr) {
+			if respErr != nil && respErr.StatusCode == http.StatusNotFound {
+				return fmt.Errorf("disablePLSNetworkPolicy: failed to get private link service subnet(%s) for service(%s)", *subnetName, serviceName)
+			}
+		}
 		return err
 	}
-	if !existsSubnet {
-		return fmt.Errorf("disablePLSNetworkPolicy: failed to get private link service subnet(%s) for service(%s)", *subnetName, serviceName)
+	if subnet.Properties == nil {
+		subnet.Properties = &armnetwork.SubnetPropertiesFormat{}
 	}
 
 	// Policy already disabled
-	if subnet.PrivateLinkServiceNetworkPolicies == network.VirtualNetworkPrivateLinkServiceNetworkPoliciesDisabled {
+	if subnet.Properties.PrivateLinkServiceNetworkPolicies != nil && *subnet.Properties.PrivateLinkServiceNetworkPolicies == armnetwork.VirtualNetworkPrivateLinkServiceNetworkPoliciesDisabled {
 		return nil
 	}
 
-	subnet.PrivateLinkServiceNetworkPolicies = network.VirtualNetworkPrivateLinkServiceNetworkPoliciesDisabled
-	err = az.CreateOrUpdateSubnet(service, subnet)
+	subnet.Properties.PrivateLinkServiceNetworkPolicies = to.Ptr(armnetwork.VirtualNetworkPrivateLinkServiceNetworkPoliciesDisabled)
+	err = az.subnetRepo.CreateOrUpdate(ctx, rg, az.VnetName, *subnetName, *subnet)
 	if err != nil {
 		return err
 	}
@@ -281,6 +295,7 @@ func (az *Cloud) getPrivateLinkServiceName(
 
 // getExpectedPrivateLinkService builds expected PLS object from service spec
 func (az *Cloud) getExpectedPrivateLinkService(
+	ctx context.Context,
 	existingPLS *armnetwork.PrivateLinkService,
 	plsName *string,
 	clusterName *string,
@@ -314,7 +329,7 @@ func (az *Cloud) getExpectedPrivateLinkService(
 		dirtyPLS = true
 	}
 
-	changed, err := az.reconcilePLSIpConfigs(existingPLS, service)
+	changed, err := az.reconcilePLSIpConfigs(ctx, existingPLS, service)
 	if err != nil {
 		return false, err
 	}
@@ -347,6 +362,7 @@ func (az *Cloud) getExpectedPrivateLinkService(
 
 // reconcile Private link service's IP configurations
 func (az *Cloud) reconcilePLSIpConfigs(
+	ctx context.Context,
 	existingPLS *armnetwork.PrivateLinkService,
 	service *v1.Service,
 ) (bool, error) {
@@ -357,13 +373,20 @@ func (az *Cloud) reconcilePLSIpConfigs(
 	if subnetName == nil {
 		subnetName = &az.SubnetName
 	}
-	subnet, existsSubnet, err := az.getSubnet("", az.VnetName, *subnetName)
+	rg := az.VnetResourceGroup
+	if rg == "" {
+		rg = az.ResourceGroup
+	}
+	subnet, err := az.subnetRepo.Get(ctx, rg, az.VnetName, *subnetName)
 	if err != nil {
+		var runtimError *azcore.ResponseError
+		if errors.As(err, &runtimError) {
+			if runtimError != nil && runtimError.StatusCode == http.StatusNotFound {
+				return false, fmt.Errorf("checkAndUpdatePLSIPConfigs: failed to get private link service subnet(%s) for service(%s)", *subnetName, serviceName)
+			}
+		}
 		return false, err
 	}
-	if !existsSubnet {
-		return false, fmt.Errorf("checkAndUpdatePLSIPConfigs: failed to get private link service subnet(%s) for service(%s)", *subnetName, serviceName)
-	}
 
 	ipConfigCount, err := getPLSIPConfigCount(service)
 	if err != nil {
diff --git a/pkg/provider/subnet/mock.go b/pkg/provider/subnet/mock.go
new file mode 100644
index 000000000..cd2fd4665
--- /dev/null
+++ b/pkg/provider/subnet/mock.go
@@ -0,0 +1,88 @@
+// /*
+// Copyright The Kubernetes Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+// */
+//
+
+// Code generated by MockGen. DO NOT EDIT.
+// Source: subnet.go
+//
+// Generated by this command:
+//
+//	mockgen -package subnet -source subnet.go -copyright_file ../../../hack/boilerplate/boilerplate.generatego.txt Repository
+//
+
+// Package subnet is a generated GoMock package.
+package subnet
+
+import (
+	context "context"
+	reflect "reflect"
+
+	armnetwork "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork/v6"
+	gomock "go.uber.org/mock/gomock"
+)
+
+// MockRepository is a mock of Repository interface.
+type MockRepository struct {
+	ctrl     *gomock.Controller
+	recorder *MockRepositoryMockRecorder
+	isgomock struct{}
+}
+
+// MockRepositoryMockRecorder is the mock recorder for MockRepository.
+type MockRepositoryMockRecorder struct {
+	mock *MockRepository
+}
+
+// NewMockRepository creates a new mock instance.
+func NewMockRepository(ctrl *gomock.Controller) *MockRepository {
+	mock := &MockRepository{ctrl: ctrl}
+	mock.recorder = &MockRepositoryMockRecorder{mock}
+	return mock
+}
+
+// EXPECT returns an object that allows the caller to indicate expected use.
+func (m *MockRepository) EXPECT() *MockRepositoryMockRecorder {
+	return m.recorder
+}
+
+// CreateOrUpdate mocks base method.
+func (m *MockRepository) CreateOrUpdate(ctx context.Context, rg, vnetName, subnetName string, subnet armnetwork.Subnet) error {
+	m.ctrl.T.Helper()
+	ret := m.ctrl.Call(m, "CreateOrUpdate", ctx, rg, vnetName, subnetName, subnet)
+	ret0, _ := ret[0].(error)
+	return ret0
+}
+
+// CreateOrUpdate indicates an expected call of CreateOrUpdate.
+func (mr *MockRepositoryMockRecorder) CreateOrUpdate(ctx, rg, vnetName, subnetName, subnet any) *gomock.Call {
+	mr.mock.ctrl.T.Helper()
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CreateOrUpdate", reflect.TypeOf((*MockRepository)(nil).CreateOrUpdate), ctx, rg, vnetName, subnetName, subnet)
+}
+
+// Get mocks base method.
+func (m *MockRepository) Get(ctx context.Context, rg, vnetName, subnetName string) (*armnetwork.Subnet, error) {
+	m.ctrl.T.Helper()
+	ret := m.ctrl.Call(m, "Get", ctx, rg, vnetName, subnetName)
+	ret0, _ := ret[0].(*armnetwork.Subnet)
+	ret1, _ := ret[1].(error)
+	return ret0, ret1
+}
+
+// Get indicates an expected call of Get.
+func (mr *MockRepositoryMockRecorder) Get(ctx, rg, vnetName, subnetName any) *gomock.Call {
+	mr.mock.ctrl.T.Helper()
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Get", reflect.TypeOf((*MockRepository)(nil).Get), ctx, rg, vnetName, subnetName)
+}
diff --git a/pkg/provider/subnet/subnet.go b/pkg/provider/subnet/subnet.go
new file mode 100644
index 000000000..8cbf903ec
--- /dev/null
+++ b/pkg/provider/subnet/subnet.go
@@ -0,0 +1,64 @@
+/*
+Copyright 2024 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package subnet
+
+import (
+	"context"
+
+	"github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork/v6"
+	"k8s.io/klog/v2"
+
+	"sigs.k8s.io/cloud-provider-azure/pkg/azclient/subnetclient"
+)
+
+type Repository interface {
+	CreateOrUpdate(ctx context.Context, rg string, vnetName string, subnetName string, subnet armnetwork.Subnet) error
+	Get(ctx context.Context, rg string, vnetName string, subnetName string) (*armnetwork.Subnet, error)
+}
+
+type repo struct {
+	SubnetsClient subnetclient.Interface
+}
+
+func NewRepo(subnetsClient subnetclient.Interface) (Repository, error) {
+	return &repo{
+		SubnetsClient: subnetsClient,
+	}, nil
+}
+
+// CreateOrUpdateSubnet invokes az.SubnetClient.CreateOrUpdate with exponential backoff retry
+func (az *repo) CreateOrUpdate(ctx context.Context, rg string, vnetName string, subnetName string, subnet armnetwork.Subnet) error {
+
+	_, rerr := az.SubnetsClient.CreateOrUpdate(ctx, rg, vnetName, subnetName, subnet)
+	klog.V(10).Infof("SubnetsClient.CreateOrUpdate(%s): end", subnetName)
+	if rerr != nil {
+		klog.Errorf("SubnetClient.CreateOrUpdate(%s) failed: %s", subnetName, rerr.Error())
+		return rerr
+	}
+
+	return nil
+}
+
+func (az *repo) Get(ctx context.Context, rg string, vnetName string, subnetName string) (*armnetwork.Subnet, error) {
+
+	subnet, err := az.SubnetsClient.Get(ctx, rg, vnetName, subnetName, nil)
+	if err != nil {
+		klog.Errorf("SubnetClient.Get(%s) failed: %s", subnetName, err.Error())
+		return nil, err
+	}
+	return subnet, nil
+}
diff --git a/vendor/modules.txt b/vendor/modules.txt
index abec322b4..96d158c25 100644
--- a/vendor/modules.txt
+++ b/vendor/modules.txt
@@ -1549,6 +1549,7 @@ sigs.k8s.io/cloud-provider-azure/pkg/azclient/securitygroupclient/mock_securityg
 sigs.k8s.io/cloud-provider-azure/pkg/azclient/snapshotclient
 sigs.k8s.io/cloud-provider-azure/pkg/azclient/sshpublickeyresourceclient
 sigs.k8s.io/cloud-provider-azure/pkg/azclient/subnetclient
+sigs.k8s.io/cloud-provider-azure/pkg/azclient/subnetclient/mock_subnetclient
 sigs.k8s.io/cloud-provider-azure/pkg/azclient/utils
 sigs.k8s.io/cloud-provider-azure/pkg/azclient/utils/armbalancer
 sigs.k8s.io/cloud-provider-azure/pkg/azclient/vaultclient
diff --git a/vendor/sigs.k8s.io/cloud-provider-azure/pkg/azclient/subnetclient/mock_subnetclient/interface.go b/vendor/sigs.k8s.io/cloud-provider-azure/pkg/azclient/subnetclient/mock_subnetclient/interface.go
new file mode 100644
index 000000000..382b6b4f8
--- /dev/null
+++ b/vendor/sigs.k8s.io/cloud-provider-azure/pkg/azclient/subnetclient/mock_subnetclient/interface.go
@@ -0,0 +1,117 @@
+// /*
+// Copyright The Kubernetes Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+// */
+
+// Code generated by MockGen. DO NOT EDIT.
+// Source: subnetclient/interface.go
+//
+// Generated by this command:
+//
+//	mockgen -package mock_subnetclient -source subnetclient/interface.go
+//
+
+// Package mock_subnetclient is a generated GoMock package.
+package mock_subnetclient
+
+import (
+	context "context"
+	reflect "reflect"
+
+	armnetwork "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork/v6"
+	gomock "go.uber.org/mock/gomock"
+)
+
+// MockInterface is a mock of Interface interface.
+type MockInterface struct {
+	ctrl     *gomock.Controller
+	recorder *MockInterfaceMockRecorder
+	isgomock struct{}
+}
+
+// MockInterfaceMockRecorder is the mock recorder for MockInterface.
+type MockInterfaceMockRecorder struct {
+	mock *MockInterface
+}
+
+// NewMockInterface creates a new mock instance.
+func NewMockInterface(ctrl *gomock.Controller) *MockInterface {
+	mock := &MockInterface{ctrl: ctrl}
+	mock.recorder = &MockInterfaceMockRecorder{mock}
+	return mock
+}
+
+// EXPECT returns an object that allows the caller to indicate expected use.
+func (m *MockInterface) EXPECT() *MockInterfaceMockRecorder {
+	return m.recorder
+}
+
+// CreateOrUpdate mocks base method.
+func (m *MockInterface) CreateOrUpdate(ctx context.Context, resourceGroupName, parentResourceName, resourceName string, resourceParam armnetwork.Subnet) (*armnetwork.Subnet, error) {
+	m.ctrl.T.Helper()
+	ret := m.ctrl.Call(m, "CreateOrUpdate", ctx, resourceGroupName, parentResourceName, resourceName, resourceParam)
+	ret0, _ := ret[0].(*armnetwork.Subnet)
+	ret1, _ := ret[1].(error)
+	return ret0, ret1
+}
+
+// CreateOrUpdate indicates an expected call of CreateOrUpdate.
+func (mr *MockInterfaceMockRecorder) CreateOrUpdate(ctx, resourceGroupName, parentResourceName, resourceName, resourceParam any) *gomock.Call {
+	mr.mock.ctrl.T.Helper()
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CreateOrUpdate", reflect.TypeOf((*MockInterface)(nil).CreateOrUpdate), ctx, resourceGroupName, parentResourceName, resourceName, resourceParam)
+}
+
+// Delete mocks base method.
+func (m *MockInterface) Delete(ctx context.Context, resourceGroupName, parentResourceName, resourceName string) error {
+	m.ctrl.T.Helper()
+	ret := m.ctrl.Call(m, "Delete", ctx, resourceGroupName, parentResourceName, resourceName)
+	ret0, _ := ret[0].(error)
+	return ret0
+}
+
+// Delete indicates an expected call of Delete.
+func (mr *MockInterfaceMockRecorder) Delete(ctx, resourceGroupName, parentResourceName, resourceName any) *gomock.Call {
+	mr.mock.ctrl.T.Helper()
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Delete", reflect.TypeOf((*MockInterface)(nil).Delete), ctx, resourceGroupName, parentResourceName, resourceName)
+}
+
+// Get mocks base method.
+func (m *MockInterface) Get(ctx context.Context, resourceGroupName, parentResourceName, resourceName string, expand *string) (*armnetwork.Subnet, error) {
+	m.ctrl.T.Helper()
+	ret := m.ctrl.Call(m, "Get", ctx, resourceGroupName, parentResourceName, resourceName, expand)
+	ret0, _ := ret[0].(*armnetwork.Subnet)
+	ret1, _ := ret[1].(error)
+	return ret0, ret1
+}
+
+// Get indicates an expected call of Get.
+func (mr *MockInterfaceMockRecorder) Get(ctx, resourceGroupName, parentResourceName, resourceName, expand any) *gomock.Call {
+	mr.mock.ctrl.T.Helper()
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Get", reflect.TypeOf((*MockInterface)(nil).Get), ctx, resourceGroupName, parentResourceName, resourceName, expand)
+}
+
+// List mocks base method.
+func (m *MockInterface) List(ctx context.Context, resourceGroupName, parentResourceName string) ([]*armnetwork.Subnet, error) {
+	m.ctrl.T.Helper()
+	ret := m.ctrl.Call(m, "List", ctx, resourceGroupName, parentResourceName)
+	ret0, _ := ret[0].([]*armnetwork.Subnet)
+	ret1, _ := ret[1].(error)
+	return ret0, ret1
+}
+
+// List indicates an expected call of List.
+func (mr *MockInterfaceMockRecorder) List(ctx, resourceGroupName, parentResourceName any) *gomock.Call {
+	mr.mock.ctrl.T.Helper()
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "List", reflect.TypeOf((*MockInterface)(nil).List), ctx, resourceGroupName, parentResourceName)
+}


Test patch:
diff --git a/pkg/provider/azure_privatelinkservice_test.go b/pkg/provider/azure_privatelinkservice_test.go
index f4a57dbea..cb8d0a6cb 100644
--- a/pkg/provider/azure_privatelinkservice_test.go
+++ b/pkg/provider/azure_privatelinkservice_test.go
@@ -22,6 +22,7 @@ import (
 	"strings"
 	"testing"
 
+	"github.com/Azure/azure-sdk-for-go/sdk/azcore"
 	"github.com/Azure/azure-sdk-for-go/sdk/azcore/to"
 	"github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork/v6"
 	"github.com/Azure/azure-sdk-for-go/services/network/mgmt/2022-07-01/network"
@@ -30,12 +31,12 @@ import (
 	v1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/utils/pointer"
+	"k8s.io/utils/ptr"
 
-	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/subnetclient/mocksubnetclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/cache"
 	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
 	"sigs.k8s.io/cloud-provider-azure/pkg/provider/privatelinkservice"
-	"sigs.k8s.io/cloud-provider-azure/pkg/retry"
+	"sigs.k8s.io/cloud-provider-azure/pkg/provider/subnet"
 )
 
 func TestReconcilePrivateLinkService(t *testing.T) {
@@ -47,7 +48,7 @@ func TestReconcilePrivateLinkService(t *testing.T) {
 		annotations       map[string]string
 		wantPLS           bool
 		expectedSubnetGet bool
-		existingSubnet    *network.Subnet
+		existingSubnet    *armnetwork.Subnet
 		expectedPLSList   bool
 		existingPLSList   []*armnetwork.PrivateLinkService
 		expectedPLSCreate bool
@@ -75,11 +76,11 @@ func TestReconcilePrivateLinkService(t *testing.T) {
 			},
 			wantPLS:           true,
 			expectedSubnetGet: true,
-			existingSubnet: &network.Subnet{
-				Name: pointer.String("subnet"),
-				ID:   pointer.String("subnetID"),
-				SubnetPropertiesFormat: &network.SubnetPropertiesFormat{
-					PrivateLinkServiceNetworkPolicies: network.VirtualNetworkPrivateLinkServiceNetworkPoliciesDisabled,
+			existingSubnet: &armnetwork.Subnet{
+				Name: ptr.To("subnet"),
+				ID:   ptr.To("subnetID"),
+				Properties: &armnetwork.SubnetPropertiesFormat{
+					PrivateLinkServiceNetworkPolicies: to.Ptr(armnetwork.VirtualNetworkPrivateLinkServiceNetworkPoliciesDisabled),
 				},
 			},
 			expectedPLSList:   true,
@@ -96,11 +97,11 @@ func TestReconcilePrivateLinkService(t *testing.T) {
 			},
 			wantPLS:           true,
 			expectedSubnetGet: true,
-			existingSubnet: &network.Subnet{
-				Name: pointer.String("subnet"),
-				ID:   pointer.String("subnetID"),
-				SubnetPropertiesFormat: &network.SubnetPropertiesFormat{
-					PrivateLinkServiceNetworkPolicies: network.VirtualNetworkPrivateLinkServiceNetworkPoliciesDisabled,
+			existingSubnet: &armnetwork.Subnet{
+				Name: ptr.To("subnet"),
+				ID:   ptr.To("subnetID"),
+				Properties: &armnetwork.SubnetPropertiesFormat{
+					PrivateLinkServiceNetworkPolicies: to.Ptr(armnetwork.VirtualNetworkPrivateLinkServiceNetworkPoliciesDisabled),
 				},
 			},
 			expectedPLSList:   true,
@@ -211,11 +212,11 @@ func TestReconcilePrivateLinkService(t *testing.T) {
 			},
 			wantPLS:           true,
 			expectedSubnetGet: true,
-			existingSubnet: &network.Subnet{
-				Name: pointer.String("subnet"),
-				ID:   pointer.String("subnetID"),
-				SubnetPropertiesFormat: &network.SubnetPropertiesFormat{
-					PrivateLinkServiceNetworkPolicies: network.VirtualNetworkPrivateLinkServiceNetworkPoliciesDisabled,
+			existingSubnet: &armnetwork.Subnet{
+				Name: ptr.To("subnet"),
+				ID:   ptr.To("subnetID"),
+				Properties: &armnetwork.SubnetPropertiesFormat{
+					PrivateLinkServiceNetworkPolicies: to.Ptr(armnetwork.VirtualNetworkPrivateLinkServiceNetworkPoliciesDisabled),
 				},
 			},
 			expectedPLSList: true,
@@ -252,11 +253,11 @@ func TestReconcilePrivateLinkService(t *testing.T) {
 			},
 			wantPLS:           true,
 			expectedSubnetGet: true,
-			existingSubnet: &network.Subnet{
-				Name: pointer.String("subnet"),
-				ID:   pointer.String("subnetID"),
-				SubnetPropertiesFormat: &network.SubnetPropertiesFormat{
-					PrivateLinkServiceNetworkPolicies: network.VirtualNetworkPrivateLinkServiceNetworkPoliciesDisabled,
+			existingSubnet: &armnetwork.Subnet{
+				Name: ptr.To("subnet"),
+				ID:   ptr.To("subnetID"),
+				Properties: &armnetwork.SubnetPropertiesFormat{
+					PrivateLinkServiceNetworkPolicies: to.Ptr(armnetwork.VirtualNetworkPrivateLinkServiceNetworkPoliciesDisabled),
 				},
 			},
 			expectedPLSList: true,
@@ -322,6 +323,7 @@ func TestReconcilePrivateLinkService(t *testing.T) {
 		},
 	}
 	for _, test := range testCases {
+		test := test
 		t.Run(test.desc, func(t *testing.T) {
 			az := GetTestCloud(ctrl)
 			service := getTestServiceWithAnnotation("test", test.annotations, false, 80)
@@ -340,10 +342,10 @@ func TestReconcilePrivateLinkService(t *testing.T) {
 			}
 			clusterName := testClusterName
 
-			mockSubnetsClient := az.SubnetsClient.(*mocksubnetclient.MockInterface)
+			mockSubnetsClient := az.subnetRepo.(*subnet.MockRepository)
 			mockPLSRepo := az.plsRepo.(*privatelinkservice.MockRepository)
 			if test.expectedSubnetGet {
-				mockSubnetsClient.EXPECT().Get(gomock.Any(), "rg", "vnet", "subnet", "").Return(*test.existingSubnet, nil).MaxTimes(2)
+				mockSubnetsClient.EXPECT().Get(gomock.Any(), "rg", "vnet", "subnet").Return(test.existingSubnet, nil).MaxTimes(2)
 			}
 			if test.expectedPLSList {
 				mockPLSRepo.EXPECT().Get(gomock.Any(), "rg", *fipConfig.ID, gomock.Any()).DoAndReturn(func(_ context.Context, _, frontendIPConfigID string, _ cache.AzureCacheReadType) (*armnetwork.PrivateLinkService, error) {
@@ -403,26 +405,26 @@ func TestDisablePLSNetworkPolicy(t *testing.T) {
 
 	testCases := []struct {
 		desc                 string
-		subnet               network.Subnet
+		subnet               *armnetwork.Subnet
 		expectedSubnetUpdate bool
 		expectedError        bool
 	}{
 		{
 			desc: "disablePLSNetworkPolicy shall not update subnet if pls-network-policy is disabled",
-			subnet: network.Subnet{
-				Name: pointer.String("plsSubnet"),
-				SubnetPropertiesFormat: &network.SubnetPropertiesFormat{
-					PrivateLinkServiceNetworkPolicies: network.VirtualNetworkPrivateLinkServiceNetworkPoliciesDisabled,
+			subnet: &armnetwork.Subnet{
+				Name: ptr.To("plsSubnet"),
+				Properties: &armnetwork.SubnetPropertiesFormat{
+					PrivateLinkServiceNetworkPolicies: to.Ptr(armnetwork.VirtualNetworkPrivateLinkServiceNetworkPoliciesDisabled),
 				},
 			},
 			expectedSubnetUpdate: false,
 		},
 		{
 			desc: "disablePLSNetworkPolicy shall update subnet if pls-network-policy is enabled",
-			subnet: network.Subnet{
-				Name: pointer.String("plsSubnet"),
-				SubnetPropertiesFormat: &network.SubnetPropertiesFormat{
-					PrivateLinkServiceNetworkPolicies: network.VirtualNetworkPrivateLinkServiceNetworkPoliciesEnabled,
+			subnet: &armnetwork.Subnet{
+				Name: ptr.To("plsSubnet"),
+				Properties: &armnetwork.SubnetPropertiesFormat{
+					PrivateLinkServiceNetworkPolicies: to.Ptr(armnetwork.VirtualNetworkPrivateLinkServiceNetworkPoliciesEnabled),
 				},
 			},
 			expectedSubnetUpdate: true,
@@ -430,22 +432,23 @@ func TestDisablePLSNetworkPolicy(t *testing.T) {
 	}
 
 	for i, test := range testCases {
+		test := test
 		az := GetTestCloud(ctrl)
 		service := &v1.Service{}
 		service.Annotations = map[string]string{
 			consts.ServiceAnnotationPLSIpConfigurationSubnet: "plsSubnet",
 		}
-		mockSubnetsClient := az.SubnetsClient.(*mocksubnetclient.MockInterface)
-		mockSubnetsClient.EXPECT().Get(gomock.Any(), "rg", "vnet", "plsSubnet", "").Return(test.subnet, nil).Times(1)
+		mockSubnetsClient := az.subnetRepo.(*subnet.MockRepository)
+		mockSubnetsClient.EXPECT().Get(gomock.Any(), "rg", "vnet", "plsSubnet").Return(test.subnet, nil).Times(1)
 		if test.expectedSubnetUpdate {
-			mockSubnetsClient.EXPECT().CreateOrUpdate(gomock.Any(), "rg", "vnet", "plsSubnet", network.Subnet{
-				Name: pointer.String("plsSubnet"),
-				SubnetPropertiesFormat: &network.SubnetPropertiesFormat{
-					PrivateLinkServiceNetworkPolicies: network.VirtualNetworkPrivateLinkServiceNetworkPoliciesDisabled,
+			mockSubnetsClient.EXPECT().CreateOrUpdate(gomock.Any(), "rg", "vnet", "plsSubnet", armnetwork.Subnet{
+				Name: ptr.To("plsSubnet"),
+				Properties: &armnetwork.SubnetPropertiesFormat{
+					PrivateLinkServiceNetworkPolicies: to.Ptr(armnetwork.VirtualNetworkPrivateLinkServiceNetworkPoliciesDisabled),
 				},
 			}).Return(nil).Times(1)
 		}
-		err := az.disablePLSNetworkPolicy(service)
+		err := az.disablePLSNetworkPolicy(context.TODO(), service)
 		assert.Equal(t, test.expectedError, err != nil, "TestCase[%d]: %s", i, test.desc)
 	}
 }
@@ -574,14 +577,14 @@ func TestGetExpectedPrivateLinkService(t *testing.T) {
 		clusterName := testClusterName
 		fipConfig := &network.FrontendIPConfiguration{ID: pointer.String("fipConfigID")}
 		pls := &armnetwork.PrivateLinkService{Properties: &armnetwork.PrivateLinkServiceProperties{}}
-		subnetClient := cloud.SubnetsClient.(*mocksubnetclient.MockInterface)
-		subnetClient.EXPECT().Get(gomock.Any(), "rg", "vnet", "subnet", gomock.Any()).Return(
-			network.Subnet{
-				ID:   pointer.String("subnetID"),
-				Name: pointer.String("subnet"),
+		subnetClient := cloud.subnetRepo.(*subnet.MockRepository)
+		subnetClient.EXPECT().Get(gomock.Any(), "rg", "vnet", "subnet").Return(
+			&armnetwork.Subnet{
+				ID:   ptr.To("subnetID"),
+				Name: ptr.To("subnet"),
 			}, nil).MaxTimes(1)
 
-		dirtyPLS, err := cloud.getExpectedPrivateLinkService(pls, &plsName, &clusterName, service, fipConfig)
+		dirtyPLS, err := cloud.getExpectedPrivateLinkService(context.TODO(), pls, &plsName, &clusterName, service, fipConfig)
 		assert.NoError(t, err)
 		assert.True(t, dirtyPLS)
 
@@ -647,7 +650,7 @@ func TestReconcilePLSIpConfigs(t *testing.T) {
 		plsName           string
 		existingIPConfigs []*armnetwork.PrivateLinkServiceIPConfiguration
 		expectedIPConfigs []*armnetwork.PrivateLinkServiceIPConfiguration
-		getSubnetError    *retry.Error
+		getSubnetError    error
 		expectedChanged   bool
 		expectedErr       bool
 	}{
@@ -657,7 +660,7 @@ func TestReconcilePLSIpConfigs(t *testing.T) {
 			annotations: map[string]string{
 				consts.ServiceAnnotationPLSIpConfigurationSubnet: "subnet",
 			},
-			getSubnetError: &retry.Error{HTTPStatusCode: http.StatusNotFound},
+			getSubnetError: &azcore.ResponseError{StatusCode: http.StatusNotFound},
 			expectedErr:    true,
 		},
 		{
@@ -1006,14 +1009,14 @@ func TestReconcilePLSIpConfigs(t *testing.T) {
 				IPConfigurations: test.existingIPConfigs,
 			},
 		}
-		subnetClient := cloud.SubnetsClient.(*mocksubnetclient.MockInterface)
-		subnetClient.EXPECT().Get(gomock.Any(), "rg", "vnet", "subnet", gomock.Any()).Return(
-			network.Subnet{
-				ID:   pointer.String("subnetID"),
-				Name: pointer.String("subnet"),
+		subnetClient := cloud.subnetRepo.(*subnet.MockRepository)
+		subnetClient.EXPECT().Get(gomock.Any(), "rg", "vnet", "subnet").Return(
+			&armnetwork.Subnet{
+				ID:   ptr.To("subnetID"),
+				Name: ptr.To("subnet"),
 			}, test.getSubnetError).MaxTimes(1)
 
-		changed, err := cloud.reconcilePLSIpConfigs(pls, service)
+		changed, err := cloud.reconcilePLSIpConfigs(context.TODO(), pls, service)
 		if test.expectedErr {
 			assert.Error(t, err, "TestCase[%d]: %s", i, test.desc)
 		} else {
diff --git a/pkg/provider/azure_subnet_repo_test.go b/pkg/provider/azure_subnet_repo_test.go
deleted file mode 100644
index 10c1c6cf7..000000000
--- a/pkg/provider/azure_subnet_repo_test.go
+++ /dev/null
@@ -1,17 +0,0 @@
-/*
-Copyright 2023 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package provider