Repository: kubernetes-sigs/cloud-provider-azure
Pull request #2699 (2022-11-04 02:37:44)

[release-1.24] Deepcopy results of cache Get()

Signed-off-by: Zhecheng Li <zhechengli@microsoft.com>

<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide/first-contribution.md#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?
/kind bug
<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature
/kind design

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->

#### What this PR does / why we need it:
[release-1.24] Deepcopy results of cache Get()
Considering thread safety, results of cache Get() should be deepcopied before used.
* Support deepcopy and add UT for all types
* Add TestGet UT
* Update current UT
* Make Entry structs exported to adjust for deepcopy
* Delete cacheKey for VMSS VM and VMSS deletion

#2445 
#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes #

#### Special notes for your reviewer:

#### Does this PR introduce a user-facing change?
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://git.k8s.io/community/contributors/guide/release-notes.md
-->
```release-note
Considering thread safety, results of cache Get() should be deepcopied before used.
```

#### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

<!--
This section can be blank if this pull request does not require a release note.

When adding links which point to resources within git repositories, like
KEPs or supporting documentation, please reference a specific commit and avoid
linking directly to the master branch. This ensures that links reference a
specific point in time, rather than a document that may change over time.

See here for guidance on getting permanent links to files: https://help.github.com/en/articles/getting-permanent-links-to-files

Please use the following format for linking documentation:
- [KEP]: <link>
- [Usage]: <link>
- [Other doc]: <link>
-->
```docs

```


k8s-ci-robot: @lzhecheng: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our [release note process](https://git.k8s.io/community/contributors/guide/release-notes.md) to remove it.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/2699#" title="Author self-approved">lzhecheng</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/release-1.24/OWNERS)~~ [lzhecheng]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
k8s-ci-robot: @lzhecheng: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-unit-1-24 | aad1c4d17a8252f225451ccdbce9a4f8458b14c7 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2699/pull-cloud-provider-azure-unit-1-24/1588359820732272640) | true | `/test pull-cloud-provider-azure-unit-1-24`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=2699). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alzhecheng). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lzhecheng: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-unit-1-24 | aad1c4d17a8252f225451ccdbce9a4f8458b14c7 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2699/pull-cloud-provider-azure-unit-1-24/1588359820732272640) | true | `/test pull-cloud-provider-azure-unit-1-24`
pull-cloud-provider-azure-check-1-24 | aad1c4d17a8252f225451ccdbce9a4f8458b14c7 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2699/pull-cloud-provider-azure-check-1-24/1588359820564500480) | true | `/test pull-cloud-provider-azure-check-1-24`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=2699). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alzhecheng). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @lzhecheng: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-check-1-24 | aad1c4d17a8252f225451ccdbce9a4f8458b14c7 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2699/pull-cloud-provider-azure-check-1-24/1588359820564500480) | true | `/test pull-cloud-provider-azure-check-1-24`
pull-cloud-provider-azure-unit-1-24 | 89c9b245f7d3fba0b7b43083efd01b56a986372e | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2699/pull-cloud-provider-azure-unit-1-24/1588361695015735296) | true | `/test pull-cloud-provider-azure-unit-1-24`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=2699). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alzhecheng). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->

Patch:
diff --git a/pkg/cache/azure_cache.go b/pkg/cache/azure_cache.go
index dd936db48..d88041de6 100644
--- a/pkg/cache/azure_cache.go
+++ b/pkg/cache/azure_cache.go
@@ -22,6 +22,8 @@ import (
 	"time"
 
 	"k8s.io/client-go/tools/cache"
+
+	"sigs.k8s.io/cloud-provider-azure/pkg/util/deepcopy"
 )
 
 // AzureCacheReadType defines the read type for cache data
@@ -120,8 +122,14 @@ func (t *TimedCache) getInternal(key string) (*AzureCacheEntry, error) {
 	return newEntry, nil
 }
 
-// Get returns the requested item by key.
+// Get returns the requested item by key with deep copy.
 func (t *TimedCache) Get(key string, crt AzureCacheReadType) (interface{}, error) {
+	data, err := t.get(key, crt)
+	copied := deepcopy.Copy(data)
+	return copied, err
+}
+
+func (t *TimedCache) get(key string, crt AzureCacheReadType) (interface{}, error) {
 	entry, err := t.getInternal(key)
 	if err != nil {
 		return nil, err
diff --git a/pkg/provider/azure_standard.go b/pkg/provider/azure_standard.go
index 2eca9b673..1b17403a2 100644
--- a/pkg/provider/azure_standard.go
+++ b/pkg/provider/azure_standard.go
@@ -449,9 +449,9 @@ type availabilitySet struct {
 	vmasCache *azcache.TimedCache
 }
 
-type availabilitySetEntry struct {
-	vmas          *compute.AvailabilitySet
-	resourceGroup string
+type AvailabilitySetEntry struct {
+	VMAS          *compute.AvailabilitySet
+	ResourceGroup string
 }
 
 func (as *availabilitySet) newVMASCache() (*azcache.TimedCache, error) {
@@ -476,9 +476,9 @@ func (as *availabilitySet) newVMASCache() (*azcache.TimedCache, error) {
 					klog.Warning("failed to get the name of the VMAS")
 					continue
 				}
-				localCache.Store(to.String(vmas.Name), &availabilitySetEntry{
-					vmas:          &vmas,
-					resourceGroup: resourceGroup,
+				localCache.Store(to.String(vmas.Name), &AvailabilitySetEntry{
+					VMAS:          &vmas,
+					ResourceGroup: resourceGroup,
 				})
 			}
 		}
@@ -1249,8 +1249,8 @@ func (as *availabilitySet) getAvailabilitySetByNodeName(nodeName string, crt azc
 
 	var result *compute.AvailabilitySet
 	vmasList.Range(func(_, value interface{}) bool {
-		vmasEntry := value.(*availabilitySetEntry)
-		vmas := vmasEntry.vmas
+		vmasEntry := value.(*AvailabilitySetEntry)
+		vmas := vmasEntry.VMAS
 		if vmas != nil && vmas.AvailabilitySetProperties != nil && vmas.VirtualMachines != nil {
 			for _, vmIDRef := range *vmas.VirtualMachines {
 				if vmIDRef.ID != nil {
diff --git a/pkg/provider/azure_vmss.go b/pkg/provider/azure_vmss.go
index 85376b7fc..0f4d01ca2 100644
--- a/pkg/provider/azure_vmss.go
+++ b/pkg/provider/azure_vmss.go
@@ -123,8 +123,8 @@ func (ss *ScaleSet) getVMSS(vmssName string, crt azcache.AzureCacheReadType) (*c
 
 		vmsses := cached.(*sync.Map)
 		if vmss, ok := vmsses.Load(vmssName); ok {
-			result := vmss.(*vmssEntry)
-			return result.vmss, nil
+			result := vmss.(*VMSSEntry)
+			return result.VMSS, nil
 		}
 
 		return nil, nil
@@ -168,13 +168,13 @@ func (ss *ScaleSet) getVmssVMByNodeIdentity(node *nodeIdentity, crt azcache.Azur
 
 		virtualMachines := cached.(*sync.Map)
 		if entry, ok := virtualMachines.Load(nodeName); ok {
-			result := entry.(*vmssVirtualMachinesEntry)
-			if result.virtualMachine == nil {
+			result := entry.(*VMSSVirtualMachinesEntry)
+			if result.VirtualMachine == nil {
 				klog.Warningf("failed to get VM with vmssVirtualMachinesEntry on Node %q", nodeName)
 				return nil, false, nil
 			}
 			found = true
-			return virtualmachine.FromVirtualMachineScaleSetVM(result.virtualMachine, virtualmachine.ByVMSS(result.vmssName)), found, nil
+			return virtualmachine.FromVirtualMachineScaleSetVM(result.VirtualMachine, virtualmachine.ByVMSS(result.VMSSName)), found, nil
 		}
 
 		return nil, found, nil
@@ -305,11 +305,11 @@ func (ss *ScaleSet) getVmssVMByInstanceID(resourceGroup, scaleSetName, instanceI
 
 		virtualMachines := cached.(*sync.Map)
 		virtualMachines.Range(func(key, value interface{}) bool {
-			vmEntry := value.(*vmssVirtualMachinesEntry)
-			if strings.EqualFold(vmEntry.resourceGroup, resourceGroup) &&
-				strings.EqualFold(vmEntry.vmssName, scaleSetName) &&
-				strings.EqualFold(vmEntry.instanceID, instanceID) {
-				vm = vmEntry.virtualMachine
+			vmEntry := value.(*VMSSVirtualMachinesEntry)
+			if strings.EqualFold(vmEntry.ResourceGroup, resourceGroup) &&
+				strings.EqualFold(vmEntry.VMSSName, scaleSetName) &&
+				strings.EqualFold(vmEntry.InstanceID, instanceID) {
+				vm = vmEntry.VirtualMachine
 				found = true
 				return false
 			}
@@ -698,21 +698,21 @@ func (ss *ScaleSet) getNodeIdentityByNodeName(nodeName string, crt azcache.Azure
 
 		vmsses := cached.(*sync.Map)
 		vmsses.Range(func(key, value interface{}) bool {
-			v := value.(*vmssEntry)
-			if v.vmss.Name == nil {
+			v := value.(*VMSSEntry)
+			if v.VMSS.Name == nil {
 				return true
 			}
 
-			vmssPrefix := *v.vmss.Name
-			if v.vmss.VirtualMachineProfile != nil &&
-				v.vmss.VirtualMachineProfile.OsProfile != nil &&
-				v.vmss.VirtualMachineProfile.OsProfile.ComputerNamePrefix != nil {
-				vmssPrefix = *v.vmss.VirtualMachineProfile.OsProfile.ComputerNamePrefix
+			vmssPrefix := *v.VMSS.Name
+			if v.VMSS.VirtualMachineProfile != nil &&
+				v.VMSS.VirtualMachineProfile.OsProfile != nil &&
+				v.VMSS.VirtualMachineProfile.OsProfile.ComputerNamePrefix != nil {
+				vmssPrefix = *v.VMSS.VirtualMachineProfile.OsProfile.ComputerNamePrefix
 			}
 
 			if strings.EqualFold(vmssPrefix, nodeName[:len(nodeName)-6]) {
-				node.vmssName = *v.vmss.Name
-				node.resourceGroup = v.resourceGroup
+				node.vmssName = *v.VMSS.Name
+				node.resourceGroup = v.ResourceGroup
 				return false
 			}
 
@@ -1533,8 +1533,8 @@ func (ss *ScaleSet) ensureBackendPoolDeletedFromVMSS(backendPoolID, vmSetName st
 		var errorList []error
 		walk := func(key, value interface{}) bool {
 			var vmss *compute.VirtualMachineScaleSet
-			if vmssEntry, ok := value.(*vmssEntry); ok {
-				vmss = vmssEntry.vmss
+			if vmssEntry, ok := value.(*VMSSEntry); ok {
+				vmss = vmssEntry.VMSS
 			} else if v, ok := value.(*compute.VirtualMachineScaleSet); ok {
 				vmss = v
 			}
@@ -1873,7 +1873,7 @@ func (ss *ScaleSet) GetAgentPoolVMSetNames(nodes []*v1.Node) (*[]string, error)
 			if err != nil {
 				return nil, fmt.Errorf("GetAgentPoolVMSetNames: failed to get availabilitySetNodesCache")
 			}
-			vms := cached.(*availabilitySetNodeEntry).vms
+			vms := cached.(*AvailabilitySetNodeEntry).VMs
 			names, err = as.getAgentPoolAvailabilitySets(vms, []*v1.Node{node})
 			if err != nil {
 				return nil, fmt.Errorf("GetAgentPoolVMSetNames: failed to execute getAgentPoolAvailabilitySets: %w", err)
diff --git a/pkg/provider/azure_vmss_cache.go b/pkg/provider/azure_vmss_cache.go
index cbf19f4e7..d12a857cf 100644
--- a/pkg/provider/azure_vmss_cache.go
+++ b/pkg/provider/azure_vmss_cache.go
@@ -33,24 +33,24 @@ import (
 	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
 )
 
-type vmssVirtualMachinesEntry struct {
-	resourceGroup  string
-	vmssName       string
-	instanceID     string
-	virtualMachine *compute.VirtualMachineScaleSetVM
-	lastUpdate     time.Time
+type VMSSVirtualMachinesEntry struct {
+	ResourceGroup  string
+	VMSSName       string
+	InstanceID     string
+	VirtualMachine *compute.VirtualMachineScaleSetVM
+	LastUpdate     time.Time
 }
 
-type vmssEntry struct {
-	vmss          *compute.VirtualMachineScaleSet
-	resourceGroup string
-	lastUpdate    time.Time
+type VMSSEntry struct {
+	VMSS          *compute.VirtualMachineScaleSet
+	ResourceGroup string
+	LastUpdate    time.Time
 }
 
-type availabilitySetNodeEntry struct {
-	vmNames   sets.String
-	nodeNames sets.String
-	vms       []compute.VirtualMachine
+type AvailabilitySetNodeEntry struct {
+	VMNames   sets.String
+	NodeNames sets.String
+	VMs       []compute.VirtualMachine
 }
 
 func (ss *ScaleSet) newVMSSCache() (*azcache.TimedCache, error) {
@@ -81,10 +81,10 @@ func (ss *ScaleSet) newVMSSCache() (*azcache.TimedCache, error) {
 					klog.Warning("failed to get the name of VMSS")
 					continue
 				}
-				localCache.Store(*scaleSet.Name, &vmssEntry{
-					vmss:          &scaleSet,
-					resourceGroup: resourceGroup,
-					lastUpdate:    time.Now().UTC(),
+				localCache.Store(*scaleSet.Name, &VMSSEntry{
+					VMSS:          &scaleSet,
+					ResourceGroup: resourceGroup,
+					LastUpdate:    time.Now().UTC(),
 				})
 			}
 		}
@@ -147,27 +147,7 @@ func (ss *ScaleSet) getVMSSVMCache(resourceGroup, vmssName string) (string, *azc
 
 // gcVMSSVMCache delete stale VMSS VMs caches from deleted VMSSes.
 func (ss *ScaleSet) gcVMSSVMCache() error {
-	cached, err := ss.vmssCache.Get(consts.VMSSKey, azcache.CacheReadTypeUnsafe)
-	if err != nil {
-		return err
-	}
-
-	vmsses := cached.(*sync.Map)
-	removed := map[string]bool{}
-	ss.vmssVMCache.Range(func(key, value interface{}) bool {
-		cacheKey := key.(string)
-		vlistIdx := cacheKey[strings.LastIndex(cacheKey, "/")+1:]
-		if _, ok := vmsses.Load(vlistIdx); !ok {
-			removed[cacheKey] = true
-		}
-		return true
-	})
-
-	for key := range removed {
-		ss.vmssVMCache.Delete(key)
-	}
-
-	return nil
+	return ss.vmssCache.Delete(consts.VMSSKey)
 }
 
 // newVMSSVirtualMachinesCache instantiates a new VMs cache for VMs belonging to the provided VMSS.
@@ -177,7 +157,7 @@ func (ss *ScaleSet) newVMSSVirtualMachinesCache(resourceGroupName, vmssName, cac
 	getter := func(key string) (interface{}, error) {
 		localCache := &sync.Map{} // [nodeName]*vmssVirtualMachinesEntry
 
-		oldCache := make(map[string]vmssVirtualMachinesEntry)
+		oldCache := make(map[string]VMSSVirtualMachinesEntry)
 
 		if vmssCache, ok := ss.vmssVMCache.Load(cacheKey); ok {
 			// get old cache before refreshing the cache
@@ -191,7 +171,7 @@ func (ss *ScaleSet) newVMSSVirtualMachinesCache(resourceGroupName, vmssName, cac
 				if cached != nil {
 					virtualMachines := cached.(*sync.Map)
 					virtualMachines.Range(func(key, value interface{}) bool {
-						oldCache[key.(string)] = *value.(*vmssVirtualMachinesEntry)
+						oldCache[key.(string)] = *value.(*VMSSVirtualMachinesEntry)
 						return true
 					})
 				}
@@ -216,18 +196,18 @@ func (ss *ScaleSet) newVMSSVirtualMachinesCache(resourceGroupName, vmssName, cac
 				continue
 			}
 
-			vmssVMCacheEntry := &vmssVirtualMachinesEntry{
-				resourceGroup:  resourceGroupName,
-				vmssName:       vmssName,
-				instanceID:     to.String(vm.InstanceID),
-				virtualMachine: &vm,
-				lastUpdate:     time.Now().UTC(),
+			vmssVMCacheEntry := &VMSSVirtualMachinesEntry{
+				ResourceGroup:  resourceGroupName,
+				VMSSName:       vmssName,
+				InstanceID:     to.String(vm.InstanceID),
+				VirtualMachine: &vm,
+				LastUpdate:     time.Now().UTC(),
 			}
 			// set cache entry to nil when the VM is under deleting.
 			if vm.VirtualMachineScaleSetVMProperties != nil &&
 				strings.EqualFold(to.String(vm.VirtualMachineScaleSetVMProperties.ProvisioningState), string(compute.ProvisioningStateDeleting)) {
 				klog.V(4).Infof("VMSS virtualMachine %q is under deleting, setting its cache to nil", computerName)
-				vmssVMCacheEntry.virtualMachine = nil
+				vmssVMCacheEntry.VirtualMachine = nil
 			}
 			localCache.Store(computerName, vmssVMCacheEntry)
 
@@ -239,24 +219,24 @@ func (ss *ScaleSet) newVMSSVirtualMachinesCache(resourceGroupName, vmssName, cac
 		for name, vmEntry := range oldCache {
 			// if the nil cache entry has existed for vmssVirtualMachinesCacheTTL in the cache
 			// then it should not be added back to the cache
-			if vmEntry.virtualMachine == nil && time.Since(vmEntry.lastUpdate) > vmssVirtualMachinesCacheTTL {
+			if vmEntry.VirtualMachine == nil && time.Since(vmEntry.LastUpdate) > vmssVirtualMachinesCacheTTL {
 				klog.V(5).Infof("ignoring expired entries from old cache for %s", name)
 				continue
 			}
-			lastUpdate := time.Now().UTC()
-			if vmEntry.virtualMachine == nil {
+			LastUpdate := time.Now().UTC()
+			if vmEntry.VirtualMachine == nil {
 				// if this is already a nil entry then keep the time the nil
 				// entry was first created, so we can cleanup unwanted entries
-				lastUpdate = vmEntry.lastUpdate
+				LastUpdate = vmEntry.LastUpdate
 			}
 
 			klog.V(5).Infof("adding old entries to new cache for %s", name)
-			localCache.Store(name, &vmssVirtualMachinesEntry{
-				resourceGroup:  vmEntry.resourceGroup,
-				vmssName:       vmEntry.vmssName,
-				instanceID:     vmEntry.instanceID,
-				virtualMachine: nil,
-				lastUpdate:     lastUpdate,
+			localCache.Store(name, &VMSSVirtualMachinesEntry{
+				ResourceGroup:  vmEntry.ResourceGroup,
+				VMSSName:       vmEntry.VMSSName,
+				InstanceID:     vmEntry.InstanceID,
+				VirtualMachine: nil,
+				LastUpdate:     LastUpdate,
 			})
 		}
 
@@ -266,6 +246,7 @@ func (ss *ScaleSet) newVMSSVirtualMachinesCache(resourceGroupName, vmssName, cac
 	return azcache.NewTimedcache(vmssVirtualMachinesCacheTTL, getter)
 }
 
+// DeleteCacheForNode deletes Node from VMSS VM and VM caches.
 func (ss *ScaleSet) DeleteCacheForNode(nodeName string) error {
 	managedByAS, err := ss.isNodeManagedByAvailabilitySet(nodeName, azcache.CacheReadTypeUnsafe)
 	if err != nil {
@@ -273,27 +254,11 @@ func (ss *ScaleSet) DeleteCacheForNode(nodeName string) error {
 		return nil
 	}
 	if managedByAS {
-		cached, err := ss.availabilitySetNodesCache.Get(consts.AvailabilitySetNodesKey, azcache.CacheReadTypeDefault)
-		if err != nil {
-			klog.Warningf("DeleteCacheForNode(%s): failed to get availabilitySetNodesCache: %s", err.Error())
-			return nil
-		}
-
 		ss.lockMap.LockEntry(consts.AvailabilitySetNodesKey)
 		defer ss.lockMap.UnlockEntry(consts.AvailabilitySetNodesKey)
 
-		entry := cached.(*availabilitySetNodeEntry)
-		entry.nodeNames.Delete(nodeName)
-		entry.vmNames.Delete(nodeName)
-		for i := len(entry.vms) - 1; i >= 0; i-- {
-			if strings.EqualFold(to.String(entry.vms[i].Name), nodeName) {
-				entry.vms = append(entry.vms[:i], entry.vms[i+1:]...)
-				break
-			}
-		}
-
+		_ = ss.availabilitySetNodesCache.Delete(consts.AvailabilitySetNodesKey)
 		_ = ss.vmCache.Delete(nodeName)
-
 		return nil
 	}
 
@@ -309,14 +274,13 @@ func (ss *ScaleSet) DeleteCacheForNode(nodeName string) error {
 		return err
 	}
 
-	vmcache, err := timedcache.Get(cacheKey, azcache.CacheReadTypeUnsafe)
+	err = timedcache.Delete(cacheKey)
 	if err != nil {
 		klog.Errorf("DeleteCacheForNode(%s) failed with error: %v", nodeName, err)
 		return err
 	}
-	virtualMachines := vmcache.(*sync.Map)
-	virtualMachines.Delete(nodeName)
 
+	// Delete in VMSS VM cache
 	if err := ss.gcVMSSVMCache(); err != nil {
 		klog.Errorf("DeleteCacheForNode(%s) failed to gc stale vmss caches: %v", nodeName, err)
 	}
@@ -352,10 +316,10 @@ func (ss *ScaleSet) newAvailabilitySetNodesCache() (*azcache.TimedCache, error)
 			return nil, err
 		}
 
-		localCache := &availabilitySetNodeEntry{
-			vmNames:   vmNames,
-			nodeNames: nodeNames,
-			vms:       vmList,
+		localCache := &AvailabilitySetNodeEntry{
+			VMNames:   vmNames,
+			NodeNames: nodeNames,
+			VMs:       vmList,
 		}
 
 		return localCache, nil
@@ -379,7 +343,7 @@ func (ss *ScaleSet) isNodeManagedByAvailabilitySet(nodeName string, crt azcache.
 		return false, err
 	}
 
-	cachedNodes := cached.(*availabilitySetNodeEntry).nodeNames
+	cachedNodes := cached.(*AvailabilitySetNodeEntry).NodeNames
 	// if the node is not in the cache, assume the node has joined after the last cache refresh and attempt to refresh the cache.
 	if !cachedNodes.Has(nodeName) {
 		klog.V(2).Infof("Node %s has joined the cluster since the last VM cache refresh, refreshing the cache", nodeName)
@@ -389,6 +353,6 @@ func (ss *ScaleSet) isNodeManagedByAvailabilitySet(nodeName string, crt azcache.
 		}
 	}
 
-	cachedVMs := cached.(*availabilitySetNodeEntry).vmNames
+	cachedVMs := cached.(*AvailabilitySetNodeEntry).VMNames
 	return cachedVMs.Has(nodeName), nil
 }
diff --git a/pkg/util/deepcopy/deepcopy.go b/pkg/util/deepcopy/deepcopy.go
new file mode 100644
index 000000000..4282ad14a
--- /dev/null
+++ b/pkg/util/deepcopy/deepcopy.go
@@ -0,0 +1,139 @@
+/*
+Copyright 2022 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package deepcopy
+
+import (
+	"reflect"
+	"sync"
+)
+
+type deepCopyInterface interface {
+	DeepCopy() interface{}
+}
+
+// Copy deepcopies from v.
+func Copy(src interface{}) interface{} {
+	if src == nil {
+		return nil
+	}
+
+	if fromSyncMap, ok := src.(*sync.Map); ok {
+		to := copySyncMap(fromSyncMap)
+		return to
+	}
+
+	return copyNormal(src)
+}
+
+// copySyncMap copies with sync.Map but not nested
+// Targets are vmssVMCache, vmssFlexVMCache, etc.
+func copySyncMap(from *sync.Map) *sync.Map {
+	to := &sync.Map{}
+
+	from.Range(func(k, v interface{}) bool {
+		vm, ok := v.(*sync.Map)
+		if ok {
+			to.Store(k, copySyncMap(vm))
+		} else {
+			to.Store(k, copyNormal(v))
+		}
+		return true
+	})
+
+	return to
+}
+
+func copyNormal(src interface{}) interface{} {
+	if src == nil {
+		return nil
+	}
+
+	from := reflect.ValueOf(src)
+
+	to := reflect.New(from.Type()).Elem()
+
+	copy(from, to)
+
+	return to.Interface()
+}
+
+func copy(from, to reflect.Value) {
+	// Check if DeepCopy() is already implemented for the interface
+	if from.CanInterface() {
+		if deepcopy, ok := from.Interface().(deepCopyInterface); ok {
+			to.Set(reflect.ValueOf(deepcopy.DeepCopy()))
+			return
+		}
+	}
+
+	switch from.Kind() {
+	case reflect.Pointer:
+		fromValue := from.Elem()
+		if !fromValue.IsValid() {
+			return
+		}
+
+		to.Set(reflect.New(fromValue.Type()))
+		copy(fromValue, to.Elem())
+
+	case reflect.Interface:
+		if from.IsNil() {
+			return
+		}
+
+		fromValue := from.Elem()
+		toValue := reflect.New(fromValue.Type()).Elem()
+		copy(fromValue, toValue)
+		to.Set(toValue)
+
+	case reflect.Struct:
+		for i := 0; i < from.NumField(); i++ {
+			if from.Type().Field(i).PkgPath != "" {
+				// It is an unexported field.
+				continue
+			}
+			copy(from.Field(i), to.Field(i))
+		}
+
+	case reflect.Slice:
+		if from.IsNil() {
+			return
+		}
+
+		to.Set(reflect.MakeSlice(from.Type(), from.Len(), from.Cap()))
+		for i := 0; i < from.Len(); i++ {
+			copy(from.Index(i), to.Index(i))
+		}
+
+	case reflect.Map:
+		if from.IsNil() {
+			return
+		}
+
+		to.Set(reflect.MakeMap(from.Type()))
+		for _, key := range from.MapKeys() {
+			fromValue := from.MapIndex(key)
+			toValue := reflect.New(fromValue.Type()).Elem()
+			copy(fromValue, toValue)
+			copiedKey := Copy(key.Interface())
+			to.SetMapIndex(reflect.ValueOf(copiedKey), toValue)
+		}
+
+	default:
+		to.Set(from)
+	}
+}


Test patch:
diff --git a/pkg/cache/azure_cache_test.go b/pkg/cache/azure_cache_test.go
index dd3a32c1e..50b110d59 100644
--- a/pkg/cache/azure_cache_test.go
+++ b/pkg/cache/azure_cache_test.go
@@ -30,7 +30,7 @@ const (
 	fakeCacheTTL = 2 * time.Second
 )
 
-type fakeDataObj struct{}
+type fakeDataObj struct{ Data string }
 
 type fakeDataSource struct {
 	called int
@@ -118,6 +118,40 @@ func TestCacheGetError(t *testing.T) {
 	assert.Nil(t, val)
 }
 
+func TestCacheGetWithDeepCopy(t *testing.T) {
+	changed, unchanged := "changed", "unchanged"
+	valFake := &fakeDataObj{unchanged}
+	cases := []struct {
+		name     string
+		data     map[string]*fakeDataObj
+		key      string
+		expected interface{}
+	}{
+		{
+			name:     "cache should return data for existing key",
+			data:     map[string]*fakeDataObj{"key1": valFake},
+			key:      "key1",
+			expected: unchanged,
+		},
+	}
+
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			dataSource, cache := newFakeCache(t)
+			dataSource.set(c.data)
+			cache.Set(c.key, valFake)
+			val, err := cache.Get(c.key, CacheReadTypeDefault)
+			assert.NoError(t, err)
+			assert.Equal(t, c.expected, val.(*fakeDataObj).Data)
+
+			// Change the value
+			valFake.Data = changed
+			cache.Set(c.key, valFake)
+			assert.Equal(t, c.expected, val.(*fakeDataObj).Data)
+		})
+	}
+}
+
 func TestCacheDelete(t *testing.T) {
 	val := &fakeDataObj{}
 	data := map[string]*fakeDataObj{
diff --git a/pkg/provider/azure_loadbalancer_test.go b/pkg/provider/azure_loadbalancer_test.go
index a4dde3ccf..99ad91221 100644
--- a/pkg/provider/azure_loadbalancer_test.go
+++ b/pkg/provider/azure_loadbalancer_test.go
@@ -4086,6 +4086,11 @@ func TestEnsurePublicIPExists(t *testing.T) {
 				Name: to.StringPtr("pip1"),
 				ID: to.StringPtr("/subscriptions/subscription/resourceGroups/rg" +
 					"/providers/Microsoft.Network/publicIPAddresses/pip1"),
+				PublicIPAddressPropertiesFormat: &network.PublicIPAddressPropertiesFormat{
+					PublicIPAddressVersion:   network.IPVersionIPv4,
+					PublicIPAllocationMethod: network.IPAllocationMethodStatic,
+				},
+				Tags: map[string]*string{},
 			},
 			shouldPutPIP: true,
 		},
@@ -4113,6 +4118,7 @@ func TestEnsurePublicIPExists(t *testing.T) {
 					},
 					PublicIPAddressVersion: "IPv4",
 				},
+				Tags: map[string]*string{consts.ServiceUsingDNSKey: to.StringPtr("default/test1")},
 			},
 			shouldPutPIP: true,
 		},
@@ -4135,6 +4141,7 @@ func TestEnsurePublicIPExists(t *testing.T) {
 					DNSSettings:            nil,
 					PublicIPAddressVersion: "IPv4",
 				},
+				Tags: map[string]*string{},
 			},
 			shouldPutPIP: true,
 		},
@@ -4181,6 +4188,7 @@ func TestEnsurePublicIPExists(t *testing.T) {
 					PublicIPAllocationMethod: "Dynamic",
 					PublicIPAddressVersion:   "IPv6",
 				},
+				Tags: map[string]*string{consts.ServiceUsingDNSKey: to.StringPtr("default/test1")},
 			},
 			shouldPutPIP: true,
 		},
@@ -4246,6 +4254,11 @@ func TestEnsurePublicIPExists(t *testing.T) {
 				Name: to.StringPtr("pip1"),
 				ID: to.StringPtr("/subscriptions/subscription/resourceGroups/rg" +
 					"/providers/Microsoft.Network/publicIPAddresses/pip1"),
+				PublicIPAddressPropertiesFormat: &network.PublicIPAddressPropertiesFormat{
+					PublicIPAddressVersion:   network.IPVersionIPv4,
+					PublicIPAllocationMethod: network.IPAllocationMethodStatic,
+				},
+				Tags: map[string]*string{},
 			},
 			shouldPutPIP: true,
 		},
@@ -4270,6 +4283,7 @@ func TestEnsurePublicIPExists(t *testing.T) {
 					PublicIPAddressVersion:   network.IPVersionIPv6,
 					PublicIPAllocationMethod: network.IPAllocationMethodDynamic,
 				},
+				Tags: map[string]*string{},
 			},
 			shouldPutPIP: true,
 		},
@@ -4277,9 +4291,14 @@ func TestEnsurePublicIPExists(t *testing.T) {
 			desc:         "shall update pip tags if there is any change",
 			existingPIPs: []network.PublicIPAddress{{Name: to.StringPtr("pip1"), Tags: map[string]*string{"a": to.StringPtr("b")}}},
 			expectedPIP: &network.PublicIPAddress{
-				Name: to.StringPtr("pip1"), Tags: map[string]*string{"a": to.StringPtr("c")},
+				Name: to.StringPtr("pip1"),
+				Tags: map[string]*string{"a": to.StringPtr("c")},
 				ID: to.StringPtr("/subscriptions/subscription/resourceGroups/rg" +
 					"/providers/Microsoft.Network/publicIPAddresses/pip1"),
+				PublicIPAddressPropertiesFormat: &network.PublicIPAddressPropertiesFormat{
+					PublicIPAddressVersion:   network.IPVersionIPv4,
+					PublicIPAllocationMethod: network.IPAllocationMethodStatic,
+				},
 			},
 			additionalAnnotations: map[string]string{
 				consts.ServiceAnnotationAzurePIPTags: "a=c",
@@ -4299,7 +4318,12 @@ func TestEnsurePublicIPExists(t *testing.T) {
 			service.ObjectMeta.Annotations = test.additionalAnnotations
 			mockPIPsClient := az.PublicIPAddressesClient.(*mockpublicipclient.MockInterface)
 			if test.shouldPutPIP {
-				mockPIPsClient.EXPECT().CreateOrUpdate(gomock.Any(), "rg", gomock.Any(), gomock.Any()).Return(nil)
+				mockPIPsClient.EXPECT().CreateOrUpdate(gomock.Any(), "rg", gomock.Any(), gomock.Any()).DoAndReturn(func(ctx context.Context, resourceGroupName string, publicIPAddressName string, parameters network.PublicIPAddress) *retry.Error {
+					if len(test.existingPIPs) != 0 {
+						test.existingPIPs[0] = parameters
+					}
+					return nil
+				}).AnyTimes()
 			}
 			mockPIPsClient.EXPECT().Get(gomock.Any(), "rg", "pip1", gomock.Any()).DoAndReturn(func(ctx context.Context, resourceGroupName string, publicIPAddressName string, expand string) (network.PublicIPAddress, *retry.Error) {
 				var basicPIP network.PublicIPAddress
diff --git a/pkg/provider/azure_routes_test.go b/pkg/provider/azure_routes_test.go
index 7a5510f44..1b1dad432 100644
--- a/pkg/provider/azure_routes_test.go
+++ b/pkg/provider/azure_routes_test.go
@@ -168,7 +168,8 @@ func TestDeleteRouteDualStack(t *testing.T) {
 			Routes: &[]network.Route{},
 		},
 	}
-	routeTableClient.EXPECT().Get(gomock.Any(), cloud.RouteTableResourceGroup, cloud.RouteTableName, "").Return(routeTables, nil).AnyTimes()
+	routeTableClient.EXPECT().Get(gomock.Any(), cloud.RouteTableResourceGroup, cloud.RouteTableName, "").Return(routeTables, nil)
+	routeTableClient.EXPECT().Get(gomock.Any(), cloud.RouteTableResourceGroup, cloud.RouteTableName, "").Return(routeTablesAfterFirstDeletion, nil)
 	routeTableClient.EXPECT().CreateOrUpdate(gomock.Any(), cloud.RouteTableResourceGroup, cloud.RouteTableName, routeTablesAfterFirstDeletion, "").Return(nil)
 	routeTableClient.EXPECT().CreateOrUpdate(gomock.Any(), cloud.RouteTableResourceGroup, cloud.RouteTableName, routeTablesAfterSecondDeletion, "").Return(nil)
 	err := cloud.DeleteRoute(context.TODO(), "cluster", &route)
diff --git a/pkg/provider/azure_test.go b/pkg/provider/azure_test.go
index d758cdd21..2a0f956e0 100644
--- a/pkg/provider/azure_test.go
+++ b/pkg/provider/azure_test.go
@@ -2545,13 +2545,14 @@ func TestIfServicesSpecifySharedRuleButDifferentPortsThenSeparateRulesAreCreated
 	svc2.Annotations[consts.ServiceAnnotationSharedSecurityRule] = consts.TrueAnnotationValue
 
 	sg := getTestSecurityGroup(az)
-	setMockSecurityGroup(az, ctrl, sg)
 
-	_, err := az.reconcileSecurityGroup(testClusterName, &svc1, to.StringPtr(getServiceLoadBalancerIP(&svc1)), nil, true)
+	setMockSecurityGroup(az, ctrl, sg)
+	sg, err := az.reconcileSecurityGroup(testClusterName, &svc1, to.StringPtr(getServiceLoadBalancerIP(&svc1)), nil, true)
 	if err != nil {
 		t.Errorf("Unexpected error adding svc1: %q", err)
 	}
 
+	setMockSecurityGroup(az, ctrl, sg)
 	sg, err = az.reconcileSecurityGroup(testClusterName, &svc2, to.StringPtr(getServiceLoadBalancerIP(&svc2)), nil, true)
 	if err != nil {
 		t.Errorf("Unexpected error adding svc2: %q", err)
@@ -2617,13 +2618,14 @@ func TestIfServicesSpecifySharedRuleButDifferentProtocolsThenSeparateRulesAreCre
 	testRuleName3 := "shared-UDP-4444-Internet"
 
 	sg := getTestSecurityGroup(az)
-	setMockSecurityGroup(az, ctrl, sg)
 
-	_, err := az.reconcileSecurityGroup(testClusterName, &svc1, to.StringPtr(getServiceLoadBalancerIP(&svc1)), nil, true)
+	setMockSecurityGroup(az, ctrl, sg)
+	sg, err := az.reconcileSecurityGroup(testClusterName, &svc1, to.StringPtr(getServiceLoadBalancerIP(&svc1)), nil, true)
 	if err != nil {
 		t.Errorf("Unexpected error adding svc1: %q", err)
 	}
 
+	setMockSecurityGroup(az, ctrl, sg)
 	sg, err = az.reconcileSecurityGroup(testClusterName, &svc2, to.StringPtr(getServiceLoadBalancerIP(&svc2)), nil, true)
 	if err != nil {
 		t.Errorf("Unexpected error adding svc2: %q", err)
@@ -2689,13 +2691,14 @@ func TestIfServicesSpecifySharedRuleButDifferentSourceAddressesThenSeparateRules
 	testRuleName3 := "shared-TCP-80-192.168.34.0_24"
 
 	sg := getTestSecurityGroup(az)
-	setMockSecurityGroup(az, ctrl, sg)
 
-	_, err := az.reconcileSecurityGroup(testClusterName, &svc1, to.StringPtr(getServiceLoadBalancerIP(&svc1)), nil, true)
+	setMockSecurityGroup(az, ctrl, sg)
+	sg, err := az.reconcileSecurityGroup(testClusterName, &svc1, to.StringPtr(getServiceLoadBalancerIP(&svc1)), nil, true)
 	if err != nil {
 		t.Errorf("Unexpected error adding svc1: %q", err)
 	}
 
+	setMockSecurityGroup(az, ctrl, sg)
 	sg, err = az.reconcileSecurityGroup(testClusterName, &svc2, to.StringPtr(getServiceLoadBalancerIP(&svc2)), nil, true)
 	if err != nil {
 		t.Errorf("Unexpected error adding svc2: %q", err)
@@ -2763,18 +2766,20 @@ func TestIfServicesSpecifySharedRuleButSomeAreOnDifferentPortsThenRulesAreSepara
 
 	testRuleName23 := testRuleName2
 	sg := getTestSecurityGroup(az)
-	setMockSecurityGroup(az, ctrl, sg)
 
-	_, err := az.reconcileSecurityGroup(testClusterName, &svc1, to.StringPtr(getServiceLoadBalancerIP(&svc1)), nil, true)
+	setMockSecurityGroup(az, ctrl, sg)
+	sg, err := az.reconcileSecurityGroup(testClusterName, &svc1, to.StringPtr(getServiceLoadBalancerIP(&svc1)), nil, true)
 	if err != nil {
 		t.Errorf("Unexpected error adding svc1: %q", err)
 	}
 
-	_, err = az.reconcileSecurityGroup(testClusterName, &svc2, to.StringPtr(getServiceLoadBalancerIP(&svc2)), nil, true)
+	setMockSecurityGroup(az, ctrl, sg)
+	sg, err = az.reconcileSecurityGroup(testClusterName, &svc2, to.StringPtr(getServiceLoadBalancerIP(&svc2)), nil, true)
 	if err != nil {
 		t.Errorf("Unexpected error adding svc2: %q", err)
 	}
 
+	setMockSecurityGroup(az, ctrl, sg)
 	sg, err = az.reconcileSecurityGroup(testClusterName, &svc3, to.StringPtr(getServiceLoadBalancerIP(&svc3)), nil, true)
 	if err != nil {
 		t.Errorf("Unexpected error adding svc3: %q", err)
@@ -2861,13 +2866,14 @@ func TestIfServiceSpecifiesSharedRuleAndServiceIsDeletedThenTheServicesPortAndAd
 	expectedRuleName := testRuleName
 
 	sg := getTestSecurityGroup(az)
-	setMockSecurityGroup(az, ctrl, sg)
 
-	_, err := az.reconcileSecurityGroup(testClusterName, &svc1, to.StringPtr(getServiceLoadBalancerIP(&svc1)), nil, true)
+	setMockSecurityGroup(az, ctrl, sg)
+	sg, err := az.reconcileSecurityGroup(testClusterName, &svc1, to.StringPtr(getServiceLoadBalancerIP(&svc1)), nil, true)
 	if err != nil {
 		t.Errorf("Unexpected error adding svc1: %q", err)
 	}
 
+	setMockSecurityGroup(az, ctrl, sg)
 	sg, err = az.reconcileSecurityGroup(testClusterName, &svc2, to.StringPtr(getServiceLoadBalancerIP(&svc2)), nil, true)
 	if err != nil {
 		t.Errorf("Unexpected error adding svc2: %q", err)
@@ -2875,6 +2881,7 @@ func TestIfServiceSpecifiesSharedRuleAndServiceIsDeletedThenTheServicesPortAndAd
 
 	validateSecurityGroup(t, sg, svc1, svc2)
 
+	setMockSecurityGroup(az, ctrl, sg)
 	sg, err = az.reconcileSecurityGroup(testClusterName, &svc1, to.StringPtr(getServiceLoadBalancerIP(&svc1)), nil, false)
 	if err != nil {
 		t.Errorf("Unexpected error removing svc1: %q", err)
@@ -2922,18 +2929,20 @@ func TestIfSomeServicesShareARuleAndOneIsDeletedItIsRemovedFromTheRightRule(t *t
 
 	testRuleName23 := testRuleName2
 	sg := getTestSecurityGroup(az)
-	setMockSecurityGroup(az, ctrl, sg)
 
-	_, err := az.reconcileSecurityGroup(testClusterName, &svc1, to.StringPtr(getServiceLoadBalancerIP(&svc1)), nil, true)
+	setMockSecurityGroup(az, ctrl, sg)
+	sg, err := az.reconcileSecurityGroup(testClusterName, &svc1, to.StringPtr(getServiceLoadBalancerIP(&svc1)), nil, true)
 	if err != nil {
 		t.Errorf("Unexpected error adding svc1: %q", err)
 	}
 
-	_, err = az.reconcileSecurityGroup(testClusterName, &svc2, to.StringPtr(getServiceLoadBalancerIP(&svc2)), nil, true)
+	setMockSecurityGroup(az, ctrl, sg)
+	sg, err = az.reconcileSecurityGroup(testClusterName, &svc2, to.StringPtr(getServiceLoadBalancerIP(&svc2)), nil, true)
 	if err != nil {
 		t.Errorf("Unexpected error adding svc2: %q", err)
 	}
 
+	setMockSecurityGroup(az, ctrl, sg)
 	sg, err = az.reconcileSecurityGroup(testClusterName, &svc3, to.StringPtr(getServiceLoadBalancerIP(&svc3)), nil, true)
 	if err != nil {
 		t.Errorf("Unexpected error adding svc3: %q", err)
@@ -2941,6 +2950,7 @@ func TestIfSomeServicesShareARuleAndOneIsDeletedItIsRemovedFromTheRightRule(t *t
 
 	validateSecurityGroup(t, sg, svc1, svc2, svc3)
 
+	setMockSecurityGroup(az, ctrl, sg)
 	sg, err = az.reconcileSecurityGroup(testClusterName, &svc1, to.StringPtr(getServiceLoadBalancerIP(&svc1)), nil, false)
 	if err != nil {
 		t.Errorf("Unexpected error removing svc1: %q", err)
@@ -3030,18 +3040,20 @@ func TestIfServiceSpecifiesSharedRuleAndLastServiceIsDeletedThenRuleIsDeleted(t
 
 	testRuleName23 := testRuleName2
 	sg := getTestSecurityGroup(az)
-	setMockSecurityGroup(az, ctrl, sg)
 
-	_, err := az.reconcileSecurityGroup(testClusterName, &svc1, to.StringPtr(getServiceLoadBalancerIP(&svc1)), nil, true)
+	setMockSecurityGroup(az, ctrl, sg)
+	sg, err := az.reconcileSecurityGroup(testClusterName, &svc1, to.StringPtr(getServiceLoadBalancerIP(&svc1)), nil, true)
 	if err != nil {
 		t.Errorf("Unexpected error adding svc1: %q", err)
 	}
 
-	_, err = az.reconcileSecurityGroup(testClusterName, &svc2, to.StringPtr(getServiceLoadBalancerIP(&svc2)), nil, true)
+	setMockSecurityGroup(az, ctrl, sg)
+	sg, err = az.reconcileSecurityGroup(testClusterName, &svc2, to.StringPtr(getServiceLoadBalancerIP(&svc2)), nil, true)
 	if err != nil {
 		t.Errorf("Unexpected error adding svc2: %q", err)
 	}
 
+	setMockSecurityGroup(az, ctrl, sg)
 	sg, err = az.reconcileSecurityGroup(testClusterName, &svc3, to.StringPtr(getServiceLoadBalancerIP(&svc3)), nil, true)
 	if err != nil {
 		t.Errorf("Unexpected error adding svc3: %q", err)
@@ -3049,11 +3061,13 @@ func TestIfServiceSpecifiesSharedRuleAndLastServiceIsDeletedThenRuleIsDeleted(t
 
 	validateSecurityGroup(t, sg, svc1, svc2, svc3)
 
-	_, err = az.reconcileSecurityGroup(testClusterName, &svc1, to.StringPtr(getServiceLoadBalancerIP(&svc1)), nil, false)
+	setMockSecurityGroup(az, ctrl, sg)
+	sg, err = az.reconcileSecurityGroup(testClusterName, &svc1, to.StringPtr(getServiceLoadBalancerIP(&svc1)), nil, false)
 	if err != nil {
 		t.Errorf("Unexpected error removing svc1: %q", err)
 	}
 
+	setMockSecurityGroup(az, ctrl, sg)
 	sg, err = az.reconcileSecurityGroup(testClusterName, &svc3, to.StringPtr(getServiceLoadBalancerIP(&svc3)), nil, false)
 	if err != nil {
 		t.Errorf("Unexpected error removing svc3: %q", err)
@@ -3125,11 +3139,11 @@ func TestCanCombineSharedAndPrivateRulesInSameGroup(t *testing.T) {
 	expectedRuleName5 := az.getSecurityRuleName(&svc5, v1.ServicePort{Port: 8888, Protocol: v1.ProtocolTCP}, "Internet")
 
 	sg := getTestSecurityGroup(az)
-	setMockSecurityGroup(az, ctrl, sg)
 
 	for i, svc := range testServices {
 		svc := svc
-		_, err := az.reconcileSecurityGroup(testClusterName, &testServices[i], to.StringPtr(getServiceLoadBalancerIP(&svc)), nil, true)
+		setMockSecurityGroup(az, ctrl, sg)
+		sg, err = az.reconcileSecurityGroup(testClusterName, &testServices[i], to.StringPtr(getServiceLoadBalancerIP(&svc)), nil, true)
 		if err != nil {
 			t.Errorf("Unexpected error adding svc%d: %q", i+1, err)
 		}
@@ -3221,11 +3235,13 @@ func TestCanCombineSharedAndPrivateRulesInSameGroup(t *testing.T) {
 		}
 	}
 
-	_, err = az.reconcileSecurityGroup(testClusterName, &svc1, to.StringPtr(getServiceLoadBalancerIP(&svc1)), nil, false)
+	setMockSecurityGroup(az, ctrl, sg)
+	sg, err = az.reconcileSecurityGroup(testClusterName, &svc1, to.StringPtr(getServiceLoadBalancerIP(&svc1)), nil, false)
 	if err != nil {
 		t.Errorf("Unexpected error removing svc1: %q", err)
 	}
 
+	setMockSecurityGroup(az, ctrl, sg)
 	sg, err = az.reconcileSecurityGroup(testClusterName, &svc5, to.StringPtr(getServiceLoadBalancerIP(&svc5)), nil, false)
 	if err != nil {
 		t.Errorf("Unexpected error removing svc5: %q", err)
diff --git a/pkg/provider/azure_vmss_cache_test.go b/pkg/provider/azure_vmss_cache_test.go
index 0f9d7b65d..518320c33 100644
--- a/pkg/provider/azure_vmss_cache_test.go
+++ b/pkg/provider/azure_vmss_cache_test.go
@@ -20,7 +20,6 @@ import (
 	"fmt"
 	"net/http"
 	"strings"
-	"sync"
 	"testing"
 
 	"github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2022-03-01/compute"
@@ -111,6 +110,7 @@ func TestVMSSVMCache(t *testing.T) {
 		vmName := to.String(vm.OsProfile.ComputerName)
 		realVM, err := ss.getVmssVM(vmName, azcache.CacheReadTypeDefault)
 		assert.NoError(t, err)
+		assert.NotNil(t, realVM)
 		assert.Equal(t, "vmss", realVM.VMSSName)
 		assert.Equal(t, to.String(vm.InstanceID), realVM.InstanceID)
 		assert.Equal(t, &vm, realVM.AsVirtualMachineScaleSetVM())
@@ -122,16 +122,7 @@ func TestVMSSVMCache(t *testing.T) {
 	err = ss.DeleteCacheForNode(vmName)
 	assert.NoError(t, err)
 
-	// the VM should be removed from cache after DeleteCacheForNode().
-	cacheKey, cache, err := ss.getVMSSVMCache("rg", testVMSSName)
-	assert.NoError(t, err)
-	cached, err := cache.Get(cacheKey, azcache.CacheReadTypeDefault)
-	assert.NoError(t, err)
-	cachedVirtualMachines := cached.(*sync.Map)
-	_, ok := cachedVirtualMachines.Load(vmName)
-	assert.Equal(t, false, ok)
-
-	// the VM should be back after another cache refresh.
+	// the VM should be in cache after refresh.
 	realVM, err := ss.getVmssVM(vmName, azcache.CacheReadTypeDefault)
 	assert.NoError(t, err)
 	assert.Equal(t, "vmss", realVM.VMSSName)
@@ -151,18 +142,20 @@ func TestDeleteCacheForAvailabilitySetNodeInVMSS(t *testing.T) {
 
 	mockVMClient.EXPECT().List(gomock.Any(), gomock.Any()).Return([]compute.VirtualMachine{
 		{Name: to.StringPtr("vm1")},
-	}, nil).AnyTimes()
+	}, nil)
+	mockVMClient.EXPECT().List(gomock.Any(), gomock.Any()).Return([]compute.VirtualMachine{}, nil).AnyTimes()
 	ss.cloud.nodeNames = sets.NewString("vm1")
 
 	err = ss.DeleteCacheForNode("vm1")
 	assert.NoError(t, err)
 
+	ss.cloud.nodeNames = sets.NewString()
 	cached, err := ss.availabilitySetNodesCache.Get(consts.AvailabilitySetNodesKey, azcache.CacheReadTypeUnsafe)
 	assert.NoError(t, err)
-	entry := cached.(*availabilitySetNodeEntry)
-	assert.Equal(t, 0, len(entry.nodeNames))
-	assert.Equal(t, 0, len(entry.vmNames))
-	assert.Equal(t, 0, len(entry.vms))
+	entry := cached.(*AvailabilitySetNodeEntry)
+	assert.Equal(t, 0, len(entry.NodeNames))
+	assert.Equal(t, 0, len(entry.VMNames))
+	assert.Equal(t, 0, len(entry.VMs))
 }
 
 func TestVMSSVMCacheWithDeletingNodes(t *testing.T) {
diff --git a/pkg/provider/azure_vmss_test.go b/pkg/provider/azure_vmss_test.go
index b2172e23d..98fac7bb4 100644
--- a/pkg/provider/azure_vmss_test.go
+++ b/pkg/provider/azure_vmss_test.go
@@ -957,12 +957,13 @@ func TestGetVmssVMByNodeIdentity(t *testing.T) {
 			assert.Nil(t, err)
 			virtualMachines := cached.(*sync.Map)
 			for _, vm := range test.goneVMList {
-				entry := vmssVirtualMachinesEntry{
-					resourceGroup: ss.ResourceGroup,
-					vmssName:      testVMSSName,
+				entry := VMSSVirtualMachinesEntry{
+					ResourceGroup: ss.ResourceGroup,
+					VMSSName:      testVMSSName,
 				}
 				virtualMachines.Store(vm, &entry)
 			}
+			cache.Set(cacheKey, cached)
 
 			for i := 0; i < len(test.vmList); i++ {
 				node := nodeIdentity{ss.ResourceGroup, testVMSSName, test.vmList[i]}
diff --git a/pkg/util/deepcopy/deepcopy_test.go b/pkg/util/deepcopy/deepcopy_test.go
new file mode 100644
index 000000000..d6515f047
--- /dev/null
+++ b/pkg/util/deepcopy/deepcopy_test.go
@@ -0,0 +1,114 @@
+/*
+Copyright 2022 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package deepcopy
+
+import (
+	"sync"
+	"testing"
+
+	"github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2022-03-01/compute"
+	"github.com/Azure/go-autorest/autorest/to"
+	"github.com/stretchr/testify/assert"
+)
+
+type fakeStruct struct {
+	Name string
+}
+
+type fakeIntf interface {
+	Get() string
+}
+
+func (f fakeStruct) Get() string {
+	return f.Name
+}
+
+// TestCopyBasic tests object with pointer, struct, map, slice, interface.
+func TestCopyBasic(t *testing.T) {
+	zones := []string{"zone0", "zone1"}
+	var vmOriginal *compute.VirtualMachine = &compute.VirtualMachine{
+		VirtualMachineProperties: &compute.VirtualMachineProperties{
+			ProvisioningState: to.StringPtr("Failed"),
+		},
+		Name:  to.StringPtr("vmOriginal"),
+		Zones: &zones,
+		Tags: map[string]*string{
+			"tag0": to.StringPtr("tagVal0"),
+		},
+	}
+	vmCopied := Copy(vmOriginal).(*compute.VirtualMachine)
+
+	psOriginal := vmOriginal.VirtualMachineProperties.ProvisioningState
+	psCopied := vmCopied.VirtualMachineProperties.ProvisioningState
+	assert.Equal(t, psOriginal, psCopied)
+	assert.Equal(t, vmOriginal.Name, vmCopied.Name)
+	assert.Equal(t, vmOriginal.Zones, vmCopied.Zones)
+	assert.Equal(t, vmOriginal.Tags, vmCopied.Tags)
+
+	var fakeOriginal fakeIntf = fakeStruct{Name: "fakeOriginal"}
+	fakeCopied := Copy(fakeOriginal).(fakeIntf)
+	assert.Equal(t, fakeOriginal.Get(), fakeCopied.Get())
+}
+
+// TestCopyVMInSyncMap tests object like compute.VirtualMachine in a sync.Map.
+func TestCopyVMInSyncMap(t *testing.T) {
+	var vmOriginal *compute.VirtualMachine = &compute.VirtualMachine{
+		VirtualMachineProperties: &compute.VirtualMachineProperties{
+			ProvisioningState: to.StringPtr("Failed"),
+		},
+		Name: to.StringPtr("vmOriginal"),
+	}
+	vmCacheOriginal := &sync.Map{}
+	vmCacheOriginal.Store("vmOriginal", vmOriginal)
+	vmCacheCopied := Copy(vmCacheOriginal).(*sync.Map)
+
+	psOriginal := vmOriginal.VirtualMachineProperties.ProvisioningState
+	vCopied, ok := vmCacheCopied.Load("vmOriginal")
+	assert.True(t, ok)
+	vmCopied := vCopied.(*compute.VirtualMachine)
+	psCopied := vmCopied.VirtualMachineProperties.ProvisioningState
+	assert.Equal(t, psOriginal, psCopied)
+	assert.Equal(t, vmOriginal.Name, vmCopied.Name)
+}
+
+type vmssEntry struct {
+	*compute.VirtualMachineScaleSet
+	Name *string
+}
+
+// TestCopyVMSSEntryInSyncMap tests object like vmssEntry in sync.Map.
+func TestCopyVMSSEntryInSyncMap(t *testing.T) {
+	vmssEntryOriginal := &vmssEntry{
+		Name: to.StringPtr("vmssEntryName"),
+		VirtualMachineScaleSet: &compute.VirtualMachineScaleSet{
+			Name: to.StringPtr("vmssOriginal"),
+		},
+	}
+	vmssCacheOriginal := &sync.Map{}
+	vmssCacheOriginal.Store("vmssEntry", vmssEntryOriginal)
+	vmssCacheCopied := Copy(vmssCacheOriginal)
+
+	vCopied, ok := vmssCacheCopied.(*sync.Map).Load("vmssEntry")
+	assert.True(t, ok)
+	vmssEntryCopied := vCopied.(*vmssEntry)
+	entryNameOriginal := vmssEntryOriginal.Name
+	entryNameCopied := vmssEntryCopied.Name
+	assert.Equal(t, entryNameOriginal, entryNameCopied)
+	vmNameOriginal := vmssEntryOriginal.VirtualMachineScaleSet.Name
+	vmNameCopied := vmssEntryCopied.VirtualMachineScaleSet.Name
+	assert.Equal(t, vmNameOriginal, vmNameCopied)
+}