Repository: kubernetes-sigs/cloud-provider-azure
Pull request #2842 (2022-12-01 12:50:52)

[e2e] Support IPv6 single stack clusters
Signed-off-by: Zhecheng Li <zhechengli@microsoft.com>

<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide/first-contribution.md#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?
/kind testing
<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature
/kind design

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->

#### What this PR does / why we need it:
[e2e] Support IPv6 single stack clusters
#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes #

#### Special notes for your reviewer:

#### Does this PR introduce a user-facing change?
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://git.k8s.io/community/contributors/guide/release-notes.md
-->
```release-note
NONE
```

#### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

<!--
This section can be blank if this pull request does not require a release note.

When adding links which point to resources within git repositories, like
KEPs or supporting documentation, please reference a specific commit and avoid
linking directly to the master branch. This ensures that links reference a
specific point in time, rather than a document that may change over time.

See here for guidance on getting permanent links to files: https://help.github.com/en/articles/getting-permanent-links-to-files

Please use the following format for linking documentation:
- [KEP]: <link>
- [Usage]: <link>
- [Other doc]: <link>
-->
```docs

```


netlify[bot]: ### <span aria-hidden="true">👷</span> Deploy Preview for *kubernetes-sigs-cloud-provide-azure* processing.


|  Name | Link |
|---------------------------------|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | e9e892d435dbe207c7d4795cf280550937be0dd5 |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/kubernetes-sigs-cloud-provide-azure/deploys/6388a32efbcdfd0009c78318 |
k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/2842#" title="Author self-approved">lzhecheng</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/master/OWNERS)~~ [lzhecheng]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
k8s-ci-robot: @lzhecheng: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cloud-provider-azure-check | e9e892d435dbe207c7d4795cf280550937be0dd5 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/2842/pull-cloud-provider-azure-check/1598298592890589184) | true | `/test pull-cloud-provider-azure-check`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cloud-provider-azure&pr=2842). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Alzhecheng). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cloud-provider-azure/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
coveralls: 
[![Coverage Status](https://coveralls.io/builds/54745533/badge)](https://coveralls.io/builds/54745533)

Coverage remained the same at 79.495% when pulling **e9e892d435dbe207c7d4795cf280550937be0dd5 on lzhecheng:fix-ipv6-only** into **11407986e99d7aede3cad3a99d9f5a665e3bdee3 on kubernetes-sigs:master**.


Patch:
diff --git a/go.mod b/go.mod
index 41fcfb123..5c4c3d864 100644
--- a/go.mod
+++ b/go.mod
@@ -19,6 +19,7 @@ require (
 	github.com/spf13/pflag v1.0.5
 	github.com/stretchr/testify v1.8.1
 	golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa
+	golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
 	golang.org/x/sys v0.2.0
 	golang.org/x/text v0.4.0
 	k8s.io/api v0.25.4
@@ -100,7 +101,6 @@ require (
 	go.uber.org/zap v1.19.0 // indirect
 	golang.org/x/net v0.2.0 // indirect
 	golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8 // indirect
-	golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4 // indirect
 	golang.org/x/term v0.2.0 // indirect
 	golang.org/x/time v0.0.0-20220210224613-90d013bbcef8 // indirect
 	google.golang.org/appengine v1.6.7 // indirect
diff --git a/pkg/provider/azure_controller_vmssflex.go b/pkg/provider/azure_controller_vmssflex.go
index 42964e2cd..753ee22fa 100644
--- a/pkg/provider/azure_controller_vmssflex.go
+++ b/pkg/provider/azure_controller_vmssflex.go
@@ -18,7 +18,6 @@ package provider
 
 import (
 	"context"
-	"fmt"
 	"net/http"
 	"strings"
 
@@ -52,17 +51,13 @@ func (fs *FlexScaleSet) AttachDisk(ctx context.Context, nodeName types.NodeName,
 		opt := v
 		attached := false
 		for _, disk := range *vm.StorageProfile.DataDisks {
-			if disk.ManagedDisk != nil && strings.EqualFold(*disk.ManagedDisk.ID, diskURI) && disk.Lun != nil {
-				if *disk.Lun == opt.lun {
-					attached = true
-					break
-				} else {
-					return nil, fmt.Errorf("disk(%s) already attached to node(%s) on LUN(%d), but target LUN is %d", diskURI, nodeName, *disk.Lun, opt.lun)
-				}
+			if disk.ManagedDisk != nil && strings.EqualFold(*disk.ManagedDisk.ID, diskURI) {
+				attached = true
+				break
 			}
 		}
 		if attached {
-			klog.V(2).Infof("azureDisk - disk(%s) already attached to node(%s) on LUN(%d)", diskURI, nodeName, opt.lun)
+			klog.V(2).Infof("azureDisk - disk(%s) already attached to node(%s)", diskURI, nodeName)
 			continue
 		}
 
diff --git a/pkg/provider/azure_loadbalancer_backendpool.go b/pkg/provider/azure_loadbalancer_backendpool.go
index a94f54eb8..cb6585035 100644
--- a/pkg/provider/azure_loadbalancer_backendpool.go
+++ b/pkg/provider/azure_loadbalancer_backendpool.go
@@ -185,13 +185,8 @@ func (bc *backendPoolTypeNodeIPConfig) ReconcileBackendPools(clusterName string,
 				for _, ipConf := range *bp.BackendIPConfigurations {
 					ipConfID := to.String(ipConf.ID)
 					nodeName, _, err := bc.VMSet.GetNodeNameByIPConfigurationID(ipConfID)
-					if err != nil {
-						if errors.Is(err, cloudprovider.InstanceNotFound) {
-							klog.V(2).Infof("bc.ReconcileBackendPools for service (%s): vm not found for ipConfID %s", serviceName, ipConfID)
-							backendIPConfigurationsToBeDeleted = append(backendIPConfigurationsToBeDeleted, ipConf)
-						} else {
-							return false, false, err
-						}
+					if err != nil && !errors.Is(err, cloudprovider.InstanceNotFound) {
+						return false, false, err
 					}
 
 					// If a node is not supposed to be included in the LB, it
diff --git a/pkg/provider/azure_utils.go b/pkg/provider/azure_utils.go
index e3542f35c..910abb56f 100644
--- a/pkg/provider/azure_utils.go
+++ b/pkg/provider/azure_utils.go
@@ -298,43 +298,36 @@ func removeDuplicatedSecurityRules(rules []network.SecurityRule) []network.Secur
 	return rules
 }
 
-func getVMSSVMCacheKey(resourceGroup, vmssName string) string {
-	cacheKey := strings.ToLower(fmt.Sprintf("%s/%s", resourceGroup, vmssName))
-	return cacheKey
-}
-
 // isNodeInVMSSVMCache check whether nodeName is in vmssVMCache
-func isNodeInVMSSVMCache(nodeName string, vmssVMCache *azcache.TimedCache) bool {
+func isNodeInVMSSVMCache(nodeName string, vmssVMCache *sync.Map) bool {
 	if vmssVMCache == nil {
 		return false
 	}
-
 	var isInCache bool
-
-	vmssVMCache.Lock.Lock()
-	defer vmssVMCache.Lock.Unlock()
-
-	for _, entry := range vmssVMCache.Store.List() {
-		if entry != nil {
-			e := entry.(*azcache.AzureCacheEntry)
-			e.Lock.Lock()
-			data := e.Data
-			if data != nil {
-				data.(*sync.Map).Range(func(vmName, _ interface{}) bool {
-					if vmName != nil && vmName.(string) == nodeName {
-						isInCache = true
-						return false
+	vmssVMCache.Range(func(_, value interface{}) bool {
+		if value != nil && value.(*azcache.TimedCache).Store != nil {
+			for _, v := range value.(*azcache.TimedCache).Store.List() {
+				if v != nil {
+					data := v.(*azcache.AzureCacheEntry).Data
+					if data != nil {
+						data.(*sync.Map).Range(func(vmName, _ interface{}) bool {
+							if vmName != nil && vmName.(string) == nodeName {
+								isInCache = true
+								return false
+							}
+							return true
+						})
 					}
-					return true
-				})
+				}
+				if isInCache {
+					return false
+				}
 			}
-			e.Lock.Unlock()
 		}
-
 		if isInCache {
-			break
+			return false
 		}
-	}
-
+		return true
+	})
 	return isInCache
 }
diff --git a/pkg/provider/azure_vmss.go b/pkg/provider/azure_vmss.go
index 3bff4627c..cd56e252a 100644
--- a/pkg/provider/azure_vmss.go
+++ b/pkg/provider/azure_vmss.go
@@ -35,6 +35,7 @@ import (
 	"k8s.io/klog/v2"
 	utilnet "k8s.io/utils/net"
 
+	"golang.org/x/sync/singleflight"
 	azcache "sigs.k8s.io/cloud-provider-azure/pkg/cache"
 	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
 	"sigs.k8s.io/cloud-provider-azure/pkg/metrics"
@@ -79,12 +80,8 @@ type ScaleSet struct {
 	// the same cluster.
 	flexScaleSet VMSet
 
-	vmssCache *azcache.TimedCache
-
-	// vmssVMCache is timed cache where the Store in the cache is a map of
-	// Key: [resourcegroup/vmssName]
-	// Value: sync.Map of [vmName]*VMSSVirtualMachinesEntry
-	vmssVMCache *azcache.TimedCache
+	vmssCache   *azcache.TimedCache
+	vmssVMCache *sync.Map // [resourcegroup/vmssname]*azcache.TimedCache
 
 	// nonVmssUniformNodesCache is used to store node names from non uniform vm.
 	// Currently, the nodes can from avset or vmss flex or individual vm.
@@ -95,6 +92,8 @@ type ScaleSet struct {
 
 	// lockMap in cache refresh
 	lockMap *lockMap
+	// group represents a class of work and units of work can be executed with duplicate suppression
+	group singleflight.Group
 }
 
 // newScaleSet creates a new ScaleSet.
@@ -117,6 +116,7 @@ func newScaleSet(az *Cloud) (VMSet, error) {
 		Cloud:           az,
 		availabilitySet: as,
 		flexScaleSet:    fs,
+		vmssVMCache:     &sync.Map{},
 		lockMap:         newLockMap(),
 	}
 
@@ -132,11 +132,6 @@ func newScaleSet(az *Cloud) (VMSet, error) {
 		return nil, err
 	}
 
-	ss.vmssVMCache, err = ss.newVMSSVirtualMachinesCache()
-	if err != nil {
-		return nil, err
-	}
-
 	return ss, nil
 }
 
@@ -179,23 +174,22 @@ func (ss *ScaleSet) getVMSS(vmssName string, crt azcache.AzureCacheReadType) (*c
 // getVmssVMByNodeIdentity find virtualMachineScaleSetVM by nodeIdentity, using node's parent VMSS cache.
 // Returns cloudprovider.InstanceNotFound if the node does not belong to the scale set named in nodeIdentity.
 func (ss *ScaleSet) getVmssVMByNodeIdentity(node *nodeIdentity, crt azcache.AzureCacheReadType) (*virtualmachine.VirtualMachine, error) {
-	// FIXME(ccc): check only if vmss is uniform.
-	_, err := getScaleSetVMInstanceID(node.nodeName)
+	cacheKey, cache, err := ss.getVMSSVMCache(node.resourceGroup, node.vmssName)
 	if err != nil {
 		return nil, err
 	}
 
-	getter := func(crt azcache.AzureCacheReadType) (*virtualmachine.VirtualMachine, bool, error) {
+	getter := func(nodeName string, crt azcache.AzureCacheReadType) (*virtualmachine.VirtualMachine, bool, error) {
 		var found bool
-		virtualMachines, err := ss.getVMSSVMsFromCache(node.resourceGroup, node.vmssName, crt)
+		cached, err := cache.Get(cacheKey, crt)
 		if err != nil {
 			return nil, found, err
 		}
-
-		if entry, ok := virtualMachines.Load(node.nodeName); ok {
+		virtualMachines := cached.(*sync.Map)
+		if entry, ok := virtualMachines.Load(nodeName); ok {
 			result := entry.(*VMSSVirtualMachinesEntry)
 			if result.VirtualMachine == nil {
-				klog.Warningf("VM is nil on Node %q, VM is in deleting state", node.nodeName)
+				klog.Warningf("VM is nil on Node %q, VM is in deleting state", nodeName)
 				return nil, true, nil
 			}
 			found = true
@@ -205,24 +199,29 @@ func (ss *ScaleSet) getVmssVMByNodeIdentity(node *nodeIdentity, crt azcache.Azur
 		return nil, found, nil
 	}
 
-	vm, found, err := getter(crt)
+	// FIXME(ccc): check only if vmss is uniform.
+	_, err = getScaleSetVMInstanceID(node.nodeName)
+	if err != nil {
+		return nil, err
+	}
+
+	vm, found, err := getter(node.nodeName, crt)
 	if err != nil {
 		return nil, err
 	}
 
 	if !found {
-		cacheKey := getVMSSVMCacheKey(node.resourceGroup, node.vmssName)
 		// lock and try find nodeName from cache again, refresh cache if still not found
 		ss.lockMap.LockEntry(cacheKey)
 		defer ss.lockMap.UnlockEntry(cacheKey)
-		vm, found, err = getter(crt)
+		vm, found, err = getter(node.nodeName, crt)
 		if err == nil && found && vm != nil {
 			klog.V(2).Infof("found VMSS VM with nodeName %s after retry", node.nodeName)
 			return vm, nil
 		}
 
 		klog.V(2).Infof("Couldn't find VMSS VM with nodeName %s, refreshing the cache(vmss: %s, rg: %s)", node.nodeName, node.vmssName, node.resourceGroup)
-		vm, found, err = getter(azcache.CacheReadTypeForceRefresh)
+		vm, found, err = getter(node.nodeName, azcache.CacheReadTypeForceRefresh)
 		if err != nil {
 			return nil, err
 		}
@@ -322,12 +321,18 @@ func (ss *ScaleSet) GetProvisioningStateByNodeName(name string) (provisioningSta
 // getCachedVirtualMachineByInstanceID gets scaleSetVMInfo from cache.
 // The node must belong to one of scale sets.
 func (ss *ScaleSet) getVmssVMByInstanceID(resourceGroup, scaleSetName, instanceID string, crt azcache.AzureCacheReadType) (*compute.VirtualMachineScaleSetVM, error) {
+	cacheKey, cache, err := ss.getVMSSVMCache(resourceGroup, scaleSetName)
+	if err != nil {
+		return nil, err
+	}
+
 	getter := func(crt azcache.AzureCacheReadType) (vm *compute.VirtualMachineScaleSetVM, found bool, err error) {
-		virtualMachines, err := ss.getVMSSVMsFromCache(resourceGroup, scaleSetName, crt)
+		cached, err := cache.Get(cacheKey, crt)
 		if err != nil {
 			return nil, false, err
 		}
 
+		virtualMachines := cached.(*sync.Map)
 		virtualMachines.Range(func(key, value interface{}) bool {
 			vmEntry := value.(*VMSSVirtualMachinesEntry)
 			if strings.EqualFold(vmEntry.ResourceGroup, resourceGroup) &&
@@ -1733,9 +1738,8 @@ func (ss *ScaleSet) ensureBackendPoolDeletedFromVMSS(backendPoolID, vmSetName st
 }
 
 func (ss *ScaleSet) ensureBackendPoolDeletedFromVmssUniform(backendPoolID, vmSetName string) error {
-	klog.V(2).Infof("ensureBackendPoolDeletedFromVmssUniform: vmSetName (%s), backendPoolID (%s)", vmSetName, backendPoolID)
-
 	vmssNamesMap := make(map[string]bool)
+
 	// the standard load balancer supports multiple vmss in its backend while the basic sku doesn't
 	if ss.useStandardLoadBalancer() && !ss.EnableMultipleStandardLoadBalancers {
 		cachedUniform, err := ss.vmssCache.Get(consts.VMSSKey, azcache.CacheReadTypeDefault)
@@ -1753,7 +1757,6 @@ func (ss *ScaleSet) ensureBackendPoolDeletedFromVmssUniform(backendPoolID, vmSet
 			} else if v, ok := value.(*compute.VirtualMachineScaleSet); ok {
 				vmss = v
 			}
-			klog.V(2).Infof("ensureBackendPoolDeletedFromVmssUniform: vmss (%s)", to.String(vmss.Name))
 
 			// When vmss is being deleted, CreateOrUpdate API would report "the vmss is being deleted" error.
 			// Since it is being deleted, we shouldn't send more CreateOrUpdate requests for it.
@@ -1787,7 +1790,6 @@ func (ss *ScaleSet) ensureBackendPoolDeletedFromVmssUniform(backendPoolID, vmSet
 				loadBalancerBackendAddressPools = *primaryIPConfig.LoadBalancerBackendAddressPools
 			}
 			for _, loadBalancerBackendAddressPool := range loadBalancerBackendAddressPools {
-				klog.V(4).Infof("ensureBackendPoolDeletedFromVMSS: loadBalancerBackendAddressPool (%s) on vmss (%s)", to.String(loadBalancerBackendAddressPool.ID), to.String(vmss.Name))
 				if strings.EqualFold(to.String(loadBalancerBackendAddressPool.ID), backendPoolID) {
 					klog.V(4).Infof("ensureBackendPoolDeletedFromVMSS: found vmss %s with backend pool %s, removing it", to.String(vmss.Name), backendPoolID)
 					vmssNamesMap[to.String(vmss.Name)] = true
diff --git a/pkg/provider/azure_vmss_cache.go b/pkg/provider/azure_vmss_cache.go
index 44ec739fc..62ec948e9 100644
--- a/pkg/provider/azure_vmss_cache.go
+++ b/pkg/provider/azure_vmss_cache.go
@@ -104,13 +104,19 @@ func (ss *ScaleSet) newVMSSCache() (*azcache.TimedCache, error) {
 
 		if resourceGroupNotFound {
 			// gc vmss vm cache when there is resource group not found
-			vmssVMKeys := ss.vmssVMCache.Store.ListKeys()
-			for _, cacheKey := range vmssVMKeys {
-				vmssName := cacheKey[strings.LastIndex(cacheKey, "/")+1:]
-				if _, ok := localCache.Load(vmssName); !ok {
-					klog.V(2).Infof("remove vmss %s from vmssVMCache due to rg not found", cacheKey)
-					_ = ss.vmssVMCache.Delete(cacheKey)
+			removed := map[string]bool{}
+			ss.vmssVMCache.Range(func(key, value interface{}) bool {
+				cacheKey := key.(string)
+				vlistIdx := cacheKey[strings.LastIndex(cacheKey, "/")+1:]
+				if _, ok := localCache.Load(vlistIdx); !ok {
+					klog.V(2).Infof("remove vmss %s from cache due to rg not found", cacheKey)
+					removed[cacheKey] = true
 				}
+				return true
+			})
+
+			for key := range removed {
+				ss.vmssVMCache.Delete(key)
 			}
 		}
 		return localCache, nil
@@ -136,20 +142,25 @@ func extractVmssVMName(name string) (string, string, error) {
 	return ssName, instanceID, nil
 }
 
-func (ss *ScaleSet) getVMSSVMsFromCache(resourceGroup, vmssName string, crt azcache.AzureCacheReadType) (*sync.Map, error) {
-	cacheKey := getVMSSVMCacheKey(resourceGroup, vmssName)
-	entry, err := ss.vmssVMCache.Get(cacheKey, crt)
-	if err != nil {
-		return nil, err
+// getVMSSVMCache returns an *azcache.TimedCache and cache key for a VMSS (creating that cache if new).
+func (ss *ScaleSet) getVMSSVMCache(resourceGroup, vmssName string) (string, *azcache.TimedCache, error) {
+	cacheKey := strings.ToLower(fmt.Sprintf("%s/%s", resourceGroup, vmssName))
+	if entry, ok := ss.vmssVMCache.Load(cacheKey); ok {
+		return cacheKey, entry.(*azcache.TimedCache), nil
 	}
 
-	if entry == nil {
-		err = fmt.Errorf("vmssVMCache entry for resourceGroup (%s), vmssName (%s) returned nil data", resourceGroup, vmssName)
-		return nil, err
+	v, err, _ := ss.group.Do(cacheKey, func() (interface{}, error) {
+		cache, err := ss.newVMSSVirtualMachinesCache(resourceGroup, vmssName, cacheKey)
+		if err != nil {
+			return nil, err
+		}
+		ss.vmssVMCache.Store(cacheKey, cache)
+		return cache, nil
+	})
+	if err != nil {
+		return "", nil, err
 	}
-
-	virtualMachines := entry.(*sync.Map)
-	return virtualMachines, nil
+	return cacheKey, v.(*azcache.TimedCache), nil
 }
 
 // gcVMSSVMCache delete stale VMSS VMs caches from deleted VMSSes.
@@ -158,37 +169,33 @@ func (ss *ScaleSet) gcVMSSVMCache() error {
 }
 
 // newVMSSVirtualMachinesCache instantiates a new VMs cache for VMs belonging to the provided VMSS.
-func (ss *ScaleSet) newVMSSVirtualMachinesCache() (*azcache.TimedCache, error) {
+func (ss *ScaleSet) newVMSSVirtualMachinesCache(resourceGroupName, vmssName, cacheKey string) (*azcache.TimedCache, error) {
 	vmssVirtualMachinesCacheTTL := time.Duration(ss.Config.VmssVirtualMachinesCacheTTLInSeconds) * time.Second
 
-	getter := func(cacheKey string) (interface{}, error) {
+	getter := func(key string) (interface{}, error) {
 		localCache := &sync.Map{} // [nodeName]*VMSSVirtualMachinesEntry
 
-		oldCache := make(map[string]*VMSSVirtualMachinesEntry)
+		oldCache := make(map[string]VMSSVirtualMachinesEntry)
 
-		entry, exists, err := ss.vmssVMCache.Store.GetByKey(cacheKey)
-		if err != nil {
-			return nil, err
-		}
-		if exists {
-			cached := entry.(*azcache.AzureCacheEntry).Data
-			if cached != nil {
-				virtualMachines := cached.(*sync.Map)
-				virtualMachines.Range(func(key, value interface{}) bool {
-					oldCache[key.(string)] = value.(*VMSSVirtualMachinesEntry)
-					return true
-				})
+		if vmssCache, ok := ss.vmssVMCache.Load(cacheKey); ok {
+			// get old cache before refreshing the cache
+			cache := vmssCache.(*azcache.TimedCache)
+			entry, exists, err := cache.Store.GetByKey(cacheKey)
+			if err != nil {
+				return nil, err
+			}
+			if exists {
+				cached := entry.(*azcache.AzureCacheEntry).Data
+				if cached != nil {
+					virtualMachines := cached.(*sync.Map)
+					virtualMachines.Range(func(key, value interface{}) bool {
+						oldCache[key.(string)] = *value.(*VMSSVirtualMachinesEntry)
+						return true
+					})
+				}
 			}
 		}
 
-		result := strings.Split(cacheKey, "/")
-		if len(result) < 2 {
-			err = fmt.Errorf("Invalid cacheKey (%s)", cacheKey)
-			return nil, err
-		}
-
-		resourceGroupName, vmssName := result[0], result[1]
-
 		vms, err := ss.listScaleSetVMs(vmssName, resourceGroupName)
 		if err != nil {
 			return nil, err
@@ -280,9 +287,15 @@ func (ss *ScaleSet) DeleteCacheForNode(nodeName string) error {
 		return err
 	}
 
-	err = ss.vmssVMCache.Delete(getVMSSVMCacheKey(node.resourceGroup, node.vmssName))
+	cacheKey, timedcache, err := ss.getVMSSVMCache(node.resourceGroup, node.vmssName)
 	if err != nil {
-		klog.Errorf("DeleteCacheForNode(%s) failed to remove from vmssVMCache with error: %v", nodeName, err)
+		klog.Errorf("DeleteCacheForNode(%s) failed with error: %v", nodeName, err)
+		return err
+	}
+
+	err = timedcache.Delete(cacheKey)
+	if err != nil {
+		klog.Errorf("DeleteCacheForNode(%s) failed with error: %v", nodeName, err)
 		return err
 	}
 
@@ -295,9 +308,9 @@ func (ss *ScaleSet) DeleteCacheForNode(nodeName string) error {
 }
 
 func (ss *ScaleSet) updateCache(nodeName, resourceGroupName, vmssName, instanceID string, updatedVM *compute.VirtualMachineScaleSetVM) error {
-	virtualMachines, err := ss.getVMSSVMsFromCache(resourceGroupName, vmssName, azcache.CacheReadTypeUnsafe)
+	cacheKey, timedCache, err := ss.getVMSSVMCache(resourceGroupName, vmssName)
 	if err != nil {
-		err = fmt.Errorf("updateCache(%s, %s, %s) failed getting vmCache with error: %v", vmssName, resourceGroupName, nodeName, err)
+		klog.Errorf("updateCache(%s, %s, %s) failed with error: %v", vmssName, resourceGroupName, nodeName, err)
 		return err
 	}
 
@@ -313,6 +326,14 @@ func (ss *ScaleSet) updateCache(nodeName, resourceGroupName, vmssName, instanceI
 
 	localCache.Store(nodeName, vmssVMCacheEntry)
 
+	vmCache, err := timedCache.Get(cacheKey, azcache.CacheReadTypeUnsafe)
+	if err != nil {
+		return err
+	}
+	if vmCache == nil {
+		return fmt.Errorf("nil vmCache")
+	}
+	virtualMachines := vmCache.(*sync.Map)
 	virtualMachines.Range(func(key, value interface{}) bool {
 		if key.(string) != nodeName {
 			localCache.Store(key.(string), value.(*VMSSVirtualMachinesEntry))
@@ -320,8 +341,7 @@ func (ss *ScaleSet) updateCache(nodeName, resourceGroupName, vmssName, instanceI
 		return true
 	})
 
-	cacheKey := getVMSSVMCacheKey(resourceGroupName, vmssName)
-	ss.vmssVMCache.Update(cacheKey, localCache)
+	timedCache.Update(cacheKey, localCache)
 	klog.V(4).Infof("updateCache(%s, %s, %s) for cacheKey(%s) updated successfully", vmssName, resourceGroupName, nodeName, cacheKey)
 	return nil
 }
diff --git a/site/package-lock.json b/site/package-lock.json
index b259169f1..33df650be 100644
--- a/site/package-lock.json
+++ b/site/package-lock.json
@@ -46,9 +46,9 @@
       }
     },
     "anymatch": {
-      "version": "3.1.3",
-      "resolved": "https://registry.npmjs.org/anymatch/-/anymatch-3.1.3.tgz",
-      "integrity": "sha512-KMReFUr0B4t+D+OBkjR3KYqvocp2XaSzO55UcB6mgQMd3KbcE+mWTyvVV7D/zsdEbNnV6acZUutkiHQXvTr1Rw==",
+      "version": "3.1.2",
+      "resolved": "https://registry.npmjs.org/anymatch/-/anymatch-3.1.2.tgz",
+      "integrity": "sha512-P43ePfOAIupkguHUycrc4qJ9kz8ZiuOUijaETwX7THt0Y/GNK7v0aa8rY816xWjZ7rJdA5XdMcpVFTKMq+RvWg==",
       "dev": true,
       "requires": {
         "normalize-path": "^3.0.0",
@@ -119,13 +119,13 @@
       }
     },
     "cliui": {
-      "version": "8.0.1",
-      "resolved": "https://registry.npmjs.org/cliui/-/cliui-8.0.1.tgz",
-      "integrity": "sha512-BSeNnyus75C4//NQ9gQt1/csTXyo/8Sb+afLAkzAptFuMsod9HFokGNudZpi/oQV73hnVK+sR+5PVRMd+Dr7YQ==",
+      "version": "7.0.4",
+      "resolved": "https://registry.npmjs.org/cliui/-/cliui-7.0.4.tgz",
+      "integrity": "sha512-OcRE68cOsVMXp1Yvonl/fzkQOyjLSu/8bhPDfQt0e0/Eb283TKP20Fs2MqoPsr9SwA595rRCA+QMzYc9nBP+JQ==",
       "dev": true,
       "requires": {
         "string-width": "^4.2.0",
-        "strip-ansi": "^6.0.1",
+        "strip-ansi": "^6.0.0",
         "wrap-ansi": "^7.0.0"
       }
     },
@@ -178,9 +178,9 @@
       "dev": true
     },
     "fast-glob": {
-      "version": "3.2.12",
-      "resolved": "https://registry.npmjs.org/fast-glob/-/fast-glob-3.2.12.tgz",
-      "integrity": "sha512-DVj4CQIYYow0BlaelwK1pHl5n5cRSJfM60UA0zK891sVInoPri2Ekj7+e1CT3/3qxXenpI+nBBmQAcJPJgaj4w==",
+      "version": "3.2.11",
+      "resolved": "https://registry.npmjs.org/fast-glob/-/fast-glob-3.2.11.tgz",
+      "integrity": "sha512-xrO3+1bxSo3ZVHAnqzyuewYT6aMFHRAd4Kcs92MAonjwQZLsK9d0SF1IyQ3k5PoirxTW0Oe/RqFgMQ6TcNE5Ew==",
       "dev": true,
       "requires": {
         "@nodelib/fs.stat": "^2.0.2",
@@ -215,9 +215,9 @@
       "dev": true
     },
     "fs-extra": {
-      "version": "11.1.0",
-      "resolved": "https://registry.npmjs.org/fs-extra/-/fs-extra-11.1.0.tgz",
-      "integrity": "sha512-0rcTq621PD5jM/e0a3EJoGC/1TC5ZBCERW82LQuwfGnCa1V8w7dpYH1yNu+SLb6E5dkeCBzKEyLGlFrnr+dUyw==",
+      "version": "10.1.0",
+      "resolved": "https://registry.npmjs.org/fs-extra/-/fs-extra-10.1.0.tgz",
+      "integrity": "sha512-oRXApq54ETRj4eMiFzGnHWGy+zo5raudjuxN0b8H7s/RU2oW0Wvsx9O0ACRN/kRq9E8Vu/ReskGB5o3ji+FzHQ==",
       "dev": true,
       "requires": {
         "graceful-fs": "^4.2.0",
@@ -264,14 +264,6 @@
         "ignore": "^5.2.0",
         "merge2": "^1.4.1",
         "slash": "^4.0.0"
-      },
-      "dependencies": {
-        "slash": {
-          "version": "4.0.0",
-          "resolved": "https://registry.npmjs.org/slash/-/slash-4.0.0.tgz",
-          "integrity": "sha512-3dOsAHXXUkQTpOYcoAxLIorMTp4gIQr5IW3iVb7A7lFIp0VHhnynm9izx6TssdrIcVIESAlVjtnO2K8bg+Coew==",
-          "dev": true
-        }
       }
     },
     "graceful-fs": {
@@ -281,9 +273,9 @@
       "dev": true
     },
     "ignore": {
-      "version": "5.2.1",
-      "resolved": "https://registry.npmjs.org/ignore/-/ignore-5.2.1.tgz",
-      "integrity": "sha512-d2qQLzTJ9WxQftPAuEQpSPmKqzxePjzVbpAVv62AQ64NTL+wR4JkrVqR/LqFsFEUsHDAiId52mJteHDFuDkElA==",
+      "version": "5.2.0",
+      "resolved": "https://registry.npmjs.org/ignore/-/ignore-5.2.0.tgz",
+      "integrity": "sha512-CmxgYGiEPCLhfLnpPp1MoRmifwEIOgjcHXxOBjv7mY96c+eWScsOP9c112ZyLdWHi0FxHjI+4uVhKYp/gcdRmQ==",
       "dev": true
     },
     "is-binary-path": {
@@ -333,9 +325,9 @@
       }
     },
     "lilconfig": {
-      "version": "2.0.6",
-      "resolved": "https://registry.npmjs.org/lilconfig/-/lilconfig-2.0.6.tgz",
-      "integrity": "sha512-9JROoBW7pobfsx+Sq2JsASvCo6Pfo6WWoUW79HuB1BCoBXD4PLWJPqDF6fNj67pqBYTbAHkE57M1kS/+L1neOg==",
+      "version": "2.0.5",
+      "resolved": "https://registry.npmjs.org/lilconfig/-/lilconfig-2.0.5.tgz",
+      "integrity": "sha512-xaYmXZtTHPAw5m+xLN8ab9C+3a8YmV3asNSPOATITbtwrfbwaLJj8h66H1WMIpALCkqsIzK3h7oQ+PdX+LQ9Eg==",
       "dev": true
     },
     "merge2": {
@@ -397,14 +389,14 @@
       "dev": true
     },
     "postcss-cli": {
-      "version": "10.1.0",
-      "resolved": "https://registry.npmjs.org/postcss-cli/-/postcss-cli-10.1.0.tgz",
-      "integrity": "sha512-Zu7PLORkE9YwNdvOeOVKPmWghprOtjFQU3srMUGbdz3pHJiFh7yZ4geiZFMkjMfB0mtTFR3h8RemR62rPkbOPA==",
+      "version": "10.0.0",
+      "resolved": "https://registry.npmjs.org/postcss-cli/-/postcss-cli-10.0.0.tgz",
+      "integrity": "sha512-Wjy/00wBBEgQqnSToznxLWDnATznokFGXsHtF/3G8glRZpz5KYlfHcBW/VMJmWAeF2x49zjgy4izjM3/Wx1dKA==",
       "dev": true,
       "requires": {
         "chokidar": "^3.3.0",
         "dependency-graph": "^0.11.0",
-        "fs-extra": "^11.0.0",
+        "fs-extra": "^10.0.0",
         "get-stdin": "^9.0.0",
         "globby": "^13.0.0",
         "picocolors": "^1.0.0",
@@ -412,7 +404,7 @@
         "postcss-reporter": "^7.0.0",
         "pretty-hrtime": "^1.0.3",
         "read-cache": "^1.0.0",
-        "slash": "^5.0.0",
+        "slash": "^4.0.0",
         "yargs": "^17.0.0"
       }
     },
@@ -494,9 +486,9 @@
       }
     },
     "slash": {
-      "version": "5.0.0",
-      "resolved": "https://registry.npmjs.org/slash/-/slash-5.0.0.tgz",
-      "integrity": "sha512-n6KkmvKS0623igEVj3FF0OZs1gYYJ0o0Hj939yc1fyxl2xt+xYpLnzJB6xBSqOfV9ZFLEWodBBN/heZJahuIJQ==",
+      "version": "4.0.0",
+      "resolved": "https://registry.npmjs.org/slash/-/slash-4.0.0.tgz",
+      "integrity": "sha512-3dOsAHXXUkQTpOYcoAxLIorMTp4gIQr5IW3iVb7A7lFIp0VHhnynm9izx6TssdrIcVIESAlVjtnO2K8bg+Coew==",
       "dev": true
     },
     "string-width": {
@@ -568,30 +560,30 @@
       "dev": true
     },
     "yaml": {
-      "version": "2.1.3",
-      "resolved": "https://registry.npmjs.org/yaml/-/yaml-2.1.3.tgz",
-      "integrity": "sha512-AacA8nRULjKMX2DvWvOAdBZMOfQlypSFkjcOcu9FalllIDJ1kvlREzcdIZmidQUqqeMv7jorHjq2HlLv/+c2lg==",
+      "version": "2.1.1",
+      "resolved": "https://registry.npmjs.org/yaml/-/yaml-2.1.1.tgz",
+      "integrity": "sha512-o96x3OPo8GjWeSLF+wOAbrPfhFOGY0W00GNaxCDv+9hkcDJEnev1yh8S7pgHF0ik6zc8sQLuL8hjHjJULZp8bw==",
       "dev": true
     },
     "yargs": {
-      "version": "17.6.2",
-      "resolved": "https://registry.npmjs.org/yargs/-/yargs-17.6.2.tgz",
-      "integrity": "sha512-1/9UrdHjDZc0eOU0HxOHoS78C69UD3JRMvzlJ7S79S2nTaWRA/whGCTV8o9e/N/1Va9YIV7Q4sOxD8VV4pCWOw==",
+      "version": "17.5.1",
+      "resolved": "https://registry.npmjs.org/yargs/-/yargs-17.5.1.tgz",
+      "integrity": "sha512-t6YAJcxDkNX7NFYiVtKvWUz8l+PaKTLiL63mJYWR2GnHq2gjEWISzsLp9wg3aY36dY1j+gfIEL3pIF+XlJJfbA==",
       "dev": true,
       "requires": {
-        "cliui": "^8.0.1",
+        "cliui": "^7.0.2",
         "escalade": "^3.1.1",
         "get-caller-file": "^2.0.5",
         "require-directory": "^2.1.1",
         "string-width": "^4.2.3",
         "y18n": "^5.0.5",
-        "yargs-parser": "^21.1.1"
+        "yargs-parser": "^21.0.0"
       }
     },
     "yargs-parser": {
-      "version": "21.1.1",
-      "resolved": "https://registry.npmjs.org/yargs-parser/-/yargs-parser-21.1.1.tgz",
-      "integrity": "sha512-tVpsJW7DdjecAiFpbIB1e3qxIQsE6NoPc5/eTdrbbIC4h0LVsWhnoa3g+m2HclBIujHzsxZ4VJVA+GUuc2/LBw==",
+      "version": "21.0.1",
+      "resolved": "https://registry.npmjs.org/yargs-parser/-/yargs-parser-21.0.1.tgz",
+      "integrity": "sha512-9BK1jFpLzJROCI5TzwZL/TU4gqjK5xiHV/RfWLOahrjAko/e4DJkRDZQXfvqAsiZzzYhgAzbgz6lg48jcm4GLg==",
       "dev": true
     }
   }
diff --git a/site/package.json b/site/package.json
index 7efdc55f4..1e469e381 100644
--- a/site/package.json
+++ b/site/package.json
@@ -19,6 +19,6 @@
   "dependencies": {},
   "devDependencies": {
     "autoprefixer": "^10.4.13",
-    "postcss-cli": "^10.1.0"
+    "postcss-cli": "^10.0.0"
   }
 }


Test patch:
diff --git a/pkg/provider/azure_controller_vmss_test.go b/pkg/provider/azure_controller_vmss_test.go
index 48b038c5c..d14779067 100644
--- a/pkg/provider/azure_controller_vmss_test.go
+++ b/pkg/provider/azure_controller_vmss_test.go
@@ -19,7 +19,6 @@ package provider
 import (
 	"fmt"
 	"net/http"
-	"strings"
 	"testing"
 
 	"github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2022-03-01/compute"
@@ -219,7 +218,7 @@ func TestDetachDiskWithVMSS(t *testing.T) {
 	}
 
 	for i, test := range testCases {
-		scaleSetName := strings.ToLower(string(test.vmssName))
+		scaleSetName := string(test.vmssName)
 		ss, err := NewTestScaleSet(ctrl)
 		assert.NoError(t, err, test.desc)
 		testCloud := ss.cloud
@@ -265,7 +264,7 @@ func TestDetachDiskWithVMSS(t *testing.T) {
 		}
 		mockVMSSVMClient := testCloud.VirtualMachineScaleSetVMsClient.(*mockvmssvmclient.MockInterface)
 		mockVMSSVMClient.EXPECT().List(gomock.Any(), testCloud.ResourceGroup, scaleSetName, gomock.Any()).Return(expectedVMSSVMs, nil).AnyTimes()
-		if scaleSetName == strings.ToLower(string(fakeStatusNotFoundVMSSName)) {
+		if scaleSetName == string(fakeStatusNotFoundVMSSName) {
 			mockVMSSVMClient.EXPECT().Update(gomock.Any(), testCloud.ResourceGroup, scaleSetName, gomock.Any(), gomock.Any(), gomock.Any()).Return(updatedVMSSVM, &retry.Error{HTTPStatusCode: http.StatusNotFound, RawError: cloudprovider.InstanceNotFound}).AnyTimes()
 		} else {
 			mockVMSSVMClient.EXPECT().Update(gomock.Any(), testCloud.ResourceGroup, scaleSetName, gomock.Any(), gomock.Any(), gomock.Any()).Return(updatedVMSSVM, nil).AnyTimes()
@@ -349,7 +348,7 @@ func TestUpdateVMWithVMSS(t *testing.T) {
 	}
 
 	for i, test := range testCases {
-		scaleSetName := strings.ToLower(string(test.vmssName))
+		scaleSetName := string(test.vmssName)
 		ss, err := NewTestScaleSet(ctrl)
 		assert.NoError(t, err, test.desc)
 		testCloud := ss.cloud
@@ -386,7 +385,7 @@ func TestUpdateVMWithVMSS(t *testing.T) {
 		}
 		mockVMSSVMClient := testCloud.VirtualMachineScaleSetVMsClient.(*mockvmssvmclient.MockInterface)
 		mockVMSSVMClient.EXPECT().List(gomock.Any(), testCloud.ResourceGroup, scaleSetName, gomock.Any()).Return(expectedVMSSVMs, nil).AnyTimes()
-		if scaleSetName == strings.ToLower(string(fakeStatusNotFoundVMSSName)) {
+		if scaleSetName == string(fakeStatusNotFoundVMSSName) {
 			mockVMSSVMClient.EXPECT().Update(gomock.Any(), testCloud.ResourceGroup, scaleSetName, gomock.Any(), gomock.Any(), gomock.Any()).Return(updatedVMSSVM, &retry.Error{HTTPStatusCode: http.StatusNotFound, RawError: cloudprovider.InstanceNotFound}).AnyTimes()
 		} else {
 			mockVMSSVMClient.EXPECT().Update(gomock.Any(), testCloud.ResourceGroup, scaleSetName, gomock.Any(), gomock.Any(), gomock.Any()).Return(updatedVMSSVM, nil).AnyTimes()
diff --git a/pkg/provider/azure_controller_vmssflex_test.go b/pkg/provider/azure_controller_vmssflex_test.go
index a55351190..e6196663e 100644
--- a/pkg/provider/azure_controller_vmssflex_test.go
+++ b/pkg/provider/azure_controller_vmssflex_test.go
@@ -43,7 +43,6 @@ func TestAttachDiskWithVmssFlex(t *testing.T) {
 		description                    string
 		nodeName                       types.NodeName
 		vmName                         string
-		inconsistentLUN                bool
 		testVMListWithoutInstanceView  []compute.VirtualMachine
 		testVMListWithOnlyInstanceView []compute.VirtualMachine
 		vmListErr                      error
@@ -79,17 +78,6 @@ func TestAttachDiskWithVmssFlex(t *testing.T) {
 			vmssFlexVMUpdateError:          &retry.Error{HTTPStatusCode: http.StatusNotFound, RawError: cloudprovider.InstanceNotFound},
 			expectedErr:                    fmt.Errorf("Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: instance not found"),
 		},
-		{
-			description:                    "error should be returned when disk lun is inconsistent",
-			nodeName:                       types.NodeName(testVM1Spec.ComputerName),
-			vmName:                         testVM1Spec.VMName,
-			inconsistentLUN:                true,
-			testVMListWithoutInstanceView:  testVMListWithoutInstanceView,
-			testVMListWithOnlyInstanceView: testVMListWithOnlyInstanceView,
-			vmListErr:                      nil,
-			vmssFlexVMUpdateError:          nil,
-			expectedErr:                    fmt.Errorf("disk(uri) already attached to node(vmssflex1000001) on LUN(1), but target LUN is 63"),
-		},
 	}
 
 	for _, tc := range testCases {
@@ -106,15 +94,12 @@ func TestAttachDiskWithVmssFlex(t *testing.T) {
 		mockVMClient.EXPECT().UpdateAsync(gomock.Any(), gomock.Any(), tc.vmName, gomock.Any(), gomock.Any()).Return(nil, tc.vmssFlexVMUpdateError).AnyTimes()
 
 		options := AttachDiskOptions{
-			lun:                     1,
-			diskName:                "diskname",
+			lun:                     0,
+			diskName:                "",
 			cachingMode:             compute.CachingTypesReadOnly,
 			diskEncryptionSetID:     "",
 			writeAcceleratorEnabled: false,
 		}
-		if tc.inconsistentLUN {
-			options.lun = 63
-		}
 		diskMap := map[string]*AttachDiskOptions{
 			"uri": &options,
 		}
@@ -296,9 +281,8 @@ func TestGetDataDisksWithVmssFlex(t *testing.T) {
 			vmListErr:                      nil,
 			expectedDataDisks: []compute.DataDisk{
 				{
-					Lun:         to.Int32Ptr(1),
-					Name:        to.StringPtr("dataDisktestvm1"),
-					ManagedDisk: &compute.ManagedDiskParameters{ID: to.StringPtr("uri")},
+					Lun:  to.Int32Ptr(1),
+					Name: to.StringPtr("dataDisktestvm1"),
 				},
 			},
 			expectedErr: nil,
diff --git a/pkg/provider/azure_loadbalancer_backendpool_test.go b/pkg/provider/azure_loadbalancer_backendpool_test.go
index 0ebc64705..ee123e166 100644
--- a/pkg/provider/azure_loadbalancer_backendpool_test.go
+++ b/pkg/provider/azure_loadbalancer_backendpool_test.go
@@ -17,7 +17,6 @@ limitations under the License.
 package provider
 
 import (
-	"errors"
 	"fmt"
 	"strings"
 	"testing"
@@ -31,7 +30,6 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/util/sets"
 	"k8s.io/client-go/kubernetes/fake"
-	cloudprovider "k8s.io/cloud-provider"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/loadbalancerclient/mockloadbalancerclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
 	"sigs.k8s.io/cloud-provider-azure/pkg/retry"
@@ -297,38 +295,6 @@ func TestReconcileBackendPoolsNodeIPConfig(t *testing.T) {
 	assert.True(t, changed)
 }
 
-func TestReconcileBackendPoolsNodeIPConfigRemoveIPConfig(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	lb := buildDefaultTestLB(testClusterName, []string{
-		"/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool1-00000000-nic-1/ipConfigurations/ipconfig1",
-		"/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool2-00000000-nic-1/ipConfigurations/ipconfig1",
-		"/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool3-00000000-nic-1/ipConfigurations/ipconfig1",
-	})
-
-	mockVMSet := NewMockVMSet(ctrl)
-	mockVMSet.EXPECT().GetNodeNameByIPConfigurationID("/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool1-00000000-nic-1/ipConfigurations/ipconfig1").Return("k8s-agentpool1-00000000", "", nil)
-	mockVMSet.EXPECT().GetNodeNameByIPConfigurationID("/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool2-00000000-nic-1/ipConfigurations/ipconfig1").Return("k8s-agentpool2-00000000", "", nil)
-	mockVMSet.EXPECT().GetNodeNameByIPConfigurationID("/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool3-00000000-nic-1/ipConfigurations/ipconfig1").Return("", "", cloudprovider.InstanceNotFound)
-	mockVMSet.EXPECT().EnsureBackendPoolDeleted(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(false, nil)
-	mockVMSet.EXPECT().GetPrimaryVMSetName().Return("k8s-agentpool1-00000000").Times(2)
-
-	az := GetTestCloud(ctrl)
-	az.VMSet = mockVMSet
-	az.nodeInformerSynced = func() bool { return true }
-	az.excludeLoadBalancerNodes = sets.NewString("k8s-agentpool1-00000000")
-
-	bc := newBackendPoolTypeNodeIPConfig(az)
-	svc := getTestService("test", v1.ProtocolTCP, nil, false, 80)
-	_, _, err := bc.ReconcileBackendPools(testClusterName, &svc, &lb)
-	assert.NoError(t, err)
-
-	mockVMSet.EXPECT().GetNodeNameByIPConfigurationID("/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool1-00000000-nic-1/ipConfigurations/ipconfig1").Return("k8s-agentpool1-00000000", "", errors.New("error"))
-	_, _, err = bc.ReconcileBackendPools(testClusterName, &svc, &lb)
-	assert.Equal(t, "error", err.Error())
-}
-
 func TestReconcileBackendPoolsNodeIPConfigPreConfigured(t *testing.T) {
 	ctrl := gomock.NewController(t)
 	defer ctrl.Finish()
diff --git a/pkg/provider/azure_utils_test.go b/pkg/provider/azure_utils_test.go
index 7308269cf..ff1a2c530 100644
--- a/pkg/provider/azure_utils_test.go
+++ b/pkg/provider/azure_utils_test.go
@@ -317,62 +317,29 @@ func TestRemoveDuplicatedSecurityRules(t *testing.T) {
 	}
 }
 
-func TestGetVMSSVMCacheKey(t *testing.T) {
-	tests := []struct {
-		description       string
-		resourceGroupName string
-		vmssName          string
-		cacheKey          string
-	}{
-		{
-			description:       "Resource group and Vmss Name are in lower case",
-			resourceGroupName: "resgrp",
-			vmssName:          "vmss",
-			cacheKey:          "resgrp/vmss",
-		},
-		{
-			description:       "Resource group has upper case and Vmss Name is in lower case",
-			resourceGroupName: "Resgrp",
-			vmssName:          "vmss",
-			cacheKey:          "resgrp/vmss",
-		},
-		{
-			description:       "Resource group is in lower case and Vmss Name has upper case",
-			resourceGroupName: "resgrp",
-			vmssName:          "Vmss",
-			cacheKey:          "resgrp/vmss",
-		},
-		{
-			description:       "Resource group and Vmss Name are both in upper case",
-			resourceGroupName: "Resgrp",
-			vmssName:          "Vmss",
-			cacheKey:          "resgrp/vmss",
-		},
-	}
-
-	for _, test := range tests {
-		result := getVMSSVMCacheKey(test.resourceGroupName, test.vmssName)
-		assert.Equal(t, result, test.cacheKey, test.description)
-	}
-}
-
 func TestIsNodeInVMSSVMCache(t *testing.T) {
+	emptyTimedCacheMap := &sync.Map{}
+	emptyTimedCacheMap.Store("key", &azcache.TimedCache{})
 
 	getter := func(key string) (interface{}, error) {
 		return nil, nil
 	}
 	emptyCacheEntryTimedCache, _ := azcache.NewTimedcache(fakeCacheTTL, getter)
 	emptyCacheEntryTimedCache.Set("key", nil)
+	emptyCacheEntryTimedCacheMap := &sync.Map{}
+	emptyCacheEntryTimedCacheMap.Store("key", emptyCacheEntryTimedCache)
 
 	cacheEntryTimedCache, _ := azcache.NewTimedcache(fakeCacheTTL, getter)
 	syncMap := &sync.Map{}
 	syncMap.Store("node", nil)
 	cacheEntryTimedCache.Set("key", syncMap)
+	cacheEntryTimedCacheMap := &sync.Map{}
+	cacheEntryTimedCacheMap.Store("key", cacheEntryTimedCache)
 
 	tests := []struct {
 		description    string
 		nodeName       string
-		vmssVMCache    *azcache.TimedCache
+		vmssVMCache    *sync.Map
 		expectedResult bool
 	}{
 		{
@@ -380,21 +347,31 @@ func TestIsNodeInVMSSVMCache(t *testing.T) {
 			vmssVMCache:    nil,
 			expectedResult: false,
 		},
+		{
+			description:    "empty cache",
+			vmssVMCache:    &sync.Map{},
+			expectedResult: false,
+		},
+		{
+			description:    "empty timed cache",
+			vmssVMCache:    emptyTimedCacheMap,
+			expectedResult: false,
+		},
 		{
 			description:    "empty CacheEntry timed cache",
-			vmssVMCache:    emptyCacheEntryTimedCache,
+			vmssVMCache:    emptyCacheEntryTimedCacheMap,
 			expectedResult: false,
 		},
 		{
 			description:    "node name in the cache",
 			nodeName:       "node",
-			vmssVMCache:    cacheEntryTimedCache,
+			vmssVMCache:    cacheEntryTimedCacheMap,
 			expectedResult: true,
 		},
 		{
 			description:    "node name not in the cache",
 			nodeName:       "node2",
-			vmssVMCache:    cacheEntryTimedCache,
+			vmssVMCache:    cacheEntryTimedCacheMap,
 			expectedResult: false,
 		},
 	}
diff --git a/pkg/provider/azure_vmss_cache_test.go b/pkg/provider/azure_vmss_cache_test.go
index 2404415b1..3895938b7 100644
--- a/pkg/provider/azure_vmss_cache_test.go
+++ b/pkg/provider/azure_vmss_cache_test.go
@@ -19,6 +19,7 @@ package provider
 import (
 	"fmt"
 	"net/http"
+	"strings"
 	"testing"
 
 	"github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2022-03-01/compute"
@@ -177,7 +178,7 @@ func TestVMSSVMCacheClearedWhenRGDeleted(t *testing.T) {
 	mockVMSSClient.EXPECT().List(gomock.Any(), gomock.Any()).Return([]compute.VirtualMachineScaleSet{expectedScaleSet}, nil).Times(1)
 
 	expectedVMs, _, _ := buildTestVirtualMachineEnv(ss.cloud, testVMSSName, "", 0, vmList, "", false)
-	mockVMSSVMClient.EXPECT().List(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(expectedVMs, nil).Times(1)
+	mockVMSSVMClient.EXPECT().List(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(expectedVMs, nil).AnyTimes()
 
 	// validate getting VMSS VM via cache.
 	vm := expectedVMs[0]
@@ -189,20 +190,19 @@ func TestVMSSVMCacheClearedWhenRGDeleted(t *testing.T) {
 	assert.Equal(t, &vm, realVM.AsVirtualMachineScaleSetVM())
 
 	// verify cache has test vmss.
-	cacheKey := getVMSSVMCacheKey("rg", testVMSSName)
-	_, err = ss.vmssVMCache.Get(cacheKey, azcache.CacheReadTypeDefault)
-	assert.Nil(t, err)
+	cacheKey := strings.ToLower(fmt.Sprintf("%s/%s", "rg", testVMSSName))
+	_, ok := ss.vmssVMCache.Load(cacheKey)
+	assert.Equal(t, true, ok)
 
 	// refresh the cache with error.
 	mockVMSSClient.EXPECT().List(gomock.Any(), gomock.Any()).Return([]compute.VirtualMachineScaleSet{}, &retry.Error{HTTPStatusCode: http.StatusNotFound}).Times(2)
-	mockVMSSVMClient.EXPECT().List(gomock.Any(), "rg", testVMSSName, gomock.Any()).Return([]compute.VirtualMachineScaleSetVM{}, &retry.Error{HTTPStatusCode: http.StatusNotFound}).Times(1)
 	realVM, err = ss.getVmssVM(vmName, azcache.CacheReadTypeForceRefresh)
 	assert.Nil(t, realVM)
 	assert.Equal(t, cloudprovider.InstanceNotFound, err)
 
 	// verify cache is cleared.
-	_, err = ss.vmssVMCache.Get(cacheKey, azcache.CacheReadTypeDefault)
-	assert.NotNil(t, err)
+	_, ok = ss.vmssVMCache.Load(cacheKey)
+	assert.Equal(t, false, ok)
 }
 
 func TestGetVMManagementTypeByNodeName(t *testing.T) {
diff --git a/pkg/provider/azure_vmss_test.go b/pkg/provider/azure_vmss_test.go
index 78b728741..cd17709cd 100644
--- a/pkg/provider/azure_vmss_test.go
+++ b/pkg/provider/azure_vmss_test.go
@@ -19,6 +19,7 @@ package provider
 import (
 	"fmt"
 	"strings"
+	"sync"
 	"testing"
 
 	"github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2022-03-01/compute"
@@ -954,9 +955,12 @@ func TestGetVmssVMByNodeIdentity(t *testing.T) {
 			mockVMSSVMClient := ss.cloud.VirtualMachineScaleSetVMsClient.(*mockvmssvmclient.MockInterface)
 			mockVMSSVMClient.EXPECT().List(gomock.Any(), ss.ResourceGroup, testVMSSName, gomock.Any()).Return(expectedVMSSVMs, nil).AnyTimes()
 
-			cacheKey := getVMSSVMCacheKey(ss.ResourceGroup, testVMSSName)
-			virtualMachines, err := ss.getVMSSVMsFromCache(ss.ResourceGroup, testVMSSName, azcache.CacheReadTypeDefault)
+			// Make some nil VMSS VM in cache.
+			cacheKey, cache, err := ss.getVMSSVMCache(ss.ResourceGroup, testVMSSName)
 			assert.Nil(t, err)
+			cached, err := cache.Get(cacheKey, azcache.CacheReadTypeDefault)
+			assert.Nil(t, err)
+			virtualMachines := cached.(*sync.Map)
 			for _, vm := range test.goneVMList {
 				entry := VMSSVirtualMachinesEntry{
 					ResourceGroup: ss.ResourceGroup,
@@ -964,7 +968,7 @@ func TestGetVmssVMByNodeIdentity(t *testing.T) {
 				}
 				virtualMachines.Store(vm, &entry)
 			}
-			ss.vmssVMCache.Update(cacheKey, virtualMachines)
+			cache.Set(cacheKey, cached)
 
 			for i := 0; i < len(test.vmList); i++ {
 				node := nodeIdentity{ss.ResourceGroup, testVMSSName, test.vmList[i]}
@@ -978,9 +982,11 @@ func TestGetVmssVMByNodeIdentity(t *testing.T) {
 				assert.Equal(t, test.goneVMExpectedErr, err)
 			}
 
-			virtualMachines, err = ss.getVMSSVMsFromCache(ss.ResourceGroup, testVMSSName, azcache.CacheReadTypeDefault)
+			cacheKey, cache, err = ss.getVMSSVMCache(ss.ResourceGroup, testVMSSName)
 			assert.Nil(t, err)
-
+			cached, err = cache.Get(cacheKey, azcache.CacheReadTypeDefault)
+			assert.Nil(t, err)
+			virtualMachines = cached.(*sync.Map)
 			for _, vm := range test.goneVMList {
 				_, ok := virtualMachines.Load(vm)
 				assert.True(t, ok)
@@ -2673,7 +2679,7 @@ func TestEnsureBackendPoolDeletedConcurrently(t *testing.T) {
 		i := i
 		id := id
 		testFunc = append(testFunc, func() error {
-			_, err := ss.EnsureBackendPoolDeleted(&v1.Service{}, id, testVMSSNames[i], backendAddressPools, true)
+			_, err = ss.EnsureBackendPoolDeleted(&v1.Service{}, id, testVMSSNames[i], backendAddressPools, true)
 			return err
 		})
 	}
diff --git a/pkg/provider/azure_vmssflex_cache_test.go b/pkg/provider/azure_vmssflex_cache_test.go
index 4c0cc99c5..8b968ca9f 100644
--- a/pkg/provider/azure_vmssflex_cache_test.go
+++ b/pkg/provider/azure_vmssflex_cache_test.go
@@ -182,9 +182,8 @@ func generateVmssFlexTestVMWithoutInstanceView(spec VmssFlexTestVMSpec) (testVMW
 				},
 				DataDisks: &[]compute.DataDisk{
 					{
-						Lun:         to.Int32Ptr(1),
-						Name:        to.StringPtr("dataDisk" + spec.VMName),
-						ManagedDisk: &compute.ManagedDiskParameters{ID: to.StringPtr("uri")},
+						Lun:  to.Int32Ptr(1),
+						Name: to.StringPtr("dataDisk" + spec.VMName),
 					},
 				},
 			},
diff --git a/tests/e2e/network/ensureloadbalancer.go b/tests/e2e/network/ensureloadbalancer.go
index 0db343793..2d32aecd6 100644
--- a/tests/e2e/network/ensureloadbalancer.go
+++ b/tests/e2e/network/ensureloadbalancer.go
@@ -159,7 +159,7 @@ var _ = Describe("Ensure LoadBalancer", Label(utils.TestSuiteLabelLB), func() {
 	It("should support BYO public IP", func() {
 		By("creating a public IP with tags")
 		ipName := basename + "-public-IP" + string(uuid.NewUUID())[0:4]
-		pip := defaultPublicIPAddress(ipName, false)
+		pip := defaultPublicIPAddress(ipName, tc.IPFamily == utils.IPv6)
 		expectedTags := map[string]*string{
 			"foo": to.StringPtr("bar"),
 		}
@@ -204,7 +204,7 @@ var _ = Describe("Ensure LoadBalancer", Label(utils.TestSuiteLabelLB), func() {
 		Expect(err).NotTo(HaveOccurred())
 		utils.Logf("Successfully created LoadBalancer service " + testServiceName + " in namespace " + ns.Name)
 
-		pip, err := utils.WaitCreatePIP(tc, ipName, tc.GetResourceGroup(), defaultPublicIPAddress(ipName, false))
+		pip, err := utils.WaitCreatePIP(tc, ipName, tc.GetResourceGroup(), defaultPublicIPAddress(ipName, tc.IPFamily == utils.IPv6))
 		Expect(err).NotTo(HaveOccurred())
 		targetIP := to.String(pip.IPAddress)
 		utils.Logf("PIP to %s", targetIP)
@@ -288,7 +288,7 @@ var _ = Describe("Ensure LoadBalancer", Label(utils.TestSuiteLabelLB), func() {
 		Expect(err).NotTo(HaveOccurred())
 		utils.Logf("Successfully created LoadBalancer service " + testServiceName + " in namespace " + ns.Name)
 
-		pip, err := utils.WaitCreatePIP(tc, ipName, tc.GetResourceGroup(), defaultPublicIPAddress(ipName, false))
+		pip, err := utils.WaitCreatePIP(tc, ipName, tc.GetResourceGroup(), defaultPublicIPAddress(ipName, tc.IPFamily == utils.IPv6))
 		Expect(err).NotTo(HaveOccurred())
 		targetIP := to.String(pip.IPAddress)
 
@@ -327,7 +327,7 @@ var _ = Describe("Ensure LoadBalancer", Label(utils.TestSuiteLabelLB), func() {
 	It("should have no operation since no change in service when update", Label(utils.TestSuiteLabelSlow), func() {
 		suffix := string(uuid.NewUUID())[0:4]
 		ipName := basename + "-public-remain" + suffix
-		pip, err := utils.WaitCreatePIP(tc, ipName, tc.GetResourceGroup(), defaultPublicIPAddress(ipName, false))
+		pip, err := utils.WaitCreatePIP(tc, ipName, tc.GetResourceGroup(), defaultPublicIPAddress(ipName, tc.IPFamily == utils.IPv6))
 		Expect(err).NotTo(HaveOccurred())
 		targetIP := to.String(pip.IPAddress)
 
@@ -358,6 +358,7 @@ var _ = Describe("Ensure LoadBalancer", Label(utils.TestSuiteLabelLB), func() {
 		Expect(err).NotTo(HaveOccurred())
 
 		// Wait for 5 minutes, there should return timeout err, since external ip should not change
+		utils.Logf("External IP should be %s", targetIP)
 		err = wait.PollImmediate(10*time.Second, 5*time.Minute, func() (bool, error) {
 			service, err = cs.CoreV1().Services(ns.Name).Get(context.TODO(), testServiceName, metav1.GetOptions{})
 			if err != nil {
@@ -369,12 +370,11 @@ var _ = Describe("Ensure LoadBalancer", Label(utils.TestSuiteLabelLB), func() {
 
 			ingressList := service.Status.LoadBalancer.Ingress
 			if len(ingressList) == 0 {
-				err = fmt.Errorf("Cannot find Ingress in limited time")
+				err = fmt.Errorf("cannot find Ingress in limited time")
 				utils.Logf("Fail to get ingress, retry it in %s seconds", 10)
 				return false, nil
 			}
 			if targetIP == ingressList[0].IP {
-				utils.Logf("External IP is still %s, as expected", targetIP)
 				return false, nil
 			}
 			utils.Logf("External IP changed, unexpected")
@@ -385,7 +385,7 @@ var _ = Describe("Ensure LoadBalancer", Label(utils.TestSuiteLabelLB), func() {
 
 	It("should support multiple external services sharing one preset public IP address", func() {
 		ipName := fmt.Sprintf("%s-public-remain-%s", basename, string(uuid.NewUUID())[0:4])
-		pip, err := utils.WaitCreatePIP(tc, ipName, tc.GetResourceGroup(), defaultPublicIPAddress(ipName, false))
+		pip, err := utils.WaitCreatePIP(tc, ipName, tc.GetResourceGroup(), defaultPublicIPAddress(ipName, tc.IPFamily == utils.IPv6))
 		defer func() {
 			err = utils.DeletePIPWithRetry(tc, ipName, "")
 			Expect(err).NotTo(HaveOccurred())
@@ -515,7 +515,7 @@ var _ = Describe("Ensure LoadBalancer", Label(utils.TestSuiteLabelLB), func() {
 			}
 
 			for _, pip := range pips {
-				if strings.EqualFold(*pip.IPAddress, sharedIP) {
+				if pip.IPAddress != nil && strings.EqualFold(*pip.IPAddress, sharedIP) {
 					utils.Logf("the public IP with address %s still exists", sharedIP)
 					return false, nil
 				}
@@ -620,7 +620,7 @@ var _ = Describe("Ensure LoadBalancer", Label(utils.TestSuiteLabelLB), func() {
 	It("should support disabling floating IP in load balancer rule with kubernetes service annotations", func() {
 		By("creating a public IP")
 		ipName := basename + "-public-IP" + string(uuid.NewUUID())[0:4]
-		pip := defaultPublicIPAddress(ipName, false)
+		pip := defaultPublicIPAddress(ipName, tc.IPFamily == utils.IPv6)
 		pip, err := utils.WaitCreatePIP(tc, ipName, tc.GetResourceGroup(), pip)
 		Expect(err).NotTo(HaveOccurred())
 		targetIP := to.String(pip.IPAddress)
@@ -723,7 +723,8 @@ var _ = Describe("EnsureLoadBalancer should not update any resources when servic
 			consts.ServiceAnnotationLoadBalancerHealthProbeNumOfProbe:  "8",
 		}
 
-		if strings.EqualFold(os.Getenv(utils.LoadBalancerSkuEnv), string(network.PublicIPAddressSkuNameStandard)) {
+		if strings.EqualFold(os.Getenv(utils.LoadBalancerSkuEnv), string(network.PublicIPAddressSkuNameStandard)) &&
+			tc.IPFamily == utils.IPv4 {
 			// Routing preference is only supported in standard public IPs
 			annotation[consts.ServiceAnnotationIPTagsForPublicIP] = "RoutingPreference=Internet"
 		}
@@ -744,7 +745,7 @@ var _ = Describe("EnsureLoadBalancer should not update any resources when servic
 	It("should respect service with BYO public IP with various configurations", func() {
 		By("Creating a BYO public IP")
 		ipName := basename + "-public-IP" + string(uuid.NewUUID())[0:4]
-		pip, err := utils.WaitCreatePIP(tc, ipName, tc.GetResourceGroup(), defaultPublicIPAddress(ipName, false))
+		pip, err := utils.WaitCreatePIP(tc, ipName, tc.GetResourceGroup(), defaultPublicIPAddress(ipName, tc.IPFamily == utils.IPv6))
 		defer func() {
 			err = utils.DeletePIPWithRetry(tc, ipName, "")
 			Expect(err).NotTo(HaveOccurred())
@@ -790,11 +791,11 @@ var _ = Describe("EnsureLoadBalancer should not update any resources when servic
 
 		By("Creating a BYO public IP prefix")
 		prefixName := "prefix"
-		prefix, err := utils.WaitCreatePIPPrefix(tc, prefixName, tc.GetResourceGroup(), defaultPublicIPPrefix(prefixName))
-		Expect(err).NotTo(HaveOccurred())
+		prefix, err := utils.WaitCreatePIPPrefix(tc, prefixName, tc.GetResourceGroup(), defaultPublicIPPrefix(prefixName, tc.IPFamily == utils.IPv6))
 		defer func() {
 			Expect(utils.DeletePIPPrefixWithRetry(tc, prefixName)).NotTo(HaveOccurred())
 		}()
+		Expect(err).NotTo(HaveOccurred())
 
 		By("Creating a service and expose it")
 		annotation := map[string]string{
@@ -897,9 +898,10 @@ func createNewSubnet(tc *utils.AzureTestClient, subnetName string) (*network.Sub
 	if subnetToReturn == nil {
 		By("Test subnet doesn't exist. Creating a new one...")
 		isNew = true
-		newSubnetCIDR, err := utils.GetNextSubnetCIDR(vNet)
+		newSubnetCIDR, err := utils.GetNextSubnetCIDR(vNet, tc.IPFamily)
 		Expect(err).NotTo(HaveOccurred())
-		newSubnet, err := tc.CreateSubnet(vNet, &subnetName, &newSubnetCIDR, true)
+		newSubnetCIDRStr := newSubnetCIDR.String()
+		newSubnet, err := tc.CreateSubnet(vNet, &subnetName, &newSubnetCIDRStr, true)
 		Expect(err).NotTo(HaveOccurred())
 		subnetToReturn = &newSubnet
 	}
@@ -1042,7 +1044,13 @@ func defaultPublicIPAddress(ipName string, isIPv6 bool) aznetwork.PublicIPAddres
 	return pip
 }
 
-func defaultPublicIPPrefix(name string) aznetwork.PublicIPPrefix {
+func defaultPublicIPPrefix(name string, isIPv6 bool) aznetwork.PublicIPPrefix {
+	pipAddrVersion := aznetwork.IPVersionIPv4
+	var prefixLen int32 = 28
+	if isIPv6 {
+		pipAddrVersion = aznetwork.IPVersionIPv6
+		prefixLen = 124
+	}
 	return aznetwork.PublicIPPrefix{
 		Name:     to.StringPtr(name),
 		Location: to.StringPtr(os.Getenv(utils.ClusterLocationEnv)),
@@ -1050,7 +1058,8 @@ func defaultPublicIPPrefix(name string) aznetwork.PublicIPPrefix {
 			Name: aznetwork.PublicIPPrefixSkuNameStandard,
 		},
 		PublicIPPrefixPropertiesFormat: &aznetwork.PublicIPPrefixPropertiesFormat{
-			PrefixLength: to.Int32Ptr(28),
+			PrefixLength:           to.Int32Ptr(prefixLen),
+			PublicIPAddressVersion: pipAddrVersion,
 		},
 	}
 }
diff --git a/tests/e2e/network/network_security_group.go b/tests/e2e/network/network_security_group.go
index 237c06bbe..f4cd83421 100644
--- a/tests/e2e/network/network_security_group.go
+++ b/tests/e2e/network/network_security_group.go
@@ -19,7 +19,6 @@ package network
 import (
 	"context"
 	"fmt"
-	"net/http"
 	"strings"
 	"time"
 
@@ -29,6 +28,7 @@ import (
 	v1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/util/intstr"
+	"k8s.io/apimachinery/pkg/util/wait"
 	clientset "k8s.io/client-go/kubernetes"
 	"k8s.io/utils/strings/slices"
 
@@ -104,42 +104,35 @@ var _ = Describe("Network security group", Label(utils.TestSuiteLabelNSG), func(
 		Expect(validateUnsharedSecurityRuleExists(nsgs, ip, port)).To(BeTrue(), "Security rule for service %s not exists", serviceName)
 
 		By("Validating network security group working")
-		var code int
-		url := fmt.Sprintf("http://%s:%d", ip, ports[0].Port)
-		for i := 1; i <= 30; i++ {
-			utils.Logf("round %d, GET %s", i, url)
-			/* #nosec G107: Potential HTTP request made with variable url */
-			resp, err := http.Get(url)
-			if err == nil {
-				defer func() {
-					if resp != nil {
-						resp.Body.Close()
-					}
-				}()
-				code = resp.StatusCode
-				if resp.StatusCode == nginxStatusCode {
-					break
-				}
-			}
-			time.Sleep(20 * time.Second)
-		}
+		// Use a hostNetwork Pod to validate Service connectivity via the cluster Node's
+		// network because the current VM running go test may not support IPv6.
+		agnhostPod := fmt.Sprintf("%s-%s", utils.ExecAgnhostPod, "nsg")
+		result, err := utils.CreateHostExecPod(cs, ns.Name, agnhostPod)
+		defer func() {
+			err = utils.DeletePod(cs, ns.Name, agnhostPod)
+			Expect(err).NotTo(HaveOccurred())
+		}()
+		Expect(result).To(BeTrue())
 		Expect(err).NotTo(HaveOccurred())
-		Expect(code).To(Equal(nginxStatusCode), "Fail to get response from the domain name")
+
+		By(fmt.Sprintf("Validating External domain name %q", ip))
+		err = utils.ValidateServiceConnectivity(ns.Name, agnhostPod, ip, int(ports[0].Port), v1.ProtocolTCP)
+		Expect(err).NotTo(HaveOccurred(), "Fail to get response from the domain name")
 
 		By("Validate automatically delete the rule, when service is deleted")
 		Expect(utils.DeleteService(cs, ns.Name, serviceName)).NotTo(HaveOccurred())
-		isDeleted := false
-		for i := 1; i <= 30; i++ {
+		err = wait.PollImmediate(20*time.Second, 10*time.Minute, func() (done bool, err error) {
 			nsgs, err := tc.GetClusterSecurityGroups()
-			Expect(err).NotTo(HaveOccurred())
+			if err != nil {
+				return false, err
+			}
 			if !validateUnsharedSecurityRuleExists(nsgs, ip, port) {
 				utils.Logf("Target rule successfully deleted")
-				isDeleted = true
-				break
+				return true, nil
 			}
-			time.Sleep(20 * time.Second)
-		}
-		Expect(isDeleted).To(BeTrue(), "Fail to automatically delete the rule")
+			return false, nil
+		})
+		Expect(err).To(BeNil(), "Fail to automatically delete the rule")
 	})
 
 	It("should support service annotation `service.beta.kubernetes.io/azure-shared-securityrule`", func() {
@@ -266,7 +259,11 @@ var _ = Describe("Network security group", Label(utils.TestSuiteLabelNSG), func(
 		Expect(err).NotTo(HaveOccurred())
 		hostExecPodIP := hostExecPod.Status.PodIP
 
-		allowCIDR := fmt.Sprintf("%s/32", hostExecPodIP)
+		mask := 32
+		if tc.IPFamily == utils.IPv6 {
+			mask = 128
+		}
+		allowCIDR := fmt.Sprintf("%s/%d", hostExecPodIP, mask)
 		service.Spec.LoadBalancerSourceRanges = []string{allowCIDR}
 		_, err = cs.CoreV1().Services(ns.Name).Create(context.TODO(), service, metav1.CreateOptions{})
 		Expect(err).NotTo(HaveOccurred())
@@ -283,7 +280,7 @@ var _ = Describe("Network security group", Label(utils.TestSuiteLabelNSG), func(
 		By("Checking if there is a LoadBalancerSourceRanges rule")
 		nsgs, err = tc.GetClusterSecurityGroups()
 		Expect(err).NotTo(HaveOccurred())
-		found = validateLoadBalancerSourceRangesRuleExists(nsgs, internalIP, allowCIDR, fmt.Sprintf("%s_32", hostExecPodIP))
+		found = validateLoadBalancerSourceRangesRuleExists(nsgs, internalIP, allowCIDR, fmt.Sprintf("%s_%d", hostExecPodIP, mask))
 		Expect(found).To(BeTrue())
 
 		By("Checking if there is a deny_all rule")
@@ -294,7 +291,7 @@ var _ = Describe("Network security group", Label(utils.TestSuiteLabelNSG), func(
 	It("should support service annotation `service.beta.kubernetes.io/azure-disable-load-balancer-floating-ip`", func() {
 		By("Creating a public IP with tags")
 		ipName := basename + "-public-IP-disable-floating-ip"
-		pip := defaultPublicIPAddress(ipName, false)
+		pip := defaultPublicIPAddress(ipName, tc.IPFamily == utils.IPv6)
 		pip, err := utils.WaitCreatePIP(tc, ipName, tc.GetResourceGroup(), pip)
 		Expect(err).NotTo(HaveOccurred())
 		targetIP := to.String(pip.IPAddress)
diff --git a/tests/e2e/network/private_link_service.go b/tests/e2e/network/private_link_service.go
index 32e5f7521..5278bfdfd 100644
--- a/tests/e2e/network/private_link_service.go
+++ b/tests/e2e/network/private_link_service.go
@@ -68,6 +68,11 @@ var _ = Describe("Private link service", Label(utils.TestSuiteLabelPrivateLinkSe
 		tc, err = utils.CreateAzureTestClient()
 		Expect(err).NotTo(HaveOccurred())
 
+		if tc.IPFamily != utils.IPv4 {
+			// https://learn.microsoft.com/en-us/azure/private-link/private-link-service-overview#limitations
+			Skip("private link service only works with IPv4")
+		}
+
 		utils.Logf("Creating deployment " + serviceName)
 		deployment := createServerDeploymentManifest(serviceName, labels)
 		_, err = cs.AppsV1().Deployments(ns.Name).Create(context.TODO(), deployment, metav1.CreateOptions{})
diff --git a/tests/e2e/network/service_annotations.go b/tests/e2e/network/service_annotations.go
index 1699b492c..b020b0be2 100644
--- a/tests/e2e/network/service_annotations.go
+++ b/tests/e2e/network/service_annotations.go
@@ -19,7 +19,7 @@ package network
 import (
 	"context"
 	"fmt"
-	"net/http"
+	"net"
 	"os"
 	"reflect"
 	"regexp"
@@ -50,15 +50,11 @@ var (
 	scalesetRE               = regexp.MustCompile(`.*/subscriptions/(?:.*)/resourceGroups/(.+)/providers/Microsoft.Compute/virtualMachineScaleSets/(.+)/virtualMachines(?:.*)`)
 	lbNameRE                 = regexp.MustCompile(`^/subscriptions/(?:.*)/resourceGroups/(?:.*)/providers/Microsoft.Network/loadBalancers/(.+)/frontendIPConfigurations(?:.*)`)
 	backendIPConfigurationRE = regexp.MustCompile(`^/subscriptions/(?:.*)/resourceGroups/(?:.*)/providers/Microsoft.Compute/virtualMachineScaleSets/(.+)/virtualMachines(?:.*)`)
-
-	ipFamily = utils.IPv4
 )
 
 const (
 	serverPort  = 80
 	testingPort = 81
-
-	nginxStatusCode = 200
 )
 
 var _ = Describe("Service with annotation", Label(utils.TestSuiteLabelServiceAnnotation), func() {
@@ -100,9 +96,6 @@ var _ = Describe("Service with annotation", Label(utils.TestSuiteLabelServiceAnn
 		utils.Logf("Waiting for backend pods to be ready")
 		err = utils.WaitPodsToBeReady(cs, ns.Name)
 		Expect(err).NotTo(HaveOccurred())
-
-		ipFamily, err = utils.GetClusterServiceIPFamily()
-		Expect(err).NotTo(HaveOccurred())
 	})
 
 	AfterEach(func() {
@@ -142,7 +135,6 @@ var _ = Describe("Service with annotation", Label(utils.TestSuiteLabelServiceAnn
 		annotation := map[string]string{
 			consts.ServiceAnnotationDNSLabelName: serviceDomainNamePrefix,
 		}
-
 		// create service with given annotation and wait it to expose
 		_ = createAndExposeDefaultServiceWithAnnotation(cs, serviceName, ns.Name, labels, annotation, ports)
 		defer func() {
@@ -151,63 +143,34 @@ var _ = Describe("Service with annotation", Label(utils.TestSuiteLabelServiceAnn
 			Expect(err).NotTo(HaveOccurred())
 		}()
 
-		By("Validating External domain name")
-		var code int
+		// Use a hostNetwork Pod to validate Service connectivity via the cluster Node's
+		// network because the current VM running go test may not support IPv6.
+		agnhostPod := fmt.Sprintf("%s-%s", utils.ExecAgnhostPod, "azure-dns-label-name")
+		result, err := utils.CreateHostExecPod(cs, ns.Name, agnhostPod)
+		defer func() {
+			err = utils.DeletePod(cs, ns.Name, agnhostPod)
+			Expect(err).NotTo(HaveOccurred())
+		}()
+		Expect(result).To(BeTrue())
+		Expect(err).NotTo(HaveOccurred())
+
 		serviceDomainName := utils.GetServiceDomainName(serviceDomainNamePrefix)
-		url := fmt.Sprintf("http://%s:%v", serviceDomainName, ports[0].Port)
-		for i := 1; i <= 30; i++ {
-			/* #nosec G107: Potential HTTP request made with variable url */
-			resp, err := http.Get(url)
-			if err == nil {
-				defer func() {
-					if resp != nil {
-						resp.Body.Close()
-					}
-				}()
-				code = resp.StatusCode
-				if code == nginxStatusCode {
-					break
-				} else {
-					utils.Logf("Received %d status code from %s", code, url)
-				}
-			} else {
-				utils.Logf("Received the following error when validating %s: %v", url, err)
-			}
-			utils.Logf("Retrying in 20 seconds")
-			time.Sleep(20 * time.Second)
-		}
-		Expect(code).To(Equal(nginxStatusCode), "Fail to get response from the domain name")
+		By(fmt.Sprintf("Validating External domain name %q", serviceDomainName))
+		err = utils.ValidateServiceConnectivity(ns.Name, agnhostPod, serviceDomainName, int(ports[0].Port), v1.ProtocolTCP)
+		Expect(err).NotTo(HaveOccurred(), "Fail to get response from the domain name")
+
 		By("Update service")
 		annotation = map[string]string{
 			consts.ServiceAnnotationDNSLabelName: serviceDomainNamePrefix + "new",
 		}
 		service := utils.CreateLoadBalancerServiceManifest(serviceName, annotation, labels, ns.Name, ports)
-		_, err := cs.CoreV1().Services(ns.Name).Update(context.TODO(), service, metav1.UpdateOptions{})
+		_, err = cs.CoreV1().Services(ns.Name).Update(context.TODO(), service, metav1.UpdateOptions{})
 		Expect(err).NotTo(HaveOccurred())
+
 		serviceDomainName = utils.GetServiceDomainName(serviceDomainNamePrefix)
-		url = fmt.Sprintf("http://%s:%v", serviceDomainName, ports[0].Port)
-		for i := 1; i <= 30; i++ {
-			/* #nosec G107: Potential HTTP request made with variable url */
-			resp, err := http.Get(url)
-			if err == nil {
-				defer func() {
-					if resp != nil {
-						resp.Body.Close()
-					}
-				}()
-				code = resp.StatusCode
-				if code == nginxStatusCode {
-					break
-				} else {
-					utils.Logf("Received %d status code from %s", code, url)
-				}
-			} else {
-				utils.Logf("Received the following error when validating %s: %v", url, err)
-			}
-			utils.Logf("Retrying in 20 seconds")
-			time.Sleep(20 * time.Second)
-		}
-		Expect(code).To(Equal(nginxStatusCode), "Fail to get response from the domain name")
+		By(fmt.Sprintf("Validating External domain name %q", serviceDomainName))
+		err = utils.ValidateServiceConnectivity(ns.Name, agnhostPod, serviceDomainName, int(ports[0].Port), v1.ProtocolTCP)
+		Expect(err).NotTo(HaveOccurred(), "Fail to get response from the domain name")
 	})
 
 	It("should support service annotation 'service.beta.kubernetes.io/azure-load-balancer-internal'", func() {
@@ -232,20 +195,48 @@ var _ = Describe("Service with annotation", Label(utils.TestSuiteLabelServiceAnn
 		vNet, err := tc.GetClusterVirtualNetwork()
 		Expect(err).NotTo(HaveOccurred())
 
-		var newSubnetCIDR string
+		var newSubnetCIDR *net.IPNet
 		for _, existingSubnet := range *vNet.Subnets {
-			if *existingSubnet.Name == subnetName {
-				By("Test subnet have existed, skip creating")
-				newSubnetCIDR = *existingSubnet.AddressPrefix
-				break
+			if *existingSubnet.Name != subnetName {
+				continue
 			}
+			utils.Logf("Test subnet have existed, skip creating")
+			if existingSubnet.AddressPrefix != nil {
+				// IPv4 picks the only AddressPrefix.
+				_, newSubnetCIDR, err = net.ParseCIDR(*existingSubnet.AddressPrefix)
+				Expect(err).NotTo(HaveOccurred())
+			} else {
+				Expect(existingSubnet.AddressPrefixes).NotTo(BeNil(),
+					"subnet AddressPrefix and AddressPrefixes shouldn't be both nil")
+
+				addrPrefixPicked := false
+				for _, addrPrefix := range *existingSubnet.AddressPrefixes {
+					var parsedIP net.IP
+					parsedIP, newSubnetCIDR, err = net.ParseCIDR(addrPrefix)
+					Expect(err).NotTo(HaveOccurred(), "failed to parse CIDR %q", addrPrefix)
+					if tc.IPFamily == utils.DualStack && parsedIP.To4() != nil {
+						// Dual-stack picks IPv4 prefix.
+						addrPrefixPicked = true
+						break
+					}
+					if tc.IPFamily == utils.IPv6 && parsedIP.To4() == nil {
+						// IPv6 picks IPv6 prefix.
+						addrPrefixPicked = true
+						break
+					}
+				}
+				Expect(addrPrefixPicked).To(BeTrue(), "there's no matching AddressPrefixes")
+			}
+			break
 		}
 
-		if newSubnetCIDR == "" {
+		if newSubnetCIDR == nil {
 			By("Test subnet doesn't exist. Creating a new one...")
-			newSubnetCIDR, err = utils.GetNextSubnetCIDR(vNet)
+			newSubnetCIDR, err = utils.GetNextSubnetCIDR(vNet, tc.IPFamily)
 			Expect(err).NotTo(HaveOccurred())
-			_, err = tc.CreateSubnet(vNet, &subnetName, &newSubnetCIDR, false)
+			newSubnetCIDRStr := newSubnetCIDR.String()
+			By(fmt.Sprintf("Creating a subnet %q", newSubnetCIDRStr))
+			_, err = tc.CreateSubnet(vNet, &subnetName, &newSubnetCIDRStr, false)
 			Expect(err).NotTo(HaveOccurred())
 			defer func() {
 				utils.Logf("cleaning up test subnet %s", subnetName)
@@ -269,9 +260,8 @@ var _ = Describe("Service with annotation", Label(utils.TestSuiteLabelServiceAnn
 		utils.Logf("Get External IP: %s", ip)
 
 		By("Validating external ip in target subnet")
-		ret, err := utils.ValidateIPInCIDR(ip, newSubnetCIDR)
-		Expect(err).NotTo(HaveOccurred())
-		Expect(ret).To(BeTrue(), "external ip %s is not in the target subnet %s", ip, newSubnetCIDR)
+		contains := newSubnetCIDR.Contains(net.ParseIP(ip))
+		Expect(contains).To(BeTrue(), "external ip %s is not in the target subnet %s", ip, newSubnetCIDR)
 	})
 
 	It("should support service annotation 'service.beta.kubernetes.io/azure-load-balancer-enable-high-availability-ports'", func() {
@@ -334,7 +324,7 @@ var _ = Describe("Service with annotation", Label(utils.TestSuiteLabelServiceAnn
 
 		By("creating test PIP in the test resource group")
 		testPIPName := "testPIP-" + string(uuid.NewUUID())[0:4]
-		pip, err := utils.WaitCreatePIP(tc, testPIPName, *rg.Name, defaultPublicIPAddress(testPIPName, false))
+		pip, err := utils.WaitCreatePIP(tc, testPIPName, *rg.Name, defaultPublicIPAddress(testPIPName, tc.IPFamily == utils.IPv6))
 		Expect(err).NotTo(HaveOccurred())
 		defer func() {
 			utils.Logf("Cleaning up service and public IP")
@@ -366,7 +356,7 @@ var _ = Describe("Service with annotation", Label(utils.TestSuiteLabelServiceAnn
 	It("should support service annotation `service.beta.kubernetes.io/azure-additional-public-ips`", func() {
 		By("creating a public IP")
 		ipName := basename + "-public-IP" + string(uuid.NewUUID())[0:4]
-		pip := defaultPublicIPAddress(ipName, false)
+		pip := defaultPublicIPAddress(ipName, tc.IPFamily == utils.IPv6)
 		pip, err := utils.WaitCreatePIP(tc, ipName, tc.GetResourceGroup(), pip)
 		Expect(err).NotTo(HaveOccurred())
 		additionalPIP := to.String(pip.IPAddress)
@@ -478,7 +468,7 @@ var _ = Describe("Service with annotation", Label(utils.TestSuiteLabelServiceAnn
 	It("should support service annotation `service.beta.kubernetes.io/azure-pip-name`", func() {
 		By("Creating two test pips")
 		pipName1 := "pip1"
-		pip1, err := utils.WaitCreatePIP(tc, pipName1, tc.GetResourceGroup(), defaultPublicIPAddress(pipName1, false))
+		pip1, err := utils.WaitCreatePIP(tc, pipName1, tc.GetResourceGroup(), defaultPublicIPAddress(pipName1, tc.IPFamily == utils.IPv6))
 		defer func() {
 			By("Cleaning up test PIP")
 			err := utils.DeletePIPWithRetry(tc, pipName1, tc.GetResourceGroup())
@@ -486,7 +476,7 @@ var _ = Describe("Service with annotation", Label(utils.TestSuiteLabelServiceAnn
 		}()
 		Expect(err).NotTo(HaveOccurred())
 		pipName2 := "pip2"
-		pip2, err := utils.WaitCreatePIP(tc, pipName2, tc.GetResourceGroup(), defaultPublicIPAddress(pipName2, false))
+		pip2, err := utils.WaitCreatePIP(tc, pipName2, tc.GetResourceGroup(), defaultPublicIPAddress(pipName2, tc.IPFamily == utils.IPv6))
 		defer func() {
 			By("Cleaning up test PIP")
 			err := utils.DeletePIPWithRetry(tc, pipName2, tc.GetResourceGroup())
@@ -534,19 +524,19 @@ var _ = Describe("Service with annotation", Label(utils.TestSuiteLabelServiceAnn
 		)
 
 		By("Creating two test PIPPrefix")
-		prefix1, err := utils.WaitCreatePIPPrefix(tc, prefix1Name, tc.GetResourceGroup(), defaultPublicIPPrefix(prefix1Name))
-		Expect(err).NotTo(HaveOccurred())
+		prefix1, err := utils.WaitCreatePIPPrefix(tc, prefix1Name, tc.GetResourceGroup(), defaultPublicIPPrefix(prefix1Name, tc.IPFamily == utils.IPv6))
 		defer func() {
 			By(fmt.Sprintf("Cleaning up pip-prefix: %s", prefix1Name))
 			Expect(utils.DeletePIPPrefixWithRetry(tc, prefix1Name)).NotTo(HaveOccurred())
 		}()
-
-		prefix2, err := utils.WaitCreatePIPPrefix(tc, prefix2Name, tc.GetResourceGroup(), defaultPublicIPPrefix(prefix2Name))
 		Expect(err).NotTo(HaveOccurred())
+
+		prefix2, err := utils.WaitCreatePIPPrefix(tc, prefix2Name, tc.GetResourceGroup(), defaultPublicIPPrefix(prefix2Name, tc.IPFamily == utils.IPv6))
 		defer func() {
 			By(fmt.Sprintf("Cleaning up pip-prefix: %s", prefix2Name))
 			Expect(utils.DeletePIPPrefixWithRetry(tc, prefix2Name)).NotTo(HaveOccurred())
 		}()
+		Expect(err).NotTo(HaveOccurred())
 
 		By("Creating a service referring to the prefix")
 		{
@@ -669,10 +659,8 @@ var _ = Describe("Service with annotation", Label(utils.TestSuiteLabelServiceAnn
 		pipName := fmt.Sprintf("%s-public-IP%s", basename, string(uuid.NewUUID())[0:4])
 		By(fmt.Sprintf("Creating a public IP %q", pipName))
 		var pip network.PublicIPAddress
-		if ipFamily == utils.IPv4 {
-			pip = defaultPublicIPAddress(pipName, false)
-		} else if ipFamily == utils.IPv6 {
-			pip = defaultPublicIPAddress(pipName, true)
+		if tc.IPFamily != utils.DualStack {
+			pip = defaultPublicIPAddress(pipName, tc.IPFamily == utils.IPv6)
 		} else {
 			// TODO: dual-stack support
 		}
@@ -688,9 +676,9 @@ var _ = Describe("Service with annotation", Label(utils.TestSuiteLabelServiceAnn
 		utils.Logf("Created pip with address %s", pipAddr)
 
 		annotation := map[string]string{}
-		if ipFamily == utils.IPv4 {
+		if tc.IPFamily == utils.IPv4 {
 			annotation[consts.ServiceAnnotationLoadBalancerIPDualStack[false]] = pipAddr
-		} else if ipFamily == utils.IPv6 {
+		} else if tc.IPFamily == utils.IPv6 {
 			annotation[consts.ServiceAnnotationLoadBalancerIPDualStack[true]] = pipAddr
 		} else {
 			// TODO: dual-stack support
diff --git a/tests/e2e/utils/azure_test_client.go b/tests/e2e/utils/azure_test_client.go
index 15031b9b0..fecffae83 100644
--- a/tests/e2e/utils/azure_test_client.go
+++ b/tests/e2e/utils/azure_test_client.go
@@ -47,6 +47,8 @@ type AzureTestClient struct {
 	acrClient      acr.BaseClient
 	authClient     azauth.BaseClient
 	computeClient  azcompute.BaseClient
+
+	IPFamily IPFamily
 }
 
 // CreateAzureTestClient makes a new AzureTestClient
@@ -94,6 +96,11 @@ func CreateAzureTestClient() (*AzureTestClient, error) {
 	}
 	location := getLocationFromNodeLabels(&nodes[0])
 
+	ipFamily, err := GetClusterServiceIPFamily()
+	if err != nil {
+		return nil, err
+	}
+
 	c := &AzureTestClient{
 		location:       location,
 		resourceGroup:  resourceGroup,
@@ -103,6 +110,7 @@ func CreateAzureTestClient() (*AzureTestClient, error) {
 		acrClient:      acrClient,
 		authClient:     authClient,
 		computeClient:  computeClient,
+		IPFamily:       ipFamily,
 	}
 
 	return c, nil
diff --git a/tests/e2e/utils/ip_utils.go b/tests/e2e/utils/ip_utils.go
index d38763fb6..5d13fe111 100644
--- a/tests/e2e/utils/ip_utils.go
+++ b/tests/e2e/utils/ip_utils.go
@@ -18,6 +18,7 @@ package utils
 
 import (
 	"fmt"
+	"net"
 	"strconv"
 	"strings"
 )
@@ -26,41 +27,82 @@ const (
 	leastPrefix = 24
 )
 
-// ValidateIPInCIDR validates whether certain ip fits CIDR
-func ValidateIPInCIDR(ip, cidr string) (bool, error) {
-	ipTemp := ip + "/24"
-	ipInt, _, err := cidrString2intArray(ipTemp)
+func getNextSubnet(vnetCIDR string, existSubnets []string) (*net.IPNet, error) {
+	// Filter existSubnets with IP family
+	isIPv6, err := isCIDRIPv6(vnetCIDR)
 	if err != nil {
-		return false, err
+		return nil, err
 	}
-	cidrInt, mask, err := cidrString2intArray(cidr)
-	if err != nil {
-		return false, err
+	existSubnetsWithSameIPFamily := []string{}
+	for _, subnet := range existSubnets {
+		isSubnetIPv6, err := isCIDRIPv6(subnet)
+		if err != nil {
+			return nil, err
+		}
+		if isSubnetIPv6 == isIPv6 {
+			existSubnetsWithSameIPFamily = append(existSubnetsWithSameIPFamily, subnet)
+		}
 	}
-	for i := 0; i < mask; i++ {
-		if cidrInt[i] != ipInt[i] {
-			return false, nil
+
+	// IPv6
+	// TODO: A better solution to replace this hack.
+	if isIPv6 {
+		ip, ipNet, _ := net.ParseCIDR(vnetCIDR)
+		mask := ipNet.Mask
+		pos := 0
+		for i := range ip {
+			if mask[i] != 255 {
+				pos = i
+				break
+			}
 		}
+		found := false
+		for i := 255; i > 0; i-- {
+			ip[pos] = byte(i)
+			// Check if the subnet is in any existing one
+			contains := false
+			for _, existSubnet := range existSubnetsWithSameIPFamily {
+				_, existSubnetIPNet, err := net.ParseCIDR(existSubnet)
+				if err != nil {
+					return nil, err
+				}
+				if existSubnetIPNet.Contains(ip) {
+					contains = true
+					break
+				}
+			}
+			if !contains {
+				found = true
+				break
+			}
+
+		}
+		if !found {
+			return nil, fmt.Errorf("failed to find the next subnet: vnetCIDR: %s, existsSubnets: %v",
+				vnetCIDR, existSubnets)
+		}
+		// Prefix length can only be 64 in case of IPv6 address prefixes in subnets.
+		_, nextSubnet, err := net.ParseCIDR(fmt.Sprintf("%s/64", ip))
+		return nextSubnet, err
 	}
-	return true, nil
-}
 
-func getNextSubnet(vnetCIDR string, existSubnets []string) (string, error) {
+	// IPv4
 	intIPArray, vNetMask, err := cidrString2intArray(vnetCIDR)
 	if err != nil {
-		return "", fmt.Errorf("Unexpected vnet address CIDR")
+		return nil, fmt.Errorf("unexpected vnet address CIDR: %v", err)
 	}
 	root := initIPTreeRoot(vNetMask)
-	for _, subnet := range existSubnets {
+	for _, subnet := range existSubnetsWithSameIPFamily {
 		subnetIPArray, subnetPrefix, err := cidrString2intArray(subnet)
 		if err != nil {
-			return "", fmt.Errorf("Unexpected subnet address prefix")
+			return nil, fmt.Errorf("unexpected subnet address prefix: %v", err)
 		}
 		setOccupiedByPrefix(root, subnetIPArray, subnetPrefix)
 	}
 	retArray, retMask := findNodeUsable(root, intIPArray)
 	ret := prefixIntArray2String(retArray, retMask)
-	return ret, nil
+	_, retSubnet, err := net.ParseCIDR(ret)
+	return retSubnet, err
 }
 
 type ipNode struct {
@@ -157,7 +199,7 @@ func prefixIntArray2String(ret []int, prefix int) (ip string) {
 func cidrString2intArray(ip string) (ret []int, prefix int, err error) {
 	splitPos := strings.Index(ip, "/")
 	if splitPos == -1 {
-		err = fmt.Errorf("It is not a valid CIDR format")
+		err = fmt.Errorf("it is not a valid CIDR format")
 	} else {
 		prefix, err = strconv.Atoi(ip[splitPos+1:])
 	}
diff --git a/tests/e2e/utils/ip_utils_test.go b/tests/e2e/utils/ip_utils_test.go
index fda8f25b3..6821e5ad9 100644
--- a/tests/e2e/utils/ip_utils_test.go
+++ b/tests/e2e/utils/ip_utils_test.go
@@ -51,24 +51,37 @@ func TestPrefixIntArray2String(t *testing.T) {
 	assert.Equal(t, cidrIP, cidrSuppose)
 }
 
-func TestValidateIPInCIDR(t *testing.T) {
-	cidr := "10.24.0.0/16"
-	ip1 := "10.24.0.100"
-	ip2 := "20.24.0.0"
-	flag1, _ := ValidateIPInCIDR(ip1, cidr)
-	assert.Equal(t, flag1, true)
-	flag2, _ := ValidateIPInCIDR(ip2, cidr)
-	assert.Equal(t, flag2, false)
-}
-
 func TestGetNextSubnet(t *testing.T) {
-	vNetCIDR := "10.24.0.0/16"
-	existSubnets := []string{
-		"10.24.0.0/24",
-		"10.24.1.0/24",
+	tests := []struct {
+		vNetCIDR     string
+		existSubnets []string
+		resultIP     string
+		resultMask   []byte
+	}{
+		{
+			vNetCIDR: "10.24.0.0/16",
+			existSubnets: []string{
+				"10.24.0.0/24",
+				"10.24.1.0/24",
+			},
+			resultIP:   "10.24.2.0",
+			resultMask: []byte{255, 255, 255, 0},
+		},
+		{
+			vNetCIDR: "2001:1234:5678:9a00::/56",
+			existSubnets: []string{
+				"2001:1234:5678:9abc::/64",
+				"2001:1234:5678:9abd::/64",
+			},
+			resultIP:   "2001:1234:5678:9aff::",
+			resultMask: []byte{255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255},
+		},
+	}
+
+	for _, tc := range tests {
+		cidr, err := getNextSubnet(tc.vNetCIDR, tc.existSubnets)
+		assert.Empty(t, err)
+		assert.Equal(t, tc.resultIP, cidr.IP.String())
+		assert.Equal(t, tc.resultMask, []byte(cidr.Mask))
 	}
-	cidrResult := "10.24.2.0/24"
-	cidr, err := getNextSubnet(vNetCIDR, existSubnets)
-	assert.Empty(t, err)
-	assert.Equal(t, cidrResult, cidr)
 }
diff --git a/tests/e2e/utils/network_utils.go b/tests/e2e/utils/network_utils.go
index c4ccb1ff7..a2b57bf57 100644
--- a/tests/e2e/utils/network_utils.go
+++ b/tests/e2e/utils/network_utils.go
@@ -132,19 +132,51 @@ func (azureTestClient *AzureTestClient) DeleteSubnet(vnetName string, subnetName
 }
 
 // GetNextSubnetCIDR obtains a new ip address which has no overlap with existing subnets.
-func GetNextSubnetCIDR(vnet aznetwork.VirtualNetwork) (string, error) {
+func GetNextSubnetCIDR(vnet aznetwork.VirtualNetwork, ipFamily IPFamily) (*net.IPNet, error) {
 	if len(*vnet.AddressSpace.AddressPrefixes) == 0 {
-		return "", fmt.Errorf("vNet has no prefix")
+		return nil, fmt.Errorf("vNet has no prefix")
 	}
+	// Because of Azure vNet limitation, underlying vNet is dual-stack for
+	// those single stack IPv6 clusters. Pods and Services are single stack
+	// IPv6 while Nodes and routes are dual-stack.
 	vnetCIDR := (*vnet.AddressSpace.AddressPrefixes)[0]
+	if ipFamily == IPv6 {
+		for i := range *vnet.AddressSpace.AddressPrefixes {
+			addrPrefix := (*vnet.AddressSpace.AddressPrefixes)[i]
+			ip, _, err := net.ParseCIDR(addrPrefix)
+			if err != nil {
+				return nil, fmt.Errorf("failed to parse address prefix CIDR: %v", err)
+			}
+			if ip.To4() == nil {
+				vnetCIDR = addrPrefix
+				break
+			}
+		}
+	}
+
 	var existSubnets []string
 	for _, subnet := range *vnet.Subnets {
-		subnet := *subnet.AddressPrefix
-		existSubnets = append(existSubnets, subnet)
+		if subnet.AddressPrefix != nil {
+			existSubnets = append(existSubnets, *subnet.AddressPrefix)
+			continue
+		}
+		if subnet.AddressPrefixes == nil {
+			return nil, fmt.Errorf("subnet AddressPrefix and AddressPrefixes shouldn't be both nil")
+		}
+		existSubnets = append(existSubnets, *subnet.AddressPrefixes...)
 	}
 	return getNextSubnet(vnetCIDR, existSubnets)
 }
 
+// isCIDRIPv6 checks if the provided CIDR is an IPv6 one.
+func isCIDRIPv6(cidr string) (bool, error) {
+	ip, _, err := net.ParseCIDR(cidr)
+	if err != nil {
+		return false, fmt.Errorf("failed to parse CIDR: %q", cidr)
+	}
+	return ip.To4() == nil, nil
+}
+
 // getSecurityGroupList returns the list of security groups in the cluster resource group.
 func (azureTestClient *AzureTestClient) getSecurityGroupList() (result aznetwork.SecurityGroupListResultPage, err error) {
 	Logf("Getting virtual network list")
@@ -187,6 +219,7 @@ func (azureTestClient *AzureTestClient) GetClusterSecurityGroups() (ret []aznetw
 
 // CreateLoadBalancerServiceManifest return the specific service to be created
 func CreateLoadBalancerServiceManifest(name string, annotation map[string]string, labels map[string]string, namespace string, ports []v1.ServicePort) *v1.Service {
+	ipFamilyPreferDS := v1.IPFamilyPolicyPreferDualStack
 	return &v1.Service{
 		ObjectMeta: metav1.ObjectMeta{
 			Name:        name,
@@ -194,9 +227,10 @@ func CreateLoadBalancerServiceManifest(name string, annotation map[string]string
 			Annotations: annotation,
 		},
 		Spec: v1.ServiceSpec{
-			Selector: labels,
-			Ports:    ports,
-			Type:     "LoadBalancer",
+			Selector:       labels,
+			Ports:          ports,
+			Type:           "LoadBalancer",
+			IPFamilyPolicy: &ipFamilyPreferDS,
 		},
 	}
 }
@@ -230,9 +264,10 @@ func WaitCreatePIPPrefix(
 ) (aznetwork.PublicIPPrefix, error) {
 	Logf("Creating PublicIPPrefix named %s", name)
 
+	var prefix aznetwork.PublicIPPrefix
+
 	resourceClient := cli.createPublicIPPrefixesClient()
 	_, err := resourceClient.CreateOrUpdate(context.Background(), rgName, name, parameter)
-	var prefix aznetwork.PublicIPPrefix
 	if err != nil {
 		return prefix, err
 	}
@@ -373,7 +408,23 @@ func SelectAvailablePrivateIP(tc *AzureTestClient) (string, error) {
 		for _, sn := range *vNet.Subnets {
 			// if there is more than one subnet, select the first one we find.
 			if !strings.Contains(*sn.Name, "controlplane") && !strings.Contains(*sn.Name, "control-plane") {
-				subnet = *sn.AddressPrefix
+				if tc.IPFamily == DualStack {
+					subnet = (*sn.AddressPrefixes)[0]
+				} else if tc.IPFamily == IPv4 {
+					subnet = *sn.AddressPrefix
+				} else {
+					for i := range *sn.AddressPrefixes {
+						addrPrefix := (*sn.AddressPrefixes)[i]
+						isIPv6, err := isCIDRIPv6(addrPrefix)
+						if err != nil {
+							return "", err
+						}
+						if isIPv6 {
+							subnet = addrPrefix
+							break
+						}
+					}
+				}
 				break
 			}
 		}
@@ -382,9 +433,15 @@ func SelectAvailablePrivateIP(tc *AzureTestClient) (string, error) {
 	if err != nil {
 		return "", fmt.Errorf("failed to parse subnet CIDR in vNet %s: %w", to.String(vNet.Name), err)
 	}
+
 	baseIP := ip.To4()
+	pos := 3
+	if tc.IPFamily == IPv6 {
+		baseIP = ip.To16()
+		pos = 15
+	}
 	for i := 0; i <= 254; i++ {
-		baseIP[3]++
+		baseIP[pos]++
 		IP := baseIP.String()
 		ret, err := vNetClient.CheckIPAddressAvailability(context.Background(), tc.GetResourceGroup(), to.String(vNet.Name), IP)
 		if err != nil {
@@ -395,7 +452,7 @@ func SelectAvailablePrivateIP(tc *AzureTestClient) (string, error) {
 			return IP, nil
 		}
 	}
-	return "", fmt.Errorf("Find no availabePrivateIP in subnet CIDR %s", subnet)
+	return "", fmt.Errorf("find no availabePrivateIP in subnet CIDR %s", subnet)
 }
 
 // GetPublicIPFromAddress finds public ip according to ip address
diff --git a/tests/e2e/utils/service_utils.go b/tests/e2e/utils/service_utils.go
index 62fa7492c..9eecaed83 100644
--- a/tests/e2e/utils/service_utils.go
+++ b/tests/e2e/utils/service_utils.go
@@ -198,7 +198,7 @@ func ValidateServiceConnectivity(ns, execPod, serviceIP string, port int, protoc
 			Logf("Expected output to contain 'succeeded', got %q; retrying...", stdout)
 			return false, nil
 		}
-		Logf("Validation succeeded: Service addr %s:%d with protocol %v", serviceIP, port, protocol)
+		Logf("Validation succeeded: Service addr %s and port %d with protocol %v", serviceIP, port, protocol)
 		return true, nil
 	})
 	return pollErr