Repository: kubernetes-sigs/cloud-provider-azure
Pull request #2413 (2022-09-27 23:40:34)

VMSS Flex Support: Cloud Node Manager should query resourceId field of IMDS endpoint to retrieve providerID
<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide/first-contribution.md#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?
/kind feature
<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature
/kind design

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->

#### What this PR does / why we need it:
Error Description:
When using VMSS Flex as K8s nodes, if the VM is created by using the VMSS default VM profile and increasing the capacity, then Cloud Node Manager (CNM) will generate the wrong ProviderID for the node.
 
Root Causes:
There are 2 issues in the existing CNM code that cause the error:
 
Issue 1: CNM UNSAFELY distinguishes Uniform VM and Non Uniform VM by just checking if the VM name contains underscore (_): if the VM name contains underscore, then CNM treats the VM as VMSS Uniform node, then further assumes the part before underscore is vmss name; if the VM name does not contain underscore, then CNM treats the VM as non Uniform node : [https://github.com/kubernetes-sigs/cloud-provider-azure/blob/master/pkg/provider/azure_instances.go#L394-L403](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubernetes-sigs%2Fcloud-provider-azure%2Fblob%2Fmaster%2Fpkg%2Fprovider%2Fazure_instances.go%23L394-L403&data=05%7C01%7CMingyang.Zheng%40microsoft.com%7Cc54c9eca910b449c47d508da9d1f9f69%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637995056425554883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Cz7VTKMgpJgJXIdLgIOYeQSOZcSLNE0yqS520NqXw2E%3D&reserved=0)
 
Issue 2: By using the node type checking result in Issue 1, CNM manually combine resourceGroupName, subId, vmName to generate a providerID: [https://github.com/kubernetes-sigs/cloud-provider-azure/blob/8752101d1949db993c918052eb27a58c439c312b/pkg/provider/azure_vmss.go#L611-L617](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubernetes-sigs%2Fcloud-provider-azure%2Fblob%2F8752101d1949db993c918052eb27a58c439c312b%2Fpkg%2Fprovider%2Fazure_vmss.go%23L611-L617&data=05%7C01%7CMingyang.Zheng%40microsoft.com%7Cc54c9eca910b449c47d508da9d1f9f69%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637995056425554883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jyftxM2BX9Fb%2FG4eDR2knqt%2FkH8y3VcY5ykw2sfFm%2Fo%3D&reserved=0)
 
Why These 2 Issues Cause Wrong ProviderID:
When using VMSS Flex VM as K8s cluster nodes, if the VM is provisioned by using the VMSS default VM profile, then the VM Name always have VMSS Flex name as prefiex (vmssFlexName_). In this case, CNM finds the underscore behind the vmssFlexName and judges it as a VMSS Uniform VM because of Issue 1. Then CNM manually generate a VMSS Uniform format ProviderID by combining the resourceGroup and subId because of Issue 2.

Proposal:
CNM can directly use the resourceId field in the IMDS response as ProviderID, no matter what the vm type it is: [https://learn.microsoft.com/en-us/azure/virtual-machines/linux/instance-metadata-service?tabs=linux#access-azure-instance-metadata-service](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flearn.microsoft.com%2Fen-us%2Fazure%2Fvirtual-machines%2Flinux%2Finstance-metadata-service%3Ftabs%3Dlinux%23access-azure-instance-metadata-service&data=05%7C01%7CMingyang.Zheng%40microsoft.com%7Cc54c9eca910b449c47d508da9d1f9f69%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637995056425554883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MwwXqfDSK6c%2F9Nc05egWTpaIX%2FIdynJiEEn3%2BoE5Xts%3D&reserved=0).

#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Refer to #639 

#### Special notes for your reviewer:

#### Does this PR introduce a user-facing change?
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://git.k8s.io/community/contributors/guide/release-notes.md
-->
```release-note
NONE
```

#### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

<!--
This section can be blank if this pull request does not require a release note.

When adding links which point to resources within git repositories, like
KEPs or supporting documentation, please reference a specific commit and avoid
linking directly to the master branch. This ensures that links reference a
specific point in time, rather than a document that may change over time.

See here for guidance on getting permanent links to files: https://help.github.com/en/articles/getting-permanent-links-to-files

Please use the following format for linking documentation:
- [KEP]: <link>
- [Usage]: <link>
- [Other doc]: <link>
-->
```docs

```


k8s-ci-robot: Hi @zmyzheng. Thanks for your PR.

I'm waiting for a [kubernetes-sigs](https://github.com/orgs/kubernetes-sigs/people) member to verify that this patch is reasonable to test. If it is, they should reply with `/ok-to-test` on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should [join the org](https://git.k8s.io/community/community-membership.md#member) to skip this step.

Once the patch is verified, the new status will be reflected by the `ok-to-test` label.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>

k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/cloud-provider-azure/pull/2413#" title="Author self-approved">zmyzheng</a>*
**Once this PR has been reviewed and has the lgtm label**, please assign jwtty for approval by writing `/assign @jwtty` in a comment. For more information see:[The Kubernetes Code Review Process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process).

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcloud-provider-azure).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["jwtty"]} -->
netlify[bot]: ### <span aria-hidden="true">👷</span> Deploy Preview for *kubernetes-sigs-cloud-provide-azure* processing.


|  Name | Link |
|---------------------------------|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | 470881350a5005396e3129e8ccba86be72551465 |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/kubernetes-sigs-cloud-provide-azure/deploys/633389f3469ccf0008998784 |
coveralls: 
[![Coverage Status](https://coveralls.io/builds/52814845/badge)](https://coveralls.io/builds/52814845)

Coverage decreased (-0.04%) to 80.313% when pulling **470881350a5005396e3129e8ccba86be72551465 on zmyzheng:vmssflex-fix-wrong-providerid** into **628624958d351069e464d5cf3fc135fc4100c1cd on kubernetes-sigs:master**.

nilo19: /ok-to-test

Patch:
diff --git a/pkg/provider/azure.go b/pkg/provider/azure.go
index bca068322..3fe132006 100644
--- a/pkg/provider/azure.go
+++ b/pkg/provider/azure.go
@@ -38,7 +38,6 @@ import (
 	clientset "k8s.io/client-go/kubernetes"
 	"k8s.io/client-go/kubernetes/scheme"
 	v1core "k8s.io/client-go/kubernetes/typed/core/v1"
-	corelisters "k8s.io/client-go/listers/core/v1"
 	"k8s.io/client-go/tools/cache"
 	"k8s.io/client-go/tools/record"
 	"k8s.io/client-go/util/flowcontrol"
@@ -144,8 +143,8 @@ type Config struct {
 	// The name of the scale set that should be used as the load balancer backend.
 	// If this is set, the Azure cloudprovider will only add nodes from that scale set to the load
 	// balancer backend pool. If this is not set, and multiple agent pools (scale sets) are used, then
-	// the cloudprovider will try to add all nodes to a single backend pool which is forbidden in the basic sku.
-	// In other words, if you use multiple agent pools (scale sets), and loadBalancerSku is set to basic, you MUST set this field.
+	// the cloudprovider will try to add all nodes to a single backend pool which is forbidden.
+	// In other words, if you use multiple agent pools (scale sets), you MUST set this field.
 	PrimaryScaleSetName string `json:"primaryScaleSetName,omitempty" yaml:"primaryScaleSetName,omitempty"`
 	// Tags determines what tags shall be applied to the shared resources managed by controller manager, which
 	// includes load balancer, security group and route table. The supported format is `a=b,c=d,...`. After updated
@@ -360,11 +359,6 @@ type Cloud struct {
 	// use LB frontEndIpConfiguration ID as the key and search for PLS attached to the frontEnd
 	plsCache *azcache.TimedCache
 
-	// Add service lister to always get latest service
-	serviceLister corelisters.ServiceLister
-	// node-sync-loop routine and service-reconcile routine should not update LoadBalancer at the same time
-	serviceReconcileLock sync.Mutex
-
 	*ManagedDiskController
 	*controllerCommon
 }
@@ -1064,14 +1058,9 @@ func (az *Cloud) SetInformers(informerFactory informers.SharedInformerFactory) {
 				}
 			}
 			az.updateNodeCaches(node, nil)
-
-			klog.V(4).Infof("Removing node %s from VMSet cache.", node.Name)
-			_ = az.VMSet.DeleteCacheForNode(node.Name)
 		},
 	})
 	az.nodeInformerSynced = nodeInformer.HasSynced
-
-	az.serviceLister = informerFactory.Core().V1().Services().Lister()
 }
 
 // updateNodeCaches updates local cache for node's zones and external resource groups.
diff --git a/pkg/provider/azure_controller_standard.go b/pkg/provider/azure_controller_standard.go
index a3868d153..7e4b3537d 100644
--- a/pkg/provider/azure_controller_standard.go
+++ b/pkg/provider/azure_controller_standard.go
@@ -97,7 +97,7 @@ func (as *availabilitySet) AttachDisk(ctx context.Context, nodeName types.NodeNa
 	klog.V(2).Infof("azureDisk - update(%s): vm(%s) - attach disk list(%s)", nodeResourceGroup, vmName, diskMap)
 	// Invalidate the cache right after updating
 	defer func() {
-		_ = as.DeleteCacheForNode(vmName)
+		_ = as.cloud.vmCache.Delete(vmName)
 	}()
 
 	future, rerr := as.VirtualMachinesClient.UpdateAsync(ctx, nodeResourceGroup, vmName, newVM, "attach_disk")
@@ -118,11 +118,6 @@ func (as *availabilitySet) AttachDisk(ctx context.Context, nodeName types.NodeNa
 	return future, nil
 }
 
-func (as *availabilitySet) DeleteCacheForNode(nodeName string) error {
-	_ = as.cloud.vmCache.Delete(nodeName)
-	return nil
-}
-
 // WaitForUpdateResult waits for the response of the update request
 func (as *availabilitySet) WaitForUpdateResult(ctx context.Context, future *azure.Future, resourceGroupName, source string) error {
 	if rerr := as.VirtualMachinesClient.WaitForUpdateResult(ctx, future, resourceGroupName, source); rerr != nil {
@@ -189,7 +184,7 @@ func (as *availabilitySet) DetachDisk(ctx context.Context, nodeName types.NodeNa
 	klog.V(2).Infof("azureDisk - update(%s): vm(%s) - detach disk list(%s)", nodeResourceGroup, vmName, nodeName, diskMap)
 	// Invalidate the cache right after updating
 	defer func() {
-		_ = as.DeleteCacheForNode(vmName)
+		_ = as.cloud.vmCache.Delete(vmName)
 	}()
 
 	rerr := as.VirtualMachinesClient.Update(ctx, nodeResourceGroup, vmName, newVM, "detach_disk")
@@ -220,7 +215,7 @@ func (as *availabilitySet) UpdateVM(ctx context.Context, nodeName types.NodeName
 	klog.V(2).Infof("azureDisk - update(%s): vm(%s)", nodeResourceGroup, vmName)
 	// Invalidate the cache right after updating
 	defer func() {
-		_ = as.DeleteCacheForNode(vmName)
+		_ = as.cloud.vmCache.Delete(vmName)
 	}()
 
 	rerr := as.VirtualMachinesClient.Update(ctx, nodeResourceGroup, vmName, compute.VirtualMachineUpdate{}, "update_vm")
diff --git a/pkg/provider/azure_controller_vmss.go b/pkg/provider/azure_controller_vmss.go
index 07c1a54ad..a8f01c1e1 100644
--- a/pkg/provider/azure_controller_vmss.go
+++ b/pkg/provider/azure_controller_vmss.go
@@ -103,7 +103,7 @@ func (ss *ScaleSet) AttachDisk(ctx context.Context, nodeName types.NodeName, dis
 
 	// Invalidate the cache right after updating
 	defer func() {
-		_ = ss.DeleteCacheForNode(vmName)
+		_ = ss.deleteCacheForNode(vmName)
 	}()
 
 	klog.V(2).Infof("azureDisk - update(%s): vm(%s) - attach disk list(%s)", nodeResourceGroup, nodeName, diskMap)
@@ -191,7 +191,7 @@ func (ss *ScaleSet) DetachDisk(ctx context.Context, nodeName types.NodeName, dis
 	}
 	// Invalidate the cache right after updating
 	defer func() {
-		_ = ss.DeleteCacheForNode(vmName)
+		_ = ss.deleteCacheForNode(vmName)
 	}()
 
 	klog.V(2).Infof("azureDisk - update(%s): vm(%s) - detach disk list(%s)", nodeResourceGroup, nodeName, diskMap)
@@ -229,7 +229,7 @@ func (ss *ScaleSet) UpdateVM(ctx context.Context, nodeName types.NodeName) error
 
 	// Invalidate the cache right after updating
 	defer func() {
-		_ = ss.DeleteCacheForNode(vmName)
+		_ = ss.deleteCacheForNode(vmName)
 	}()
 
 	klog.V(2).Infof("azureDisk - update(%s): vm(%s)", nodeResourceGroup, nodeName)
diff --git a/pkg/provider/azure_controller_vmssflex.go b/pkg/provider/azure_controller_vmssflex.go
index 27b21d05e..a8efe97ec 100644
--- a/pkg/provider/azure_controller_vmssflex.go
+++ b/pkg/provider/azure_controller_vmssflex.go
@@ -94,7 +94,7 @@ func (fs *FlexScaleSet) AttachDisk(ctx context.Context, nodeName types.NodeName,
 	}
 
 	defer func() {
-		_ = fs.DeleteCacheForNode(vmName)
+		_ = fs.deleteCacheForNode(vmName)
 	}()
 
 	klog.V(2).Infof("azureDisk - update(%s): vm(%s) - attach disk list(%s)", nodeResourceGroup, vmName, diskMap)
@@ -174,7 +174,7 @@ func (fs *FlexScaleSet) DetachDisk(ctx context.Context, nodeName types.NodeName,
 	}
 
 	defer func() {
-		_ = fs.DeleteCacheForNode(vmName)
+		_ = fs.deleteCacheForNode(vmName)
 	}()
 
 	klog.V(2).Infof("azureDisk - update(%s): vm(%s) - detach disk list(%s)", nodeResourceGroup, vmName, nodeName, diskMap)
@@ -214,7 +214,7 @@ func (fs *FlexScaleSet) UpdateVM(ctx context.Context, nodeName types.NodeName) e
 	}
 
 	defer func() {
-		_ = fs.DeleteCacheForNode(vmName)
+		_ = fs.deleteCacheForNode(vmName)
 	}()
 
 	klog.V(2).Infof("azureDisk - update(%s): vm(%s)", nodeResourceGroup, vmName)
diff --git a/pkg/provider/azure_instance_metadata.go b/pkg/provider/azure_instance_metadata.go
index 708804a19..5e7f916c7 100644
--- a/pkg/provider/azure_instance_metadata.go
+++ b/pkg/provider/azure_instance_metadata.go
@@ -72,6 +72,7 @@ type ComputeMetadata struct {
 	ResourceGroup          string `json:"resourceGroupName,omitempty"`
 	VMScaleSetName         string `json:"vmScaleSetName,omitempty"`
 	SubscriptionID         string `json:"subscriptionId,omitempty"`
+	ResourceID             string `json:"resourceId,omitempty"`
 }
 
 // InstanceMetadata represents instance information.
diff --git a/pkg/provider/azure_instances.go b/pkg/provider/azure_instances.go
index e89630f32..141ada6fe 100644
--- a/pkg/provider/azure_instances.go
+++ b/pkg/provider/azure_instances.go
@@ -385,22 +385,16 @@ func (az *Cloud) getLocalInstanceProviderID(metadata *InstanceMetadata, nodeName
 	resourceGroup := strings.ToLower(metadata.Compute.ResourceGroup)
 	subscriptionID := strings.ToLower(metadata.Compute.SubscriptionID)
 
-	// Compose instanceID based on nodeName for standard instance.
-	if metadata.Compute.VMScaleSetName == "" {
-		return az.getStandardMachineID(subscriptionID, resourceGroup, nodeName), nil
+	if metadata.Compute.ResourceID == "" {
+		// No ResourceID is got from instance metadata service, clean up cache and report errors.
+		_ = az.Metadata.imsCache.Delete(consts.MetadataCacheKey)
+		return "", fmt.Errorf("get empty ResoureceID from instance metadata service")
 	}
 
-	// Get scale set name and instanceID from vmName for vmss.
-	ssName, instanceID, err := extractVmssVMName(metadata.Compute.Name)
-	if err != nil {
-		if errors.Is(err, ErrorNotVmssInstance) {
-			// Compose machineID for standard Node.
-			return az.getStandardMachineID(subscriptionID, resourceGroup, nodeName), nil
-		}
-		return "", err
-	}
-	// Compose instanceID based on ssName and instanceID for vmss instance.
-	return az.getVmssMachineID(subscriptionID, resourceGroup, ssName, instanceID), nil
+	providerID := strings.Replace(metadata.Compute.ResourceID, metadata.Compute.SubscriptionID, subscriptionID, -1)
+	providerID = strings.Replace(providerID, metadata.Compute.ResourceGroup, resourceGroup, -1)
+
+	return providerID, nil
 }
 
 // InstanceTypeByProviderID returns the cloudprovider instance type of the node with the specified unique providerID
diff --git a/pkg/provider/azure_loadbalancer.go b/pkg/provider/azure_loadbalancer.go
index 4c55c90e6..706cf31df 100644
--- a/pkg/provider/azure_loadbalancer.go
+++ b/pkg/provider/azure_loadbalancer.go
@@ -31,7 +31,6 @@ import (
 	"github.com/Azure/go-autorest/autorest/to"
 
 	v1 "k8s.io/api/core/v1"
-	apierrors "k8s.io/apimachinery/pkg/api/errors"
 	utilerrors "k8s.io/apimachinery/pkg/util/errors"
 	"k8s.io/apimachinery/pkg/util/sets"
 	cloudprovider "k8s.io/cloud-provider"
@@ -145,11 +144,6 @@ func (az *Cloud) EnsureLoadBalancer(ctx context.Context, clusterName string, ser
 	// When a client updates the internal load balancer annotation,
 	// the service may be switched from an internal LB to a public one, or vise versa.
 	// Here we'll firstly ensure service do not lie in the opposite LB.
-
-	// Serialize service reconcile process
-	az.serviceReconcileLock.Lock()
-	defer az.serviceReconcileLock.Unlock()
-
 	var err error
 	serviceName := getServiceName(service)
 	mc := metrics.NewMetricContext("services", "ensure_loadbalancer", az.ResourceGroup, az.getNetworkResourceSubscriptionID(), serviceName)
@@ -170,25 +164,8 @@ func (az *Cloud) EnsureLoadBalancer(ctx context.Context, clusterName string, ser
 	return lbStatus, nil
 }
 
-func (az *Cloud) getLatestService(service *v1.Service) (*v1.Service, bool, error) {
-	latestService, err := az.serviceLister.Services(service.Namespace).Get(service.Name)
-	switch {
-	case apierrors.IsNotFound(err):
-		// service absence in store means the service deletion is caught by watcher
-		return nil, false, nil
-	case err != nil:
-		return nil, false, err
-	default:
-		return latestService.DeepCopy(), true, nil
-	}
-}
-
 // UpdateLoadBalancer updates hosts under the specified load balancer.
 func (az *Cloud) UpdateLoadBalancer(ctx context.Context, clusterName string, service *v1.Service, nodes []*v1.Node) error {
-	// Serialize service reconcile process
-	az.serviceReconcileLock.Lock()
-	defer az.serviceReconcileLock.Unlock()
-
 	var err error
 	serviceName := getServiceName(service)
 	mc := metrics.NewMetricContext("services", "update_loadbalancer", az.ResourceGroup, az.getNetworkResourceSubscriptionID(), serviceName)
@@ -199,17 +176,6 @@ func (az *Cloud) UpdateLoadBalancer(ctx context.Context, clusterName string, ser
 		klog.V(5).InfoS("UpdateLoadBalancer Finish", "service", serviceName, "cluster", clusterName, "service_spec", service, "error", err)
 	}()
 
-	// In case UpdateLoadBalancer gets stale service spec, retrieve the latest from lister
-	service, serviceExists, err := az.getLatestService(service)
-	if err != nil {
-		return fmt.Errorf("UpdateLoadBalancer: failed to get latest service %s: %w", service.Name, err)
-	}
-	if !serviceExists {
-		isOperationSucceeded = true
-		klog.V(2).Infof("UpdateLoadBalancer: skipping service %s because service is going to be deleted", service.Name)
-		return nil
-	}
-
 	shouldUpdateLB, err := az.shouldUpdateLoadBalancer(clusterName, service, nodes)
 	if err != nil {
 		return err
@@ -237,10 +203,6 @@ func (az *Cloud) UpdateLoadBalancer(ctx context.Context, clusterName string, ser
 // have multiple underlying components, meaning a Get could say that the LB
 // doesn't exist even if some part of it is still laying around.
 func (az *Cloud) EnsureLoadBalancerDeleted(ctx context.Context, clusterName string, service *v1.Service) error {
-	// Serialize service reconcile process
-	az.serviceReconcileLock.Lock()
-	defer az.serviceReconcileLock.Unlock()
-
 	var err error
 	isInternal := requiresInternalLoadBalancer(service)
 	serviceName := getServiceName(service)
@@ -2637,7 +2599,7 @@ func (az *Cloud) shouldUpdateLoadBalancer(clusterName string, service *v1.Servic
 	}
 
 	_, _, existsLb, _ := az.getServiceLoadBalancer(service, clusterName, nodes, false, existingManagedLBs)
-	return existsLb && service.ObjectMeta.DeletionTimestamp == nil && service.Spec.Type == v1.ServiceTypeLoadBalancer, nil
+	return existsLb && service.ObjectMeta.DeletionTimestamp == nil, nil
 }
 
 func logSafe(s *string) string {
diff --git a/pkg/provider/azure_mock_vmsets.go b/pkg/provider/azure_mock_vmsets.go
index f3f5c6cfc..9af0bf47c 100644
--- a/pkg/provider/azure_mock_vmsets.go
+++ b/pkg/provider/azure_mock_vmsets.go
@@ -30,331 +30,313 @@ import (
 	cache "sigs.k8s.io/cloud-provider-azure/pkg/cache"
 )
 
-// MockVMSet is a mock of VMSet interface.
+// MockVMSet is a mock of VMSet interface
 type MockVMSet struct {
 	ctrl     *gomock.Controller
 	recorder *MockVMSetMockRecorder
 }
 
-// MockVMSetMockRecorder is the mock recorder for MockVMSet.
+// MockVMSetMockRecorder is the mock recorder for MockVMSet
 type MockVMSetMockRecorder struct {
 	mock *MockVMSet
 }
 
-// NewMockVMSet creates a new mock instance.
+// NewMockVMSet creates a new mock instance
 func NewMockVMSet(ctrl *gomock.Controller) *MockVMSet {
 	mock := &MockVMSet{ctrl: ctrl}
 	mock.recorder = &MockVMSetMockRecorder{mock}
 	return mock
 }
 
-// EXPECT returns an object that allows the caller to indicate expected use.
+// EXPECT returns an object that allows the caller to indicate expected use
 func (m *MockVMSet) EXPECT() *MockVMSetMockRecorder {
 	return m.recorder
 }
 
-// AttachDisk mocks base method.
-func (m *MockVMSet) AttachDisk(ctx context.Context, nodeName types.NodeName, diskMap map[string]*AttachDiskOptions) (*azure.Future, error) {
+// GetInstanceIDByNodeName mocks base method
+func (m *MockVMSet) GetInstanceIDByNodeName(name string) (string, error) {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "AttachDisk", ctx, nodeName, diskMap)
-	ret0, _ := ret[0].(*azure.Future)
+	ret := m.ctrl.Call(m, "GetInstanceIDByNodeName", name)
+	ret0, _ := ret[0].(string)
 	ret1, _ := ret[1].(error)
 	return ret0, ret1
 }
 
-// AttachDisk indicates an expected call of AttachDisk.
-func (mr *MockVMSetMockRecorder) AttachDisk(ctx, nodeName, diskMap interface{}) *gomock.Call {
+// GetInstanceIDByNodeName indicates an expected call of GetInstanceIDByNodeName
+func (mr *MockVMSetMockRecorder) GetInstanceIDByNodeName(name interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "AttachDisk", reflect.TypeOf((*MockVMSet)(nil).AttachDisk), ctx, nodeName, diskMap)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetInstanceIDByNodeName", reflect.TypeOf((*MockVMSet)(nil).GetInstanceIDByNodeName), name)
 }
 
-// DeleteCacheForNode mocks base method.
-func (m *MockVMSet) DeleteCacheForNode(nodeName string) error {
+// GetInstanceTypeByNodeName mocks base method
+func (m *MockVMSet) GetInstanceTypeByNodeName(name string) (string, error) {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "DeleteCacheForNode", nodeName)
-	ret0, _ := ret[0].(error)
-	return ret0
+	ret := m.ctrl.Call(m, "GetInstanceTypeByNodeName", name)
+	ret0, _ := ret[0].(string)
+	ret1, _ := ret[1].(error)
+	return ret0, ret1
 }
 
-// DeleteCacheForNode indicates an expected call of DeleteCacheForNode.
-func (mr *MockVMSetMockRecorder) DeleteCacheForNode(nodeName interface{}) *gomock.Call {
+// GetInstanceTypeByNodeName indicates an expected call of GetInstanceTypeByNodeName
+func (mr *MockVMSetMockRecorder) GetInstanceTypeByNodeName(name interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "DeleteCacheForNode", reflect.TypeOf((*MockVMSet)(nil).DeleteCacheForNode), nodeName)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetInstanceTypeByNodeName", reflect.TypeOf((*MockVMSet)(nil).GetInstanceTypeByNodeName), name)
 }
 
-// DetachDisk mocks base method.
-func (m *MockVMSet) DetachDisk(ctx context.Context, nodeName types.NodeName, diskMap map[string]string) error {
+// GetIPByNodeName mocks base method
+func (m *MockVMSet) GetIPByNodeName(name string) (string, string, error) {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "DetachDisk", ctx, nodeName, diskMap)
-	ret0, _ := ret[0].(error)
-	return ret0
+	ret := m.ctrl.Call(m, "GetIPByNodeName", name)
+	ret0, _ := ret[0].(string)
+	ret1, _ := ret[1].(string)
+	ret2, _ := ret[2].(error)
+	return ret0, ret1, ret2
 }
 
-// DetachDisk indicates an expected call of DetachDisk.
-func (mr *MockVMSetMockRecorder) DetachDisk(ctx, nodeName, diskMap interface{}) *gomock.Call {
+// GetIPByNodeName indicates an expected call of GetIPByNodeName
+func (mr *MockVMSetMockRecorder) GetIPByNodeName(name interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "DetachDisk", reflect.TypeOf((*MockVMSet)(nil).DetachDisk), ctx, nodeName, diskMap)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetIPByNodeName", reflect.TypeOf((*MockVMSet)(nil).GetIPByNodeName), name)
 }
 
-// EnsureBackendPoolDeleted mocks base method.
-func (m *MockVMSet) EnsureBackendPoolDeleted(service *v1.Service, backendPoolID, vmSetName string, backendAddressPools *[]network.BackendAddressPool, deleteFromVMSet bool) error {
+// GetPrimaryInterface mocks base method
+func (m *MockVMSet) GetPrimaryInterface(nodeName string) (network.Interface, error) {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "EnsureBackendPoolDeleted", service, backendPoolID, vmSetName, backendAddressPools, deleteFromVMSet)
-	ret0, _ := ret[0].(error)
-	return ret0
+	ret := m.ctrl.Call(m, "GetPrimaryInterface", nodeName)
+	ret0, _ := ret[0].(network.Interface)
+	ret1, _ := ret[1].(error)
+	return ret0, ret1
 }
 
-// EnsureBackendPoolDeleted indicates an expected call of EnsureBackendPoolDeleted.
-func (mr *MockVMSetMockRecorder) EnsureBackendPoolDeleted(service, backendPoolID, vmSetName, backendAddressPools, deleteFromVMSet interface{}) *gomock.Call {
+// GetPrimaryInterface indicates an expected call of GetPrimaryInterface
+func (mr *MockVMSetMockRecorder) GetPrimaryInterface(nodeName interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "EnsureBackendPoolDeleted", reflect.TypeOf((*MockVMSet)(nil).EnsureBackendPoolDeleted), service, backendPoolID, vmSetName, backendAddressPools, deleteFromVMSet)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetPrimaryInterface", reflect.TypeOf((*MockVMSet)(nil).GetPrimaryInterface), nodeName)
 }
 
-// EnsureBackendPoolDeletedFromVMSets mocks base method.
-func (m *MockVMSet) EnsureBackendPoolDeletedFromVMSets(vmSetNamesMap map[string]bool, backendPoolID string) error {
+// GetNodeNameByProviderID mocks base method
+func (m *MockVMSet) GetNodeNameByProviderID(providerID string) (types.NodeName, error) {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "EnsureBackendPoolDeletedFromVMSets", vmSetNamesMap, backendPoolID)
-	ret0, _ := ret[0].(error)
-	return ret0
+	ret := m.ctrl.Call(m, "GetNodeNameByProviderID", providerID)
+	ret0, _ := ret[0].(types.NodeName)
+	ret1, _ := ret[1].(error)
+	return ret0, ret1
 }
 
-// EnsureBackendPoolDeletedFromVMSets indicates an expected call of EnsureBackendPoolDeletedFromVMSets.
-func (mr *MockVMSetMockRecorder) EnsureBackendPoolDeletedFromVMSets(vmSetNamesMap, backendPoolID interface{}) *gomock.Call {
+// GetNodeNameByProviderID indicates an expected call of GetNodeNameByProviderID
+func (mr *MockVMSetMockRecorder) GetNodeNameByProviderID(providerID interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "EnsureBackendPoolDeletedFromVMSets", reflect.TypeOf((*MockVMSet)(nil).EnsureBackendPoolDeletedFromVMSets), vmSetNamesMap, backendPoolID)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetNodeNameByProviderID", reflect.TypeOf((*MockVMSet)(nil).GetNodeNameByProviderID), providerID)
 }
 
-// EnsureHostInPool mocks base method.
-func (m *MockVMSet) EnsureHostInPool(service *v1.Service, nodeName types.NodeName, backendPoolID, vmSetName string) (string, string, string, *compute.VirtualMachineScaleSetVM, error) {
+// GetZoneByNodeName mocks base method
+func (m *MockVMSet) GetZoneByNodeName(name string) (cloud_provider.Zone, error) {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "EnsureHostInPool", service, nodeName, backendPoolID, vmSetName)
-	ret0, _ := ret[0].(string)
-	ret1, _ := ret[1].(string)
-	ret2, _ := ret[2].(string)
-	ret3, _ := ret[3].(*compute.VirtualMachineScaleSetVM)
-	ret4, _ := ret[4].(error)
-	return ret0, ret1, ret2, ret3, ret4
+	ret := m.ctrl.Call(m, "GetZoneByNodeName", name)
+	ret0, _ := ret[0].(cloud_provider.Zone)
+	ret1, _ := ret[1].(error)
+	return ret0, ret1
 }
 
-// EnsureHostInPool indicates an expected call of EnsureHostInPool.
-func (mr *MockVMSetMockRecorder) EnsureHostInPool(service, nodeName, backendPoolID, vmSetName interface{}) *gomock.Call {
+// GetZoneByNodeName indicates an expected call of GetZoneByNodeName
+func (mr *MockVMSetMockRecorder) GetZoneByNodeName(name interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "EnsureHostInPool", reflect.TypeOf((*MockVMSet)(nil).EnsureHostInPool), service, nodeName, backendPoolID, vmSetName)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetZoneByNodeName", reflect.TypeOf((*MockVMSet)(nil).GetZoneByNodeName), name)
 }
 
-// EnsureHostsInPool mocks base method.
-func (m *MockVMSet) EnsureHostsInPool(service *v1.Service, nodes []*v1.Node, backendPoolID, vmSetName string) error {
+// GetPrimaryVMSetName mocks base method
+func (m *MockVMSet) GetPrimaryVMSetName() string {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "EnsureHostsInPool", service, nodes, backendPoolID, vmSetName)
-	ret0, _ := ret[0].(error)
+	ret := m.ctrl.Call(m, "GetPrimaryVMSetName")
+	ret0, _ := ret[0].(string)
 	return ret0
 }
 
-// EnsureHostsInPool indicates an expected call of EnsureHostsInPool.
-func (mr *MockVMSetMockRecorder) EnsureHostsInPool(service, nodes, backendPoolID, vmSetName interface{}) *gomock.Call {
+// GetPrimaryVMSetName indicates an expected call of GetPrimaryVMSetName
+func (mr *MockVMSetMockRecorder) GetPrimaryVMSetName() *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "EnsureHostsInPool", reflect.TypeOf((*MockVMSet)(nil).EnsureHostsInPool), service, nodes, backendPoolID, vmSetName)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetPrimaryVMSetName", reflect.TypeOf((*MockVMSet)(nil).GetPrimaryVMSetName))
 }
 
-// GetAgentPoolVMSetNames mocks base method.
-func (m *MockVMSet) GetAgentPoolVMSetNames(nodes []*v1.Node) (*[]string, error) {
+// GetVMSetNames mocks base method
+func (m *MockVMSet) GetVMSetNames(service *v1.Service, nodes []*v1.Node) (*[]string, error) {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "GetAgentPoolVMSetNames", nodes)
+	ret := m.ctrl.Call(m, "GetVMSetNames", service, nodes)
 	ret0, _ := ret[0].(*[]string)
 	ret1, _ := ret[1].(error)
 	return ret0, ret1
 }
 
-// GetAgentPoolVMSetNames indicates an expected call of GetAgentPoolVMSetNames.
-func (mr *MockVMSetMockRecorder) GetAgentPoolVMSetNames(nodes interface{}) *gomock.Call {
-	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetAgentPoolVMSetNames", reflect.TypeOf((*MockVMSet)(nil).GetAgentPoolVMSetNames), nodes)
-}
-
-// GetDataDisks mocks base method.
-func (m *MockVMSet) GetDataDisks(nodeName types.NodeName, crt cache.AzureCacheReadType) ([]compute.DataDisk, *string, error) {
-	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "GetDataDisks", nodeName, crt)
-	ret0, _ := ret[0].([]compute.DataDisk)
-	ret1, _ := ret[1].(*string)
-	ret2, _ := ret[2].(error)
-	return ret0, ret1, ret2
-}
-
-// GetDataDisks indicates an expected call of GetDataDisks.
-func (mr *MockVMSetMockRecorder) GetDataDisks(nodeName, crt interface{}) *gomock.Call {
+// GetVMSetNames indicates an expected call of GetVMSetNames
+func (mr *MockVMSetMockRecorder) GetVMSetNames(service, nodes interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetDataDisks", reflect.TypeOf((*MockVMSet)(nil).GetDataDisks), nodeName, crt)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetVMSetNames", reflect.TypeOf((*MockVMSet)(nil).GetVMSetNames), service, nodes)
 }
 
-// GetIPByNodeName mocks base method.
-func (m *MockVMSet) GetIPByNodeName(name string) (string, string, error) {
+// GetNodeVMSetName mocks base method
+func (m *MockVMSet) GetNodeVMSetName(node *v1.Node) (string, error) {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "GetIPByNodeName", name)
+	ret := m.ctrl.Call(m, "GetNodeVMSetName", node)
 	ret0, _ := ret[0].(string)
-	ret1, _ := ret[1].(string)
-	ret2, _ := ret[2].(error)
-	return ret0, ret1, ret2
+	ret1, _ := ret[1].(error)
+	return ret0, ret1
 }
 
-// GetIPByNodeName indicates an expected call of GetIPByNodeName.
-func (mr *MockVMSetMockRecorder) GetIPByNodeName(name interface{}) *gomock.Call {
+// GetNodeVMSetName indicates an expected call of GetNodeVMSetName
+func (mr *MockVMSetMockRecorder) GetNodeVMSetName(node interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetIPByNodeName", reflect.TypeOf((*MockVMSet)(nil).GetIPByNodeName), name)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetNodeVMSetName", reflect.TypeOf((*MockVMSet)(nil).GetNodeVMSetName), node)
 }
 
-// GetInstanceIDByNodeName mocks base method.
-func (m *MockVMSet) GetInstanceIDByNodeName(name string) (string, error) {
+// EnsureHostsInPool mocks base method
+func (m *MockVMSet) EnsureHostsInPool(service *v1.Service, nodes []*v1.Node, backendPoolID, vmSetName string) error {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "GetInstanceIDByNodeName", name)
-	ret0, _ := ret[0].(string)
-	ret1, _ := ret[1].(error)
-	return ret0, ret1
+	ret := m.ctrl.Call(m, "EnsureHostsInPool", service, nodes, backendPoolID, vmSetName)
+	ret0, _ := ret[0].(error)
+	return ret0
 }
 
-// GetInstanceIDByNodeName indicates an expected call of GetInstanceIDByNodeName.
-func (mr *MockVMSetMockRecorder) GetInstanceIDByNodeName(name interface{}) *gomock.Call {
+// EnsureHostsInPool indicates an expected call of EnsureHostsInPool
+func (mr *MockVMSetMockRecorder) EnsureHostsInPool(service, nodes, backendPoolID, vmSetName interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetInstanceIDByNodeName", reflect.TypeOf((*MockVMSet)(nil).GetInstanceIDByNodeName), name)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "EnsureHostsInPool", reflect.TypeOf((*MockVMSet)(nil).EnsureHostsInPool), service, nodes, backendPoolID, vmSetName)
 }
 
-// GetInstanceTypeByNodeName mocks base method.
-func (m *MockVMSet) GetInstanceTypeByNodeName(name string) (string, error) {
+// EnsureHostInPool mocks base method
+func (m *MockVMSet) EnsureHostInPool(service *v1.Service, nodeName types.NodeName, backendPoolID, vmSetName string) (string, string, string, *compute.VirtualMachineScaleSetVM, error) {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "GetInstanceTypeByNodeName", name)
+	ret := m.ctrl.Call(m, "EnsureHostInPool", service, nodeName, backendPoolID, vmSetName)
 	ret0, _ := ret[0].(string)
-	ret1, _ := ret[1].(error)
-	return ret0, ret1
+	ret1, _ := ret[1].(string)
+	ret2, _ := ret[2].(string)
+	ret3, _ := ret[3].(*compute.VirtualMachineScaleSetVM)
+	ret4, _ := ret[4].(error)
+	return ret0, ret1, ret2, ret3, ret4
 }
 
-// GetInstanceTypeByNodeName indicates an expected call of GetInstanceTypeByNodeName.
-func (mr *MockVMSetMockRecorder) GetInstanceTypeByNodeName(name interface{}) *gomock.Call {
+// EnsureHostInPool indicates an expected call of EnsureHostInPool
+func (mr *MockVMSetMockRecorder) EnsureHostInPool(service, nodeName, backendPoolID, vmSetName interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetInstanceTypeByNodeName", reflect.TypeOf((*MockVMSet)(nil).GetInstanceTypeByNodeName), name)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "EnsureHostInPool", reflect.TypeOf((*MockVMSet)(nil).EnsureHostInPool), service, nodeName, backendPoolID, vmSetName)
 }
 
-// GetNodeCIDRMasksByProviderID mocks base method.
-func (m *MockVMSet) GetNodeCIDRMasksByProviderID(providerID string) (int, int, error) {
+// EnsureBackendPoolDeleted mocks base method
+func (m *MockVMSet) EnsureBackendPoolDeleted(service *v1.Service, backendPoolID, vmSetName string, backendAddressPools *[]network.BackendAddressPool, deleteFromVMSet bool) error {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "GetNodeCIDRMasksByProviderID", providerID)
-	ret0, _ := ret[0].(int)
-	ret1, _ := ret[1].(int)
-	ret2, _ := ret[2].(error)
-	return ret0, ret1, ret2
+	ret := m.ctrl.Call(m, "EnsureBackendPoolDeleted", service, backendPoolID, vmSetName, backendAddressPools, deleteFromVMSet)
+	ret0, _ := ret[0].(error)
+	return ret0
 }
 
-// GetNodeCIDRMasksByProviderID indicates an expected call of GetNodeCIDRMasksByProviderID.
-func (mr *MockVMSetMockRecorder) GetNodeCIDRMasksByProviderID(providerID interface{}) *gomock.Call {
+// EnsureBackendPoolDeleted indicates an expected call of EnsureBackendPoolDeleted
+func (mr *MockVMSetMockRecorder) EnsureBackendPoolDeleted(service, backendPoolID, vmSetName, backendAddressPools, deleteFromVMSet interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetNodeCIDRMasksByProviderID", reflect.TypeOf((*MockVMSet)(nil).GetNodeCIDRMasksByProviderID), providerID)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "EnsureBackendPoolDeleted", reflect.TypeOf((*MockVMSet)(nil).EnsureBackendPoolDeleted), service, backendPoolID, vmSetName, backendAddressPools, deleteFromVMSet)
 }
 
-// GetNodeNameByIPConfigurationID mocks base method.
-func (m *MockVMSet) GetNodeNameByIPConfigurationID(ipConfigurationID string) (string, string, error) {
+// EnsureBackendPoolDeletedFromVMSets mocks base method
+func (m *MockVMSet) EnsureBackendPoolDeletedFromVMSets(vmSetNamesMap map[string]bool, backendPoolID string) error {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "GetNodeNameByIPConfigurationID", ipConfigurationID)
-	ret0, _ := ret[0].(string)
-	ret1, _ := ret[1].(string)
-	ret2, _ := ret[2].(error)
-	return ret0, ret1, ret2
+	ret := m.ctrl.Call(m, "EnsureBackendPoolDeletedFromVMSets", vmSetNamesMap, backendPoolID)
+	ret0, _ := ret[0].(error)
+	return ret0
 }
 
-// GetNodeNameByIPConfigurationID indicates an expected call of GetNodeNameByIPConfigurationID.
-func (mr *MockVMSetMockRecorder) GetNodeNameByIPConfigurationID(ipConfigurationID interface{}) *gomock.Call {
+// EnsureBackendPoolDeletedFromVMSets indicates an expected call of EnsureBackendPoolDeletedFromVMSets
+func (mr *MockVMSetMockRecorder) EnsureBackendPoolDeletedFromVMSets(vmSetNamesMap, backendPoolID interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetNodeNameByIPConfigurationID", reflect.TypeOf((*MockVMSet)(nil).GetNodeNameByIPConfigurationID), ipConfigurationID)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "EnsureBackendPoolDeletedFromVMSets", reflect.TypeOf((*MockVMSet)(nil).EnsureBackendPoolDeletedFromVMSets), vmSetNamesMap, backendPoolID)
 }
 
-// GetNodeNameByProviderID mocks base method.
-func (m *MockVMSet) GetNodeNameByProviderID(providerID string) (types.NodeName, error) {
+// AttachDisk mocks base method
+func (m *MockVMSet) AttachDisk(ctx context.Context, nodeName types.NodeName, diskMap map[string]*AttachDiskOptions) (*azure.Future, error) {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "GetNodeNameByProviderID", providerID)
-	ret0, _ := ret[0].(types.NodeName)
+	ret := m.ctrl.Call(m, "AttachDisk", nodeName, diskMap)
+	ret0, _ := ret[0].(*azure.Future)
 	ret1, _ := ret[1].(error)
 	return ret0, ret1
 }
 
-// GetNodeNameByProviderID indicates an expected call of GetNodeNameByProviderID.
-func (mr *MockVMSetMockRecorder) GetNodeNameByProviderID(providerID interface{}) *gomock.Call {
+// AttachDisk indicates an expected call of AttachDisk
+func (mr *MockVMSetMockRecorder) AttachDisk(ctx, nodeName, diskMap interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetNodeNameByProviderID", reflect.TypeOf((*MockVMSet)(nil).GetNodeNameByProviderID), providerID)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "AttachDisk", reflect.TypeOf((*MockVMSet)(nil).AttachDisk), nodeName, diskMap)
 }
 
-// GetNodeVMSetName mocks base method.
-func (m *MockVMSet) GetNodeVMSetName(node *v1.Node) (string, error) {
+// DetachDisk mocks base method
+func (m *MockVMSet) DetachDisk(ctx context.Context, nodeName types.NodeName, diskMap map[string]string) error {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "GetNodeVMSetName", node)
-	ret0, _ := ret[0].(string)
-	ret1, _ := ret[1].(error)
-	return ret0, ret1
+	ret := m.ctrl.Call(m, "DetachDisk", nodeName, diskMap)
+	ret0, _ := ret[0].(error)
+	return ret0
 }
 
-// GetNodeVMSetName indicates an expected call of GetNodeVMSetName.
-func (mr *MockVMSetMockRecorder) GetNodeVMSetName(node interface{}) *gomock.Call {
+// DetachDisk indicates an expected call of DetachDisk
+func (mr *MockVMSetMockRecorder) DetachDisk(ctx, nodeName, diskMap interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetNodeVMSetName", reflect.TypeOf((*MockVMSet)(nil).GetNodeVMSetName), node)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "DetachDisk", reflect.TypeOf((*MockVMSet)(nil).DetachDisk), nodeName, diskMap)
 }
 
-// GetPowerStatusByNodeName mocks base method.
-func (m *MockVMSet) GetPowerStatusByNodeName(name string) (string, error) {
+// WaitForUpdateResult mocks base method
+func (m *MockVMSet) WaitForUpdateResult(ctx context.Context, future *azure.Future, resourceGroupName, source string) error {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "GetPowerStatusByNodeName", name)
-	ret0, _ := ret[0].(string)
-	ret1, _ := ret[1].(error)
-	return ret0, ret1
+	ret := m.ctrl.Call(m, "WaitForUpdateResult", ctx, future, resourceGroupName, source)
+	ret0, _ := ret[0].(error)
+	return ret0
 }
 
-// GetPowerStatusByNodeName indicates an expected call of GetPowerStatusByNodeName.
-func (mr *MockVMSetMockRecorder) GetPowerStatusByNodeName(name interface{}) *gomock.Call {
+// WaitForUpdateResult indicates an expected call of WaitForUpdateResult
+func (mr *MockVMSetMockRecorder) WaitForUpdateResult(ctx, future, resourceGroupName, source interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetPowerStatusByNodeName", reflect.TypeOf((*MockVMSet)(nil).GetPowerStatusByNodeName), name)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "WaitForUpdateResult", reflect.TypeOf((*MockVMSet)(nil).WaitForUpdateResult), ctx, future, resourceGroupName, source)
 }
 
-// GetPrimaryInterface mocks base method.
-func (m *MockVMSet) GetPrimaryInterface(nodeName string) (network.Interface, error) {
+// GetDataDisks mocks base method
+func (m *MockVMSet) GetDataDisks(nodeName types.NodeName, crt cache.AzureCacheReadType) ([]compute.DataDisk, *string, error) {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "GetPrimaryInterface", nodeName)
-	ret0, _ := ret[0].(network.Interface)
-	ret1, _ := ret[1].(error)
-	return ret0, ret1
+	ret := m.ctrl.Call(m, "GetDataDisks", nodeName, crt)
+	ret0, _ := ret[0].([]compute.DataDisk)
+	ret1, _ := ret[1].(*string)
+	ret2, _ := ret[2].(error)
+	return ret0, ret1, ret2
 }
 
-// GetPrimaryInterface indicates an expected call of GetPrimaryInterface.
-func (mr *MockVMSetMockRecorder) GetPrimaryInterface(nodeName interface{}) *gomock.Call {
+// GetDataDisks indicates an expected call of GetDataDisks
+func (mr *MockVMSetMockRecorder) GetDataDisks(nodeName, crt interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetPrimaryInterface", reflect.TypeOf((*MockVMSet)(nil).GetPrimaryInterface), nodeName)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetDataDisks", reflect.TypeOf((*MockVMSet)(nil).GetDataDisks), nodeName, crt)
 }
 
-// GetPrimaryVMSetName mocks base method.
-func (m *MockVMSet) GetPrimaryVMSetName() string {
+// UpdateVM mocks base method
+func (m *MockVMSet) UpdateVM(ctx context.Context, nodeName types.NodeName) error {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "GetPrimaryVMSetName")
-	ret0, _ := ret[0].(string)
+	ret := m.ctrl.Call(m, "UpdateVM", nodeName)
+	ret0, _ := ret[0].(error)
 	return ret0
 }
 
-// GetPrimaryVMSetName indicates an expected call of GetPrimaryVMSetName.
-func (mr *MockVMSetMockRecorder) GetPrimaryVMSetName() *gomock.Call {
+// UpdateVM indicates an expected call of UpdateVM
+func (mr *MockVMSetMockRecorder) UpdateVM(ctx, nodeName interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetPrimaryVMSetName", reflect.TypeOf((*MockVMSet)(nil).GetPrimaryVMSetName))
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "UpdateVM", reflect.TypeOf((*MockVMSet)(nil).UpdateVM), nodeName)
 }
 
-// GetPrivateIPsByNodeName mocks base method.
-func (m *MockVMSet) GetPrivateIPsByNodeName(name string) ([]string, error) {
+// GetPowerStatusByNodeName mocks base method
+func (m *MockVMSet) GetPowerStatusByNodeName(name string) (string, error) {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "GetPrivateIPsByNodeName", name)
-	ret0, _ := ret[0].([]string)
+	ret := m.ctrl.Call(m, "GetPowerStatusByNodeName", name)
+	ret0, _ := ret[0].(string)
 	ret1, _ := ret[1].(error)
 	return ret0, ret1
 }
 
-// GetPrivateIPsByNodeName indicates an expected call of GetPrivateIPsByNodeName.
-func (mr *MockVMSetMockRecorder) GetPrivateIPsByNodeName(name interface{}) *gomock.Call {
+// GetPowerStatusByNodeName indicates an expected call of GetPowerStatusByNodeName
+func (mr *MockVMSetMockRecorder) GetPowerStatusByNodeName(name interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetPrivateIPsByNodeName", reflect.TypeOf((*MockVMSet)(nil).GetPrivateIPsByNodeName), name)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetPowerStatusByNodeName", reflect.TypeOf((*MockVMSet)(nil).GetPowerStatusByNodeName), name)
 }
 
-// GetProvisioningStateByNodeName mocks base method.
+// GetProvisioningStateByNodeName mocks base method
 func (m *MockVMSet) GetProvisioningStateByNodeName(name string) (string, error) {
 	m.ctrl.T.Helper()
 	ret := m.ctrl.Call(m, "GetProvisioningStateByNodeName", name)
@@ -363,66 +345,70 @@ func (m *MockVMSet) GetProvisioningStateByNodeName(name string) (string, error)
 	return ret0, ret1
 }
 
-// GetProvisioningStateByNodeName indicates an expected call of GetProvisioningStateByNodeName.
+// GetProvisioningStateByNodeName indicates an expected call of GetProvisioningStateByNodeName
 func (mr *MockVMSetMockRecorder) GetProvisioningStateByNodeName(name interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
 	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetProvisioningStateByNodeName", reflect.TypeOf((*MockVMSet)(nil).GetProvisioningStateByNodeName), name)
 }
 
-// GetVMSetNames mocks base method.
-func (m *MockVMSet) GetVMSetNames(service *v1.Service, nodes []*v1.Node) (*[]string, error) {
+// GetPrivateIPsByNodeName mocks base method
+func (m *MockVMSet) GetPrivateIPsByNodeName(name string) ([]string, error) {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "GetVMSetNames", service, nodes)
-	ret0, _ := ret[0].(*[]string)
+	ret := m.ctrl.Call(m, "GetPrivateIPsByNodeName", name)
+	ret0, _ := ret[0].([]string)
 	ret1, _ := ret[1].(error)
 	return ret0, ret1
 }
 
-// GetVMSetNames indicates an expected call of GetVMSetNames.
-func (mr *MockVMSetMockRecorder) GetVMSetNames(service, nodes interface{}) *gomock.Call {
+// GetPrivateIPsByNodeName indicates an expected call of GetPrivateIPsByNodeName
+func (mr *MockVMSetMockRecorder) GetPrivateIPsByNodeName(name interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetVMSetNames", reflect.TypeOf((*MockVMSet)(nil).GetVMSetNames), service, nodes)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetPrivateIPsByNodeName", reflect.TypeOf((*MockVMSet)(nil).GetPrivateIPsByNodeName), name)
 }
 
-// GetZoneByNodeName mocks base method.
-func (m *MockVMSet) GetZoneByNodeName(name string) (cloud_provider.Zone, error) {
+// GetNodeNameByIPConfigurationID mocks base method
+func (m *MockVMSet) GetNodeNameByIPConfigurationID(ipConfigurationID string) (string, string, error) {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "GetZoneByNodeName", name)
-	ret0, _ := ret[0].(cloud_provider.Zone)
-	ret1, _ := ret[1].(error)
-	return ret0, ret1
+	ret := m.ctrl.Call(m, "GetNodeNameByIPConfigurationID", ipConfigurationID)
+	ret0, _ := ret[0].(string)
+	ret1, _ := ret[1].(string)
+	ret2, _ := ret[2].(error)
+	return ret0, ret1, ret2
 }
 
-// GetZoneByNodeName indicates an expected call of GetZoneByNodeName.
-func (mr *MockVMSetMockRecorder) GetZoneByNodeName(name interface{}) *gomock.Call {
+// GetNodeNameByIPConfigurationID indicates an expected call of GetNodeNameByIPConfigurationID
+func (mr *MockVMSetMockRecorder) GetNodeNameByIPConfigurationID(ipConfigurationID interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetZoneByNodeName", reflect.TypeOf((*MockVMSet)(nil).GetZoneByNodeName), name)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetNodeNameByIPConfigurationID", reflect.TypeOf((*MockVMSet)(nil).GetNodeNameByIPConfigurationID), ipConfigurationID)
 }
 
-// UpdateVM mocks base method.
-func (m *MockVMSet) UpdateVM(ctx context.Context, nodeName types.NodeName) error {
+// GetNodeCIDRMasksByProviderID mocks base method
+func (m *MockVMSet) GetNodeCIDRMasksByProviderID(providerID string) (int, int, error) {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "UpdateVM", ctx, nodeName)
-	ret0, _ := ret[0].(error)
-	return ret0
+	ret := m.ctrl.Call(m, "GetNodeCIDRMasksByProviderID", providerID)
+	ret0, _ := ret[0].(int)
+	ret1, _ := ret[1].(int)
+	ret2, _ := ret[2].(error)
+	return ret0, ret1, ret2
 }
 
-// UpdateVM indicates an expected call of UpdateVM.
-func (mr *MockVMSetMockRecorder) UpdateVM(ctx, nodeName interface{}) *gomock.Call {
+// GetNodeCIDRMasksByProviderID indicates an expected call of GetNodeCIDRMasksByProviderID
+func (mr *MockVMSetMockRecorder) GetNodeCIDRMasksByProviderID(providerID interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "UpdateVM", reflect.TypeOf((*MockVMSet)(nil).UpdateVM), ctx, nodeName)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetNodeCIDRMasksByProviderID", reflect.TypeOf((*MockVMSet)(nil).GetNodeCIDRMasksByProviderID), providerID)
 }
 
-// WaitForUpdateResult mocks base method.
-func (m *MockVMSet) WaitForUpdateResult(ctx context.Context, future *azure.Future, resourceGroupName, source string) error {
+// GetAgentPoolVMSetNames mocks base method
+func (m *MockVMSet) GetAgentPoolVMSetNames(nodes []*v1.Node) (*[]string, error) {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "WaitForUpdateResult", ctx, future, resourceGroupName, source)
-	ret0, _ := ret[0].(error)
-	return ret0
+	ret := m.ctrl.Call(m, "GetAgentPoolVMSetNames", nodes)
+	ret0, _ := ret[0].(*[]string)
+	ret1, _ := ret[1].(error)
+	return ret0, ret1
 }
 
-// WaitForUpdateResult indicates an expected call of WaitForUpdateResult.
-func (mr *MockVMSetMockRecorder) WaitForUpdateResult(ctx, future, resourceGroupName, source interface{}) *gomock.Call {
+// GetAgentPoolVMSetNames indicates an expected call of GetAgentPoolVMSetNames
+func (mr *MockVMSetMockRecorder) GetAgentPoolVMSetNames(nodes interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "WaitForUpdateResult", reflect.TypeOf((*MockVMSet)(nil).WaitForUpdateResult), ctx, future, resourceGroupName, source)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetAgentPoolVMSetNames", reflect.TypeOf((*MockVMSet)(nil).GetAgentPoolVMSetNames), nodes)
 }
diff --git a/pkg/provider/azure_vmsets.go b/pkg/provider/azure_vmsets.go
index 26e556847..9ebe27533 100644
--- a/pkg/provider/azure_vmsets.go
+++ b/pkg/provider/azure_vmsets.go
@@ -105,7 +105,4 @@ type VMSet interface {
 
 	// GetAgentPoolVMSetNames returns all vmSet names according to the nodes
 	GetAgentPoolVMSetNames(nodes []*v1.Node) (*[]string, error)
-
-	// DeleteCacheForNode removes the node entry from cache.
-	DeleteCacheForNode(nodeName string) error
 }
diff --git a/pkg/provider/azure_vmss.go b/pkg/provider/azure_vmss.go
index 929cf4149..72a8fdd44 100644
--- a/pkg/provider/azure_vmss.go
+++ b/pkg/provider/azure_vmss.go
@@ -487,7 +487,7 @@ func (ss *ScaleSet) GetZoneByNodeName(name string) (cloudprovider.Zone, error) {
 	} else {
 		err = fmt.Errorf("failed to get zone info")
 		klog.Errorf("GetZoneByNodeName: got unexpected error %v", err)
-		_ = ss.DeleteCacheForNode(name)
+		_ = ss.deleteCacheForNode(name)
 		return cloudprovider.Zone{}, err
 	}
 
@@ -1347,7 +1347,7 @@ func (ss *ScaleSet) EnsureHostsInPool(service *v1.Service, nodes []*v1.Node, bac
 
 		// Invalidate the cache since the VMSS VM would be updated.
 		defer func() {
-			_ = ss.DeleteCacheForNode(localNodeName)
+			_ = ss.deleteCacheForNode(localNodeName)
 		}()
 	}
 
@@ -1634,7 +1634,7 @@ func (ss *ScaleSet) EnsureBackendPoolDeleted(service *v1.Service, backendPoolID,
 
 		// Invalidate the cache since the VMSS VM would be updated.
 		defer func() {
-			_ = ss.DeleteCacheForNode(nodeName)
+			_ = ss.deleteCacheForNode(nodeName)
 		}()
 	}
 
@@ -1823,7 +1823,7 @@ func (ss *ScaleSet) GetAgentPoolVMSetNames(nodes []*v1.Node) (*[]string, error)
 			if err != nil {
 				return nil, fmt.Errorf("GetAgentPoolVMSetNames: failed to get availabilitySetNodesCache")
 			}
-			vms := cached.(*availabilitySetNodeEntry).vms
+			vms := cached.(availabilitySetNodeEntry).vms
 			names, err = as.getAgentPoolAvailabilitySets(vms, []*v1.Node{node})
 			if err != nil {
 				return nil, fmt.Errorf("GetAgentPoolVMSetNames: failed to execute getAgentPoolAvailabilitySets: %w", err)
diff --git a/pkg/provider/azure_vmss_cache.go b/pkg/provider/azure_vmss_cache.go
index 62e4ad831..5721be88e 100644
--- a/pkg/provider/azure_vmss_cache.go
+++ b/pkg/provider/azure_vmss_cache.go
@@ -266,59 +266,29 @@ func (ss *ScaleSet) newVMSSVirtualMachinesCache(resourceGroupName, vmssName, cac
 	return azcache.NewTimedcache(vmssVirtualMachinesCacheTTL, getter)
 }
 
-func (ss *ScaleSet) DeleteCacheForNode(nodeName string) error {
-	managedByAS, err := ss.isNodeManagedByAvailabilitySet(nodeName, azcache.CacheReadTypeUnsafe)
-	if err != nil {
-		klog.Warningf("DeleteCacheForNode(%s): failed to check if the node is managed by AvailabilitySet", nodeName)
-		return nil
-	}
-	if managedByAS {
-		cached, err := ss.availabilitySetNodesCache.Get(consts.AvailabilitySetNodesKey, azcache.CacheReadTypeDefault)
-		if err != nil {
-			klog.Warningf("DeleteCacheForNode(%s): failed to get availabilitySetNodesCache: %s", err.Error())
-			return nil
-		}
-
-		ss.lockMap.LockEntry(consts.AvailabilitySetNodesKey)
-		defer ss.lockMap.UnlockEntry(consts.AvailabilitySetNodesKey)
-
-		entry := cached.(*availabilitySetNodeEntry)
-		entry.nodeNames.Delete(nodeName)
-		entry.vmNames.Delete(nodeName)
-		for i := len(entry.vms) - 1; i >= 0; i-- {
-			if strings.EqualFold(to.String(entry.vms[i].Name), nodeName) {
-				entry.vms = append(entry.vms[:i], entry.vms[i+1:]...)
-				break
-			}
-		}
-
-		_ = ss.vmCache.Delete(nodeName)
-
-		return nil
-	}
-
+func (ss *ScaleSet) deleteCacheForNode(nodeName string) error {
 	node, err := ss.getNodeIdentityByNodeName(nodeName, azcache.CacheReadTypeUnsafe)
 	if err != nil {
-		klog.Errorf("DeleteCacheForNode(%s) failed with error: %v", nodeName, err)
+		klog.Errorf("deleteCacheForNode(%s) failed with error: %v", nodeName, err)
 		return err
 	}
 
 	cacheKey, timedcache, err := ss.getVMSSVMCache(node.resourceGroup, node.vmssName)
 	if err != nil {
-		klog.Errorf("DeleteCacheForNode(%s) failed with error: %v", nodeName, err)
+		klog.Errorf("deleteCacheForNode(%s) failed with error: %v", nodeName, err)
 		return err
 	}
 
 	vmcache, err := timedcache.Get(cacheKey, azcache.CacheReadTypeUnsafe)
 	if err != nil {
-		klog.Errorf("DeleteCacheForNode(%s) failed with error: %v", nodeName, err)
+		klog.Errorf("deleteCacheForNode(%s) failed with error: %v", nodeName, err)
 		return err
 	}
 	virtualMachines := vmcache.(*sync.Map)
 	virtualMachines.Delete(nodeName)
 
 	if err := ss.gcVMSSVMCache(); err != nil {
-		klog.Errorf("DeleteCacheForNode(%s) failed to gc stale vmss caches: %v", nodeName, err)
+		klog.Errorf("deleteCacheForNode(%s) failed to gc stale vmss caches: %v", nodeName, err)
 	}
 
 	return nil
@@ -352,7 +322,7 @@ func (ss *ScaleSet) newAvailabilitySetNodesCache() (*azcache.TimedCache, error)
 			return nil, err
 		}
 
-		localCache := &availabilitySetNodeEntry{
+		localCache := availabilitySetNodeEntry{
 			vmNames:   vmNames,
 			nodeNames: nodeNames,
 			vms:       vmList,
@@ -379,7 +349,7 @@ func (ss *ScaleSet) isNodeManagedByAvailabilitySet(nodeName string, crt azcache.
 		return false, err
 	}
 
-	cachedNodes := cached.(*availabilitySetNodeEntry).nodeNames
+	cachedNodes := cached.(availabilitySetNodeEntry).nodeNames
 	// if the node is not in the cache, assume the node has joined after the last cache refresh and attempt to refresh the cache.
 	if !cachedNodes.Has(nodeName) {
 		klog.V(2).Infof("Node %s has joined the cluster since the last VM cache refresh, refreshing the cache", nodeName)
@@ -389,6 +359,6 @@ func (ss *ScaleSet) isNodeManagedByAvailabilitySet(nodeName string, crt azcache.
 		}
 	}
 
-	cachedVMs := cached.(*availabilitySetNodeEntry).vmNames
+	cachedVMs := cached.(availabilitySetNodeEntry).vmNames
 	return cachedVMs.Has(nodeName), nil
 }
diff --git a/pkg/provider/azure_vmssflex_cache.go b/pkg/provider/azure_vmssflex_cache.go
index 1b86b50cb..66e47f394 100644
--- a/pkg/provider/azure_vmssflex_cache.go
+++ b/pkg/provider/azure_vmssflex_cache.go
@@ -321,7 +321,7 @@ func (fs *FlexScaleSet) getVmssFlexByName(vmssFlexName string) (*compute.Virtual
 	return nil, cloudprovider.InstanceNotFound
 }
 
-func (fs *FlexScaleSet) DeleteCacheForNode(nodeName string) error {
+func (fs *FlexScaleSet) deleteCacheForNode(nodeName string) error {
 	vmssFlexID, err := fs.getNodeVmssFlexID(nodeName)
 	if err != nil {
 		return err


Test patch:
diff --git a/pkg/provider/azure_controller_vmss_test.go b/pkg/provider/azure_controller_vmss_test.go
index 70c59e33a..d736b462f 100644
--- a/pkg/provider/azure_controller_vmss_test.go
+++ b/pkg/provider/azure_controller_vmss_test.go
@@ -29,7 +29,6 @@ import (
 	"k8s.io/apimachinery/pkg/types"
 	cloudprovider "k8s.io/cloud-provider"
 
-	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/vmclient/mockvmclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/vmssclient/mockvmssclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/vmssvmclient/mockvmssvmclient"
 	azcache "sigs.k8s.io/cloud-provider-azure/pkg/cache"
@@ -91,8 +90,6 @@ func TestAttachDiskWithVMSS(t *testing.T) {
 		mockVMSSClient.EXPECT().List(gomock.Any(), testCloud.ResourceGroup).Return([]compute.VirtualMachineScaleSet{expectedVMSS}, nil).AnyTimes()
 		mockVMSSClient.EXPECT().Get(gomock.Any(), testCloud.ResourceGroup, scaleSetName).Return(expectedVMSS, nil).MaxTimes(1)
 		mockVMSSClient.EXPECT().CreateOrUpdate(gomock.Any(), testCloud.ResourceGroup, scaleSetName, gomock.Any()).Return(nil).MaxTimes(1)
-		mockVMClient := testCloud.VirtualMachinesClient.(*mockvmclient.MockInterface)
-		mockVMClient.EXPECT().List(gomock.Any(), ss.ResourceGroup).Return([]compute.VirtualMachine{}, nil).AnyTimes()
 
 		expectedVMSSVMs, _, _ := buildTestVirtualMachineEnv(testCloud, scaleSetName, "", 0, test.vmssVMList, "succeeded", false)
 		for _, vmssvm := range expectedVMSSVMs {
@@ -250,9 +247,6 @@ func TestDetachDiskWithVMSS(t *testing.T) {
 			mockVMSSVMClient.EXPECT().Update(gomock.Any(), testCloud.ResourceGroup, scaleSetName, gomock.Any(), gomock.Any(), gomock.Any()).Return(nil).AnyTimes()
 		}
 
-		mockVMClient := testCloud.VirtualMachinesClient.(*mockvmclient.MockInterface)
-		mockVMClient.EXPECT().List(gomock.Any(), ss.ResourceGroup).Return([]compute.VirtualMachine{}, nil).AnyTimes()
-
 		diskMap := map[string]string{}
 		for _, diskName := range test.disks {
 			diskURI := fmt.Sprintf("/subscriptions/%s/resourceGroups/%s/providers/Microsoft.Compute/disks/%s",
@@ -365,9 +359,6 @@ func TestUpdateVMWithVMSS(t *testing.T) {
 			mockVMSSVMClient.EXPECT().Update(gomock.Any(), testCloud.ResourceGroup, scaleSetName, gomock.Any(), gomock.Any(), gomock.Any()).Return(nil).AnyTimes()
 		}
 
-		mockVMClient := testCloud.VirtualMachinesClient.(*mockvmclient.MockInterface)
-		mockVMClient.EXPECT().List(gomock.Any(), ss.ResourceGroup).Return([]compute.VirtualMachine{}, nil).AnyTimes()
-
 		err = ss.UpdateVM(ctx, test.vmssvmName)
 		assert.Equal(t, test.expectedErr, err != nil, "TestCase[%d]: %s, err: %v", i, test.desc, err)
 		if test.expectedErr {
diff --git a/pkg/provider/azure_instances_test.go b/pkg/provider/azure_instances_test.go
index aed4bb8fc..2de63d923 100644
--- a/pkg/provider/azure_instances_test.go
+++ b/pkg/provider/azure_instances_test.go
@@ -117,6 +117,7 @@ func TestInstanceID(t *testing.T) {
 		nodeName            string
 		vmssName            string
 		metadataName        string
+		resourceID          string
 		metadataTemplate    string
 		vmType              string
 		expectedID          string
@@ -130,6 +131,7 @@ func TestInstanceID(t *testing.T) {
 			vmList:              []string{"vm1"},
 			nodeName:            "vm1",
 			metadataName:        "vm1",
+			resourceID:          "/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Compute/virtualMachines/vm1",
 			vmType:              consts.VMTypeStandard,
 			useInstanceMetadata: true,
 			expectedID:          "/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Compute/virtualMachines/vm1",
@@ -140,6 +142,7 @@ func TestInstanceID(t *testing.T) {
 			vmssName:            "vmss1",
 			nodeName:            "vmss1_0",
 			metadataName:        "vmss1_0",
+			resourceID:          "/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Compute/virtualMachineScaleSets/vmss1/virtualMachines/0",
 			vmType:              consts.VMTypeStandard,
 			useInstanceMetadata: true,
 			expectedID:          "/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Compute/virtualMachineScaleSets/vmss1/virtualMachines/0",
@@ -150,6 +153,7 @@ func TestInstanceID(t *testing.T) {
 			vmssName:            "vmss1",
 			nodeName:            "vmss1-0",
 			metadataName:        "vmss1-0",
+			resourceID:          "/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Compute/virtualMachines/vmss1-0",
 			vmType:              consts.VMTypeStandard,
 			useInstanceMetadata: true,
 			expectedID:          "/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Compute/virtualMachines/vmss1-0",
@@ -225,7 +229,7 @@ func TestInstanceID(t *testing.T) {
 			if test.metadataTemplate != "" {
 				fmt.Fprintf(w, test.metadataTemplate)
 			} else {
-				fmt.Fprintf(w, "{\"compute\":{\"name\":\"%s\",\"VMScaleSetName\":\"%s\",\"subscriptionId\":\"subscription\",\"resourceGroupName\":\"rg\"}}", test.metadataName, test.vmssName)
+				fmt.Fprintf(w, "{\"compute\":{\"name\":\"%s\",\"VMScaleSetName\":\"%s\",\"subscriptionId\":\"subscription\",\"resourceGroupName\":\"rg\", \"resourceId\":\"%s\"}}", test.metadataName, test.vmssName, test.resourceID)
 			}
 		}))
 		go func() {
diff --git a/pkg/provider/azure_loadbalancer_test.go b/pkg/provider/azure_loadbalancer_test.go
index 78278438d..a024c127a 100644
--- a/pkg/provider/azure_loadbalancer_test.go
+++ b/pkg/provider/azure_loadbalancer_test.go
@@ -920,7 +920,7 @@ func TestShouldReleaseExistingOwnedPublicIP(t *testing.T) {
 			expectedShouldRelease: false,
 		},
 		{
-			desc:           "existing public ip with no format properties (unit test only?), tags required by annotation, expect release",
+			desc:           "existing public ip with no format properties (unit test only?), tags required by annotation, no release",
 			existingPip:    existingPipWithNoPublicIPAddressFormatProperties,
 			lbShouldExist:  true,
 			lbIsInternal:   false,
@@ -2028,7 +2028,7 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 		{
 			desc: "getExpectedLBRules shall return corresponding probe and lbRule (slb with IPv6)",
 			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
-				consts.ServiceAnnotationLoadBalancerInternal: "true",
+				"service.beta.kubernetes.io/azure-load-balancer-internal": "true",
 			}, true, 80),
 			loadBalancerSku: "standard",
 			expectedProbes:  getDefaultTestProbes("Tcp", ""),
@@ -2037,8 +2037,8 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 		{
 			desc: "getExpectedLBRules shall return corresponding probe and lbRule (slb with HA enabled)",
 			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
-				consts.ServiceAnnotationLoadBalancerEnableHighAvailabilityPorts: "true",
-				consts.ServiceAnnotationLoadBalancerInternal:                    "true",
+				"service.beta.kubernetes.io/azure-load-balancer-enable-high-availability-ports": "true",
+				"service.beta.kubernetes.io/azure-load-balancer-internal":                       "true",
 			}, false, 80),
 			loadBalancerSku: "standard",
 			expectedProbes:  getDefaultTestProbes("Tcp", ""),
@@ -2047,8 +2047,8 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 		{
 			desc: "getExpectedLBRules shall return corresponding probe and lbRule (slb with HA mode and SCTP)",
 			service: getTestService("test1", v1.ProtocolSCTP, map[string]string{
-				consts.ServiceAnnotationLoadBalancerEnableHighAvailabilityPorts: "true",
-				consts.ServiceAnnotationLoadBalancerInternal:                    "true",
+				"service.beta.kubernetes.io/azure-load-balancer-enable-high-availability-ports": "true",
+				"service.beta.kubernetes.io/azure-load-balancer-internal":                       "true",
 			}, false, 80),
 			loadBalancerSku: "standard",
 			expectedRules:   getHATestRules(true, false, v1.ProtocolSCTP),
@@ -2056,8 +2056,8 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 		{
 			desc: "getExpectedLBRules shall return corresponding probe and lbRule (slb with HA enabled multi-ports services)",
 			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
-				consts.ServiceAnnotationLoadBalancerEnableHighAvailabilityPorts: "true",
-				consts.ServiceAnnotationLoadBalancerInternal:                    "true",
+				"service.beta.kubernetes.io/azure-load-balancer-enable-high-availability-ports": "true",
+				"service.beta.kubernetes.io/azure-load-balancer-internal":                       "true",
 			}, false, 80, 8080),
 			loadBalancerSku: "standard",
 			expectedProbes:  getDefaultTestProbes("Tcp", ""),
@@ -2074,7 +2074,7 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 		{
 			desc: "getExpectedLBRules should return tcp probe when invalid protocol is defined",
 			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsRequestPath): "/healthy1",
+				"service.beta.kubernetes.io/port_80_health-probe_request-path": "/healthy1",
 			}, false, 80),
 			loadBalancerSku: "standard",
 			probeProtocol:   "TCP1",
@@ -2084,7 +2084,7 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 		{
 			desc: "getExpectedLBRules should return tcp probe when invalid protocol is defined",
 			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsRequestPath): "/healthy1",
+				"service.beta.kubernetes.io/port_80_health-probe_request-path": "/healthy1",
 			}, false, 80),
 			loadBalancerSku: "basic",
 			probeProtocol:   "TCP1",
@@ -2094,8 +2094,8 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 		{
 			desc: "getExpectedLBRules should return correct rule when deprecated annotations are added",
 			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsRequestPath): "/healthy1",
-				consts.ServiceAnnotationLoadBalancerHealthProbeProtocol:                              "https",
+				"service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path": "/healthy1",
+				"service.beta.kubernetes.io/azure-load-balancer-health-probe-protocol":     "https",
 			}, false, 80),
 			loadBalancerSku: "standard",
 			expectedProbes:  getDefaultTestProbes("Https", "/healthy1"),
@@ -2104,8 +2104,8 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 		{
 			desc: "getExpectedLBRules should return correct rule when deprecated annotations are added",
 			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsRequestPath): "/healthy1",
-				consts.ServiceAnnotationLoadBalancerHealthProbeProtocol:                              "http",
+				"service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path": "/healthy1",
+				"service.beta.kubernetes.io/azure-load-balancer-health-probe-protocol":     "http",
 			}, false, 80),
 			loadBalancerSku: "standard",
 			expectedProbes:  getDefaultTestProbes("Http", "/healthy1"),
@@ -2114,8 +2114,8 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 		{
 			desc: "getExpectedLBRules should return correct rule when deprecated annotations are added",
 			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsRequestPath): "/healthy1",
-				consts.ServiceAnnotationLoadBalancerHealthProbeProtocol:                              "tcp",
+				"service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path": "/healthy1",
+				"service.beta.kubernetes.io/azure-load-balancer-health-probe-protocol":     "tcp",
 			}, false, 80),
 			loadBalancerSku: "standard",
 			expectedProbes:  getDefaultTestProbes("Tcp", ""),
@@ -2124,7 +2124,7 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 		{
 			desc: "getExpectedLBRules should return correct rule when deprecated annotations are added",
 			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsRequestPath): "/healthy1",
+				"service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path": "/healthy1",
 			}, false, 80),
 			loadBalancerSku: "standard",
 			probeProtocol:   "Https",
@@ -2134,8 +2134,8 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 		{
 			desc: "getExpectedLBRules should overwrite value defined in deprecated annotation when deprecated annotations and probe path are defined",
 			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
-				consts.ServiceAnnotationLoadBalancerHealthProbeRequestPath:                           "/healthy1",
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsRequestPath): "/healthy2",
+				"service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path": "/healthy1",
+				"service.beta.kubernetes.io/port_80_health-probe_request-path":             "/healthy2",
 			}, false, 80),
 			loadBalancerSku: "standard",
 			probeProtocol:   "Https",
@@ -2145,9 +2145,9 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 		{
 			desc: "getExpectedLBRules should return error when deprecated tcp health probe annotations and protocols are added and config is not valid",
 			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsProbeInterval): "10",
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsNumOfProbe):    "20",
-				consts.ServiceAnnotationLoadBalancerHealthProbeProtocol:                                "https",
+				"service.beta.kubernetes.io/port_80_health-probe_interval":             "10",
+				"service.beta.kubernetes.io/port_80_health-probe_num-of-probe":         "20",
+				"service.beta.kubernetes.io/azure-load-balancer-health-probe-protocol": "https",
 			}, false, 80),
 			loadBalancerSku: "standard",
 			probeProtocol:   "Tcp",
@@ -2156,9 +2156,9 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 		{
 			desc: "getExpectedLBRules should return error when deprecated tcp health probe annotations and protocols are added and config is not valid",
 			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsProbeInterval): "10",
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsNumOfProbe):    "20",
-				consts.ServiceAnnotationLoadBalancerHealthProbeProtocol:                                "tcp",
+				"service.beta.kubernetes.io/port_80_health-probe_interval":             "10",
+				"service.beta.kubernetes.io/port_80_health-probe_num-of-probe":         "20",
+				"service.beta.kubernetes.io/azure-load-balancer-health-probe-protocol": "tcp",
 			}, false, 80),
 			loadBalancerSku: "standard",
 			probeProtocol:   "Tcp",
@@ -2167,8 +2167,8 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 		{
 			desc: "getExpectedLBRules should return correct rule when health probe annotations are added",
 			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsProbeInterval): "20",
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsNumOfProbe):    "5",
+				"service.beta.kubernetes.io/port_80_health-probe_interval":     "20",
+				"service.beta.kubernetes.io/port_80_health-probe_num-of-probe": "5",
 			}, false, 80),
 			loadBalancerSku: "standard",
 			probeProtocol:   "Https",
@@ -2179,8 +2179,8 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 		{
 			desc: "getExpectedLBRules should return correct rule when health probe annotations are added,default path should be /healthy",
 			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsProbeInterval): "20",
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsNumOfProbe):    "5",
+				"service.beta.kubernetes.io/port_80_health-probe_interval":     "20",
+				"service.beta.kubernetes.io/port_80_health-probe_num-of-probe": "5",
 			}, false, 80),
 			loadBalancerSku: "standard",
 			probeProtocol:   "Http",
@@ -2190,8 +2190,8 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 		{
 			desc: "getExpectedLBRules should return correct rule when tcp health probe annotations are added",
 			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsProbeInterval): "20",
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsNumOfProbe):    "5",
+				"service.beta.kubernetes.io/port_80_health-probe_interval":     "20",
+				"service.beta.kubernetes.io/port_80_health-probe_num-of-probe": "5",
 			}, false, 80),
 			loadBalancerSku: "standard",
 			probeProtocol:   "Tcp",
@@ -2201,8 +2201,8 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 		{
 			desc: "getExpectedLBRules should return error when invalid tcp health probe annotations are added",
 			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsProbeInterval): "20",
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsNumOfProbe):    "5a",
+				"service.beta.kubernetes.io/port_80_health-probe_interval":     "20",
+				"service.beta.kubernetes.io/port_80_health-probe_num-of-probe": "5a",
 			}, false, 80),
 			loadBalancerSku: "standard",
 			probeProtocol:   "Tcp",
@@ -2211,8 +2211,8 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 		{
 			desc: "getExpectedLBRules should return error when invalid tcp health probe annotations are added",
 			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsProbeInterval): "1",
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsNumOfProbe):    "5",
+				"service.beta.kubernetes.io/port_80_health-probe_interval":     "1",
+				"service.beta.kubernetes.io/port_80_health-probe_num-of-probe": "5",
 			}, false, 80),
 			loadBalancerSku: "standard",
 			probeProtocol:   "Tcp",
@@ -2221,8 +2221,8 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 		{
 			desc: "getExpectedLBRules should return error when invalid tcp health probe annotations are added",
 			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsProbeInterval): "10",
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsNumOfProbe):    "1",
+				"service.beta.kubernetes.io/port_80_health-probe_interval":     "10",
+				"service.beta.kubernetes.io/port_80_health-probe_num-of-probe": "1",
 			}, false, 80),
 			loadBalancerSku: "standard",
 			probeProtocol:   "Tcp",
@@ -2231,8 +2231,8 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 		{
 			desc: "getExpectedLBRules should return error when invalid tcp health probe annotations are added",
 			service: getTestService("test1", v1.ProtocolTCP, map[string]string{
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsProbeInterval): "10",
-				consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsNumOfProbe):    "20",
+				"service.beta.kubernetes.io/port_80_health-probe_interval":     "10",
+				"service.beta.kubernetes.io/port_80_health-probe_num-of-probe": "20",
 			}, false, 80),
 			loadBalancerSku: "standard",
 			probeProtocol:   "Tcp",
@@ -2262,9 +2262,9 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 	}{
 		desc: "getExpectedLBRules should expected rules when timeout are added",
 		service: getTestService("test1", v1.ProtocolTCP, map[string]string{
-			consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsProbeInterval): "10",
-			consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsNumOfProbe):    "10",
-			consts.ServiceAnnotationLoadBalancerIdleTimeout:                                        "5",
+			"service.beta.kubernetes.io/port_80_health-probe_interval":        "10",
+			"service.beta.kubernetes.io/port_80_health-probe_num-of-probe":    "10",
+			"service.beta.kubernetes.io/azure-load-balancer-tcp-idle-timeout": "5",
 		}, false, 80),
 		loadBalancerSku: "standard",
 		probeProtocol:   "Tcp",
@@ -2280,8 +2280,8 @@ func TestReconcileLoadBalancerRule(t *testing.T) {
 	rules1[1].Probe.ID = to.StringPtr("/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/loadBalancers/lbname/probes/atest1-TCP-34567")
 	rules1[2].Probe.ID = to.StringPtr("/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/loadBalancers/lbname/probes/atest1-TCP-34567")
 	svc := getTestService("test1", v1.ProtocolTCP, map[string]string{
-		consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsProbeInterval): "10",
-		consts.BuildHealthProbeAnnotationKeyForPort(80, consts.HealthProbeParamsNumOfProbe):    "10",
+		"service.beta.kubernetes.io/port_80_health-probe_interval":     "10",
+		"service.beta.kubernetes.io/port_80_health-probe_num-of-probe": "10",
 	}, false, 80, 443, 421)
 	svc.Spec.ExternalTrafficPolicy = v1.ServiceExternalTrafficPolicyTypeLocal
 	svc.Spec.HealthCheckNodePort = 34567
@@ -4221,42 +4221,30 @@ func TestShouldUpdateLoadBalancer(t *testing.T) {
 	testCases := []struct {
 		desc                   string
 		lbHasDeletionTimestamp bool
-		serviceType            v1.ServiceType
 		existsLb               bool
 		expectedOutput         bool
 	}{
 		{
 			desc:                   "should update a load balancer that does not have a deletion timestamp and exists in Azure",
 			lbHasDeletionTimestamp: false,
-			serviceType:            v1.ServiceTypeLoadBalancer,
 			existsLb:               true,
 			expectedOutput:         true,
 		},
 		{
 			desc:                   "should not update a load balancer that is being deleted / already deleted in K8s",
 			lbHasDeletionTimestamp: true,
-			serviceType:            v1.ServiceTypeLoadBalancer,
-			existsLb:               true,
-			expectedOutput:         false,
-		},
-		{
-			desc:                   "should not update a load balancer that is no longer LoadBalancer type in K8s",
-			lbHasDeletionTimestamp: false,
-			serviceType:            v1.ServiceTypeClusterIP,
 			existsLb:               true,
 			expectedOutput:         false,
 		},
 		{
 			desc:                   "should not update a load balancer that does not exist in Azure",
 			lbHasDeletionTimestamp: false,
-			serviceType:            v1.ServiceTypeLoadBalancer,
 			existsLb:               false,
 			expectedOutput:         false,
 		},
 		{
 			desc:                   "should not update a load balancer that has a deletion timestamp and does not exist in Azure",
 			lbHasDeletionTimestamp: true,
-			serviceType:            v1.ServiceTypeLoadBalancer,
 			existsLb:               false,
 			expectedOutput:         false,
 		},
@@ -4265,7 +4253,6 @@ func TestShouldUpdateLoadBalancer(t *testing.T) {
 	for i, test := range testCases {
 		az := GetTestCloud(ctrl)
 		service := getTestService("test1", v1.ProtocolTCP, nil, false, 80)
-		service.Spec.Type = test.serviceType
 		setMockPublicIPs(az, ctrl, 1)
 		mockLBsClient := mockloadbalancerclient.NewMockInterface(ctrl)
 		az.LoadBalancerClient = mockLBsClient
@@ -4499,8 +4486,8 @@ func TestIsFrontendIPConfigIsUnsafeToDelete(t *testing.T) {
 			unsafe: true,
 		},
 		{
-			desc: "isFrontendIPConfigUnsafeToDelete should return true if there is a " +
-				"outbound rule referencing the frontend IP config",
+			desc: "isFrontendIPConfigUnsafeToDelete should return false if there is a " +
+				"loadBalancing rule from this service referencing the frontend IP config",
 			existingLB: &network.LoadBalancer{
 				Name: to.StringPtr("lb"),
 				LoadBalancerPropertiesFormat: &network.LoadBalancerPropertiesFormat{
@@ -4520,7 +4507,7 @@ func TestIsFrontendIPConfigIsUnsafeToDelete(t *testing.T) {
 		},
 		{
 			desc: "isFrontendIPConfigUnsafeToDelete should return false if there is a " +
-				"loadBalancing rule from this service referencing the frontend IP config",
+				"outbound rule referencing the frontend IP config",
 			existingLB: &network.LoadBalancer{
 				Name: to.StringPtr("lb"),
 				LoadBalancerPropertiesFormat: &network.LoadBalancerPropertiesFormat{
diff --git a/pkg/provider/azure_vmss_cache_test.go b/pkg/provider/azure_vmss_cache_test.go
index d0a82df9e..7dce58274 100644
--- a/pkg/provider/azure_vmss_cache_test.go
+++ b/pkg/provider/azure_vmss_cache_test.go
@@ -28,14 +28,11 @@ import (
 	"github.com/golang/mock/gomock"
 	"github.com/stretchr/testify/assert"
 
-	"k8s.io/apimachinery/pkg/util/sets"
 	cloudprovider "k8s.io/cloud-provider"
 
-	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/vmclient/mockvmclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/vmssclient/mockvmssclient"
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/vmssvmclient/mockvmssvmclient"
 	azcache "sigs.k8s.io/cloud-provider-azure/pkg/cache"
-	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
 	"sigs.k8s.io/cloud-provider-azure/pkg/retry"
 )
 
@@ -88,11 +85,8 @@ func TestVMSSVMCache(t *testing.T) {
 	defer ctrl.Finish()
 
 	vmList := []string{"vmssee6c2000000", "vmssee6c2000001", "vmssee6c2000002"}
-	c := GetTestCloud(ctrl)
-	c.DisableAvailabilitySetNodes = true
-	vmSet, err := newScaleSet(c)
+	ss, err := NewTestScaleSet(ctrl)
 	assert.NoError(t, err)
-	ss := vmSet.(*ScaleSet)
 
 	mockVMSSClient := mockvmssclient.NewMockInterface(ctrl)
 	mockVMSSVMClient := mockvmssvmclient.NewMockInterface(ctrl)
@@ -116,13 +110,13 @@ func TestVMSSVMCache(t *testing.T) {
 		assert.Equal(t, &vm, realVM.AsVirtualMachineScaleSetVM())
 	}
 
-	// validate DeleteCacheForNode().
+	// validate deleteCacheForNode().
 	vm := expectedVMs[0]
 	vmName := to.String(vm.OsProfile.ComputerName)
-	err = ss.DeleteCacheForNode(vmName)
+	err = ss.deleteCacheForNode(vmName)
 	assert.NoError(t, err)
 
-	// the VM should be removed from cache after DeleteCacheForNode().
+	// the VM should be removed from cache after deleteCacheForNode().
 	cacheKey, cache, err := ss.getVMSSVMCache("rg", testVMSSName)
 	assert.NoError(t, err)
 	cached, err := cache.Get(cacheKey, azcache.CacheReadTypeDefault)
@@ -139,32 +133,6 @@ func TestVMSSVMCache(t *testing.T) {
 	assert.Equal(t, &vm, realVM.AsVirtualMachineScaleSetVM())
 }
 
-func TestDeleteCacheForAvailabilitySetNodeInVMSS(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	ss, err := NewTestScaleSet(ctrl)
-	assert.NoError(t, err)
-
-	mockVMClient := mockvmclient.NewMockInterface(ctrl)
-	ss.cloud.VirtualMachinesClient = mockVMClient
-
-	mockVMClient.EXPECT().List(gomock.Any(), gomock.Any()).Return([]compute.VirtualMachine{
-		{Name: to.StringPtr("vm1")},
-	}, nil).AnyTimes()
-	ss.cloud.nodeNames = sets.NewString("vm1")
-
-	err = ss.DeleteCacheForNode("vm1")
-	assert.NoError(t, err)
-
-	cached, err := ss.availabilitySetNodesCache.Get(consts.AvailabilitySetNodesKey, azcache.CacheReadTypeUnsafe)
-	assert.NoError(t, err)
-	entry := cached.(*availabilitySetNodeEntry)
-	assert.Equal(t, 0, len(entry.nodeNames))
-	assert.Equal(t, 0, len(entry.vmNames))
-	assert.Equal(t, 0, len(entry.vms))
-}
-
 func TestVMSSVMCacheWithDeletingNodes(t *testing.T) {
 	ctrl := gomock.NewController(t)
 	defer ctrl.Finish()
diff --git a/pkg/provider/azure_vmss_test.go b/pkg/provider/azure_vmss_test.go
index ea2752273..94dbcc9f0 100644
--- a/pkg/provider/azure_vmss_test.go
+++ b/pkg/provider/azure_vmss_test.go
@@ -2585,9 +2585,6 @@ func TestEnsureBackendPoolDeleted(t *testing.T) {
 		mockVMSSVMClient.EXPECT().List(gomock.Any(), ss.ResourceGroup, testVMSSName, gomock.Any()).Return(expectedVMSSVMs, nil).AnyTimes()
 		mockVMSSVMClient.EXPECT().UpdateVMs(gomock.Any(), ss.ResourceGroup, testVMSSName, gomock.Any(), gomock.Any(), gomock.Any()).Return(test.vmClientErr).Times(test.expectedVMSSVMPutTimes)
 
-		mockVMClient := ss.cloud.VirtualMachinesClient.(*mockvmclient.MockInterface)
-		mockVMClient.EXPECT().List(gomock.Any(), ss.ResourceGroup).Return([]compute.VirtualMachine{}, nil).AnyTimes()
-
 		err = ss.EnsureBackendPoolDeleted(&v1.Service{}, test.backendpoolID, testVMSSName, test.backendAddressPools, true)
 		assert.Equal(t, test.expectedErr, err != nil, test.description+", but an error occurs")
 	}
@@ -2648,9 +2645,6 @@ func TestEnsureBackendPoolDeletedConcurrently(t *testing.T) {
 		*lbBackendpools = append(*lbBackendpools, compute.SubResource{ID: to.StringPtr(testLBBackendpoolID1)})
 	}
 
-	mockVMClient := ss.cloud.VirtualMachinesClient.(*mockvmclient.MockInterface)
-	mockVMClient.EXPECT().List(gomock.Any(), ss.ResourceGroup).Return([]compute.VirtualMachine{}, nil).AnyTimes()
-
 	mockVMSSClient := ss.cloud.VirtualMachineScaleSetsClient.(*mockvmssclient.MockInterface)
 	mockVMSSClient.EXPECT().List(gomock.Any(), ss.ResourceGroup).Return([]compute.VirtualMachineScaleSet{vmss0, vmss1}, nil).AnyTimes()
 	mockVMSSClient.EXPECT().List(gomock.Any(), "rg1").Return(nil, nil).AnyTimes()
diff --git a/tests/e2e/network/ensureloadbalancer.go b/tests/e2e/network/ensureloadbalancer.go
index ace96e6d2..df90f765e 100644
--- a/tests/e2e/network/ensureloadbalancer.go
+++ b/tests/e2e/network/ensureloadbalancer.go
@@ -617,7 +617,7 @@ var _ = Describe("Ensure LoadBalancer", Label(utils.TestSuiteLabelLB), func() {
 	})
 
 	It("should support disabling floating IP in load balancer rule with kubernetes service annotations", func() {
-		By("creating a public IP")
+		By("creating a public IP with tags")
 		ipName := basename + "-public-IP" + string(uuid.NewUUID())[0:4]
 		pip := defaultPublicIPAddress(ipName)
 		pip, err := utils.WaitCreatePIP(tc, ipName, tc.GetResourceGroup(), pip)
diff --git a/tests/e2e/network/service_annotations.go b/tests/e2e/network/service_annotations.go
index 3aaaf6003..0d0ca67c3 100644
--- a/tests/e2e/network/service_annotations.go
+++ b/tests/e2e/network/service_annotations.go
@@ -238,31 +238,6 @@ var _ = Describe("Service with annotation", Label(utils.TestSuiteLabelServiceAnn
 		Expect(ret).To(BeTrue(), "external ip %s is not in the target subnet %s", ip, newSubnetCIDR)
 	})
 
-	It("should support service annotation 'service.beta.kubernetes.io/azure-load-balancer-enable-high-availability-ports'", func() {
-		if !strings.EqualFold(os.Getenv(utils.LoadBalancerSkuEnv), string(network.PublicIPAddressSkuNameStandard)) {
-			Skip("azure-load-balancer-enable-high-availability-ports only work with Standard Load Balancer")
-		}
-
-		annotation := map[string]string{
-			consts.ServiceAnnotationLoadBalancerEnableHighAvailabilityPorts: "true",
-			consts.ServiceAnnotationLoadBalancerInternal:                    "true",
-		}
-		// create service with given annotation and wait it to expose
-		ip := createAndExposeDefaultServiceWithAnnotation(cs, serviceName, ns.Name, labels, annotation, ports)
-		defer func() {
-			utils.Logf("cleaning up test service %s", serviceName)
-			err := utils.DeleteService(cs, ns.Name, serviceName)
-			Expect(err).NotTo(HaveOccurred())
-		}()
-
-		lb := getAzureInternalLoadBalancerFromPrivateIP(tc, ip, "")
-
-		Expect(len(*lb.LoadBalancingRules)).To(Equal(1))
-		rule := (*lb.LoadBalancingRules)[0]
-		Expect(*rule.FrontendPort).To(Equal(int32(0)))
-		Expect(*rule.BackendPort).To(Equal(int32(0)))
-	})
-
 	It("should support service annotation 'service.beta.kubernetes.io/azure-load-balancer-tcp-idle-timeout'", func() {
 		annotation := map[string]string{
 			consts.ServiceAnnotationLoadBalancerIdleTimeout: "5",
@@ -270,11 +245,6 @@ var _ = Describe("Service with annotation", Label(utils.TestSuiteLabelServiceAnn
 
 		// create service with given annotation and wait it to expose
 		publicIP := createAndExposeDefaultServiceWithAnnotation(cs, serviceName, ns.Name, labels, annotation, ports)
-		defer func() {
-			By("Cleaning up service")
-			err := utils.DeleteService(cs, ns.Name, serviceName)
-			Expect(err).NotTo(HaveOccurred())
-		}()
 
 		// get lb from azure client
 		lb := getAzureLoadBalancerFromPIP(tc, publicIP, tc.GetResourceGroup(), "")
@@ -353,62 +323,6 @@ var _ = Describe("Service with annotation", Label(utils.TestSuiteLabelServiceAnn
 		Expect(validateSharedSecurityRuleExists(nsgs, ipList, port)).To(BeTrue(), "Security rule for service %s not exists", serviceName)
 	})
 
-	It("should support service annotation `service.beta.kubernetes.io/azure-additional-public-ips`", func() {
-		By("creating a public IP")
-		ipName := basename + "-public-IP" + string(uuid.NewUUID())[0:4]
-		pip := defaultPublicIPAddress(ipName)
-		pip, err := utils.WaitCreatePIP(tc, ipName, tc.GetResourceGroup(), pip)
-		Expect(err).NotTo(HaveOccurred())
-		additionalPIP := to.String(pip.IPAddress)
-		utils.Logf("created pip with address %s", additionalPIP)
-
-		By("Exposing service with additional pip")
-		annotation := map[string]string{
-			consts.ServiceAnnotationAdditionalPublicIPs: additionalPIP,
-		}
-		ip := createAndExposeDefaultServiceWithAnnotation(cs, serviceName, ns.Name, labels, annotation, ports)
-		defer func() {
-			By("cleaning up")
-			err := utils.DeleteServiceIfExists(cs, ns.Name, serviceName)
-			Expect(err).NotTo(HaveOccurred())
-			err = utils.DeletePIPWithRetry(tc, ipName, "")
-			Expect(err).NotTo(HaveOccurred())
-		}()
-
-		err = wait.PollImmediate(10*time.Second, 2*time.Minute, func() (bool, error) {
-			service, err := cs.CoreV1().Services(ns.Name).Get(context.TODO(), serviceName, metav1.GetOptions{})
-			if err != nil {
-				if utils.IsRetryableAPIError(err) {
-					return false, nil
-				}
-				return false, err
-			}
-
-			utils.Logf("Checking additionalPIP in ingress")
-			foundInIngress := false
-			ingressList := service.Status.LoadBalancer.Ingress
-			Expect(ingressList).NotTo(BeNil())
-			for _, ingress := range ingressList {
-				if additionalPIP == ingress.IP {
-					foundInIngress = true
-					break
-				}
-			}
-
-			utils.Logf("Checking additionalPIP in nsg")
-			ipList := []string{ip, additionalPIP}
-			port := fmt.Sprintf("%d", serverPort)
-			nsgs, err := tc.GetClusterSecurityGroups()
-			if err != nil {
-				return false, err
-			}
-			foundInNsg := validateSharedSecurityRuleExists(nsgs, ipList, port)
-
-			return foundInIngress && foundInNsg, nil
-		})
-		Expect(err).NotTo(HaveOccurred())
-	})
-
 	It("should support service annotation `service.beta.kubernetes.io/azure-pip-tags`", func() {
 		By("Creating a service with custom tags")
 		annotation := map[string]string{
@@ -590,23 +504,21 @@ var _ = Describe("Service with annotation", Label(utils.TestSuiteLabelServiceAnn
 		}
 	})
 
-	It("should support service annotation 'service.beta.kubernetes.io/azure-load-balancer-health-probe-num-of-probe', 'service.beta.kubernetes.io/azure-load-balancer-health-probe-interval', 'service.beta.kubernetes.io/azure-load-balancer-health-probe-protocol' and port specific configs", func() {
+	It("should support service annotation 'service.beta.kubernetes.io/azure-load-balancer-health-probe-num-of-probe' and port specific configs", func() {
 		By("Creating a service with health probe annotations")
 		annotation := map[string]string{
-			consts.ServiceAnnotationLoadBalancerHealthProbeNumOfProbe:                                      "5",
-			consts.BuildHealthProbeAnnotationKeyForPort(serverPort, consts.HealthProbeParamsNumOfProbe):    "3",
-			consts.ServiceAnnotationLoadBalancerHealthProbeInterval:                                        "15",
-			consts.BuildHealthProbeAnnotationKeyForPort(serverPort, consts.HealthProbeParamsProbeInterval): "10",
-			consts.ServiceAnnotationLoadBalancerHealthProbeProtocol:                                        "Http",
-			consts.ServiceAnnotationLoadBalancerHealthProbeRequestPath:                                     "/",
+			consts.ServiceAnnotationLoadBalancerHealthProbeNumOfProbe:                                   "5",
+			consts.BuildHealthProbeAnnotationKeyForPort(serverPort, consts.HealthProbeParamsNumOfProbe): "3",
 		}
 
 		// create service with given annotation and wait it to expose
 		publicIP := createAndExposeDefaultServiceWithAnnotation(cs, serviceName, ns.Name, labels, annotation, ports)
 		defer func() {
-			By("Cleaning up service")
+			By("Cleaning up service and public IP")
 			err := utils.DeleteService(cs, ns.Name, serviceName)
 			Expect(err).NotTo(HaveOccurred())
+			err = utils.DeletePIPWithRetry(tc, publicIP, tc.GetResourceGroup())
+			Expect(err).NotTo(HaveOccurred())
 		}()
 		pipFrontendConfigID := getPIPFrontendConfigurationID(tc, publicIP, tc.GetResourceGroup(), "")
 		pipFrontendConfigIDSplit := strings.Split(pipFrontendConfigID, "/")
@@ -634,20 +546,53 @@ var _ = Describe("Service with annotation", Label(utils.TestSuiteLabelServiceAnn
 
 		By("Validating health probe configs")
 		var numberOfProbes *int32
-		var intervalInSeconds *int32
 		for _, probe := range targetProbes {
 			if probe.NumberOfProbes != nil {
 				numberOfProbes = probe.NumberOfProbes
 			}
-			if probe.IntervalInSeconds != nil {
-				intervalInSeconds = probe.IntervalInSeconds
-			}
 		}
-		utils.Logf("Validating health probe config numberOfProbes")
 		Expect(*numberOfProbes).To(Equal(int32(3)))
-		utils.Logf("Validating health probe config intervalInSeconds")
-		Expect(*intervalInSeconds).To(Equal(int32(10)))
-		utils.Logf("Validating health probe config protocol")
+	})
+	It("should support service annotation 'service.beta.kubernetes.io/azure-load-balancer-health-probe-protocol' and port specific configs", func() {
+		By("Creating a service with health probe annotations")
+		annotation := map[string]string{
+			consts.ServiceAnnotationLoadBalancerHealthProbeProtocol:    "Http",
+			consts.ServiceAnnotationLoadBalancerHealthProbeRequestPath: "/",
+		}
+
+		// create service with given annotation and wait it to expose
+		publicIP := createAndExposeDefaultServiceWithAnnotation(cs, serviceName, ns.Name, labels, annotation, ports)
+		defer func() {
+			By("Cleaning up service and public IP")
+			err := utils.DeleteService(cs, ns.Name, serviceName)
+			Expect(err).NotTo(HaveOccurred())
+			err = utils.DeletePIPWithRetry(tc, publicIP, tc.GetResourceGroup())
+			Expect(err).NotTo(HaveOccurred())
+		}()
+		pipFrontendConfigID := getPIPFrontendConfigurationID(tc, publicIP, tc.GetResourceGroup(), "")
+		pipFrontendConfigIDSplit := strings.Split(pipFrontendConfigID, "/")
+		Expect(len(pipFrontendConfigIDSplit)).NotTo(Equal(0))
+
+		var lb *network.LoadBalancer
+		var targetProbes []*network.Probe
+		//wait for backend update
+		err := wait.PollImmediate(5*time.Second, 60*time.Second, func() (bool, error) {
+			lb = getAzureLoadBalancerFromPIP(tc, publicIP, tc.GetResourceGroup(), "")
+			targetProbes = []*network.Probe{}
+			for i := range *lb.LoadBalancerPropertiesFormat.Probes {
+				probe := (*lb.LoadBalancerPropertiesFormat.Probes)[i]
+				utils.Logf("One probe of LB is %q", *probe.Name)
+				probeSplit := strings.Split(*probe.Name, "-")
+				Expect(len(probeSplit)).NotTo(Equal(0))
+				if pipFrontendConfigIDSplit[len(pipFrontendConfigIDSplit)-1] == probeSplit[0] {
+					targetProbes = append(targetProbes, &probe)
+				}
+			}
+			return len(targetProbes) == 1, nil
+		})
+		Expect(err).NotTo(HaveOccurred())
+		// get lb from azure client
+		By("Validating health probe configs")
 		Expect((len(targetProbes))).To(Equal(1))
 		Expect(targetProbes[0].Protocol).To(Equal(network.ProbeProtocolHTTP))
 	})