Repository: kubernetes-sigs/cloud-provider-azure (go)

Issue:
Azure load balancer is never updated when node status changes
(this is the same as https://github.com/kubernetes/kubernetes/issues/108283)
### What happened?

Nodes that are no longer in the `Ready` state (i.e. due to a reboot) do not get removed from backend pools of Azure load balancers.

In the current code, a node gets removed from the backend pool of an Azure load balancer only if:

- it is labeled with `node.kubernetes.io/exclude-from-external-load-balancers` or
- it is labeled with `"kubernetes.azure.com/managed" ="false"`

This is due to https://github.com/kubernetes-sigs/cloud-provider-azure/issues/851, first merged in 1.23. ([1][2])

However with the above PR, nodes that are no longer ready (i.e. due to a reboot) do _not_ get removed backend pools of Azure load balancers, which consequently keep considering such nodes as valid backends.



[1] https://github.com/kubernetes-sigs/cloud-provider-azure/blob/507dd8975b175b4aa6f5e3bfcc2870ddacb91bd7/pkg/provider/azure.go#L1077-L1088
[2] https://github.com/kubernetes-sigs/cloud-provider-azure/blob/507dd8975b175b4aa6f5e3bfcc2870ddacb91bd7/pkg/provider/azure.go#L1204-L1222

### What did you expect to happen?

Nodes that are no longer in the `Ready` state should be removed from Azure load balancers. 

The (cloud-agnostic) cloud provider controller already updates the list of nodes to include in backend pools based on
- `node.kubernetes.io/exclude-from-external-load-balancers` label or
- node Status != Ready
```
func (s *Controller) getNodeConditionPredicate() NodeConditionPredicate {
	return func(node *v1.Node) bool {
		if _, hasExcludeBalancerLabel := node.Labels[v1.LabelNodeExcludeBalancers]; hasExcludeBalancerLabel {
			return false
		}

		// If we have no info, don't accept
		if len(node.Status.Conditions) == 0 {
			return false
		}
		for _, cond := range node.Status.Conditions {
			// We consider the node for load balancing only when its NodeReady condition status
			// is ConditionTrue
			if cond.Type == v1.NodeReady && cond.Status != v1.ConditionTrue {
				klog.V(4).Infof("Ignoring node %v with %v condition status %v", node.Name, cond.Type, cond.Status)
				return false
			}
		}
		return true
	}
}
```
(https://github.com/kubernetes/kubernetes/blob/b93fd35dd030519c24dd02f8bc2a7f873d2928cd/staging/src/k8s.io/cloud-provider/controllers/service/controller.go#L668-L688)

... and the result of that is passed over to cloud-specific code.

Azure-specific code ignores that piece of information when it comes to deleting nodes from a load balancer and uses its own logic introduced in https://github.com/kubernetes-sigs/cloud-provider-azure/issues/851. **We whould extend this logic to include node status changes.**



### How can we reproduce it (as minimally and precisely as possible)?

- Have a service backed by UDP server pods
- Have a cloud load balancer deployed in Azure pointing to the service in the previous point
- Dump the backend pools on azure side: `az network lb list -g $RESOURCE_GROUP`
- Reboot a node
- the Azure cloud provider code won't react to that node becoming `NotReady`; in particular it will not issue a DELETE operation in order to remove that node from the corresponding backend pool on the Azure load balancer. This is evident in the logs because this line does /not/ appear (while it does appear in kubernetes 1.22!):
```
I0221 13:23:05.213367       1 azure_loadbalancer.go:1143] reconcileLoadBalancer for service (e2e-k8s-service-load-balancer-with-pdb-new-2741/service-test)(true): lb backendpool - found unwanted node
ci-ln-2mnyxbb-002ac-9wktq-master-1, decouple it from the LB
```
- Dump again the backend pools on azure side: `az network lb list -g $RESOURCE_GROUP` and compare it with the previous output from the same command: the output is exactly the same, showing that the rebooting node didn't get removed from its backend pool.


### Anything else we need to know?

This causes high disruption times to PDB-based apps during upgrade jobs:
we went from about 5s in kubernetes 1.22 to about 13s with kubernetes 1.23, as reported in https://github.com/kubernetes/kubernetes/issues/108112

### Kubernetes version

since 1.23

### Cloud provider

Azure

### OS version

_No response_

### Install tools

_No response_

### Container runtime (CRI) and and version (if applicable)

_No response_

### Related plugins (CNI, CSI, ...) and versions (if applicable)

_No response_

Pull request:
fix: exclude non-ready nodes from azure load balancer
/kind bug
/kind regression


#### What this PR does / why we need it:
This PR makes sure that nodes that are not in the ready state get removed from Azure load balancer backend pools.

#### Which issue(s) this PR fixes:
Fixes #1194

```release-note
NONE
```

Patch:
diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml
index f7d6b6e29..657689671 100644
--- a/.github/workflows/release.yaml
+++ b/.github/workflows/release.yaml
@@ -186,6 +186,7 @@ jobs:
 
           GO111MODULE=on go install k8s.io/release/cmd/release-notes@latest
           
+          RELEASE_NOTE=site/content/en/blog/releases/${{github.ref_name}}.md
           
           release-notes --repo=cloud-provider-azure \
           	--org=kubernetes-sigs \
@@ -193,14 +194,10 @@ jobs:
           	--start-sha=${START_SHA} \
           	--end-sha=${END_SHA} \
           	--markdown-links=true \
-          	--output=release-notes.md
+          	--output=${RELEASE_NOTE}
           
-          # Generate release note for site
-          # Prepend site header
-          SITE_RELEASE_NOTE=site/content/en/blog/releases/${{github.ref_name}}.md
-          SITE_HEAD="---\ntitle: "v${VERSION}"\nlinkTitle: "v${VERSION}"\ndate: $(date '+%Y-%m-%d')\ndescription: Cloud Provider Azure v${VERSION}\n---\n\n"
-          cp release-notes.md ${SITE_RELEASE_NOTE}
-          sed -i "1i${SITE_HEAD}" ${SITE_RELEASE_NOTE}
+          NOTE_HEAD="---\ntitle: "v${VERSION}"\nlinkTitle: "v${VERSION}"\ndate: $(date '+%Y-%m-%d')\ndescription: Cloud Provider Azure v${VERSION}\n---\n\n"
+          sed -i "1i${NOTE_HEAD}" ${RELEASE_NOTE}
       - name: Download artifacts
         uses: actions/download-artifact@v2
         with:
@@ -208,7 +205,7 @@ jobs:
       - name: Publish release
         uses: softprops/action-gh-release@v1
         with:
-          body_path: release-notes.md
+          body_path: site/content/en/blog/releases/${{github.ref_name}}.md
           files: |
             ./artifacts/cloud-node-manager-*-*/*
             ./artifacts/cloud-controller-manager-*-*/*
diff --git a/go.mod b/go.mod
index 37e994e41..dabab7b09 100644
--- a/go.mod
+++ b/go.mod
@@ -3,7 +3,7 @@ module sigs.k8s.io/cloud-provider-azure
 go 1.17
 
 require (
-	github.com/Azure/azure-sdk-for-go v62.0.0+incompatible
+	github.com/Azure/azure-sdk-for-go v61.6.0+incompatible
 	github.com/Azure/go-autorest/autorest v0.11.24
 	github.com/Azure/go-autorest/autorest/adal v0.9.18
 	github.com/Azure/go-autorest/autorest/mocks v0.4.1
diff --git a/go.sum b/go.sum
index c5325f755..af6f23cbb 100644
--- a/go.sum
+++ b/go.sum
@@ -17,8 +17,8 @@ cloud.google.com/go/storage v1.6.0/go.mod h1:N7U0C8pVQ/+NIKOBQyamJIeKQKkZ+mxpohl
 cloud.google.com/go/storage v1.8.0/go.mod h1:Wv1Oy7z6Yz3DshWRJFhqM/UCfaWIRTdp0RXyy7KQOVs=
 cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9ullr3+Kg0=
 dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
-github.com/Azure/azure-sdk-for-go v62.0.0+incompatible h1:8N2k27SYtc12qj5nTsuFMFJPZn5CGmgMWqTy4y9I7Jw=
-github.com/Azure/azure-sdk-for-go v62.0.0+incompatible/go.mod h1:9XXNKU+eRnpl9moKnB4QOLf1HestfXbmab5FXxiDBjc=
+github.com/Azure/azure-sdk-for-go v61.6.0+incompatible h1:jdHWEqRK9boUrdUPIWDE9dKLmxbHmz+PFk3jRQ9s1C0=
+github.com/Azure/azure-sdk-for-go v61.6.0+incompatible/go.mod h1:9XXNKU+eRnpl9moKnB4QOLf1HestfXbmab5FXxiDBjc=
 github.com/Azure/go-ansiterm v0.0.0-20210608223527-2377c96fe795/go.mod h1:LmzpDX56iTiv29bbRTIsUNlaFfuhWRQBWjQdVyAevI8=
 github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 h1:UQHMgLO+TxOElx5B5HZ4hJQsoJ/PvUvKRhJHDQXO8P8=
 github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E=
diff --git a/hack/ensure-clusterctl.sh b/hack/ensure-clusterctl.sh
index 69d8fe272..b4eaa6cf1 100755
--- a/hack/ensure-clusterctl.sh
+++ b/hack/ensure-clusterctl.sh
@@ -20,7 +20,7 @@ set -o pipefail
 REPO_ROOT=$(dirname "${BASH_SOURCE[0]}")/..
 TOOLS_BIN="${REPO_ROOT}/bin"
 CLUSTERCTL="${TOOLS_BIN}/clusterctl"
-MINIMUM_CLUSTERCTL_VERSION=v1.1.1
+MINIMUM_CLUSTERCTL_VERSION=v1.0.2
 goarch="$(go env GOARCH)"
 goos="$(go env GOOS)"
 
diff --git a/pkg/azureclients/fileclient/azure_fileclient.go b/pkg/azureclients/fileclient/azure_fileclient.go
index 59f4770a2..6b6a4d34c 100644
--- a/pkg/azureclients/fileclient/azure_fileclient.go
+++ b/pkg/azureclients/fileclient/azure_fileclient.go
@@ -44,10 +44,6 @@ type ShareOptions struct {
 	RequestGiB int
 	// supported values: ""(by default), "TransactionOptimized", "Cool", "Hot", "Premium"
 	AccessTier string
-	// supported values: ""(by default), "AllSquash", "NoRootSquash", "RootSquash"
-	RootSquash string
-	// Metadata - A name-value pair to associate with the share as metadata.
-	Metadata map[string]*string
 }
 
 // New creates a azure file client
@@ -81,12 +77,6 @@ func (c *Client) CreateFileShare(resourceGroupName, accountName string, shareOpt
 	if shareOptions.AccessTier != "" {
 		fileShareProperties.AccessTier = storage.ShareAccessTier(shareOptions.AccessTier)
 	}
-	if shareOptions.RootSquash != "" {
-		fileShareProperties.RootSquash = storage.RootSquashType(shareOptions.RootSquash)
-	}
-	if shareOptions.Metadata != nil {
-		fileShareProperties.Metadata = shareOptions.Metadata
-	}
 	fileShare := storage.FileShare{
 		Name:                &shareOptions.Name,
 		FileShareProperties: fileShareProperties,
diff --git a/pkg/nodemanager/nodemanager.go b/pkg/nodemanager/nodemanager.go
index 05e7729ca..c1cf90c54 100644
--- a/pkg/nodemanager/nodemanager.go
+++ b/pkg/nodemanager/nodemanager.go
@@ -244,7 +244,7 @@ func (cnc *CloudNodeController) reconcileNodeLabels(node *v1.Node) error {
 // UpdateNodeAddress updates the nodeAddress of a single node
 func (cnc *CloudNodeController) updateNodeAddress(ctx context.Context, node *v1.Node) error {
 	// Do not process nodes that are still tainted
-	cloudTaint := getCloudTaint(node.Spec.Taints)
+	cloudTaint := GetCloudTaint(node.Spec.Taints)
 	if cloudTaint != nil {
 		klog.V(5).Infof("This node %s is still tainted. Will not process.", node.Name)
 		return nil
@@ -325,7 +325,7 @@ func (cnc *CloudNodeController) UpdateCloudNode(ctx context.Context, _, newObj i
 		return
 	}
 
-	cloudTaint := getCloudTaint(node.Spec.Taints)
+	cloudTaint := GetCloudTaint(node.Spec.Taints)
 	if cloudTaint == nil {
 		// The node has already been initialized so nothing to do.
 		return
@@ -343,7 +343,7 @@ func (cnc *CloudNodeController) AddCloudNode(ctx context.Context, obj interface{
 		return
 	}
 
-	cloudTaint := getCloudTaint(node.Spec.Taints)
+	cloudTaint := GetCloudTaint(node.Spec.Taints)
 	if cloudTaint == nil {
 		klog.V(2).Infof("This node %s is registered without the cloud taint. Will not process.", node.Name)
 		return
@@ -361,7 +361,7 @@ func (cnc *CloudNodeController) initializeNode(ctx context.Context, node *v1.Nod
 		return
 	}
 
-	cloudTaint := getCloudTaint(curNode.Spec.Taints)
+	cloudTaint := GetCloudTaint(curNode.Spec.Taints)
 	if cloudTaint == nil {
 		// Node object received from event had the cloud taint but was outdated,
 		// the node has actually already been initialized.
@@ -378,16 +378,9 @@ func (cnc *CloudNodeController) initializeNode(ctx context.Context, node *v1.Nod
 		}
 	}
 
-	var nodeModifiers []nodeModifier
-	err = clientretry.OnError(UpdateNodeSpecBackoff, func(err error) bool {
-		return err != nil && strings.HasPrefix(err.Error(), "failed to set node provider id")
-	}, func() error {
-		nodeModifiers, err = cnc.getNodeModifiersFromCloudProvider(ctx, curNode)
-		return err
-	})
+	nodeModifiers, err := cnc.getNodeModifiersFromCloudProvider(ctx, curNode)
 	if err != nil {
-		// Instead of just logging the error, panic and node manager can restart
-		utilruntime.Must(fmt.Errorf("failed to initialize node %s at cloudprovider: %w", node.Name, err))
+		utilruntime.HandleError(fmt.Errorf("failed to initialize node %s at cloudprovider: %w", node.Name, err))
 		return
 	}
 
@@ -442,9 +435,10 @@ func (cnc *CloudNodeController) getNodeModifiersFromCloudProvider(ctx context.Co
 				}
 			})
 		} else {
-			// if we are not able to get node provider id,
-			// we return error here and retry in the caller initializeNode()
-			return nil, fmt.Errorf("failed to set node provider id: %w", err)
+			// we should attempt to set providerID on node, but
+			// we can continue if we fail since we will attempt to set
+			// node addresses given the node name in getNodeAddressesByName
+			klog.Errorf("failed to set node provider id: %v", err)
 		}
 	}
 
@@ -519,7 +513,7 @@ func (cnc *CloudNodeController) getNodeModifiersFromCloudProvider(ctx context.Co
 	return nodeModifiers, nil
 }
 
-func getCloudTaint(taints []v1.Taint) *v1.Taint {
+func GetCloudTaint(taints []v1.Taint) *v1.Taint {
 	for _, taint := range taints {
 		if taint.Key == cloudproviderapi.TaintExternalCloudProvider {
 			return &taint
diff --git a/pkg/provider/azure.go b/pkg/provider/azure.go
index 45fa3984f..3e32a65ea 100644
--- a/pkg/provider/azure.go
+++ b/pkg/provider/azure.go
@@ -70,6 +70,7 @@ import (
 	"sigs.k8s.io/cloud-provider-azure/pkg/azureclients/zoneclient"
 	azcache "sigs.k8s.io/cloud-provider-azure/pkg/cache"
 	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
+	nodemanager "sigs.k8s.io/cloud-provider-azure/pkg/nodemanager"
 	"sigs.k8s.io/cloud-provider-azure/pkg/retry"
 
 	// ensure the newly added package from azure-sdk-for-go is in vendor/
@@ -1036,16 +1037,20 @@ func (az *Cloud) updateNodeCaches(prevNode, newNode *v1.Node) {
 			delete(az.nodeResourceGroups, prevNode.ObjectMeta.Name)
 		}
 
-		// Remove from unmanagedNodes cache.
 		managed, ok := prevNode.ObjectMeta.Labels[consts.ManagedByAzureLabel]
-		if ok && strings.EqualFold(managed, consts.NotManagedByAzureLabelValue) {
+		isNodeManagedByCloudProvider := !ok || !strings.EqualFold(managed, consts.NotManagedByAzureLabelValue)
+
+		klog.Infof("managed=%v, ok=%v, isNodeManagedByCloudProvider=%v",
+			managed, ok, isNodeManagedByCloudProvider)
+
+		// Remove from unmanagedNodes cache
+		if !isNodeManagedByCloudProvider {
 			az.unmanagedNodes.Delete(prevNode.ObjectMeta.Name)
-			az.excludeLoadBalancerNodes.Delete(prevNode.ObjectMeta.Name)
 		}
 
-		// Remove from excludeLoadBalancerNodes cache.
-		if _, hasExcludeBalancerLabel := prevNode.ObjectMeta.Labels[v1.LabelNodeExcludeBalancers]; hasExcludeBalancerLabel {
-			az.excludeLoadBalancerNodes.Delete(prevNode.ObjectMeta.Name)
+		// if the node is being deleted from the cluster, exclude it from load balancers
+		if newNode == nil {
+			az.excludeLoadBalancerNodes.Insert(prevNode.ObjectMeta.Name)
 		}
 
 		// Remove from nodePrivateIPs cache.
@@ -1074,17 +1079,35 @@ func (az *Cloud) updateNodeCaches(prevNode, newNode *v1.Node) {
 			az.nodeResourceGroups[newNode.ObjectMeta.Name] = strings.ToLower(newRG)
 		}
 
-		// Add to unmanagedNodes cache.
+		_, hasExcludeBalancerLabel := newNode.ObjectMeta.Labels[v1.LabelNodeExcludeBalancers]
 		managed, ok := newNode.ObjectMeta.Labels[consts.ManagedByAzureLabel]
-		if ok && strings.EqualFold(managed, consts.NotManagedByAzureLabelValue) {
+		isNodeManagedByCloudProvider := !ok || !strings.EqualFold(managed, consts.NotManagedByAzureLabelValue)
+
+		// Update unmanagedNodes cache
+		if !isNodeManagedByCloudProvider {
 			az.unmanagedNodes.Insert(newNode.ObjectMeta.Name)
-			az.excludeLoadBalancerNodes.Insert(newNode.ObjectMeta.Name)
 		}
 
-		// Add to excludeLoadBalancerNodes cache.
-		if _, hasExcludeBalancerLabel := newNode.ObjectMeta.Labels[v1.LabelNodeExcludeBalancers]; hasExcludeBalancerLabel {
-			klog.V(4).Infof("adding node %s from the exclude-from-lb list because the label %s is found", newNode.Name, v1.LabelNodeExcludeBalancers)
+		// Update excludeLoadBalancerNodes cache
+		switch {
+		case !isNodeManagedByCloudProvider:
 			az.excludeLoadBalancerNodes.Insert(newNode.ObjectMeta.Name)
+
+		case hasExcludeBalancerLabel:
+			az.excludeLoadBalancerNodes.Insert(newNode.ObjectMeta.Name)
+
+		case !isNodeReady(newNode) && nodemanager.GetCloudTaint(newNode.Spec.Taints) == nil:
+			// If not in ready state and not a newly created node, add to excludeLoadBalancerNodes cache.
+			// New nodes (tainted with "node.cloudprovider.kubernetes.io/uninitialized") should not be
+			// excluded from load balancers regardless of their state, so as to reduce the number of
+			// VMSS API calls and not provoke VMScaleSetActiveModelsCountLimitReached.
+			// (https://github.com/kubernetes-sigs/cloud-provider-azure/issues/851)
+			az.excludeLoadBalancerNodes.Insert(newNode.ObjectMeta.Name)
+
+		default:
+			// Nodes not falling into the three cases above are valid backends and
+			// should not appear in excludeLoadBalancerNodes cache.
+			az.excludeLoadBalancerNodes.Delete(newNode.ObjectMeta.Name)
 		}
 
 		// Add to nodePrivateIPs cache
@@ -1220,3 +1243,12 @@ func (az *Cloud) ShouldNodeExcludedFromLoadBalancer(nodeName string) (bool, erro
 
 	return az.excludeLoadBalancerNodes.Has(nodeName), nil
 }
+
+func isNodeReady(node *v1.Node) bool {
+	for _, cond := range node.Status.Conditions {
+		if cond.Type == v1.NodeReady && cond.Status == v1.ConditionTrue {
+			return true
+		}
+	}
+	return false
+}
diff --git a/pkg/provider/azure_loadbalancer_backendpool.go b/pkg/provider/azure_loadbalancer_backendpool.go
index 707ccdaf8..b774accc9 100644
--- a/pkg/provider/azure_loadbalancer_backendpool.go
+++ b/pkg/provider/azure_loadbalancer_backendpool.go
@@ -154,21 +154,20 @@ func (bc *backendPoolTypeNodeIPConfig) ReconcileBackendPools(clusterName string,
 			// If the LB backend pool type is configured from nodeIP or podIP
 			// to nodeIPConfiguration, we need to decouple the VM NICs from the LB
 			// before attaching nodeIPs/podIPs to the LB backend pool.
-			// TODO(nilo19): uncomment this when we find a new way to determine the backend pool type
-			//if bp.BackendAddressPoolPropertiesFormat != nil &&
-			//	(bp.BackendIPConfigurations == nil || len(*bp.BackendIPConfigurations) == 0) {
-			//	// It is not allowed to change the type of the backend pool, so we delete it
-			//	// and create a new empty backend pool.
-			//	if err := bc.DeleteLBBackendPool(lbName, to.String(bp.Name)); err != nil {
-			//		klog.Errorf("bc.ReconcileBackendPools for service (%s): failed to DeleteLBBackendPool: %s", serviceName, err.Error())
-			//		return false, false, err
-			//	}
-			//	newBackendPools = append(newBackendPools[:i], newBackendPools[i+1:]...)
-			//	lb.BackendAddressPools = &newBackendPools
-			//	lb.Etag = nil
-			//	foundBackendPool = false
-			//	break
-			//}
+			if bp.BackendAddressPoolPropertiesFormat != nil &&
+				(bp.BackendIPConfigurations == nil || len(*bp.BackendIPConfigurations) == 0) {
+				// It is not allowed to change the type of the backend pool, so we delete it
+				// and create a new empty backend pool.
+				if err := bc.DeleteLBBackendPool(lbName, to.String(bp.Name)); err != nil {
+					klog.Errorf("bc.ReconcileBackendPools for service (%s): failed to DeleteLBBackendPool: %s", serviceName, err.Error())
+					return false, false, err
+				}
+				newBackendPools = append(newBackendPools[:i], newBackendPools[i+1:]...)
+				lb.BackendAddressPools = &newBackendPools
+				lb.Etag = nil
+				foundBackendPool = false
+				break
+			}
 
 			var backendIPConfigurationsToBeDeleted []network.InterfaceIPConfiguration
 			if bp.BackendAddressPoolPropertiesFormat != nil && bp.BackendIPConfigurations != nil {
@@ -386,9 +385,8 @@ func (bi *backendPoolTypeNodeIP) ReconcileBackendPools(clusterName string, servi
 	lbName := *lb.Name
 	serviceName := getServiceName(service)
 	lbBackendPoolName := getBackendPoolName(clusterName, service)
-	// TODO(nilo19): uncomment this when we find a new way to determine the backend pool type
-	//vmSetName := bi.mapLoadBalancerNameToVMSet(lbName, clusterName)
-	//lbBackendPoolID := bi.getBackendPoolID(to.String(lb.Name), bi.getLoadBalancerResourceGroup(), getBackendPoolName(clusterName, service))
+	vmSetName := bi.mapLoadBalancerNameToVMSet(lbName, clusterName)
+	lbBackendPoolID := bi.getBackendPoolID(to.String(lb.Name), bi.getLoadBalancerResourceGroup(), getBackendPoolName(clusterName, service))
 
 	for i := len(newBackendPools) - 1; i >= 0; i-- {
 		bp := newBackendPools[i]
@@ -399,28 +397,27 @@ func (bi *backendPoolTypeNodeIP) ReconcileBackendPools(clusterName string, servi
 			// If the LB backend pool type is configured from nodeIPConfiguration
 			// to nodeIP, we need to decouple the VM NICs from the LB
 			// before attaching nodeIPs/podIPs to the LB backend pool.
-			// TODO(nilo19): uncomment this when we find a new way to determine the backend pool type
-			//if bp.BackendAddressPoolPropertiesFormat != nil &&
-			//	bp.BackendIPConfigurations != nil &&
-			//	len(*bp.BackendIPConfigurations) > 0 {
-			//	// It is not allowed to change the type of the backend pool, so we delete it
-			//	// and create a new empty backend pool. Note that for IP configuration based
-			//	// backend pools, we must decouple it from the vmSets before deleting.
-			//	klog.V(2).Infof("bi.ReconcileBackendPools for service (%s): ensuring the LB is decoupled from the VMSet", serviceName)
-			//	if err := bi.VMSet.EnsureBackendPoolDeleted(service, lbBackendPoolID, vmSetName, lb.BackendAddressPools, true); err != nil {
-			//		klog.Errorf("bi.ReconcileBackendPools for service (%s): failed to EnsureBackendPoolDeleted: %s", serviceName, err.Error())
-			//		return false, false, err
-			//	}
-			//	if err := bi.DeleteLBBackendPool(lbName, to.String(bp.Name)); err != nil {
-			//		klog.Errorf("bi.ReconcileBackendPools for service (%s): failed to DeleteLBBackendPool: %s", serviceName, err.Error())
-			//		return false, false, err
-			//	}
-			//	newBackendPools = append(newBackendPools[:i], newBackendPools[i+1:]...)
-			//	lb.BackendAddressPools = &newBackendPools
-			//	lb.Etag = nil
-			//	foundBackendPool = false
-			//	break
-			//}
+			if bp.BackendAddressPoolPropertiesFormat != nil &&
+				bp.BackendIPConfigurations != nil &&
+				len(*bp.BackendIPConfigurations) > 0 {
+				// It is not allowed to change the type of the backend pool, so we delete it
+				// and create a new empty backend pool. Note that for IP configuration based
+				// backend pools, we must decouple it from the vmSets before deleting.
+				klog.V(2).Infof("bi.ReconcileBackendPools for service (%s): ensuring the LB is decoupled from the VMSet", serviceName)
+				if err := bi.VMSet.EnsureBackendPoolDeleted(service, lbBackendPoolID, vmSetName, lb.BackendAddressPools, true); err != nil {
+					klog.Errorf("bi.ReconcileBackendPools for service (%s): failed to EnsureBackendPoolDeleted: %s", serviceName, err.Error())
+					return false, false, err
+				}
+				if err := bi.DeleteLBBackendPool(lbName, to.String(bp.Name)); err != nil {
+					klog.Errorf("bi.ReconcileBackendPools for service (%s): failed to DeleteLBBackendPool: %s", serviceName, err.Error())
+					return false, false, err
+				}
+				newBackendPools = append(newBackendPools[:i], newBackendPools[i+1:]...)
+				lb.BackendAddressPools = &newBackendPools
+				lb.Etag = nil
+				foundBackendPool = false
+				break
+			}
 
 			var nodeIPAddressesToBeDeleted []string
 			for nodeName := range bi.excludeLoadBalancerNodes {
diff --git a/site/content/en/development/deploy-cluster.md b/site/content/en/development/deploy-cluster.md
index ed1944b6e..1295f947b 100644
--- a/site/content/en/development/deploy-cluster.md
+++ b/site/content/en/development/deploy-cluster.md
@@ -41,7 +41,7 @@ Customizations are supported by environment variables:
 | AZURE_NODE_MACHINE_TYPE                 | false    | VM SKU of the worker nodes                                                         | Standard_D2s_v3                                                         |
 | AZURE_LOCATION                          | false    | region of the cluster resources                                                    | westus2                                                                 |
 | AZURE_CLOUD_CONTROLLER_MANAGER_IMG      | false    | image of the cloud-controller-manager                                              | mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.23.1 |
-| AZURE_CLOUD_NODE_MANAGER_IMG            | false    | image of the cloud-node-manager                                                    | mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.23.1       |
+| AZURE_CLOUD_CONTROLLER_MANAGER_IMG      | false    | image of the cloud-node-manager                                                    | mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.23.1       |
 | KUBERNETES_VERSION                      | false    | Kubernetes components version                                                      | v1.23.0                                                                 |
 | LB_SKU                                  | false    | LoadBalancer SKU, Standard or Basic                                                | Standard                                                                |
 | ENABLE_MULTI_SLB                        | false    | Enable multiple standard LoadBalancers per cluster                                 | false                                                                   |
diff --git a/site/content/en/topics/loadbalancer.md b/site/content/en/topics/loadbalancer.md
index 53f016537..6ad49849a 100644
--- a/site/content/en/topics/loadbalancer.md
+++ b/site/content/en/topics/loadbalancer.md
@@ -227,8 +227,6 @@ The backend pool type can be configured by specifying `loadBalancerBackendPoolCo
 2. `nodeIP`. In this case we attach nodes to the LB by calling the LB API to add the node private IP addresses to the LB backend pool.
 3. `podIP` (not supported yet). In this case we do not attach nodes to the LB. Instead we directly adding pod IPs to the LB backend pool.
 
-> Currently, we do NOT support changing the backend pool type in existing clusters
-
 ## Load balancer limits
 
 The limits of the load balancer related resources are listed below:
diff --git a/vendor/github.com/Azure/azure-sdk-for-go/version/version.go b/vendor/github.com/Azure/azure-sdk-for-go/version/version.go
index 431bf47dd..0d76aec56 100644
--- a/vendor/github.com/Azure/azure-sdk-for-go/version/version.go
+++ b/vendor/github.com/Azure/azure-sdk-for-go/version/version.go
@@ -4,4 +4,4 @@ package version
 // Licensed under the MIT License. See License.txt in the project root for license information.
 
 // Number contains the semantic version of this SDK.
-const Number = "v62.0.0"
+const Number = "v61.6.0"
diff --git a/vendor/modules.txt b/vendor/modules.txt
index 6c1f49610..63f9ec0a3 100644
--- a/vendor/modules.txt
+++ b/vendor/modules.txt
@@ -1,4 +1,4 @@
-# github.com/Azure/azure-sdk-for-go v62.0.0+incompatible
+# github.com/Azure/azure-sdk-for-go v61.6.0+incompatible
 ## explicit
 github.com/Azure/azure-sdk-for-go/services/authorization/mgmt/2015-07-01/authorization
 github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2021-07-01/compute


Test patch:
diff --git a/pkg/nodemanager/nodemanager_test.go b/pkg/nodemanager/nodemanager_test.go
index e666e2969..1441c4112 100644
--- a/pkg/nodemanager/nodemanager_test.go
+++ b/pkg/nodemanager/nodemanager_test.go
@@ -1149,78 +1149,3 @@ func TestNodeProviderIDAlreadySet(t *testing.T) {
 	// CCM node controller should not overwrite provider if it's already set
 	assert.Equal(t, "test-provider-id", fnh.UpdatedNodes[0].Spec.ProviderID, "Node ProviderID not set correctly")
 }
-
-// This test checks that a node manager should retry 20 times when failing to get providerID and then panic
-func TestNodeProviderIDNotSet(t *testing.T) {
-	ctrl := gomock.NewController(t)
-	defer ctrl.Finish()
-
-	fnh := &testutil.FakeNodeHandler{
-		Existing: []*v1.Node{
-			{
-				ObjectMeta: metav1.ObjectMeta{
-					Name:              "node0",
-					CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC),
-					Labels:            map[string]string{},
-				},
-				Status: v1.NodeStatus{
-					Conditions: []v1.NodeCondition{
-						{
-							Type:               v1.NodeReady,
-							Status:             v1.ConditionUnknown,
-							LastHeartbeatTime:  metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC),
-							LastTransitionTime: metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC),
-						},
-					},
-				},
-				Spec: v1.NodeSpec{
-					Taints: []v1.Taint{
-						{
-							Key:    "ImproveCoverageTaint",
-							Value:  "true",
-							Effect: v1.TaintEffectNoSchedule,
-						},
-						{
-							Key:    cloudproviderapi.TaintExternalCloudProvider,
-							Value:  "true",
-							Effect: v1.TaintEffectNoSchedule,
-						},
-					},
-				},
-			},
-		},
-		Clientset:      fake.NewSimpleClientset(&v1.PodList{}),
-		DeleteWaitChan: make(chan struct{}),
-	}
-
-	ctx := context.TODO()
-	factory := informers.NewSharedInformerFactory(fnh, 0)
-	mockNP := mocknodeprovider.NewMockNodeProvider(ctrl)
-	// InstanceID function should be retried for 20 times when error happens consistently
-	mockNP.EXPECT().InstanceID(ctx, types.NodeName("node0")).Return("", cloudprovider.InstanceNotFound).MinTimes(20).MaxTimes(20)
-
-	eventBroadcaster := record.NewBroadcaster()
-	cloudNodeController := &CloudNodeController{
-		kubeClient:                fnh,
-		nodeInformer:              factory.Core().V1().Nodes(),
-		nodeName:                  "node0",
-		nodeProvider:              mockNP,
-		nodeStatusUpdateFrequency: 1 * time.Second,
-		recorder:                  eventBroadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "cloud-node-controller"}),
-	}
-	eventBroadcaster.StartLogging(klog.Infof)
-
-	// Expect AddCloudNode() to panic when providerID is not set properly
-	func() {
-		defer func() {
-			err := recover()
-			assert.NotNil(t, err, "AddCloudNode() didn't panic")
-		}()
-
-		cloudNodeController.AddCloudNode(context.TODO(), fnh.Existing[0])
-		t.Errorf("AddCloudNode() didn't panic when providerID not found")
-	}()
-
-	// Node update should fail
-	assert.Equal(t, 0, len(fnh.UpdatedNodes), "Node was updated (unexpected)")
-}
diff --git a/pkg/provider/azure_loadbalancer_backendpool_test.go b/pkg/provider/azure_loadbalancer_backendpool_test.go
index 646508ea4..e1f3345a9 100644
--- a/pkg/provider/azure_loadbalancer_backendpool_test.go
+++ b/pkg/provider/azure_loadbalancer_backendpool_test.go
@@ -17,6 +17,7 @@ limitations under the License.
 package provider
 
 import (
+	"fmt"
 	"strings"
 	"testing"
 
@@ -315,28 +316,27 @@ func TestReconcileBackendPoolsNodeIPConfig(t *testing.T) {
 	assert.True(t, changed)
 }
 
-// TODO(nilo19): uncomment this when we find a new way to determine the backend pool type
-//func TestReconcileBackendPoolsNodeIPToIPConfig(t *testing.T) {
-//	ctrl := gomock.NewController(t)
-//	defer ctrl.Finish()
-//
-//	lb := buildLBWithVMIPs(testClusterName, []string{"10.0.0.1", "10.0.0.2"})
-//	mockLBClient := mockloadbalancerclient.NewMockInterface(ctrl)
-//	mockLBClient.EXPECT().DeleteLBBackendPool(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(retry.NewError(false, fmt.Errorf("delete LB backend pool error")))
-//
-//	az := GetTestCloud(ctrl)
-//	az.LoadBalancerClient = mockLBClient
-//	bc := newBackendPoolTypeNodeIPConfig(az)
-//	svc := getTestService("test", v1.ProtocolTCP, nil, false, 80)
-//	_, _, err := bc.ReconcileBackendPools(testClusterName, &svc, lb)
-//	assert.Contains(t, err.Error(), "delete LB backend pool error")
-//
-//	lb = buildLBWithVMIPs(testClusterName, []string{"10.0.0.1", "10.0.0.2"})
-//	mockLBClient.EXPECT().DeleteLBBackendPool(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(nil)
-//	_, _, err = bc.ReconcileBackendPools(testClusterName, &svc, lb)
-//	assert.NoError(t, err)
-//	assert.Empty(t, (*lb.BackendAddressPools)[0].LoadBalancerBackendAddresses)
-//}
+func TestReconcileBackendPoolsNodeIPToIPConfig(t *testing.T) {
+	ctrl := gomock.NewController(t)
+	defer ctrl.Finish()
+
+	lb := buildLBWithVMIPs(testClusterName, []string{"10.0.0.1", "10.0.0.2"})
+	mockLBClient := mockloadbalancerclient.NewMockInterface(ctrl)
+	mockLBClient.EXPECT().DeleteLBBackendPool(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(retry.NewError(false, fmt.Errorf("delete LB backend pool error")))
+
+	az := GetTestCloud(ctrl)
+	az.LoadBalancerClient = mockLBClient
+	bc := newBackendPoolTypeNodeIPConfig(az)
+	svc := getTestService("test", v1.ProtocolTCP, nil, false, 80)
+	_, _, err := bc.ReconcileBackendPools(testClusterName, &svc, lb)
+	assert.Contains(t, err.Error(), "delete LB backend pool error")
+
+	lb = buildLBWithVMIPs(testClusterName, []string{"10.0.0.1", "10.0.0.2"})
+	mockLBClient.EXPECT().DeleteLBBackendPool(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(nil)
+	_, _, err = bc.ReconcileBackendPools(testClusterName, &svc, lb)
+	assert.NoError(t, err)
+	assert.Empty(t, (*lb.BackendAddressPools)[0].LoadBalancerBackendAddresses)
+}
 
 func TestReconcileBackendPoolsNodeIP(t *testing.T) {
 	ctrl := gomock.NewController(t)
@@ -402,48 +402,47 @@ func TestReconcileBackendPoolsNodeIP(t *testing.T) {
 	assert.True(t, changed)
 }
 
-// TODO(nilo19): uncomment this when we find a new way to determine the backend pool type
-//func TestReconcileBackendPoolsNodeIPConfigToIP(t *testing.T) {
-//	ctrl := gomock.NewController(t)
-//	defer ctrl.Finish()
-//
-//	lb := buildDefaultTestLB(testClusterName, []string{
-//		"/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool1-00000000-nic-1/ipConfigurations/ipconfig1",
-//		"/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool2-00000000-nic-1/ipConfigurations/ipconfig1",
-//	})
-//	mockVMSet := NewMockVMSet(ctrl)
-//	mockVMSet.EXPECT().EnsureBackendPoolDeleted(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(fmt.Errorf("delete LB backend pool error"))
-//	mockVMSet.EXPECT().GetPrimaryVMSetName().Return("k8s-agentpool1-00000000").AnyTimes()
-//
-//	az := GetTestCloud(ctrl)
-//	az.VMSet = mockVMSet
-//	//az.LoadBalancerClient = mockLBClient
-//	bi := newBackendPoolTypeNodeIP(az)
-//	svc := getTestService("test", v1.ProtocolTCP, nil, false, 80)
-//	_, _, err := bi.ReconcileBackendPools(testClusterName, &svc, &lb)
-//	assert.Contains(t, err.Error(), "delete LB backend pool error")
-//
-//	lb = buildDefaultTestLB(testClusterName, []string{
-//		"/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool1-00000000-nic-1/ipConfigurations/ipconfig1",
-//		"/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool2-00000000-nic-1/ipConfigurations/ipconfig1",
-//	})
-//	mockLBClient := mockloadbalancerclient.NewMockInterface(ctrl)
-//	mockLBClient.EXPECT().DeleteLBBackendPool(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(retry.NewError(false, fmt.Errorf("error2")))
-//	mockVMSet.EXPECT().EnsureBackendPoolDeleted(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(nil)
-//	az.LoadBalancerClient = mockLBClient
-//	_, _, err = bi.ReconcileBackendPools(testClusterName, &svc, &lb)
-//	assert.Contains(t, err.Error(), "error2")
-//
-//	lb = buildDefaultTestLB(testClusterName, []string{
-//		"/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool1-00000000-nic-1/ipConfigurations/ipconfig1",
-//		"/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool2-00000000-nic-1/ipConfigurations/ipconfig1",
-//	})
-//	mockLBClient.EXPECT().DeleteLBBackendPool(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(nil)
-//	mockVMSet.EXPECT().EnsureBackendPoolDeleted(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(nil)
-//	_, _, err = bi.ReconcileBackendPools(testClusterName, &svc, &lb)
-//	assert.NoError(t, err)
-//	assert.Empty(t, (*lb.BackendAddressPools)[0].LoadBalancerBackendAddresses)
-//}
+func TestReconcileBackendPoolsNodeIPConfigToIP(t *testing.T) {
+	ctrl := gomock.NewController(t)
+	defer ctrl.Finish()
+
+	lb := buildDefaultTestLB(testClusterName, []string{
+		"/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool1-00000000-nic-1/ipConfigurations/ipconfig1",
+		"/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool2-00000000-nic-1/ipConfigurations/ipconfig1",
+	})
+	mockVMSet := NewMockVMSet(ctrl)
+	mockVMSet.EXPECT().EnsureBackendPoolDeleted(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(fmt.Errorf("delete LB backend pool error"))
+	mockVMSet.EXPECT().GetPrimaryVMSetName().Return("k8s-agentpool1-00000000").AnyTimes()
+
+	az := GetTestCloud(ctrl)
+	az.VMSet = mockVMSet
+	//az.LoadBalancerClient = mockLBClient
+	bi := newBackendPoolTypeNodeIP(az)
+	svc := getTestService("test", v1.ProtocolTCP, nil, false, 80)
+	_, _, err := bi.ReconcileBackendPools(testClusterName, &svc, &lb)
+	assert.Contains(t, err.Error(), "delete LB backend pool error")
+
+	lb = buildDefaultTestLB(testClusterName, []string{
+		"/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool1-00000000-nic-1/ipConfigurations/ipconfig1",
+		"/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool2-00000000-nic-1/ipConfigurations/ipconfig1",
+	})
+	mockLBClient := mockloadbalancerclient.NewMockInterface(ctrl)
+	mockLBClient.EXPECT().DeleteLBBackendPool(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(retry.NewError(false, fmt.Errorf("error2")))
+	mockVMSet.EXPECT().EnsureBackendPoolDeleted(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(nil)
+	az.LoadBalancerClient = mockLBClient
+	_, _, err = bi.ReconcileBackendPools(testClusterName, &svc, &lb)
+	assert.Contains(t, err.Error(), "error2")
+
+	lb = buildDefaultTestLB(testClusterName, []string{
+		"/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool1-00000000-nic-1/ipConfigurations/ipconfig1",
+		"/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool2-00000000-nic-1/ipConfigurations/ipconfig1",
+	})
+	mockLBClient.EXPECT().DeleteLBBackendPool(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(nil)
+	mockVMSet.EXPECT().EnsureBackendPoolDeleted(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(nil)
+	_, _, err = bi.ReconcileBackendPools(testClusterName, &svc, &lb)
+	assert.NoError(t, err)
+	assert.Empty(t, (*lb.BackendAddressPools)[0].LoadBalancerBackendAddresses)
+}
 
 func TestRemoveNodeIPAddressFromBackendPool(t *testing.T) {
 	nodeIPAddresses := []string{"1.2.3.4", "4.3.2.1"}
diff --git a/pkg/provider/azure_test.go b/pkg/provider/azure_test.go
index 1936be1eb..f8ab8b377 100644
--- a/pkg/provider/azure_test.go
+++ b/pkg/provider/azure_test.go
@@ -38,6 +38,7 @@ import (
 	"k8s.io/apimachinery/pkg/util/sets"
 	"k8s.io/client-go/informers"
 	cloudprovider "k8s.io/cloud-provider"
+	cloudproviderapi "k8s.io/cloud-provider/api"
 	servicehelpers "k8s.io/cloud-provider/service/helpers"
 
 	"sigs.k8s.io/cloud-provider-azure/pkg/auth"
@@ -3324,7 +3325,7 @@ func TestUpdateNodeCaches(t *testing.T) {
 	ctrl := gomock.NewController(t)
 	defer ctrl.Finish()
 	az := GetTestCloud(ctrl)
-
+	// delete node appearing in unmanagedNodes and excludeLoadBalancerNodes
 	zone := fmt.Sprintf("%s-0", az.Location)
 	nodesInZone := sets.NewString("prevNode")
 	az.nodeZones = map[string]sets.String{zone: nodesInZone}
@@ -3348,9 +3349,10 @@ func TestUpdateNodeCaches(t *testing.T) {
 	assert.Equal(t, 0, len(az.nodeZones[zone]))
 	assert.Equal(t, 0, len(az.nodeResourceGroups))
 	assert.Equal(t, 0, len(az.unmanagedNodes))
-	assert.Equal(t, 0, len(az.excludeLoadBalancerNodes))
+	assert.Equal(t, 1, len(az.excludeLoadBalancerNodes))
 	assert.Equal(t, 0, len(az.nodeNames))
 
+	// add new (unmanaged and to-be-excluded) node
 	newNode := v1.Node{
 		ObjectMeta: metav1.ObjectMeta{
 			Labels: map[string]string{
@@ -3367,10 +3369,91 @@ func TestUpdateNodeCaches(t *testing.T) {
 	assert.Equal(t, 1, len(az.nodeZones[zone]))
 	assert.Equal(t, 1, len(az.nodeResourceGroups))
 	assert.Equal(t, 1, len(az.unmanagedNodes))
-	assert.Equal(t, 1, len(az.excludeLoadBalancerNodes))
+	assert.Equal(t, 2, len(az.excludeLoadBalancerNodes))
 	assert.Equal(t, 1, len(az.nodeNames))
 }
 
+func TestUpdateNodeCacheExcludeLoadBalancer(t *testing.T) {
+	ctrl := gomock.NewController(t)
+	defer ctrl.Finish()
+	az := GetTestCloud(ctrl)
+
+	zone := fmt.Sprintf("%s-0", az.Location)
+	nodesInZone := sets.NewString("aNode")
+	az.nodeZones = map[string]sets.String{zone: nodesInZone}
+	az.nodeResourceGroups = map[string]string{"aNode": "rg"}
+
+	// a non-ready node should be excluded
+	az.unmanagedNodes = sets.NewString()
+	az.excludeLoadBalancerNodes = sets.NewString()
+	az.nodeNames = sets.NewString()
+	nonReadyNode := v1.Node{
+		ObjectMeta: metav1.ObjectMeta{
+			Labels: map[string]string{
+				v1.LabelTopologyZone: zone,
+			},
+			Name: "aNode",
+		},
+		Status: v1.NodeStatus{
+			Conditions: []v1.NodeCondition{
+				{
+					Type:   v1.NodeReady,
+					Status: v1.ConditionFalse,
+				},
+			},
+		},
+	}
+	az.updateNodeCaches(nil, &nonReadyNode)
+	assert.Equal(t, 1, len(az.excludeLoadBalancerNodes))
+
+	// node becomes ready, it should be removed from az.excludeLoadBalancerNodes
+	readyNode := v1.Node{
+		ObjectMeta: metav1.ObjectMeta{
+			Labels: map[string]string{
+				v1.LabelTopologyZone: zone,
+			},
+			Name: "aNode",
+		},
+		Status: v1.NodeStatus{
+			Conditions: []v1.NodeCondition{
+				{
+					Type:   v1.NodeReady,
+					Status: v1.ConditionTrue,
+				},
+			},
+		},
+	}
+	az.updateNodeCaches(&nonReadyNode, &readyNode)
+	assert.Equal(t, 0, len(az.excludeLoadBalancerNodes))
+
+	// new non-ready node with taint is added to the cluster: don't exclude it
+	nonReadyTaintedNode := v1.Node{
+		ObjectMeta: metav1.ObjectMeta{
+			Labels: map[string]string{
+				v1.LabelTopologyZone: zone,
+			},
+			Name: "anotherNode",
+		},
+		Spec: v1.NodeSpec{
+			Taints: []v1.Taint{{
+				Key:    cloudproviderapi.TaintExternalCloudProvider,
+				Value:  "aValue",
+				Effect: v1.TaintEffectNoSchedule},
+			},
+		},
+		Status: v1.NodeStatus{
+			Conditions: []v1.NodeCondition{
+				{
+					Type:   v1.NodeReady,
+					Status: v1.ConditionFalse,
+				},
+			},
+		},
+	}
+	az.updateNodeCaches(nil, &nonReadyTaintedNode)
+	assert.Equal(t, 0, len(az.excludeLoadBalancerNodes))
+}
+
 func TestGetActiveZones(t *testing.T) {
 	ctrl := gomock.NewController(t)
 	defer ctrl.Finish()
diff --git a/tests/k8s-azure/manifest/cluster-api/vmss-multi-nodepool.yaml b/tests/k8s-azure/manifest/cluster-api/vmss-multi-nodepool.yaml
index 048b991dd..623d8c82b 100644
--- a/tests/k8s-azure/manifest/cluster-api/vmss-multi-nodepool.yaml
+++ b/tests/k8s-azure/manifest/cluster-api/vmss-multi-nodepool.yaml
@@ -55,6 +55,7 @@ spec:
       apiServer:
         extraArgs:
           cloud-config: /etc/kubernetes/azure.json
+          cloud-provider: azure
           feature-gates: MixedProtocolLBService=true
         extraVolumes:
           - hostPath: /etc/kubernetes/azure.json
@@ -200,7 +201,7 @@ spec:
     nodeRegistration:
       name: '{{ ds.meta_data["local_hostname"] }}'
       kubeletExtraArgs:
-        cloud-provider: external
+        cloud-provider: azure
         cloud-config: /etc/kubernetes/azure.json
         azure-container-registry-config: /etc/kubernetes/azure.json
   files:
@@ -263,7 +264,7 @@ spec:
     nodeRegistration:
       name: '{{ ds.meta_data["local_hostname"] }}'
       kubeletExtraArgs:
-        cloud-provider: external
+        cloud-provider: azure
         cloud-config: /etc/kubernetes/azure.json
         azure-container-registry-config: /etc/kubernetes/azure.json
   files:
@@ -606,7 +607,6 @@ data:
               command:
                 - cloud-node-manager
                 - --node-name=$(NODE_NAME)
-                - --v=4
               env:
                 - name: NODE_NAME
                   valueFrom: