Repository: Azure/aad-pod-identity
Pull request #776 (2020-08-31 18:46:37)

chore: set dnspolicy to clusterfirstwithhostnet for NMI
<!-- Thank you for helping AAD Pod Identity with a pull request! Please make sure you read the [contributing guidelines](https://github.com/Azure/aad-pod-identity/blob/master/CONTRIBUTING.md). -->

**Reason for Change**:
<!-- What does this PR improve or fix in AAD Pod Identity? Why is it needed? -->
Adds `dnsPolicy: ClusterFirstWithHostNet` for NMI daemonset as that's the recommended policy for pods running on host network

**Requirements**

- [x] squashed commits
- [ ] included documentation
- [ ] added unit tests and e2e tests (if applicable). See [test standard](https://github.com/Azure/aad-pod-identity/blob/master/CONTRIBUTING.md#test-standard) for more details.
- [x] ran `make precommit`

**Issue Fixed**:
<!-- If this PR fixes GitHub issue 1234, add "Fixes #1234" to the next line. -->
fixes https://github.com/Azure/aad-pod-identity/issues/775

**Please answer the following questions with yes/no**:

Does this change contain code from or inspired by another project? If so, did you notify the maintainers and provide attribution?


**Notes for Reviewers**:


codecov-commenter: # [Codecov](https://codecov.io/gh/Azure/aad-pod-identity/pull/776?src=pr&el=h1) Report
> Merging [#776](https://codecov.io/gh/Azure/aad-pod-identity/pull/776?src=pr&el=desc) into [master](https://codecov.io/gh/Azure/aad-pod-identity/commit/c88836ed0a7bb91b3011e37d0f27eb4cadd5149a?el=desc) will **not change** coverage.
> The diff coverage is `n/a`.


```diff
@@           Coverage Diff           @@
##           master     #776   +/-   ##
=======================================
  Coverage   36.70%   36.70%           
=======================================
  Files          25       25           
  Lines        3098     3098           
=======================================
  Hits         1137     1137           
  Misses       1887     1887           
  Partials       74       74           
```


Patch:
diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
index 0d865b45..29e367e6 100644
--- a/.github/ISSUE_TEMPLATE/bug_report.md
+++ b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -4,9 +4,14 @@ about: Create a report to help AAD Pod Identity improve
 title: ''
 labels: bug
 assignees: ''
-
 ---
 
+**Have you**
+
+- [ ] Read [Troubleshooting Guide](../../docs/readmes/README.troubleshooting.md)
+- [ ] Read [Known Issues](../../docs/readmes/README.known-issues.md)
+- [ ] Searched on [GitHub issues](https://github.com/Azure/aad-pod-identity/issues)
+
 **Describe the bug**
 
 **Steps To Reproduce**
diff --git a/README.md b/README.md
index d072317d..fa5338eb 100644
--- a/README.md
+++ b/README.md
@@ -97,21 +97,45 @@ For more information, please refer to the [design documentation](./docs/design/c
 
 ## Role Assignment
 
-Your cluster will need the correct role assignment configuration to perform Azure-related operations such as assigning and un-assigning the identity on the underlying VM/VMSS. Please refer to the [role assignment](./docs/readmes/README.role-assignment.md) documentation to review and set required role assignments.
+Your cluster will need the correct role assignment configuration to perform Azure-related operations such as assigning and un-assigning the identity on the underlying VM/VMSS. You can run the following commands to help you set up the appropriate role assignments for your cluster identity before deploying aad-pod-identity (assuming you are running an AKS cluster):
+
+```bash
+export SUBSCRIPTION_ID="<SubscriptionID>"
+export RESOURCE_GROUP="<AKSResourceGroup>"
+export CLUSTER_NAME="<AKSClusterName>"
+export CLUSTER_LOCATION="<AKSClusterLocation>"
+
+# if you are planning to deploy your user-assigned identities in a separate resource group
+export IDENTITY_RESOURCE_GROUP="<IdentityResourceGroup>"
+
+./hack/role-assignment.sh
+```
+
+> Note: `<AKSResourceGroup>` is where your AKS cluster is deployed to.
+
+For more details, please refer to the [role assignment](./docs/readmes/README.role-assignment.md) documentation.
 
 ## Demo
 
 You will need [Azure CLI] installed and a Kubernetes cluster running on Azure, either managed by [AKS] or provisioned with [AKS Engine].
 
-Set the following Azure-related environment variables before getting started:
+Run the following commands to set Azure-related environment variables and login to Azure via `az login`:
 
 ```bash
-export SUBSCRIPTION_ID="<SubscriptionId>"
-export RESOURCE_GROUP="<ResourceGroup>"
+export SUBSCRIPTION_ID="<SubscriptionID>"
+export RESOURCE_GROUP="<AKSResourceGroup>"
+export CLUSTER_NAME="<AKSClusterName>"
+export CLUSTER_LOCATION="<AKSClusterLocation>"
+
+export IDENTITY_RESOURCE_GROUP="MC_${RESOURCE_GROUP}_${CLUSTER_NAME}_${CLUSTER_LOCATION}"
 export IDENTITY_NAME="demo"
+
+# login as a user and set the appropriate subscription ID
+az login
+az account set -s "${SUBSCRIPTION_ID}"
 ```
 
-> For AKS cluster, there are two resource groups that you need to be aware of - the resource group that contains the AKS cluster itself, and the cluster resource group (`MC_<AKSClusterName>_<AKSResourceGroup>_<Location>`). The latter contains all of the infrastructure resources associated with the cluster like VM/VMSS and VNet. Depending on where you deploy your user-assigned identities, you might need additional role assignments. Please refer to [Role Assignment](#role-assignment) for more information. For this demo, it is recommended to use the cluster resource group (the one with `MC_` prefix) as the `RESOURCE_GROUP` environment variable.
+> For AKS clusters, there are two resource groups that you need to be aware of - the resource group where you deploy your AKS cluster to (denoted by the environment variable `RESOURCE_GROUP`), and the cluster resource group (`MC_<AKSResourceGroup>_<AKSClusterName>_<AKSClusterLocation>`). The latter contains all of the infrastructure resources associated with the cluster like VM/VMSS and VNet. Depending on where you deploy your user-assigned identities, you might need additional role assignments. Please refer to [Role Assignment](#role-assignment) for more information. For this demo, it is recommended to deploy the demo identity to your cluster resource group (the one with `MC_` prefix).
 
 ### 1. Deploy aad-pod-identity
 
@@ -144,20 +168,24 @@ For a list of overwritable values when installing with Helm, please refer to [th
 
 > Important: For AKS clusters with limited [egress-traffic], Please install pod-identity in `kube-system` namespace using the [helm charts].
 
+```bash
+helm install aad-pod-identity aad-pod-identity/aad-pod-identity --namespace=kube-system
+```
+
 ### 2. Create an identity on Azure
 
 Create an identity on Azure and store the client ID and resource ID of the identity as environment variables:
 
 ```bash
-az identity create -g $RESOURCE_GROUP -n $IDENTITY_NAME --subscription $SUBSCRIPTION_ID
-export IDENTITY_CLIENT_ID="$(az identity show -g $RESOURCE_GROUP -n $IDENTITY_NAME --subscription $SUBSCRIPTION_ID --query clientId -otsv)"
-export IDENTITY_RESOURCE_ID="$(az identity show -g $RESOURCE_GROUP -n $IDENTITY_NAME --subscription $SUBSCRIPTION_ID --query id -otsv)"
+az identity create -g ${IDENTITY_RESOURCE_GROUP} -n ${IDENTITY_NAME}
+export IDENTITY_CLIENT_ID="$(az identity show -g ${IDENTITY_RESOURCE_GROUP} -n ${IDENTITY_NAME} --query clientId -otsv)"
+export IDENTITY_RESOURCE_ID="$(az identity show -g ${IDENTITY_RESOURCE_GROUP} -n ${IDENTITY_NAME} --query id -otsv)"
 ```
 
 Assign the role "Reader" to the identity so it has read access to the resource group. At the same time, store the identity assignment ID as an environment variable.
 
 ```bash
-export IDENTITY_ASSIGNMENT_ID="$(az role assignment create --role Reader --assignee $IDENTITY_CLIENT_ID --scope /subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP --query id -otsv)"
+export IDENTITY_ASSIGNMENT_ID="$(az role assignment create --role Reader --assignee ${IDENTITY_CLIENT_ID} --scope /subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${IDENTITY_RESOURCE_GROUP} --query id -otsv)"
 ```
 
 ### 3. Deploy `AzureIdentity`
@@ -169,11 +197,11 @@ cat <<EOF | kubectl apply -f -
 apiVersion: "aadpodidentity.k8s.io/v1"
 kind: AzureIdentity
 metadata:
-  name: $IDENTITY_NAME
+  name: ${IDENTITY_NAME}
 spec:
   type: 0
-  resourceID: $IDENTITY_RESOURCE_ID
-  clientID: $IDENTITY_CLIENT_ID
+  resourceID: ${IDENTITY_RESOURCE_ID}
+  clientID: ${IDENTITY_CLIENT_ID}
 EOF
 ```
 
@@ -181,7 +209,7 @@ EOF
 
 ### 4. (Optional) Match pods in the namespace
 
-For matching pods in the namespace, please refer to namespaced [README](docs/readmes/README.namespaced.md).
+For matching pods in the namespace, please refer to the [namespaced documentation](docs/readmes/README.namespaced.md).
 
 ### 5. Deploy `AzureIdentityBinding`
 
@@ -192,10 +220,10 @@ cat <<EOF | kubectl apply -f -
 apiVersion: "aadpodidentity.k8s.io/v1"
 kind: AzureIdentityBinding
 metadata:
-  name: $IDENTITY_NAME-binding
+  name: ${IDENTITY_NAME}-binding
 spec:
-  azureIdentity: $IDENTITY_NAME
-  selector: $IDENTITY_NAME
+  azureIdentity: ${IDENTITY_NAME}
+  selector: ${IDENTITY_NAME}
 EOF
 ```
 
@@ -216,9 +244,9 @@ spec:
   - name: demo
     image: mcr.microsoft.com/k8s/aad-pod-identity/demo:1.2
     args:
-      - --subscriptionid=$SUBSCRIPTION_ID
-      - --clientid=$IDENTITY_CLIENT_ID
-      - --resourcegroup=$RESOURCE_GROUP
+      - --subscriptionid=${SUBSCRIPTION_ID}
+      - --clientid=${IDENTITY_CLIENT_ID}
+      - --resourcegroup=${IDENTITY_RESOURCE_GROUP}
     env:
       - name: MY_POD_NAME
         valueFrom:
@@ -246,7 +274,8 @@ kubectl logs demo
 ```
 
 If successful, the log output would be similar to the following output:
-```
+
+```log
 ...
 successfully doARMOperations vm count 1
 successfully acquired a token using the MSI, msiEndpoint(http://169.254.169.254/metadata/identity/oauth2/token)
@@ -259,10 +288,10 @@ Once you are done with the demo, clean up your resources:
 
 ```bash
 kubectl delete pod demo
-kubectl delete azureidentity $IDENTITY_NAME
-kubectl delete azureidentitybinding $IDENTITY_NAME-binding
-az role assignment delete --id $IDENTITY_ASSIGNMENT_ID
-az identity delete -g $RESOURCE_GROUP -n $IDENTITY_NAME
+kubectl delete azureidentity ${IDENTITY_NAME}
+kubectl delete azureidentitybinding ${IDENTITY_NAME}-binding
+az role assignment delete --id ${IDENTITY_ASSIGNMENT_ID}
+az identity delete -g ${IDENTITY_RESOURCE_GROUP} -n ${IDENTITY_NAME}
 ```
 
 ## Uninstall Notes
diff --git a/cmd/mic/main.go b/cmd/mic/main.go
index 819ccb70..23673b4d 100644
--- a/cmd/mic/main.go
+++ b/cmd/mic/main.go
@@ -19,22 +19,23 @@ import (
 )
 
 var (
-	kubeconfig          string
-	cloudconfig         string
-	forceNamespaced     bool
-	versionInfo         bool
-	syncRetryDuration   time.Duration
-	leaderElectionCfg   mic.LeaderElectionConfig
-	httpProbePort       string
-	enableProfile       bool
-	enableScaleFeatures bool
-	createDeleteBatch   int64
-	clientQPS           float64
-	prometheusPort      string
-	immutableUserMSIs   string
-	cmConfig            mic.CMConfig
-	typeUpgradeConfig   mic.TypeUpgradeConfig
-	updateUserMSIConfig mic.UpdateUserMSIConfig
+	kubeconfig                          string
+	cloudconfig                         string
+	forceNamespaced                     bool
+	versionInfo                         bool
+	syncRetryDuration                   time.Duration
+	leaderElectionCfg                   mic.LeaderElectionConfig
+	httpProbePort                       string
+	enableProfile                       bool
+	enableScaleFeatures                 bool
+	createDeleteBatch                   int64
+	clientQPS                           float64
+	prometheusPort                      string
+	immutableUserMSIs                   string
+	cmConfig                            mic.CMConfig
+	typeUpgradeConfig                   mic.TypeUpgradeConfig
+	updateUserMSIConfig                 mic.UpdateUserMSIConfig
+	identityAssignmentReconcileInterval time.Duration
 )
 
 func main() {
@@ -87,6 +88,9 @@ func main() {
 	flag.IntVar(&updateUserMSIConfig.MaxRetry, "update-user-msi-max-retry", 2, "The maximum retry of UpdateUserMSI call")
 	flag.DurationVar(&updateUserMSIConfig.RetryInterval, "update-user-msi-retry-interval", 1*time.Second, "The duration to wait before retrying UpdateUserMSI")
 
+	// Parameters for reconciling identity assignment on Azure
+	flag.DurationVar(&identityAssignmentReconcileInterval, "identity-assignment-reconcile-interval", 3*time.Minute, "The interval between reconciling identity assignment on Azure based on an existing list of AzureAssignedIdentities")
+
 	flag.Parse()
 
 	podns := os.Getenv("MIC_POD_NAMESPACE")
@@ -140,17 +144,18 @@ func main() {
 	}
 
 	micConfig := &mic.Config{
-		CloudCfgPath:          cloudconfig,
-		RestConfig:            config,
-		IsNamespaced:          forceNamespaced,
-		SyncRetryInterval:     syncRetryDuration,
-		LeaderElectionCfg:     &leaderElectionCfg,
-		EnableScaleFeatures:   enableScaleFeatures,
-		CreateDeleteBatch:     createDeleteBatch,
-		ImmutableUserMSIsList: immutableUserMSIsList,
-		CMcfg:                 &cmConfig,
-		TypeUpgradeCfg:        &typeUpgradeConfig,
-		UpdateUserMSICfg:      &updateUserMSIConfig,
+		CloudCfgPath:                        cloudconfig,
+		RestConfig:                          config,
+		IsNamespaced:                        forceNamespaced,
+		SyncRetryInterval:                   syncRetryDuration,
+		LeaderElectionCfg:                   &leaderElectionCfg,
+		EnableScaleFeatures:                 enableScaleFeatures,
+		CreateDeleteBatch:                   createDeleteBatch,
+		ImmutableUserMSIsList:               immutableUserMSIsList,
+		CMcfg:                               &cmConfig,
+		TypeUpgradeCfg:                      &typeUpgradeConfig,
+		UpdateUserMSICfg:                    &updateUserMSIConfig,
+		IdentityAssignmentReconcileInterval: identityAssignmentReconcileInterval,
 	}
 
 	micClient, err := mic.NewMICClient(micConfig)
diff --git a/docs/readmes/README.role-assignment.md b/docs/readmes/README.role-assignment.md
index 2b9058f1..bb8a1736 100644
--- a/docs/readmes/README.role-assignment.md
+++ b/docs/readmes/README.role-assignment.md
@@ -2,6 +2,24 @@
 
 AKS and aks-engine clusters require an identity to communicate with Azure. This identity can be either a **managed identity** (in the form of system-assigned identity or user-assigned identity) or a **service principal**. This section explains various role assignments that need to be performed before using AAD Pod Identity. Without the proper role assignments, your Azure cluster will not have the correct permission to assign and un-assign identities from the underlying virtual machines (VM) or virtual machine scale sets (VMSS).
 
+## TL;DR
+
+Run the following commands to help you set up the appropriate role assignments for your cluster identity before deploying aad-pod-identity (assuming you are running an AKS cluster):
+
+```bash
+export SUBSCRIPTION_ID="<SubscriptionID>"
+export RESOURCE_GROUP="<AKSResourceGroup>"
+export CLUSTER_NAME="<AKSClusterName>"
+export CLUSTER_LOCATION="<AKSClusterLocation>"
+
+# if you are planning to deploy your user-assigned identities in a separate resource group
+export IDENTITY_RESOURCE_GROUP="<IdentityResourceGroup>"
+
+./hack/role-assignment.sh
+```
+
+> Note: `<AKSResourceGroup>` is where your AKS cluster is deployed to.
+
 ## Introduction
 
 Currently, [MIC](../../README.md#managed-identity-controller) uses one of the following two ways to authenticate with Azure:
@@ -18,7 +36,7 @@ Currently, [MIC](../../README.md#managed-identity-controller) uses one of the fo
 | Authentication method                  | `/etc/kubernetes/azure.json` fields used                                                     |
 |----------------------------------------|---------------------------------------------------------------------------------------------|
 | System-assigned managed identity       | `useManagedIdentityExtension: true` and `userAssignedIdentityID:""`                         |
-| User-assigned managed identity cluster | `useManagedIdentityExtension: true` and `userAssignedIdentityID:"<UserAssignedIdentityID>"` |
+| User-assigned managed identity         | `useManagedIdentityExtension: true` and `userAssignedIdentityID:"<UserAssignedIdentityID>"` |
 | Service principal (default)            | `aadClientID: "<AADClientID>"` and `aadClientSecret: "<AADClientSecret>"`                   |
 
 ## Obtaining the ID of the managed identity / service principal
@@ -44,6 +62,8 @@ az role assignment create --role "Managed Identity Operator" --assignee <ID> --s
 az role assignment create --role "Virtual Machine Contributor" --assignee <ID> --scope /subscriptions/<SubscriptionID>/resourcegroups/<ClusterResourceGroup>
 ```
 
+> RBAC and non-RBAC clusters require the same role assignments.
+
 ## User-assigned identities that are not within the cluster resource group
 
 There are additional role assignments required if you wish to assign user-assigned identities that are not within the cluster resource group. You can run the following command to assign the **Managed Identity Operator** role with the identity resource group scope:
diff --git a/docs/readmes/README.troubleshooting.md b/docs/readmes/README.troubleshooting.md
index ffbb6984..f0861492 100644
--- a/docs/readmes/README.troubleshooting.md
+++ b/docs/readmes/README.troubleshooting.md
@@ -32,6 +32,66 @@ The expected output should be:
 -A aad-metadata -j RETURN
 ```
 
+### Run a pod to validate your identity setup
+
+You could run the following commands to validate your identity setup (assuming you have the proper `AzureIdentity` and `AzureIdentityBinding` deployed):
+
+```bash
+kubectl run azure-cli -it --image=mcr.microsoft.com/azure-cli --labels=aadpodidbinding=<selector defined in AzureIdentityBinding> /bin/bash
+
+# within the azure-cli shell
+az login -i --debug
+```
+
+`az login -i` will use the Azure identity bound to the `azure-cli` pod and perform a login to Azure via Azure CLI. If succeeded, you would have an output as below:
+
+```log
+urllib3.connectionpool : Starting new HTTP connection (1): 169.254.169.254:80
+urllib3.connectionpool : http://169.254.169.254:80 "GET /metadata/identity/oauth2/token?resource=https%3A%2F%2Fmanagement.core.windows.net%2F&api-version=2018-02-01 HTTP/1.1" 200 1667
+msrestazure.azure_active_directory : MSI: Retrieving a token from http://169.254.169.254/metadata/identity/oauth2/token, with payload {'resource': 'https://management.core.windows.net/', 'api-version': '2018-02-01'}
+msrestazure.azure_active_directory : MSI: Token retrieved
+MSI: token was retrieved. Now trying to initialize local accounts...
+...
+[
+  {
+    "environmentName": "AzureCloud",
+    "homeTenantId": "<REDACTED>",
+    "id": "<REDACTED>",
+    "isDefault": true,
+    "managedByTenants": [],
+    "name": "<REDACTED>",
+    "state": "Enabled",
+    "tenantId": "<REDACTED>",
+    "user": {
+      "assignedIdentityInfo": "MSI",
+      "name": "systemAssignedIdentity",
+      "type": "servicePrincipal"
+    }
+  }
+]
+```
+
+Based on the logs above, Azure CLI was able to retrieve a token from `http://169.254.169.254:80/metadata/identity/oauth2/token`. Its request is routed to the NMI pod that is running within the same node. Identify which node the Azure CLI pod is scheduled to by running the following command:
+
+```bash
+kubectl get pods -owide
+
+NAME                                    READY   STATUS    RESTARTS   AGE   IP             NODE                                 NOMINATED NODE   READINESS GATES
+azure-cli                               1/1     Running   1          12s   10.240.0.117   k8s-agentpool1-95854893-vmss000002   <none>           <none>
+```
+
+Take a note at the node the pod is scheduled to and its IP address. Check the logs of the NMI pod that is scheduled to the same node. You should be able to see a token requested by the azure-cli pod, identified by its pod IP address `10.240.0.117`:
+
+```bash
+kubectl logs <nmi pod name>
+
+...
+I0821 18:22:50.810806       1 standard.go:72] no clientID or resourceID in request. default/azure-cli has been matched with azure identity default/demo
+I0821 18:22:50.810895       1 standard.go:178] matched identityType:0 clientid:7eb6##### REDACTED #####a6a9 resource:https://management.core.windows.net/
+I0821 18:22:51.348117       1 server.go:190] status (200) took 537597287 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=10.240.0.117
+...
+```
+
 ## Common Issues
 
 Common issues or questions that users have run into when using pod identity are detailed below.
@@ -67,7 +127,7 @@ kubectl delete azureassignedidentity --all
 To delete only a specific `AzureAssignedIdentity`, run the following command:
 ```bash
 kubectl get azureassignedidentity <name> -n <namespace> -o=json | jq '.items[].metadata.finalizers=null' | kubectl apply -f -
-kubectl delete azureassignedidentity <name> -n <namespace> 
+kubectl delete azureassignedidentity <name> -n <namespace>
 ```
 
 Past issues:
diff --git a/hack/role-assignment.sh b/hack/role-assignment.sh
new file mode 100755
index 00000000..4feec8d9
--- /dev/null
+++ b/hack/role-assignment.sh
@@ -0,0 +1,32 @@
+#!/bin/bash
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+[[ ! -z "${SUBSCRIPTION_ID:-}" ]] || (echo 'Must specify SUBSCRIPTION_ID' && exit 1)
+[[ ! -z "${RESOURCE_GROUP:-}" ]] || (echo 'Must specify RESOURCE_GROUP' && exit 1)
+[[ ! -z "${CLUSTER_NAME:-}" ]] || (echo 'Must specify CLUSTER_NAME' && exit 1)
+[[ ! -z "${CLUSTER_LOCATION:-}" ]] || (echo 'Must specify CLUSTER_LOCATION' && exit 1)
+
+echo "az login as a user and set the appropriate subscription ID"
+az login
+az account set -s "${SUBSCRIPTION_ID}"
+
+echo "Retrieving your cluster identity ID, which will be used for role assignment"
+ID="$(az aks show -g ${RESOURCE_GROUP} -n ${CLUSTER_NAME} --query servicePrincipalProfile.clientId -otsv)"
+
+echo "Checking if the aks cluster is using managed identity"
+if [[ "${ID:-}" == "msi" ]]; then
+  ID="$(az aks show -g ${RESOURCE_GROUP} -n ${CLUSTER_NAME} --query identityProfile.kubeletidentity.clientId -otsv)"
+fi
+
+echo "Performing role assignments"
+az role assignment create --role "Managed Identity Operator" --assignee "${ID}" --scope "/subscriptions/${SUBSCRIPTION_ID}/resourcegroups/MC_${RESOURCE_GROUP}_${CLUSTER_NAME}_${CLUSTER_LOCATION}"
+az role assignment create --role "Virtual Machine Contributor" --assignee "${ID}" --scope "/subscriptions/${SUBSCRIPTION_ID}/resourcegroups/MC_${RESOURCE_GROUP}_${CLUSTER_NAME}_${CLUSTER_LOCATION}"
+
+# your resource group that is used to store your user-assigned identities
+# assuming it is within the same subscription as your AKS cluster resource group
+if [[ -n "${IDENTITY_RESOURCE_GROUP:-}" ]]; then
+  az role assignment create --role "Managed Identity Operator" --assignee "${ID}" --scope "/subscriptions/${SUBSCRIPTION_ID}/resourcegroups/${IDENTITY_RESOURCE_GROUP}"
+fi
diff --git a/manifest_staging/charts/aad-pod-identity/README.md b/manifest_staging/charts/aad-pod-identity/README.md
index e10bfed8..fcb73bab 100755
--- a/manifest_staging/charts/aad-pod-identity/README.md
+++ b/manifest_staging/charts/aad-pod-identity/README.md
@@ -17,7 +17,7 @@ helm install aad-pod-identity/aad-pod-identity --set=installCRDs=true
 ## Helm chart and aad-pod-identity versions
 
 | Helm Chart Version | AAD Pod Identity Version |
-| ------------------ | ------------------------ |
+|--------------------|--------------------------|
 | `1.5.2`            | `1.5.2`                  |
 | `1.5.3`            | `1.5.3`                  |
 | `1.5.4`            | `1.5.4`                  |
@@ -181,56 +181,57 @@ The following is a basic example of the required change in the user-supplied val
 
 The following tables list the configurable parameters of the aad-pod-identity chart and their default values.
 
-| Parameter                                | Description                                                                                                                                                                                                                                                                                                                   | Default                                                        |
-| ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- |
-| `nameOverride`                           | String to partially override aad-pod-identity.fullname template with a string (will prepend the release name)                                                                                                                                                                                                                 | `""`                                                           |
-| `fullnameOverride`                       | String to fully override aad-pod-identity.fullname template with a string                                                                                                                                                                                                                                                     | `""`                                                           |
-| `image.repository`                       | Image repository                                                                                                                                                                                                                                                                                                              | `mcr.microsoft.com/k8s/aad-pod-identity`                       |
-| `image.pullPolicy`                       | Image pull policy                                                                                                                                                                                                                                                                                                             | `Always`                                                       |
-| `forceNameSpaced`                        | By default, AAD Pod Identity matches pods to identities across namespaces. To match only pods in the namespace containing AzureIdentity set this to true.                                                                                                                                                                     | `false`                                                        |
-| `installMICException`                    | When NMI runs on a node where MIC is running, then MIC token request call is also intercepted by NMI. MIC can't get a valid token to initialize and then assign the identity. Installing an exception for MIC would ensure all token requests for MIC pods directly go to IMDS and not go through the pod-identity validation | `true`                                                         |
-| `adminsecret.cloud`                      | Azure cloud environment name                                                                                                                                                                                                                                                                                                  | ` `                                                            |
-| `adminsecret.subscriptionID`             | Azure subscription ID                                                                                                                                                                                                                                                                                                         | ` `                                                            |
-| `adminsecret.resourceGroup`              | Azure resource group                                                                                                                                                                                                                                                                                                          | ` `                                                            |
-| `adminsecret.vmType`                     | `standard` for normal virtual machine nodes, and `vmss` for cluster deployed with a virtual machine scale set                                                                                                                                                                                                                 | ` `                                                            |
-| `adminsecret.tenantID`                   | Azure service principal tenantID                                                                                                                                                                                                                                                                                              | ` `                                                            |
-| `adminsecret.clientID`                   | Azure service principal clientID                                                                                                                                                                                                                                                                                              | ` `                                                            |
-| `adminsecret.clientSecret`               | Azure service principal clientSecret                                                                                                                                                                                                                                                                                          | ` `                                                            |
-| `mic.image`                              | MIC image name                                                                                                                                                                                                                                                                                                                | `mic`                                                          |
-| `mic.tag`                                | MIC image tag                                                                                                                                                                                                                                                                                                                 | `1.6.2`                                                        |
-| `mic.PriorityClassName`                  | MIC priority class (can only be set when deploying to kube-system namespace)                                                                                                                                                                                                                                                  |                                                                |
-| `mic.logVerbosity`                       | Log level. Uses V logs (glog)                                                                                                                                                                                                                                                                                                 | `0`                                                            |
-| `mic.resources`                          | Resource limit for MIC                                                                                                                                                                                                                                                                                                        | `{}`                                                           |
-| `mic.podAnnotations`                     | Pod annotations for MIC                                                                                                                                                                                                                                                                                                       | `{}`                                                           |
-| `mic.tolerations`                        | Affinity settings                                                                                                                                                                                                                                                                                                             | `{}`                                                           |
-| `mic.affinity`                           | List of node taints to tolerate                                                                                                                                                                                                                                                                                               | `[]`                                                           |
-| `mic.leaderElection.instance`            | Override leader election instance name                                                                                                                                                                                                                                                                                        | If not provided, default value is `hostname`                   |
-| `mic.leaderElection.namespace`           | Override the namespace to create leader election objects                                                                                                                                                                                                                                                                      | `default`                                                      |
-| `mic.leaderElection.name`                | Override leader election name                                                                                                                                                                                                                                                                                                 | If not provided, default value is `aad-pod-identity-mic`       |
-| `mic.leaderElection.duration`            | Override leader election duration                                                                                                                                                                                                                                                                                             | If not provided, default value is `15s`                        |
-| `mic.probePort`                          | Override http liveliness probe port                                                                                                                                                                                                                                                                                           | If not provided, default port is `8080`                        |
-| `mic.syncRetryDuration`                  | Override interval in seconds at which sync loop should periodically check for errors and reconcile                                                                                                                                                                                                                            | If not provided, default value is `3600s`                      |
-| `mic.immutableUserMSIs`                  | List of  user-defined identities that shouldn't be deleted from VM/VMSS.                                                                                                                                                                                                                                                      | If not provided, default value is empty                        |
-| `mic.cloudConfig`                        | The cloud configuration used to authenticate with Azure                                                                                                                                                                                                                                                                       | If not provided, default value is `/etc/kubernetes/azure.json` |
-| `mic.updateUserMSIMaxRetry`              | The maximum retry of UpdateUserMSI call in case of assignment errors                                                                                                                                                                                                                                                          | If not provided, default value is `2`                          |
-| `mic.updateUserMSIRetryInterval`         | The duration to wait before retrying UpdateUserMSI (batch assigning/un-assigning identity from VM/VMSS) in case of errors                                                                                                                                                                                                     | If not provided, default value is `1s`                         |
-| `nmi.image`                              | NMI image name                                                                                                                                                                                                                                                                                                                | `nmi`                                                          |
-| `nmi.tag`                                | NMI image tag                                                                                                                                                                                                                                                                                                                 | `1.6.2`                                                        |
-| `nmi.PriorityClassName`                  | NMI priority class (can only be set when deploying to kube-system namespace)                                                                                                                                                                                                                                                  |                                                                |
-| `nmi.resources`                          | Resource limit for NMI                                                                                                                                                                                                                                                                                                        | `{}`                                                           |
-| `nmi.podAnnotations`                     | Pod annotations for NMI                                                                                                                                                                                                                                                                                                       | `{}`                                                           |
-| `nmi.tolerations`                        | Affinity settings                                                                                                                                                                                                                                                                                                             | `{}`                                                           |
-| `nmi.affinity`                           | List of node taints to tolerate                                                                                                                                                                                                                                                                                               | `[]`                                                           |
-| `nmi.ipTableUpdateTimeIntervalInSeconds` | Override iptables update interval in seconds                                                                                                                                                                                                                                                                                  | `60`                                                           |
-| `nmi.micNamespace`                       | Override mic namespace to short circuit MIC token requests                                                                                                                                                                                                                                                                    | If not provided, default is `default` namespace                |
-| `nmi.probePort`                          | Override http liveliness probe port                                                                                                                                                                                                                                                                                           | If not provided, default is `8085`                             |
-| `nmi.retryAttemptsForCreated`            | Override number of retries in NMI to find assigned identity in CREATED state                                                                                                                                                                                                                                                  | If not provided, default is  `16`                              |
-| `nmi.retryAttemptsForAssigned`           | Override number of retries in NMI to find assigned identity in ASSIGNED state                                                                                                                                                                                                                                                 | If not provided, default is  `4`                               |
-| `nmi.findIdentityRetryIntervalInSeconds` | Override retry interval to find assigned identities in seconds                                                                                                                                                                                                                                                                | If not provided, default is  `5`                               |
-| `rbac.enabled`                           | Create and use RBAC for all aad-pod-identity resources                                                                                                                                                                                                                                                                        | `true`                                                         |
-| `rbac.allowAccessToSecrets`              | NMI requires permissions to get secrets when service principal (type: 1) is used in AzureIdentity. If using only MSI (type: 0) in AzureIdentity, secret get permission can be disabled by setting this to false.                                                                                                              | `true`                                                         |
-| `azureIdentities`                        | List of azure identities and azure identity bindings resources to create                                                                                                                                                                                                                                                      | `[]`                                                           |
-| `installCRDs`                            | If true, install necessary custom resources                                                                                                                                                                                                                                                                                   | `false`                                                        |
+| Parameter                                 | Description                                                                                                                                                                                                                                                                                                                   | Default                                                        |
+|-------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------|
+| `nameOverride`                            | String to partially override aad-pod-identity.fullname template with a string (will prepend the release name)                                                                                                                                                                                                                 | `""`                                                           |
+| `fullnameOverride`                        | String to fully override aad-pod-identity.fullname template with a string                                                                                                                                                                                                                                                     | `""`                                                           |
+| `image.repository`                        | Image repository                                                                                                                                                                                                                                                                                                              | `mcr.microsoft.com/k8s/aad-pod-identity`                       |
+| `image.pullPolicy`                        | Image pull policy                                                                                                                                                                                                                                                                                                             | `Always`                                                       |
+| `forceNameSpaced`                         | By default, AAD Pod Identity matches pods to identities across namespaces. To match only pods in the namespace containing AzureIdentity set this to true.                                                                                                                                                                     | `false`                                                        |
+| `installMICException`                     | When NMI runs on a node where MIC is running, then MIC token request call is also intercepted by NMI. MIC can't get a valid token to initialize and then assign the identity. Installing an exception for MIC would ensure all token requests for MIC pods directly go to IMDS and not go through the pod-identity validation | `true`                                                         |
+| `adminsecret.cloud`                       | Azure cloud environment name                                                                                                                                                                                                                                                                                                  | ` `                                                            |
+| `adminsecret.subscriptionID`              | Azure subscription ID                                                                                                                                                                                                                                                                                                         | ` `                                                            |
+| `adminsecret.resourceGroup`               | Azure resource group                                                                                                                                                                                                                                                                                                          | ` `                                                            |
+| `adminsecret.vmType`                      | `standard` for normal virtual machine nodes, and `vmss` for cluster deployed with a virtual machine scale set                                                                                                                                                                                                                 | ` `                                                            |
+| `adminsecret.tenantID`                    | Azure service principal tenantID                                                                                                                                                                                                                                                                                              | ` `                                                            |
+| `adminsecret.clientID`                    | Azure service principal clientID                                                                                                                                                                                                                                                                                              | ` `                                                            |
+| `adminsecret.clientSecret`                | Azure service principal clientSecret                                                                                                                                                                                                                                                                                          | ` `                                                            |
+| `mic.image`                               | MIC image name                                                                                                                                                                                                                                                                                                                | `mic`                                                          |
+| `mic.tag`                                 | MIC image tag                                                                                                                                                                                                                                                                                                                 | `1.6.2`                                                        |
+| `mic.PriorityClassName`                   | MIC priority class (can only be set when deploying to kube-system namespace)                                                                                                                                                                                                                                                  |                                                                |
+| `mic.logVerbosity`                        | Log level. Uses V logs (glog)                                                                                                                                                                                                                                                                                                 | `0`                                                            |
+| `mic.resources`                           | Resource limit for MIC                                                                                                                                                                                                                                                                                                        | `{}`                                                           |
+| `mic.podAnnotations`                      | Pod annotations for MIC                                                                                                                                                                                                                                                                                                       | `{}`                                                           |
+| `mic.tolerations`                         | Affinity settings                                                                                                                                                                                                                                                                                                             | `{}`                                                           |
+| `mic.affinity`                            | List of node taints to tolerate                                                                                                                                                                                                                                                                                               | `[]`                                                           |
+| `mic.leaderElection.instance`             | Override leader election instance name                                                                                                                                                                                                                                                                                        | If not provided, default value is `hostname`                   |
+| `mic.leaderElection.namespace`            | Override the namespace to create leader election objects                                                                                                                                                                                                                                                                      | `default`                                                      |
+| `mic.leaderElection.name`                 | Override leader election name                                                                                                                                                                                                                                                                                                 | If not provided, default value is `aad-pod-identity-mic`       |
+| `mic.leaderElection.duration`             | Override leader election duration                                                                                                                                                                                                                                                                                             | If not provided, default value is `15s`                        |
+| `mic.probePort`                           | Override http liveliness probe port                                                                                                                                                                                                                                                                                           | If not provided, default port is `8080`                        |
+| `mic.syncRetryDuration`                   | Override interval in seconds at which sync loop should periodically check for errors and reconcile                                                                                                                                                                                                                            | If not provided, default value is `3600s`                      |
+| `mic.immutableUserMSIs`                   | List of  user-defined identities that shouldn't be deleted from VM/VMSS.                                                                                                                                                                                                                                                      | If not provided, default value is empty                        |
+| `mic.cloudConfig`                         | The cloud configuration used to authenticate with Azure                                                                                                                                                                                                                                                                       | If not provided, default value is `/etc/kubernetes/azure.json` |
+| `mic.updateUserMSIMaxRetry`               | The maximum retry of UpdateUserMSI call in case of assignment errors                                                                                                                                                                                                                                                          | If not provided, default value is `2`                          |
+| `mic.updateUserMSIRetryInterval`          | The duration to wait before retrying UpdateUserMSI (batch assigning/un-assigning identity from VM/VMSS) in case of errors                                                                                                                                                                                                     | If not provided, default value is `1s`                         |
+| `mic.identityAssignmentReconcileInterval` | The interval between reconciling identity assignment on Azure based on an existing list of AzureAssignedIdentities                                                                                                                                                                                                            | If not provided, default value is `3m`                         |
+| `nmi.image`                               | NMI image name                                                                                                                                                                                                                                                                                                                | `nmi`                                                          |
+| `nmi.tag`                                 | NMI image tag                                                                                                                                                                                                                                                                                                                 | `1.6.2`                                                        |
+| `nmi.PriorityClassName`                   | NMI priority class (can only be set when deploying to kube-system namespace)                                                                                                                                                                                                                                                  |                                                                |
+| `nmi.resources`                           | Resource limit for NMI                                                                                                                                                                                                                                                                                                        | `{}`                                                           |
+| `nmi.podAnnotations`                      | Pod annotations for NMI                                                                                                                                                                                                                                                                                                       | `{}`                                                           |
+| `nmi.tolerations`                         | Affinity settings                                                                                                                                                                                                                                                                                                             | `{}`                                                           |
+| `nmi.affinity`                            | List of node taints to tolerate                                                                                                                                                                                                                                                                                               | `[]`                                                           |
+| `nmi.ipTableUpdateTimeIntervalInSeconds`  | Override iptables update interval in seconds                                                                                                                                                                                                                                                                                  | `60`                                                           |
+| `nmi.micNamespace`                        | Override mic namespace to short circuit MIC token requests                                                                                                                                                                                                                                                                    | If not provided, default is `default` namespace                |
+| `nmi.probePort`                           | Override http liveliness probe port                                                                                                                                                                                                                                                                                           | If not provided, default is `8085`                             |
+| `nmi.retryAttemptsForCreated`             | Override number of retries in NMI to find assigned identity in CREATED state                                                                                                                                                                                                                                                  | If not provided, default is  `16`                              |
+| `nmi.retryAttemptsForAssigned`            | Override number of retries in NMI to find assigned identity in ASSIGNED state                                                                                                                                                                                                                                                 | If not provided, default is  `4`                               |
+| `nmi.findIdentityRetryIntervalInSeconds`  | Override retry interval to find assigned identities in seconds                                                                                                                                                                                                                                                                | If not provided, default is  `5`                               |
+| `rbac.enabled`                            | Create and use RBAC for all aad-pod-identity resources                                                                                                                                                                                                                                                                        | `true`                                                         |
+| `rbac.allowAccessToSecrets`               | NMI requires permissions to get secrets when service principal (type: 1) is used in AzureIdentity. If using only MSI (type: 0) in AzureIdentity, secret get permission can be disabled by setting this to false.                                                                                                              | `true`                                                         |
+| `azureIdentities`                         | List of azure identities and azure identity bindings resources to create                                                                                                                                                                                                                                                      | `[]`                                                           |
+| `installCRDs`                             | If true, install necessary custom resources                                                                                                                                                                                                                                                                                   | `false`                                                        |
 
 ## Troubleshooting
 
diff --git a/manifest_staging/charts/aad-pod-identity/templates/mic-deployment.yaml b/manifest_staging/charts/aad-pod-identity/templates/mic-deployment.yaml
index 2eaebccd..76acabb7 100644
--- a/manifest_staging/charts/aad-pod-identity/templates/mic-deployment.yaml
+++ b/manifest_staging/charts/aad-pod-identity/templates/mic-deployment.yaml
@@ -76,6 +76,9 @@ spec:
           {{- if .Values.mic.updateUserMSIRetryInterval }}
           - --update-user-msi-retry-interval={{ .Values.mic.updateUserMSIRetryInterval }}
           {{- end }}
+          {{- if .Values.mic.identityAssignmentReconcileInterval }}
+          - --identity-assignment-reconcile-interval={{ .Values.mic.identityAssignmentReconcileInterval }}
+          {{- end }}
         env:
           - name: MIC_POD_NAMESPACE
             valueFrom:
diff --git a/manifest_staging/charts/aad-pod-identity/templates/nmi-daemonset.yaml b/manifest_staging/charts/aad-pod-identity/templates/nmi-daemonset.yaml
index 8f0db6be..45fbf216 100644
--- a/manifest_staging/charts/aad-pod-identity/templates/nmi-daemonset.yaml
+++ b/manifest_staging/charts/aad-pod-identity/templates/nmi-daemonset.yaml
@@ -32,6 +32,7 @@ spec:
       priorityClassName: {{ .Values.nmi.priorityClassName | quote }}
       {{- end }}
       hostNetwork: true
+      dnsPolicy: ClusterFirstWithHostNet
       volumes:
       - hostPath:
           path: /run/xtables.lock
diff --git a/manifest_staging/charts/aad-pod-identity/values.yaml b/manifest_staging/charts/aad-pod-identity/values.yaml
index 4fc2bc92..490426fb 100644
--- a/manifest_staging/charts/aad-pod-identity/values.yaml
+++ b/manifest_staging/charts/aad-pod-identity/values.yaml
@@ -111,6 +111,10 @@ mic:
   # Default value is 1s
   updateUserMSIRetryInterval: ""
 
+  # The interval between reconciling identity assignment on Azure based on an existing list of AzureAssignedIdentities
+  # Default value is 3m
+  identityAssignmentReconcileInterval: ""
+
 nmi:
   image: nmi
   tag: 1.6.2
diff --git a/manifest_staging/deploy/infra/deployment-rbac.yaml b/manifest_staging/deploy/infra/deployment-rbac.yaml
index 067798e8..1bd394c1 100644
--- a/manifest_staging/deploy/infra/deployment-rbac.yaml
+++ b/manifest_staging/deploy/infra/deployment-rbac.yaml
@@ -114,6 +114,7 @@ spec:
     spec:
       serviceAccountName: aad-pod-id-nmi-service-account
       hostNetwork: true
+      dnsPolicy: ClusterFirstWithHostNet
       volumes:
       - hostPath:
           path: /run/xtables.lock
diff --git a/manifest_staging/deploy/infra/deployment.yaml b/manifest_staging/deploy/infra/deployment.yaml
index 6c5710c2..440b0dbc 100644
--- a/manifest_staging/deploy/infra/deployment.yaml
+++ b/manifest_staging/deploy/infra/deployment.yaml
@@ -70,6 +70,7 @@ spec:
         tier: node
     spec:
       hostNetwork: true
+      dnsPolicy: ClusterFirstWithHostNet
       volumes:
       - hostPath:
           path: /run/xtables.lock
diff --git a/manifest_staging/deploy/infra/managed-mode-deployment.yaml b/manifest_staging/deploy/infra/managed-mode-deployment.yaml
index e20e1fd2..77bd6fab 100644
--- a/manifest_staging/deploy/infra/managed-mode-deployment.yaml
+++ b/manifest_staging/deploy/infra/managed-mode-deployment.yaml
@@ -99,6 +99,7 @@ spec:
     spec:
       serviceAccountName: aad-pod-id-nmi-service-account
       hostNetwork: true
+      dnsPolicy: ClusterFirstWithHostNet
       volumes:
       - hostPath:
           path: /run/xtables.lock
diff --git a/manifest_staging/deploy/infra/noazurejson/deployment-rbac.yaml b/manifest_staging/deploy/infra/noazurejson/deployment-rbac.yaml
index 75058bfa..8b959b07 100644
--- a/manifest_staging/deploy/infra/noazurejson/deployment-rbac.yaml
+++ b/manifest_staging/deploy/infra/noazurejson/deployment-rbac.yaml
@@ -112,6 +112,7 @@ spec:
     spec:
       serviceAccountName: aad-pod-id-nmi-service-account
       hostNetwork: true
+      dnsPolicy: ClusterFirstWithHostNet
       volumes:
       - hostPath:
           path: /run/xtables.lock
diff --git a/manifest_staging/deploy/infra/noazurejson/deployment.yaml b/manifest_staging/deploy/infra/noazurejson/deployment.yaml
index cb71e8bf..80aa3b0a 100644
--- a/manifest_staging/deploy/infra/noazurejson/deployment.yaml
+++ b/manifest_staging/deploy/infra/noazurejson/deployment.yaml
@@ -48,7 +48,7 @@ spec:
     plural: azurepodidentityexceptions
   scope: Namespaced
 ---
-apiVersion: extensions/v1beta1
+apiVersion: apps/v1
 kind: DaemonSet
 metadata:
   labels:
@@ -68,6 +68,7 @@ spec:
         tier: node
     spec:
       hostNetwork: true
+      dnsPolicy: ClusterFirstWithHostNet
       volumes:
       - hostPath:
           path: /run/xtables.lock
@@ -126,7 +127,7 @@ metadata:
   name: aadpodidentity-admin-secret
   namespace: default
 ---
-apiVersion: extensions/v1beta1
+apiVersion: apps/v1
 kind: Deployment
 metadata:
   labels:
diff --git a/pkg/cloudprovider/cloudprovider.go b/pkg/cloudprovider/cloudprovider.go
index f2cf8621..6d64c95d 100644
--- a/pkg/cloudprovider/cloudprovider.go
+++ b/pkg/cloudprovider/cloudprovider.go
@@ -3,9 +3,11 @@ package cloudprovider
 import (
 	"fmt"
 	"io/ioutil"
+	"net/http"
 	"os"
 	"path"
 	"regexp"
+	"strconv"
 	"strings"
 	"time"
 
@@ -45,6 +47,8 @@ const (
 	linkedAuthorizationFailed retry.RetriableError = "LinkedAuthorizationFailed"
 	// Occurs when the user-assigned identity does not exist.
 	failedIdentityOperation retry.RetriableError = "FailedIdentityOperation"
+	// retryAfterHeaderKey is the retry-after header key in ARM responses.
+	retryAfterHeaderKey = "Retry-After"
 )
 
 // NewCloudProvider returns a azure cloud provider client
@@ -148,6 +152,16 @@ func (c *Client) Init() error {
 		return fmt.Errorf("failed to create VM client, error: %+v", err)
 	}
 
+	// We explicitly removes http.StatusTooManyRequests from autorest.StatusCodesForRetry.
+	// Refer https://github.com/Azure/go-autorest/issues/398.
+	statusCodesForRetry := make([]int, 0)
+	for _, code := range autorest.StatusCodesForRetry {
+		if code != http.StatusTooManyRequests {
+			statusCodesForRetry = append(statusCodesForRetry, code)
+		}
+	}
+	autorest.StatusCodesForRetry = statusCodesForRetry
+
 	return nil
 }
 
@@ -155,11 +169,11 @@ func (c *Client) Init() error {
 func (c *Client) GetUserMSIs(name string, isvmss bool) ([]string, error) {
 	idH, _, err := c.getIdentityResource(name, isvmss)
 	if err != nil {
-		return nil, fmt.Errorf("failed to get identity resource, error %v", err)
+		return nil, fmt.Errorf("failed to get identity resource, error: %v", err)
 	}
 	info := idH.IdentityInfo()
 	if info == nil {
-		return nil, fmt.Errorf("identity info is nil")
+		return []string{}, nil
 	}
 	idList := info.GetUserIdentityList()
 	return idList, nil
@@ -169,7 +183,7 @@ func (c *Client) GetUserMSIs(name string, isvmss bool) ([]string, error) {
 func (c *Client) UpdateUserMSI(addUserAssignedMSIIDs, removeUserAssignedMSIIDs []string, name string, isvmss bool) error {
 	idH, updateFunc, err := c.getIdentityResource(name, isvmss)
 	if err != nil {
-		return fmt.Errorf("failed to get identity resource, error %v", err)
+		return fmt.Errorf("failed to get identity resource, error: %v", err)
 	}
 
 	info := idH.IdentityInfo()
@@ -329,3 +343,24 @@ func extractIdentitiesFromError(err error) []string {
 
 	return extracted
 }
+
+// getRetryAfter gets the retryAfter from http response.
+// The value of Retry-After can be either the number of seconds or a date in RFC1123 format.
+func getRetryAfter(resp *http.Response) time.Duration {
+	if resp == nil {
+		return 0
+	}
+
+	ra := resp.Header.Get(retryAfterHeaderKey)
+	if ra == "" {
+		return 0
+	}
+
+	var dur time.Duration
+	if retryAfter, _ := strconv.Atoi(ra); retryAfter > 0 {
+		dur = time.Duration(retryAfter) * time.Second
+	} else if t, err := time.Parse(time.RFC1123, ra); err == nil {
+		dur = time.Until(t)
+	}
+	return dur
+}
diff --git a/pkg/cloudprovider/vm.go b/pkg/cloudprovider/vm.go
index 7d1e3fd0..58df7d86 100644
--- a/pkg/cloudprovider/vm.go
+++ b/pkg/cloudprovider/vm.go
@@ -22,6 +22,9 @@ import (
 type VMClient struct {
 	client   compute.VirtualMachinesClient
 	reporter *metrics.Reporter
+	// ARM throttling configures.
+	retryAfterReader time.Time
+	retryAfterWriter time.Time
 }
 
 // VMClientInt is the interface used by "cloudprovider" for interacting with Azure vmas
@@ -77,8 +80,16 @@ func (c *VMClient) Get(rgName string, nodeName string) (compute.VirtualMachine,
 		}
 	}()
 
+	// Report errors if the client is throttled.
+	if c.retryAfterReader.After(time.Now()) {
+		return compute.VirtualMachine{}, fmt.Errorf("VMGet client throttled, retry after: %v", c.retryAfterReader)
+	}
+
 	vm, err := c.client.Get(ctx, rgName, nodeName, "")
 	if err != nil {
+		resp := vm.Response.Response
+		// Update RetryAfterReader so that no more requests would be sent until RetryAfter expires.
+		c.retryAfterReader = time.Now().Add(getRetryAfter(resp))
 		return vm, fmt.Errorf("failed to get vm %s in resource group %s, error: %+v", nodeName, rgName, err)
 	}
 	stats.Increment(stats.TotalGetCalls, 1)
@@ -107,7 +118,15 @@ func (c *VMClient) UpdateIdentities(rg, nodeName string, vm compute.VirtualMachi
 		}
 	}()
 
+	// Report errors if the client is throttled.
+	if c.retryAfterWriter.After(time.Now()) {
+		return fmt.Errorf("VMUpdate client throttled, retry after: %v", c.retryAfterWriter)
+	}
+
 	if future, err = c.client.Update(ctx, rg, nodeName, compute.VirtualMachineUpdate{Identity: vm.Identity}); err != nil {
+		resp := future.Response()
+		// Update RetryAfterWriter so that no more requests would be sent until RetryAfter expires.
+		c.retryAfterWriter = time.Now().Add(getRetryAfter(resp))
 		return fmt.Errorf("failed to update identities for %s in %s, error: %+v", nodeName, rg, err)
 	}
 	if err = future.WaitForCompletionRef(ctx, c.client.Client); err != nil {
diff --git a/pkg/cloudprovider/vmss.go b/pkg/cloudprovider/vmss.go
index 3e567027..64fbd28f 100644
--- a/pkg/cloudprovider/vmss.go
+++ b/pkg/cloudprovider/vmss.go
@@ -22,6 +22,9 @@ import (
 type VMSSClient struct {
 	client   compute.VirtualMachineScaleSetsClient
 	reporter *metrics.Reporter
+	// ARM throttling configures.
+	retryAfterReader time.Time
+	retryAfterWriter time.Time
 }
 
 // VMSSClientInt is the interface used by "cloudprovider" for interacting with Azure vmss
@@ -78,7 +81,15 @@ func (c *VMSSClient) UpdateIdentities(rg, vmssName string, vmssIdentities comput
 		}
 	}()
 
+	// Report errors if the client is throttled.
+	if c.retryAfterWriter.After(time.Now()) {
+		return fmt.Errorf("VMSSUpdate client throttled, retry after: %v", c.retryAfterWriter)
+	}
+
 	if future, err = c.client.Update(ctx, rg, vmssName, compute.VirtualMachineScaleSetUpdate{Identity: vmssIdentities.Identity}); err != nil {
+		resp := future.Response()
+		// Update RetryAfterWriter so that no more requests would be sent until RetryAfter expires.
+		c.retryAfterWriter = time.Now().Add(getRetryAfter(resp))
 		return fmt.Errorf("failed to update identities for %s in %s, error: %+v", vmssName, rg, err)
 	}
 	if err = future.WaitForCompletionRef(ctx, c.client.Client); err != nil {
@@ -107,8 +118,17 @@ func (c *VMSSClient) Get(rgName string, vmssName string) (ret compute.VirtualMac
 			klog.Warningf("failed to report metrics, error: %+v", err)
 		}
 	}()
+
+	// Report errors if the client is throttled.
+	if c.retryAfterReader.After(time.Now()) {
+		return compute.VirtualMachineScaleSet{}, fmt.Errorf("VMSSGet client throttled, retry after: %v", c.retryAfterReader)
+	}
+
 	vmss, err := c.client.Get(ctx, rgName, vmssName)
 	if err != nil {
+		resp := vmss.Response.Response
+		// Update RetryAfterReader so that no more requests would be sent until RetryAfter expires.
+		c.retryAfterReader = time.Now().Add(getRetryAfter(resp))
 		return vmss, fmt.Errorf("failed to get vmss %s in resource group %s, error: %+v", vmssName, rgName, err)
 	}
 	stats.Increment(stats.TotalGetCalls, 1)
diff --git a/pkg/mic/mic.go b/pkg/mic/mic.go
index b5229d0a..463e0d76 100644
--- a/pkg/mic/mic.go
+++ b/pkg/mic/mic.go
@@ -77,19 +77,20 @@ type UpdateUserMSIConfig struct {
 // Client has the required pointers to talk to the api server
 // and interact with the CRD related datastructure.
 type Client struct {
-	CRDClient            crd.ClientInt
-	CloudClient          cloudprovider.ClientInt
-	PodClient            pod.ClientInt
-	CloudConfigWatcher   filewatcher.ClientInt
-	EventRecorder        record.EventRecorder
-	EventChannel         chan aadpodid.EventType
-	NodeClient           NodeGetter
-	IsNamespaced         bool
-	SyncLoopStarted      bool
-	syncRetryInterval    time.Duration
-	enableScaleFeatures  bool
-	createDeleteBatch    int64
-	ImmutableUserMSIsMap map[string]bool
+	CRDClient                           crd.ClientInt
+	CloudClient                         cloudprovider.ClientInt
+	PodClient                           pod.ClientInt
+	CloudConfigWatcher                  filewatcher.ClientInt
+	EventRecorder                       record.EventRecorder
+	EventChannel                        chan aadpodid.EventType
+	NodeClient                          NodeGetter
+	IsNamespaced                        bool
+	SyncLoopStarted                     bool
+	syncRetryInterval                   time.Duration
+	enableScaleFeatures                 bool
+	createDeleteBatch                   int64
+	ImmutableUserMSIsMap                map[string]bool
+	identityAssignmentReconcileInterval time.Duration
 
 	syncing int32 // protect against conucrrent sync's
 
@@ -103,17 +104,18 @@ type Client struct {
 
 // Config - MIC Config
 type Config struct {
-	CloudCfgPath          string
-	RestConfig            *rest.Config
-	IsNamespaced          bool
-	SyncRetryInterval     time.Duration
-	LeaderElectionCfg     *LeaderElectionConfig
-	EnableScaleFeatures   bool
-	CreateDeleteBatch     int64
-	ImmutableUserMSIsList []string
-	CMcfg                 *CMConfig
-	TypeUpgradeCfg        *TypeUpgradeConfig
-	UpdateUserMSICfg      *UpdateUserMSIConfig
+	CloudCfgPath                        string
+	RestConfig                          *rest.Config
+	IsNamespaced                        bool
+	SyncRetryInterval                   time.Duration
+	LeaderElectionCfg                   *LeaderElectionConfig
+	EnableScaleFeatures                 bool
+	CreateDeleteBatch                   int64
+	ImmutableUserMSIsList               []string
+	CMcfg                               *CMConfig
+	TypeUpgradeCfg                      *TypeUpgradeConfig
+	UpdateUserMSICfg                    *UpdateUserMSIConfig
+	IdentityAssignmentReconcileInterval time.Duration
 }
 
 // ClientInt is an abstraction used to perform an MIC sync cycle.
@@ -200,21 +202,22 @@ func NewMICClient(cfg *Config) (*Client, error) {
 	}
 
 	c := &Client{
-		CRDClient:            crdClient,
-		CloudClient:          cloudClient,
-		PodClient:            podClient,
-		CloudConfigWatcher:   cloudConfigWatcher,
-		EventRecorder:        recorder,
-		EventChannel:         eventCh,
-		NodeClient:           &NodeClient{informer.Core().V1().Nodes()},
-		IsNamespaced:         cfg.IsNamespaced,
-		syncRetryInterval:    cfg.SyncRetryInterval,
-		enableScaleFeatures:  cfg.EnableScaleFeatures,
-		createDeleteBatch:    cfg.CreateDeleteBatch,
-		ImmutableUserMSIsMap: immutableUserMSIsMap,
-		TypeUpgradeCfg:       cfg.TypeUpgradeCfg,
-		CMCfg:                cfg.CMcfg,
-		CMClient:             cmClient,
+		CRDClient:                           crdClient,
+		CloudClient:                         cloudClient,
+		PodClient:                           podClient,
+		CloudConfigWatcher:                  cloudConfigWatcher,
+		EventRecorder:                       recorder,
+		EventChannel:                        eventCh,
+		NodeClient:                          &NodeClient{informer.Core().V1().Nodes()},
+		IsNamespaced:                        cfg.IsNamespaced,
+		syncRetryInterval:                   cfg.SyncRetryInterval,
+		enableScaleFeatures:                 cfg.EnableScaleFeatures,
+		createDeleteBatch:                   cfg.CreateDeleteBatch,
+		ImmutableUserMSIsMap:                immutableUserMSIsMap,
+		TypeUpgradeCfg:                      cfg.TypeUpgradeCfg,
+		CMCfg:                               cfg.CMcfg,
+		CMClient:                            cmClient,
+		identityAssignmentReconcileInterval: cfg.IdentityAssignmentReconcileInterval,
 	}
 
 	leaderElector, err := c.NewLeaderElector(clientSet, recorder, cfg.LeaderElectionCfg)
@@ -388,6 +391,9 @@ func (c *Client) Sync(exit <-chan struct{}) {
 	ticker := time.NewTicker(c.syncRetryInterval)
 	defer ticker.Stop()
 
+	identityAssignmentReconcileTicker := time.NewTicker(c.identityAssignmentReconcileInterval)
+	defer identityAssignmentReconcileTicker.Stop()
+
 	klog.Info("sync thread started.")
 	c.SyncLoopStarted = true
 	var event aadpodid.EventType
@@ -402,6 +408,10 @@ func (c *Client) Sync(exit <-chan struct{}) {
 			klog.V(6).Infof("received event: %v", event)
 		case <-ticker.C:
 			klog.V(6).Infof("running periodic sync loop")
+		case <-identityAssignmentReconcileTicker.C:
+			klog.V(6).Infof("reconciling identity assignment on Azure")
+			c.reconcileIdentityAssignment()
+			continue
 		}
 		totalSyncCycles++
 		stats.Init()
@@ -1054,7 +1064,7 @@ func (c *Client) updateUserMSI(newAssignedIDs map[string]aadpodid.AzureAssignedI
 				assignedID.Status.Status = aadpodid.AssignedIDCreated
 				err := c.updateAssignedIdentity(&assignedID)
 				if err != nil {
-					message := fmt.Sprintf("failed to update AzureAssignedIdentity %s/%s for pod %s/%s, error %+v", assignedID.Namespace, assignedID.Name, assignedID.Spec.Pod, assignedID.Spec.PodNamespace, err)
+					message := fmt.Sprintf("failed to update AzureAssignedIdentity %s/%s for pod %s/%s, error: %+v", assignedID.Namespace, assignedID.Name, assignedID.Spec.Pod, assignedID.Spec.PodNamespace, err)
 					c.EventRecorder.Event(binding, corev1.EventTypeWarning, "binding apply error", message)
 					klog.Error(message)
 				}
@@ -1236,7 +1246,7 @@ func (c *Client) cleanUpAllAssignedIdentitiesOnNode(node string, nodeTrackList t
 
 		err := c.removeAssignedIdentity(&deleteID)
 		if err != nil {
-			message := fmt.Sprintf("failed to remove AzureIdentityBinding %s/%s from node %s for pod %s/%s, error %v", binding.Namespace, binding.Name, deleteID.Spec.NodeName, deleteID.Spec.PodNamespace, deleteID.Spec.Pod, err)
+			message := fmt.Sprintf("failed to remove AzureIdentityBinding %s/%s from node %s for pod %s/%s, error: %v", binding.Namespace, binding.Name, deleteID.Spec.NodeName, deleteID.Spec.PodNamespace, deleteID.Spec.Pod, err)
 			c.EventRecorder.Event(binding, corev1.EventTypeWarning, "binding remove error", message)
 			klog.Error(message)
 			continue
@@ -1255,7 +1265,7 @@ func (c *Client) consolidateVMSSNodes(nodeMap map[string]trackUserAssignedMSIIds
 	for nodeName, nodeTrackList := range nodeMap {
 		node, err := c.NodeClient.Get(nodeName)
 		if err != nil && !strings.Contains(err.Error(), "not found") {
-			klog.Errorf("failed to get node %s, error %+v", nodeName, err)
+			klog.Errorf("failed to get node %s, error: %+v", nodeName, err)
 			continue
 		}
 		if err != nil && strings.Contains(err.Error(), "not found") {
@@ -1317,3 +1327,119 @@ func (c *Client) checkIfIdentityImmutable(id string) bool {
 	}
 	return false
 }
+
+// generateIdentityAssignmentState generates the current and desired state of each node's identity
+// assignments based on an existing list of AzureAssignedIdentity as the source of truth.
+func (c *Client) generateIdentityAssignmentState() (currentState map[string]map[string]bool, desiredState map[string]map[string]bool, isVMSSMap map[string]bool, err error) {
+	type nodeMetadata struct {
+		nodeName string
+		isVMSS   bool
+	}
+
+	assignedIDs, err := c.CRDClient.ListAssignedIDs()
+	if err != nil {
+		return nil, nil, nil, fmt.Errorf("failed to list AzureAssignedIdentities, error: %+v", err)
+	}
+
+	nodeMetadataCache := make(map[string]nodeMetadata)
+	isVMSSMap = make(map[string]bool)
+	currentState = make(map[string]map[string]bool)
+	desiredState = make(map[string]map[string]bool)
+	for _, assignedID := range *assignedIDs {
+		if _, ok := nodeMetadataCache[assignedID.Spec.NodeName]; !ok {
+			node, err := c.NodeClient.Get(assignedID.Spec.NodeName)
+			if err != nil {
+				return nil, nil, nil, fmt.Errorf("failed to get node %s, error: %+v", assignedID.Spec.NodeName, err)
+			}
+
+			nodeName, isVMSS, err := isVMSS(node)
+			if err != nil {
+				return nil, nil, nil, fmt.Errorf("failed to check if node %s is VMSS, error: %+v", assignedID.Spec.NodeName, err)
+			} else if isVMSS {
+				nodeName = getVMSSName(nodeName)
+			} else {
+				// VM node name does not require conversion
+				nodeName = assignedID.Spec.NodeName
+			}
+
+			// cache node metadata to avoid excessive GET calls
+			nodeMetadataCache[assignedID.Spec.NodeName] = nodeMetadata{
+				nodeName: nodeName,
+				isVMSS:   isVMSS,
+			}
+		}
+
+		nodeName := nodeMetadataCache[assignedID.Spec.NodeName].nodeName
+		isVMSS := nodeMetadataCache[assignedID.Spec.NodeName].isVMSS
+		isVMSSMap[nodeName] = isVMSS
+
+		// only consider AzureAssignedIdentities in ASSIGNED state
+		// do not consider AzureAssignedIdentities in CREATED state because they are either:
+		// 1. in the process of assigning the identities on Azure or
+		// 2. encountering errors when assigning identities on Azure
+		if assignedID.Status.Status == aadpodid.AssignedIDAssigned && assignedID.Spec.AzureIdentityRef.Spec.Type == aadpodid.UserAssignedMSI {
+			if _, ok := desiredState[nodeName]; !ok {
+				desiredState[nodeName] = make(map[string]bool)
+			}
+			desiredState[nodeName][assignedID.Spec.AzureIdentityRef.Spec.ResourceID] = true
+		}
+
+		if _, ok := currentState[nodeName]; !ok {
+			currentState[nodeName] = make(map[string]bool)
+			idList, err := c.getUserMSIListForNode(nodeName, isVMSS)
+			if err != nil {
+				return nil, nil, nil, fmt.Errorf("failed to get a list of user-assigned identites from node %s, error: %+v", nodeName, err)
+			}
+
+			for _, identityResourceID := range idList {
+				currentState[nodeName][identityResourceID] = true
+			}
+		}
+	}
+
+	return currentState, desiredState, isVMSSMap, nil
+}
+
+// generateIdentityAssignmentDiff perform a diff between current
+// and desired state of identity assignment on Azure and returns
+// a map with the node name as the key and a list of user-assigned
+// identities we should assign to the node as the value.
+func generateIdentityAssignmentDiff(currentState map[string]map[string]bool, desiredState map[string]map[string]bool) map[string][]string {
+	diff := make(map[string][]string)
+	for nodeName, identityResourceIDs := range desiredState {
+		var identitiesToAssign []string
+		for identityResourceID := range identityResourceIDs {
+			if _, ok := currentState[nodeName]; ok && currentState[nodeName][identityResourceID] {
+				continue
+			}
+			identitiesToAssign = append(identitiesToAssign, identityResourceID)
+		}
+
+		if len(identitiesToAssign) > 0 {
+			diff[nodeName] = identitiesToAssign
+		}
+	}
+
+	return diff
+}
+
+// reconcileIdentityAssignment uses the existing list of AzureAssignedIdentities
+// as the single source of truth and reconciles identity assignment on Azure.
+func (c *Client) reconcileIdentityAssignment() {
+	currentState, desiredState, isVMSSMap, err := c.generateIdentityAssignmentState()
+	if err != nil {
+		klog.Errorf("failed to generate identity assignment state, error: %+v", err)
+		return
+	}
+
+	klog.V(6).Infof("current state of identity assignment on Azure: %+v", currentState)
+	klog.V(6).Infof("desired state of identity assignment on Azure: %+v", desiredState)
+
+	diff := generateIdentityAssignmentDiff(currentState, desiredState)
+	for nodeNameOnAzure, identitiesToAssign := range diff {
+		klog.Infof("reconciling identity assignment for %v on node %s", identitiesToAssign, nodeNameOnAzure)
+		if err := c.CloudClient.UpdateUserMSI(identitiesToAssign, nil, nodeNameOnAzure, isVMSSMap[nodeNameOnAzure]); err != nil {
+			klog.Errorf("failed to update user-assigned identities on node %s, error: %+v", nodeNameOnAzure, err)
+		}
+	}
+}
diff --git a/pkg/nmi/server/server.go b/pkg/nmi/server/server.go
index 27173ca8..6feb00b4 100644
--- a/pkg/nmi/server/server.go
+++ b/pkg/nmi/server/server.go
@@ -340,7 +340,7 @@ func (s *Server) msiHandler(w http.ResponseWriter, r *http.Request) (ns string)
 
 	podns, podname, rsName, selectors, err := s.KubeClient.GetPodInfo(podIP)
 	if err != nil {
-		klog.Errorf("failed to get pod info from pod IP: %s, error %+v", podIP, err)
+		klog.Errorf("failed to get pod info from pod IP: %s, error: %+v", podIP, err)
 		http.Error(w, err.Error(), http.StatusInternalServerError)
 		return
 	}


Test patch:
diff --git a/pkg/cloudprovider/cloudprovider_test.go b/pkg/cloudprovider/cloudprovider_test.go
index 4326939d..461aca60 100644
--- a/pkg/cloudprovider/cloudprovider_test.go
+++ b/pkg/cloudprovider/cloudprovider_test.go
@@ -2,10 +2,12 @@ package cloudprovider
 
 import (
 	"errors"
+	"net/http"
 	"reflect"
 	"sort"
 	"strings"
 	"testing"
+	"time"
 
 	"github.com/Azure/aad-pod-identity/pkg/config"
 	"github.com/Azure/aad-pod-identity/pkg/retry"
@@ -535,3 +537,45 @@ func isSliceEqual(s1, s2 []string) bool {
 	}
 	return true
 }
+
+func TestGetRetryAfter(t *testing.T) {
+	cases := []struct {
+		desc               string
+		resp               *http.Response
+		expectedRetryAfter time.Duration
+	}{
+		{
+			desc:               "response is nil",
+			expectedRetryAfter: 0,
+		},
+		{
+			desc:               "no Retry-After header in the response",
+			resp:               &http.Response{},
+			expectedRetryAfter: 0,
+		},
+		{
+			desc:               "Retry-After in response is unknown format",
+			resp:               &http.Response{Header: http.Header{"Retry-After": []string{time.Now().Add(180 * time.Second).Format(time.RFC822)}}},
+			expectedRetryAfter: 0,
+		},
+		{
+			desc:               "Retry-After in response is 180",
+			resp:               &http.Response{Header: http.Header{"Retry-After": []string{"180"}}},
+			expectedRetryAfter: 3 * time.Minute,
+		},
+		{
+			desc:               "Retry-After in response is in RFC1123 format",
+			resp:               &http.Response{Header: http.Header{"Retry-After": []string{time.Now().Add(180 * time.Second).Format(time.RFC1123)}}},
+			expectedRetryAfter: 3 * time.Minute,
+		},
+	}
+
+	for _, tc := range cases {
+		t.Run(tc.desc, func(t *testing.T) {
+			retryAfterDuration := getRetryAfter(tc.resp)
+			if tc.expectedRetryAfter != retryAfterDuration.Round(time.Minute) {
+				t.Fatalf("expected retry after to be: %v, got: %v", tc.expectedRetryAfter, retryAfterDuration)
+			}
+		})
+	}
+}
diff --git a/pkg/mic/mic_test.go b/pkg/mic/mic_test.go
index 9e10d842..31e18089 100644
--- a/pkg/mic/mic_test.go
+++ b/pkg/mic/mic_test.go
@@ -18,6 +18,7 @@ import (
 	"github.com/Azure/aad-pod-identity/pkg/retry"
 
 	"github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2019-12-01/compute"
+	"github.com/stretchr/testify/assert"
 	api "k8s.io/api/core/v1"
 	corev1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
@@ -264,6 +265,23 @@ func (c *TestVMSSClient) CompareMSI(nodeName string, expectedUserIDs []string) b
 	return true
 }
 
+func (c *TestCloudClient) GetUserMSIs(name string, isvmss bool) ([]string, error) {
+	var ret []string
+	if isvmss {
+		vmss, _ := c.testVMSSClient.Get("", name)
+		for id := range vmss.Identity.UserAssignedIdentities {
+			ret = append(ret, id)
+		}
+	} else {
+		vm, _ := c.testVMClient.Get("", name)
+		for id := range vm.Identity.UserAssignedIdentities {
+			ret = append(ret, id)
+		}
+	}
+
+	return ret, nil
+}
+
 func (c *TestCloudClient) ListMSI() (ret map[string]*[]string) {
 	if c.Client.Config.VMType == "vmss" {
 		return c.testVMSSClient.ListMSI()
@@ -726,17 +744,18 @@ func NewMICTestClient(eventCh chan internalaadpodid.EventType,
 	reporter, _ := metrics.NewReporter()
 
 	realMICClient := &Client{
-		CloudClient:          cpClient,
-		CRDClient:            crdClient,
-		EventRecorder:        eventRecorder,
-		PodClient:            podClient,
-		EventChannel:         eventCh,
-		NodeClient:           nodeClient,
-		syncRetryInterval:    120 * time.Second,
-		IsNamespaced:         isNamespaced,
-		createDeleteBatch:    createDeleteBatch,
-		ImmutableUserMSIsMap: immutableUserMSIs,
-		Reporter:             reporter,
+		CloudClient:                         cpClient,
+		CRDClient:                           crdClient,
+		EventRecorder:                       eventRecorder,
+		PodClient:                           podClient,
+		EventChannel:                        eventCh,
+		NodeClient:                          nodeClient,
+		syncRetryInterval:                   120 * time.Second,
+		IsNamespaced:                        isNamespaced,
+		createDeleteBatch:                   createDeleteBatch,
+		ImmutableUserMSIsMap:                immutableUserMSIs,
+		Reporter:                            reporter,
+		identityAssignmentReconcileInterval: 3 * time.Minute,
 	}
 
 	return &TestMICClient{
@@ -1761,6 +1780,231 @@ func TestCloudProviderRetryLoop(t *testing.T) {
 	}
 }
 
+func TestGenerateIdentityAssignmentStateVM(t *testing.T) {
+	eventCh := make(chan internalaadpodid.EventType)
+	cloudClient := NewTestCloudClient(config.AzureConfig{VMType: "vmss"})
+	crdClient := NewTestCrdClient(nil)
+	podClient := NewTestPodClient()
+	nodeClient := NewTestNodeClient()
+	var evtRecorder TestEventRecorder
+	evtRecorder.lastEvent = new(LastEvent)
+	evtRecorder.eventChannel = make(chan bool)
+
+	micClient := NewMICTestClient(eventCh, cloudClient, crdClient, podClient, nodeClient, &evtRecorder, false, 4, nil)
+	currentState, desiredState, isVMSSMap, err := micClient.generateIdentityAssignmentState()
+	assert.Empty(t, currentState)
+	assert.Empty(t, desiredState)
+	assert.Empty(t, isVMSSMap)
+	assert.NoError(t, err)
+
+	nodeClient.AddNode("node-0", func(n *corev1.Node) {
+		n.Spec.ProviderID = "azure:///subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Compute/virtualMachines/node-0"
+	})
+
+	_ = crdClient.CreateAssignedIdentity(&internalaadpodid.AzureAssignedIdentity{
+		Spec: internalaadpodid.AzureAssignedIdentitySpec{
+			NodeName: "node-0",
+			AzureIdentityRef: &internalaadpodid.AzureIdentity{
+				Spec: internalaadpodid.AzureIdentitySpec{
+					ResourceID: testResourceID,
+				},
+			},
+		},
+		Status: internalaadpodid.AzureAssignedIdentityStatus{
+			Status: aadpodid.AssignedIDAssigned,
+		},
+	})
+
+	// the user-assigned identity isn't assigned to a VMSS instance on Azure
+	currentState, desiredState, isVMSSMap, err = micClient.generateIdentityAssignmentState()
+	assert.Equal(t, currentState, map[string]map[string]bool{
+		"node-0": {},
+	})
+	assert.Equal(t, desiredState, map[string]map[string]bool{
+		"node-0": {
+			testResourceID: true,
+		},
+	})
+	assert.Equal(t, isVMSSMap, map[string]bool{
+		"node-0": false,
+	})
+	assert.NoError(t, err)
+
+	// the user-assigned identity is now assigned to a VM instance on Azure
+	vm, _ := cloudClient.testVMClient.Get("", "node-0")
+	vm.Identity = &compute.VirtualMachineIdentity{
+		UserAssignedIdentities: map[string]*compute.VirtualMachineIdentityUserAssignedIdentitiesValue{
+			testResourceID: {},
+		},
+	}
+	_ = cloudClient.testVMClient.UpdateIdentities("", "node-0", vm)
+
+	currentState, desiredState, isVMSSMap, err = micClient.generateIdentityAssignmentState()
+	assert.Equal(t, currentState, map[string]map[string]bool{
+		"node-0": {
+			testResourceID: true,
+		},
+	})
+	assert.Equal(t, desiredState, map[string]map[string]bool{
+		"node-0": {
+			testResourceID: true,
+		},
+	})
+	assert.Equal(t, isVMSSMap, map[string]bool{
+		"node-0": false,
+	})
+	assert.NoError(t, err)
+}
+
+func TestGenerateIdentityAssignmentStateVMSS(t *testing.T) {
+	eventCh := make(chan internalaadpodid.EventType)
+	cloudClient := NewTestCloudClient(config.AzureConfig{VMType: "vmss"})
+	crdClient := NewTestCrdClient(nil)
+	podClient := NewTestPodClient()
+	nodeClient := NewTestNodeClient()
+	var evtRecorder TestEventRecorder
+	evtRecorder.lastEvent = new(LastEvent)
+	evtRecorder.eventChannel = make(chan bool)
+
+	micClient := NewMICTestClient(eventCh, cloudClient, crdClient, podClient, nodeClient, &evtRecorder, false, 4, nil)
+	currentState, desiredState, isVMSSMap, err := micClient.generateIdentityAssignmentState()
+	assert.Empty(t, currentState)
+	assert.Empty(t, desiredState)
+	assert.Empty(t, isVMSSMap)
+	assert.NoError(t, err)
+
+	nodeClient.AddNode("node-0", func(n *corev1.Node) {
+		n.Spec.ProviderID = "azure:///subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Compute/virtualMachineScaleSets/node-0/virtualMachines/0"
+	})
+
+	_ = crdClient.CreateAssignedIdentity(&internalaadpodid.AzureAssignedIdentity{
+		Spec: internalaadpodid.AzureAssignedIdentitySpec{
+			NodeName: "node-0",
+			AzureIdentityRef: &internalaadpodid.AzureIdentity{
+				Spec: internalaadpodid.AzureIdentitySpec{
+					Type:       internalaadpodid.UserAssignedMSI,
+					ResourceID: testResourceID,
+				},
+			},
+		},
+		Status: internalaadpodid.AzureAssignedIdentityStatus{
+			Status: aadpodid.AssignedIDAssigned,
+		},
+	})
+
+	// the user-assigned identity isn't assigned to a VMSS instance on Azure
+	currentState, desiredState, isVMSSMap, err = micClient.generateIdentityAssignmentState()
+	assert.Equal(t, currentState, map[string]map[string]bool{
+		"node-0": {},
+	})
+	assert.Equal(t, desiredState, map[string]map[string]bool{
+		"node-0": {
+			testResourceID: true,
+		},
+	})
+	assert.Equal(t, isVMSSMap, map[string]bool{
+		"node-0": true,
+	})
+	assert.NoError(t, err)
+
+	// the user-assigned identity is now assigned to a VMSS instance on Azure
+	vmss, _ := cloudClient.testVMSSClient.Get("", "node-0")
+	vmss.Identity = &compute.VirtualMachineScaleSetIdentity{
+		UserAssignedIdentities: map[string]*compute.VirtualMachineScaleSetIdentityUserAssignedIdentitiesValue{
+			testResourceID: {},
+		},
+	}
+	_ = cloudClient.testVMSSClient.UpdateIdentities("", "node-0", vmss)
+
+	currentState, desiredState, isVMSSMap, err = micClient.generateIdentityAssignmentState()
+	assert.Equal(t, currentState, map[string]map[string]bool{
+		"node-0": {
+			testResourceID: true,
+		},
+	})
+	assert.Equal(t, desiredState, map[string]map[string]bool{
+		"node-0": {
+			testResourceID: true,
+		},
+	})
+	assert.Equal(t, isVMSSMap, map[string]bool{
+		"node-0": true,
+	})
+	assert.NoError(t, err)
+}
+
+func TestGenerateIdentityAssignmentDiff(t *testing.T) {
+	testCases := []struct {
+		currentState map[string]map[string]bool
+		desiredState map[string]map[string]bool
+		expectedDiff map[string][]string
+	}{
+		{
+			currentState: map[string]map[string]bool{
+				"node-0": {
+					"id-0": true,
+				},
+			},
+			desiredState: map[string]map[string]bool{
+				"node-0": {
+					"id-0": true,
+				},
+			},
+			expectedDiff: map[string][]string{},
+		},
+		{
+			currentState: map[string]map[string]bool{
+				"node-1": {
+					"id-1": true,
+				},
+			},
+			desiredState: map[string]map[string]bool{
+				"node-0": {
+					"id-0": true,
+				},
+				"node-1": {
+					"id-0": true,
+					"id-1": true,
+				},
+			},
+			expectedDiff: map[string][]string{
+				"node-0": {
+					"id-0",
+				},
+				"node-1": {
+					"id-0",
+				},
+			},
+		},
+		{
+			currentState: nil,
+			desiredState: map[string]map[string]bool{
+				"node-0": {
+					"id-0": true,
+				},
+			},
+			expectedDiff: map[string][]string{
+				"node-0": {
+					"id-0",
+				},
+			},
+		},
+		{
+			currentState: map[string]map[string]bool{
+				"node-0": {
+					"id-0": true,
+				},
+			},
+			desiredState: nil,
+			expectedDiff: map[string][]string{},
+		},
+	}
+
+	for _, tc := range testCases {
+		assert.Equal(t, tc.expectedDiff, generateIdentityAssignmentDiff(tc.currentState, tc.desiredState))
+	}
+}
+
 func findAssignedIDByName(name string, assignedIDs *[]internalaadpodid.AzureAssignedIdentity) *internalaadpodid.AzureAssignedIdentity {
 	for _, assignedID := range *assignedIDs {
 		if assignedID.Name == name {
diff --git a/test/e2e/e2e_suite_test.go b/test/e2e/e2e_suite_test.go
index c40653fa..56992604 100644
--- a/test/e2e/e2e_suite_test.go
+++ b/test/e2e/e2e_suite_test.go
@@ -85,22 +85,24 @@ var _ = AfterSuite(func() {
 		Namespace: iptablesNamespace,
 	})
 
-	dumpLogs()
+	defer func() {
+		if !config.IsSoakTest {
+			By("Uninstalling AAD Pod Identity via Helm")
+			helm.Uninstall()
+
+			iptables.WaitForRules(iptables.WaitForRulesInput{
+				Creator:         kubeClient,
+				Getter:          kubeClient,
+				Lister:          kubeClient,
+				Namespace:       iptablesNamespace.Name,
+				KubeconfigPath:  clusterProxy.GetKubeconfigPath(),
+				CreateDaemonSet: false,
+				ShouldExist:     false,
+			})
+		}
+	}()
 
-	if !config.IsSoakTest {
-		By("Uninstalling AAD Pod Identity via Helm")
-		helm.Uninstall()
-
-		iptables.WaitForRules(iptables.WaitForRulesInput{
-			Creator:         kubeClient,
-			Getter:          kubeClient,
-			Lister:          kubeClient,
-			Namespace:       iptablesNamespace.Name,
-			KubeconfigPath:  clusterProxy.GetKubeconfigPath(),
-			CreateDaemonSet: false,
-			ShouldExist:     false,
-		})
-	}
+	dumpLogs()
 })
 
 func initScheme() *runtime.Scheme {
@@ -121,7 +123,8 @@ func dumpLogs() {
 
 		for _, pod := range podList.Items {
 			By(fmt.Sprintf("Dumping logs for %s scheduled to %s", pod.Name, pod.Spec.NodeName))
-			_, _ = exec.KubectlLogs(kubeconfigPath, pod.Name, framework.NamespaceKubeSystem)
+			_, err := exec.KubectlLogs(kubeconfigPath, pod.Name, framework.NamespaceKubeSystem)
+			Expect(err).To(BeNil())
 		}
 	}
 }
diff --git a/test/e2e/framework/helm/helm_helpers.go b/test/e2e/framework/helm/helm_helpers.go
index a038f62e..c92665aa 100644
--- a/test/e2e/framework/helm/helm_helpers.go
+++ b/test/e2e/framework/helm/helm_helpers.go
@@ -41,12 +41,14 @@ func Install(input InstallInput) {
 		chartName,
 		"manifest_staging/charts/aad-pod-identity",
 		"--wait",
+		"--debug",
 		fmt.Sprintf("--namespace=%s", framework.NamespaceKubeSystem),
 		"--debug",
 	})
 	args = append(args, generateValueArgs(input.Config)...)
 
-	helm(args)
+	err = helm(args)
+	Expect(err).To(BeNil())
 }
 
 // Uninstall uninstalls aad-pod-identity via Helm 3.
@@ -58,7 +60,8 @@ func Uninstall() {
 		"--debug",
 	}
 
-	helm(args)
+	// ignore error to allow cleanup completion
+	_ = helm(args)
 }
 
 // UpgradeInput is the input for Upgrade.
@@ -83,12 +86,14 @@ func Upgrade(input UpgradeInput) {
 		chartName,
 		"manifest_staging/charts/aad-pod-identity",
 		"--wait",
+		"--debug",
 		fmt.Sprintf("--namespace=%s", framework.NamespaceKubeSystem),
 		"--debug",
 	})
 	args = append(args, generateValueArgs(input.Config)...)
 
-	helm(args)
+	err = helm(args)
+	Expect(err).To(BeNil())
 }
 
 func generateValueArgs(config *framework.Config) []string {
@@ -113,12 +118,12 @@ func generateValueArgs(config *framework.Config) []string {
 	return args
 }
 
-func helm(args []string) {
+func helm(args []string) error {
 	By(fmt.Sprintf("helm %s", strings.Join(args, " ")))
 
 	cmd := exec.Command("helm", args...)
 	stdoutStderr, err := cmd.CombinedOutput()
 	fmt.Printf("%s", stdoutStderr)
 
-	Expect(err).To(BeNil())
+	return err
 }
diff --git a/test/e2e/framework/iptables/iptables_helpers.go b/test/e2e/framework/iptables/iptables_helpers.go
index ff72932c..a86f0d3c 100644
--- a/test/e2e/framework/iptables/iptables_helpers.go
+++ b/test/e2e/framework/iptables/iptables_helpers.go
@@ -63,6 +63,7 @@ func WaitForRules(input WaitForRulesInput) {
 					},
 					Spec: corev1.PodSpec{
 						HostNetwork:                   true,
+						DNSPolicy:                     corev1.DNSClusterFirstWithHostNet,
 						TerminationGracePeriodSeconds: to.Int64Ptr(int64(0)),
 						Containers: []corev1.Container{
 							{
diff --git a/test/e2e/framework/timeout.go b/test/e2e/framework/timeout.go
index 5cffe625..326bec3c 100644
--- a/test/e2e/framework/timeout.go
+++ b/test/e2e/framework/timeout.go
@@ -12,10 +12,10 @@ const (
 	CreatePolling = 1 * time.Second
 
 	// DeleteTimeout represents the duration it waits until a delete operation times out.
-	DeleteTimeout = 10 * time.Second
+	DeleteTimeout = 1 * time.Minute
 
 	// DeletePolling represents the polling interval for a delete operation.
-	DeletePolling = 1 * time.Second
+	DeletePolling = 5 * time.Second
 
 	// ListTimeout represents the duration it waits until a list operation times out.
 	ListTimeout = 10 * time.Second
diff --git a/test/e2e/node_test.go b/test/e2e/node_test.go
index b947e272..914e1294 100644
--- a/test/e2e/node_test.go
+++ b/test/e2e/node_test.go
@@ -7,6 +7,7 @@ import (
 	"strings"
 
 	aadpodv1 "github.com/Azure/aad-pod-identity/pkg/apis/aadpodidentity/v1"
+	"github.com/Azure/aad-pod-identity/test/e2e/framework"
 	"github.com/Azure/aad-pod-identity/test/e2e/framework/azure"
 	"github.com/Azure/aad-pod-identity/test/e2e/framework/azureassignedidentity"
 	"github.com/Azure/aad-pod-identity/test/e2e/framework/azureidentity"
@@ -353,6 +354,56 @@ var _ = Describe("When managing identities from the underlying nodes", func() {
 		userAssignedIdentities := azureClient.ListUserAssignedIdentities(node.Spec.ProviderID)
 		Expect(isUserAssignedIdentityExist(userAssignedIdentities, immutableIdentity)).To(BeTrue())
 	})
+
+	It("should reconcile identity assignment on Azure if the user-assigned identity is manually unassigned from the underlying node", func() {
+		// Schedule identity-validator to this node
+		node := nodes.Items[0]
+
+		identityValidator := identityvalidator.Create(identityvalidator.CreateInput{
+			Creator:         kubeClient,
+			Config:          config,
+			Namespace:       ns.Name,
+			IdentityBinding: azureIdentityBinding.Spec.Selector,
+			NodeName:        node.Name,
+		})
+
+		azureassignedidentity.Wait(azureassignedidentity.WaitInput{
+			Getter:            kubeClient,
+			PodName:           identityValidator.Name,
+			Namespace:         ns.Name,
+			AzureIdentityName: azureIdentity.Name,
+			StateToWaitFor:    aadpodv1.AssignedIDAssigned,
+		})
+
+		identityvalidator.Validate(identityvalidator.ValidateInput{
+			Getter:             kubeClient,
+			Config:             config,
+			KubeconfigPath:     kubeconfigPath,
+			PodName:            identityValidator.Name,
+			Namespace:          ns.Name,
+			IdentityClientID:   azureIdentity.Spec.ClientID,
+			IdentityResourceID: azureIdentity.Spec.ResourceID,
+		})
+
+		err := azureClient.UnassignUserAssignedIdentity(node.Spec.ProviderID, keyvaultIdentity)
+		Expect(err).To(BeNil())
+
+		By("Waiting for identity assignment to be reconciled")
+		Eventually(func() (bool, error) {
+			userAssignedIdentities := azureClient.ListUserAssignedIdentities(node.Spec.ProviderID)
+			return isUserAssignedIdentityExist(userAssignedIdentities, keyvaultIdentity), nil
+		}, framework.WaitTimeout, framework.WaitPolling).Should(BeTrue())
+
+		identityvalidator.Validate(identityvalidator.ValidateInput{
+			Getter:             kubeClient,
+			Config:             config,
+			KubeconfigPath:     kubeconfigPath,
+			PodName:            identityValidator.Name,
+			Namespace:          ns.Name,
+			IdentityClientID:   azureIdentity.Spec.ClientID,
+			IdentityResourceID: azureIdentity.Spec.ResourceID,
+		})
+	})
 })
 
 func getVMSSNodes(nodes *corev1.NodeList) []corev1.Node {