Repository: Azure/aad-pod-identity
Pull request #431 (2019-10-23 09:09:26)

Adding support for whitelisting of user-defined managed identities
<!-- Thank you for helping AAD Pod Identity with a pull request! -->

**Reason for Change**:
In some cases a user-defined managed identity could be removed from the AAD


**Issue Fixed**:
This PR is adding whitelist of user defined managed identities what will never be removed from AAD

**Notes for Reviewers**:
The change is as following:
- mic main  accepts new cmd parameter *whiteListedIdentitiesParam* - comma separated list of identities that shouldn't be deleted from AAD
- at *updateUserMSI* function we compare *removeUserAssignedMSIIDs* and *whiteListedIdentities* and only missing values are removed from AAD
- helm chart and Readme are changed accordingly

This feature is extremely important for migrating existing production clusters to aad-pod-identity.
In particular, all our existing deployments will use user-defined identity as one per team.
Reuse of the same identity to multiple pods caused the identity to be deleted from AAD when it still in use. 
This change will solve the issue.

Best regards,
Dmitry


Patch:
diff --git a/charts/aad-pod-identity/README.md b/charts/aad-pod-identity/README.md
index acb68ce7..e5f89419 100644
--- a/charts/aad-pod-identity/README.md
+++ b/charts/aad-pod-identity/README.md
@@ -162,6 +162,7 @@ The following tables list the configurable parameters of the aad-pod-identity ch
 | `mic.leaderElection.duration`            | Override leader election duration                                                                                                                                                                                | If not provided, default value is `15s`                  |
 | `mic.probePort`                          | Override http liveliness probe port                                                                                                                                                                              | If not provided, default port is `8080`                  |
 | `mic.syncRetryDuration`                  | Override interval in seconds at which sync loop should periodically check for errors and reconcile                                                                                                               | If not provided, default value is `3600s`                |
+| `mic.immutableUserMSIs`                  | List of  user-defined identities that shouldn't be deleted from VM/VMSS.                                                                                                                                         | If not provided, default value is empty           |
 | `nmi.image`                              | NMI image name                                                                                                                                                                                                   | `nmi`                                                    |
 | `nmi.tag`                                | NMI image tag                                                                                                                                                                                                    | `1.5.2`                                                  |
 | `nmi.resources`                          | Resource limit for NMI                                                                                                                                                                                           | `{}`                                                     |
diff --git a/charts/aad-pod-identity/templates/mic-deployment.yaml b/charts/aad-pod-identity/templates/mic-deployment.yaml
index 8ea58dbe..0618e0d3 100644
--- a/charts/aad-pod-identity/templates/mic-deployment.yaml
+++ b/charts/aad-pod-identity/templates/mic-deployment.yaml
@@ -61,6 +61,9 @@ spec:
           {{- end }}
           {{- if .Values.mic.clientQps }}
           - --clientQps={{ .Values.mic.clientQps }}
+          {{- end }}          
+          {{- if .Values.mic.immutableUserMSIs }}
+          - "--immutableUserMSIs={{- join "," .Values.mic.immutableUserMSIs}}"
           {{- end }}
         env:
           - name: FORCENAMESPACED
diff --git a/charts/aad-pod-identity/values.yaml b/charts/aad-pod-identity/values.yaml
index 9f948fc1..836971b2 100644
--- a/charts/aad-pod-identity/values.yaml
+++ b/charts/aad-pod-identity/values.yaml
@@ -62,6 +62,12 @@ mic:
   # Override interval in seconds at which sync loop should periodically check for errors and reconcile (default is 3600s)
   syncRetryDuration: ""
 
+  # Override the defult value of immutable identities. 
+  immutableUserMSIs: []
+    # Example of MSIs (should be replaced with the real ids)
+    #- "00000000-0000-0000-0000-000000000000"
+    #- "11111111-1111-1111-1111-111111111111"
+
   # https://github.com/Azure/aad-pod-identity/blob/master/docs/readmes/README.featureflags.md#batch-create-delete-flag
   # default value is 20
   createDeleteBatch: ""
diff --git a/cmd/mic/main.go b/cmd/mic/main.go
index d6e81a7a..019e64aa 100644
--- a/cmd/mic/main.go
+++ b/cmd/mic/main.go
@@ -5,6 +5,7 @@ import (
 	"net/http"
 	_ "net/http/pprof"
 	"os"
+	"strings"
 	"time"
 
 	"github.com/Azure/aad-pod-identity/pkg/mic"
@@ -27,6 +28,7 @@ var (
 	enableScaleFeatures bool
 	createDeleteBatch   int64
 	clientQPS           float64
+	immutableUserMSIs   string
 )
 
 func main() {
@@ -62,6 +64,9 @@ func main() {
 	// Client QPS is used to configure the client-go QPS throttling and bursting.
 	flag.Float64Var(&clientQPS, "clientQps", 5, "Client QPS used for throttling of calls to kube-api server")
 
+	//Identities that should be never removed from Azure AD (used defined managed identities)
+	flag.StringVar(&immutableUserMSIs, "immutableUserMSIs", "", "prevent deletion of these IDs from the underlying VM/VMSS")
+
 	flag.Parse()
 	if versionInfo {
 		version.PrintVersionAndExit()
@@ -98,7 +103,10 @@ func main() {
 	config.Burst = int(clientQPS)
 	glog.Infof("Client QPS set to: %v. Burst to: %v", config.QPS, config.Burst)
 
-	micClient, err := mic.NewMICClient(cloudconfig, config, forceNamespaced, syncRetryDuration, &leaderElectionCfg, enableScaleFeatures, createDeleteBatch)
+	immutableUserMSIsList := strings.Split(immutableUserMSIs, ",")
+	glog.Infof("immutable identities are %v", immutableUserMSIsList)
+
+	micClient, err := mic.NewMICClient(cloudconfig, config, forceNamespaced, syncRetryDuration, &leaderElectionCfg, enableScaleFeatures, createDeleteBatch, immutableUserMSIsList)
 	if err != nil {
 		glog.Fatalf("Could not get the MIC client: %+v", err)
 	}
diff --git a/cmd/nmi/main.go b/cmd/nmi/main.go
index 83f23eef..19686c23 100644
--- a/cmd/nmi/main.go
+++ b/cmd/nmi/main.go
@@ -1,6 +1,7 @@
 package main
 
 import (
+	goflag "flag"
 	"os"
 
 	"net/http"
@@ -45,6 +46,9 @@ var (
 )
 
 func main() {
+	// this is done for glog used by client-go underneath
+	pflag.CommandLine.AddGoFlagSet(goflag.CommandLine)
+
 	pflag.Parse()
 	if *versionInfo {
 		version.PrintVersionAndExit()
diff --git a/docs/readmes/README.featureflags.md b/docs/readmes/README.featureflags.md
index 95772d3e..4e8a121a 100644
--- a/docs/readmes/README.featureflags.md
+++ b/docs/readmes/README.featureflags.md
@@ -42,3 +42,9 @@ endpoint from Pods which are not using host networking and return an HTTP 403
 Forbidden response. This flag is disabled by default to maximize compatibility.
 Users are encouraged to determine if this option is relevant and beneficial for
 their use cases.
+
+## ImmutableUserMSIs flag
+> Available from 1.5.4 release
+
+Aad-pod-identity has a new flag immutableUserMSIs which can be used to prevent deletetion of spcecified identitites from VM/VMSS.
+The list is comma separated. Example: 00000000-0000-0000-0000-000000000000,11111111-1111-1111-1111-111111111111
diff --git a/pkg/mic/mic.go b/pkg/mic/mic.go
index df1260ee..1a0ee10f 100644
--- a/pkg/mic/mic.go
+++ b/pkg/mic/mic.go
@@ -62,6 +62,7 @@ type Client struct {
 	syncRetryInterval   time.Duration
 	enableScaleFeatures bool
 	createDeleteBatch   int64
+	ImmutableUserMSIs   map[string]bool
 
 	syncing int32 // protect against conucrrent sync's
 
@@ -85,7 +86,7 @@ type trackUserAssignedMSIIds struct {
 
 // NewMICClient returnes new mic client
 func NewMICClient(cloudconfig string, config *rest.Config, isNamespaced bool, syncRetryInterval time.Duration,
-	leaderElectionConfig *LeaderElectionConfig, enableScaleFeatures bool, createDeleteBatch int64) (*Client, error) {
+	leaderElectionConfig *LeaderElectionConfig, enableScaleFeatures bool, createDeleteBatch int64, immutableUserMSIsList []string) (*Client, error) {
 	glog.Infof("Starting to create the pod identity client. Version: %v. Build date: %v", version.MICVersion, version.BuildDate)
 
 	clientSet := kubernetes.NewForConfigOrDie(config)
@@ -117,6 +118,15 @@ func NewMICClient(cloudconfig string, config *rest.Config, isNamespaced bool, sy
 	eventBroadcaster.StartRecordingToSink(&typedcorev1.EventSinkImpl{Interface: clientSet.CoreV1().Events("")})
 	recorder := eventBroadcaster.NewRecorder(scheme.Scheme, corev1.EventSource{Component: aadpodid.CRDGroup})
 
+	var immutableUserMSIsMap map[string]bool
+
+	if immutableUserMSIsList != nil {
+		immutableUserMSIsMap = make(map[string]bool)
+		for _, item := range immutableUserMSIsList {
+			immutableUserMSIsMap[strings.ToLower(item)] = true
+		}
+	}
+
 	c := &Client{
 		CRDClient:           crdClient,
 		CloudClient:         cloudClient,
@@ -128,6 +138,7 @@ func NewMICClient(cloudconfig string, config *rest.Config, isNamespaced bool, sy
 		syncRetryInterval:   syncRetryInterval,
 		enableScaleFeatures: enableScaleFeatures,
 		createDeleteBatch:   createDeleteBatch,
+		ImmutableUserMSIs:   immutableUserMSIsMap,
 	}
 	leaderElector, err := c.NewLeaderElector(clientSet, recorder, leaderElectionConfig)
 	if err != nil {
@@ -794,7 +805,20 @@ func (c *Client) updateUserMSI(newAssignedIDs map[string]aadpodid.AzureAssignedI
 	addUserAssignedMSIIDs := c.getUniqueIDs(nodeTrackList.addUserAssignedMSIIDs)
 	removeUserAssignedMSIIDs := c.getUniqueIDs(nodeTrackList.removeUserAssignedMSIIDs)
 
-	err := c.CloudClient.UpdateUserMSI(addUserAssignedMSIIDs, removeUserAssignedMSIIDs, nodeOrVMSSName, nodeTrackList.isvmss)
+	// The the list of identities (user-defined managed identities) shouldn't be deleted from VM/VMSS, but only from Kubernetes
+	var approvedRemoveUserAssignedMSIIDs []string
+	if c.ImmutableUserMSIs != nil {
+		approvedRemoveUserAssignedMSIIDs := make([]string, 0)
+		for _, item := range removeUserAssignedMSIIDs {
+			if _, ok := c.ImmutableUserMSIs[item]; !ok {
+				approvedRemoveUserAssignedMSIIDs = append(approvedRemoveUserAssignedMSIIDs, item)
+			}
+		}
+	} else {
+		approvedRemoveUserAssignedMSIIDs = removeUserAssignedMSIIDs
+	}
+
+	err := c.CloudClient.UpdateUserMSI(addUserAssignedMSIIDs, approvedRemoveUserAssignedMSIIDs, nodeOrVMSSName, nodeTrackList.isvmss)
 	if err != nil {
 		glog.Errorf("Updating msis on node %s, add [%d], del [%d] failed with error %v", nodeOrVMSSName, len(nodeTrackList.assignedIDsToCreate), len(nodeTrackList.assignedIDsToDelete), err)
 		idList, getErr := c.getUserMSIListForNode(nodeOrVMSSName, nodeTrackList.isvmss)
diff --git a/pkg/nmi/server/server.go b/pkg/nmi/server/server.go
index 60caf936..1f08d5f6 100644
--- a/pkg/nmi/server/server.go
+++ b/pkg/nmi/server/server.go
@@ -152,6 +152,9 @@ func (fn appHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 		"req.path":   r.URL.Path,
 		"req.remote": parseRemoteAddr(r.RemoteAddr),
 	})
+	// Set the header in advance so that both success as well
+	// as error paths have it set as application/json content type.
+	w.Header().Set("Content-Type", "application/json")
 	start := time.Now()
 	defer func() {
 		var err error
@@ -459,7 +462,6 @@ func (s *Server) defaultPathHandler(logger *log.Entry, w http.ResponseWriter, r
 		logger.Errorf("failed io operation of reading response body, %+v", err)
 		http.Error(w, err.Error(), http.StatusInternalServerError)
 	}
-	w.Header().Set("Content-Type", "application/json")
 	w.Write(body)
 }
 
@@ -557,7 +559,7 @@ func (s *Server) listPodIDsWithRetry(ctx context.Context, kubeClient k8s.Client,
 			err = ctx.Err()
 			return nil, true, err
 		}
-		logger.Warningf("failed to get assigned ids for pod:%s/%s in ASSIGNED state, retrying attempt: %d", podns, podname, attempt)
+		logger.Debugf("failed to get assigned ids for pod:%s/%s in ASSIGNED state, retrying attempt: %d", podns, podname, attempt)
 	}
 	return nil, true, fmt.Errorf("getting assigned identities for pod %s/%s in ASSIGNED state failed after %d attempts, retry duration [%d]s. Error: %v",
 		podns, podname, s.ListPodIDsRetryAttemptsForCreated+s.ListPodIDsRetryAttemptsForAssigned, s.ListPodIDsRetryIntervalInSeconds, err)


Test patch:
diff --git a/pkg/mic/mic_test.go b/pkg/mic/mic_test.go
index b7c72f95..df33e6c6 100644
--- a/pkg/mic/mic_test.go
+++ b/pkg/mic/mic_test.go
@@ -601,7 +601,8 @@ func NewMICTestClient(eventCh chan aadpodid.EventType,
 	podClient *TestPodClient,
 	nodeClient *TestNodeClient,
 	eventRecorder *TestEventRecorder, isNamespaced bool,
-	createDeleteBatch int64) *TestMICClient {
+	createDeleteBatch int64,
+	immutableUserMSIs map[string]bool) *TestMICClient {
 
 	realMICClient := &Client{
 		CloudClient:       cpClient,
@@ -613,6 +614,7 @@ func NewMICTestClient(eventCh chan aadpodid.EventType,
 		syncRetryInterval: 120 * time.Second,
 		IsNamespaced:      isNamespaced,
 		createDeleteBatch: createDeleteBatch,
+		ImmutableUserMSIs: immutableUserMSIs,
 	}
 
 	return &TestMICClient{
@@ -719,7 +721,7 @@ func TestSimpleMICClient(t *testing.T) {
 	evtRecorder.lastEvent = new(LastEvent)
 	evtRecorder.eventChannel = make(chan bool, 100)
 
-	micClient := NewMICTestClient(eventCh, cloudClient, crdClient, podClient, nodeClient, &evtRecorder, false, 4)
+	micClient := NewMICTestClient(eventCh, cloudClient, crdClient, podClient, nodeClient, &evtRecorder, false, 4, nil)
 
 	crdClient.CreateID("test-id", "default", aadpodid.UserAssignedMSI, "test-user-msi-resourceid", "test-user-msi-clientid", nil, "", "", "", "")
 	crdClient.CreateBinding("testbinding", "default", "test-id", "test-select", "")
@@ -840,7 +842,7 @@ func TestAddDelMICClient(t *testing.T) {
 	evtRecorder.lastEvent = new(LastEvent)
 	evtRecorder.eventChannel = make(chan bool, 100)
 
-	micClient := NewMICTestClient(eventCh, cloudClient, crdClient, podClient, nodeClient, &evtRecorder, false, 4)
+	micClient := NewMICTestClient(eventCh, cloudClient, crdClient, podClient, nodeClient, &evtRecorder, false, 4, nil)
 
 	// Test to add and delete at the same time.
 	// Add a pod, identity and binding.
@@ -932,7 +934,7 @@ func TestMicAddDelVMSS(t *testing.T) {
 	evtRecorder.lastEvent = new(LastEvent)
 	evtRecorder.eventChannel = make(chan bool, 100)
 
-	micClient := NewMICTestClient(eventCh, cloudClient, crdClient, podClient, nodeClient, &evtRecorder, false, 4)
+	micClient := NewMICTestClient(eventCh, cloudClient, crdClient, podClient, nodeClient, &evtRecorder, false, 4, nil)
 
 	// Test to add and delete at the same time.
 	// Add a pod, identity and binding.
@@ -1035,7 +1037,7 @@ func TestMICStateFlow(t *testing.T) {
 	evtRecorder.lastEvent = new(LastEvent)
 	evtRecorder.eventChannel = make(chan bool, 100)
 
-	micClient := NewMICTestClient(eventCh, cloudClient, crdClient, podClient, nodeClient, &evtRecorder, false, 4)
+	micClient := NewMICTestClient(eventCh, cloudClient, crdClient, podClient, nodeClient, &evtRecorder, false, 4, nil)
 
 	// Add a pod, identity and binding.
 	crdClient.CreateID("test-id1", "default", aadpodid.UserAssignedMSI, "test-user-msi-resourceid", "test-user-msi-clientid", nil, "", "", "", "")
@@ -1149,7 +1151,7 @@ func TestForceNamespaced(t *testing.T) {
 	evtRecorder.lastEvent = new(LastEvent)
 	evtRecorder.eventChannel = make(chan bool, 100)
 
-	micClient := NewMICTestClient(eventCh, cloudClient, crdClient, podClient, nodeClient, &evtRecorder, true, 4)
+	micClient := NewMICTestClient(eventCh, cloudClient, crdClient, podClient, nodeClient, &evtRecorder, true, 4, nil)
 
 	crdClient.CreateID("test-id1", "default", aadpodid.UserAssignedMSI, "test-user-msi-resourceid", "test-user-msi-clientid", nil, "", "", "", "idrv1")
 	crdClient.CreateBinding("testbinding1", "default", "test-id1", "test-select1", "bindingrv1")
@@ -1213,7 +1215,7 @@ func TestSyncRetryLoop(t *testing.T) {
 	evtRecorder.lastEvent = new(LastEvent)
 	evtRecorder.eventChannel = make(chan bool, 100)
 
-	micClient := NewMICTestClient(eventCh, cloudClient, crdClient, podClient, nodeClient, &evtRecorder, false, 4)
+	micClient := NewMICTestClient(eventCh, cloudClient, crdClient, podClient, nodeClient, &evtRecorder, false, 4, nil)
 	syncRetryInterval, err := time.ParseDuration("10s")
 	if err != nil {
 		t.Errorf("error parsing duration: %v", err)
@@ -1292,7 +1294,7 @@ func TestSyncNodeNotFound(t *testing.T) {
 	evtRecorder.lastEvent = new(LastEvent)
 	evtRecorder.eventChannel = make(chan bool, 100)
 
-	micClient := NewMICTestClient(eventCh, cloudClient, crdClient, podClient, nodeClient, &evtRecorder, false, 4)
+	micClient := NewMICTestClient(eventCh, cloudClient, crdClient, podClient, nodeClient, &evtRecorder, false, 4, nil)
 
 	// Add a pod, identity and binding.
 	crdClient.CreateID("test-id1", "default", aadpodid.UserAssignedMSI, "test-user-msi-resourceid", "test-user-msi-clientid", nil, "", "", "", "")
@@ -1364,7 +1366,7 @@ func TestProcessingTimeForScale(t *testing.T) {
 	evtRecorder.lastEvent = new(LastEvent)
 	evtRecorder.eventChannel = make(chan bool, 20000)
 
-	micClient := NewMICTestClient(eventCh, cloudClient, crdClient, podClient, nodeClient, &evtRecorder, false, 4)
+	micClient := NewMICTestClient(eventCh, cloudClient, crdClient, podClient, nodeClient, &evtRecorder, false, 4, nil)
 
 	// Add a pod, identity and binding.
 	crdClient.CreateID("test-id1", "default", aadpodid.UserAssignedMSI, "test-user-msi-resourceid", "test-user-msi-clientid", nil, "", "", "", "")
@@ -1419,7 +1421,114 @@ func TestSyncExit(t *testing.T) {
 	evtRecorder.lastEvent = new(LastEvent)
 	evtRecorder.eventChannel = make(chan bool)
 
-	micClient := NewMICTestClient(eventCh, cloudClient, crdClient, podClient, nodeClient, &evtRecorder, false, 4)
+	micClient := NewMICTestClient(eventCh, cloudClient, crdClient, podClient, nodeClient, &evtRecorder, false, 4, nil)
 
 	micClient.testRunSync()(t)
 }
+
+func TestMicAddDelVMSSwithImmutableIdentities(t *testing.T) {
+	eventCh := make(chan aadpodid.EventType, 100)
+	cloudClient := NewTestCloudClient(config.AzureConfig{VMType: "vmss"})
+	crdClient := NewTestCrdClient(nil)
+	podClient := NewTestPodClient()
+	nodeClient := NewTestNodeClient()
+	var evtRecorder TestEventRecorder
+	evtRecorder.lastEvent = new(LastEvent)
+	evtRecorder.eventChannel = make(chan bool, 100)
+	var immutableUserMSIs = map[string]bool{
+		"zero-test":                true,
+		"test-user-msi-resourceid": true,
+	}
+
+	micClient := NewMICTestClient(eventCh, cloudClient, crdClient, podClient, nodeClient, &evtRecorder, false, 4, immutableUserMSIs)
+
+	// Test to add and delete at the same time.
+	// Add a pod, identity and binding.
+	crdClient.CreateID("test-id1", "default", aadpodid.UserAssignedMSI, "test-user-msi-resourceid", "test-user-msi-clientid", nil, "", "", "", "")
+	crdClient.CreateBinding("testbinding1", "default", "test-id1", "test-select1", "")
+
+	nodeClient.AddNode("test-node1", func(n *corev1.Node) {
+		n.Spec.ProviderID = "azure:///subscriptions/fakeSub/resourceGroups/fakeGroup/providers/Microsoft.Compute/virtualMachineScaleSets/testvmss1/virtualMachines/0"
+	})
+
+	nodeClient.AddNode("test-node2", func(n *corev1.Node) {
+		n.Spec.ProviderID = "azure:///subscriptions/fakeSub/resourceGroups/fakeGroup/providers/Microsoft.Compute/virtualMachineScaleSets/testvmss1/virtualMachines/1"
+	})
+
+	nodeClient.AddNode("test-node3", func(n *corev1.Node) {
+		n.Spec.ProviderID = "azure:///subscriptions/fakeSub/resourceGroups/fakeGroup/providers/Microsoft.Compute/virtualMachineScaleSets/testvmss2/virtualMachines/0"
+	})
+
+	podClient.AddPod("test-pod1", "default", "test-node1", "test-select1")
+	podClient.AddPod("test-pod2", "default", "test-node2", "test-select1")
+	podClient.AddPod("test-pod3", "default", "test-node3", "test-select1")
+
+	defer micClient.testRunSync()(t)
+
+	eventCh <- aadpodid.PodCreated
+	eventCh <- aadpodid.PodCreated
+	eventCh <- aadpodid.PodCreated
+	if !evtRecorder.WaitForEvents(3) {
+		t.Fatalf("Timeout waiting for mic sync cycles")
+	}
+	listAssignedIDs, err := crdClient.ListAssignedIDs()
+	if err != nil {
+		glog.Error(err)
+		t.Errorf("list assigned failed")
+	}
+	if !(len(*listAssignedIDs) == 3) {
+		t.Fatalf("expected assigned identities len: %d, got: %d", 3, len(*listAssignedIDs))
+	}
+
+	if !cloudClient.CompareMSI("testvmss1", []string{"test-user-msi-resourceid"}) {
+		t.Fatalf("missing identity: %+v", cloudClient.ListMSI()["testvmss1"])
+	}
+	if !cloudClient.CompareMSI("testvmss2", []string{"test-user-msi-resourceid"}) {
+		t.Fatalf("missing identity: %+v", cloudClient.ListMSI()["testvmss2"])
+	}
+
+	podClient.DeletePod("test-pod1", "default")
+	eventCh <- aadpodid.PodDeleted
+
+	if !evtRecorder.WaitForEvents(1) {
+		t.Fatal("Timeout waiting for mic sync cycles")
+	}
+	listAssignedIDs, err = crdClient.ListAssignedIDs()
+	if err != nil {
+		glog.Error(err)
+		t.Errorf("list assigned failed")
+	}
+	if !(len(*listAssignedIDs) == 2) {
+		t.Fatalf("expected assigned identities len: %d, got: %d", 2, len(*listAssignedIDs))
+	}
+
+	if !cloudClient.CompareMSI("testvmss1", []string{"test-user-msi-resourceid"}) {
+		t.Fatalf("missing identity: %+v", cloudClient.ListMSI()["testvmss1"])
+	}
+	if !cloudClient.CompareMSI("testvmss2", []string{"test-user-msi-resourceid"}) {
+		t.Fatalf("missing identity: %+v", cloudClient.ListMSI()["testvmss2"])
+	}
+
+	podClient.DeletePod("test-pod2", "default")
+
+	eventCh <- aadpodid.PodDeleted
+
+	if !evtRecorder.WaitForEvents(1) {
+		t.Fatal("Timeout waiting for mic sync cycles")
+	}
+	listAssignedIDs, err = crdClient.ListAssignedIDs()
+	if err != nil {
+		glog.Error(err)
+		t.Errorf("list assigned failed")
+	}
+	if !(len(*listAssignedIDs) == 1) {
+		t.Fatalf("expected assigned identities len: %d, got: %d", 1, len(*listAssignedIDs))
+	}
+
+	if !cloudClient.CompareMSI("testvmss1", []string{"test-user-msi-resourceid"}) {
+		t.Fatalf("missing identity: %+v", cloudClient.ListMSI()["testvmss1"])
+	}
+	if !cloudClient.CompareMSI("testvmss2", []string{"test-user-msi-resourceid"}) {
+		t.Fatalf("missing identity: %+v", cloudClient.ListMSI()["testvmss2"])
+	}
+}
diff --git a/test/common/k8s/infra/infra.go b/test/common/k8s/infra/infra.go
index d322c1e7..fbf4a2e3 100644
--- a/test/common/k8s/infra/infra.go
+++ b/test/common/k8s/infra/infra.go
@@ -11,7 +11,7 @@ import (
 )
 
 // CreateInfra will deploy all the infrastructure components (nmi and mic) on a Kubernetes cluster
-func CreateInfra(namespace, registry, nmiVersion, micVersion, templateOutputPath string, old, enableScaleFeatures bool) error {
+func CreateInfra(namespace, registry, nmiVersion, micVersion, templateOutputPath string, old, enableScaleFeatures bool, immutableUserMSIs string) error {
 	var err error
 	var t *template.Template
 	if !old {
@@ -44,6 +44,7 @@ func CreateInfra(namespace, registry, nmiVersion, micVersion, templateOutputPath
 		MICArg              bool
 		NMIArg              bool
 		EnableScaleFeatures bool
+		ImmutableUserMSIs   string
 	}{
 		namespace,
 		registry,
@@ -52,6 +53,7 @@ func CreateInfra(namespace, registry, nmiVersion, micVersion, templateOutputPath
 		micArg,
 		nmiArg,
 		enableScaleFeatures,
+		immutableUserMSIs,
 	}
 	if err := t.Execute(deployFile, deployData); err != nil {
 		return errors.Wrap(err, "Failed to create a deployment file from deployment-rbac.yaml")
diff --git a/test/e2e/README.md b/test/e2e/README.md
index 64a6c66d..85facf83 100644
--- a/test/e2e/README.md
+++ b/test/e2e/README.md
@@ -109,6 +109,7 @@ To ensure consistency across all tests, they generally follow the format below:
 | With AzureIdentity, AzureBinding and identity validator deployed, remove the AzureIdentity | AzureAssignedIdentity should get removed and identity validator should not be able to access Azure resources | Basic |
 | With AzureIdentity, AzureBinding and identity validator deployed, remove the AzureIdentityBinding | AzureAssignedIdentity should get removed and identity validator should not be able to access Azure resources | Basic |
 | With AzureIdentity, AzureBinding and identity validator deployed, remove the identity validator deployment | AzureAssignedIdentity should get removed | Basic |
+| With AzureIdentity, AzureBinding and identity validator deployed, remove the immutable identity based deployment | Immutable identity should be removed from the cluster but shouldn't be deleted from VM/VMSS | Basic |
 | Add an AzureIdentity and AzureBinding, deploy identity validator with the label marked in binding, then drain the node containing the identity validator deployment | A new AzureAssignedIdentity should be established with the new pod and the old one should be removed | Advanced |
 | Add a number of AzureIdentities and AzureIdentityBindings in order and remove them in random order | The correct identities and identity binding should be removed and the rest should remain untouched | Random |
 | Enable a user assigned identity on VMs, then assign a different user assigned identity to a pod | Pod identity should work as expected and the user assigned identity on VMs should not be altered after deleting the pod identity | Advanced |
diff --git a/test/e2e/aadpodidentity_test.go b/test/e2e/aadpodidentity_test.go
index b8a08991..3b05f7e4 100644
--- a/test/e2e/aadpodidentity_test.go
+++ b/test/e2e/aadpodidentity_test.go
@@ -39,6 +39,7 @@ const (
 	keyvaultIdentity  = "keyvault-identity"
 	identityValidator = "identity-validator"
 	nmiDaemonSet      = "nmi"
+	immutableIdentity = "immutable-identity"
 )
 
 var (
@@ -60,7 +61,7 @@ var _ = BeforeSuite(func() {
 	Expect(err).NotTo(HaveOccurred())
 	cfg = *c
 	fmt.Printf("System MSI enabled: %v\n", cfg.SystemMSICluster)
-	setupInfra(cfg.Registry, cfg.NMIVersion, cfg.MICVersion, cfg.EnableScaleFeatures)
+	setupInfra(cfg.Registry, cfg.NMIVersion, cfg.MICVersion, cfg.EnableScaleFeatures, cfg.ImmutableUserMSIs)
 })
 
 var _ = AfterSuite(func() {
@@ -498,7 +499,7 @@ var _ = Describe("Kubernetes cluster using aad-pod-identity", func() {
 		}
 
 		// reset the infra to previous state
-		setupInfra(cfg.Registry, cfg.NMIVersion, cfg.MICVersion, cfg.EnableScaleFeatures)
+		setupInfra(cfg.Registry, cfg.NMIVersion, cfg.MICVersion, cfg.EnableScaleFeatures, cfg.ImmutableUserMSIs)
 	})
 
 	It("should not alter the system assigned identity after creating and deleting pod identity", func() {
@@ -594,7 +595,7 @@ var _ = Describe("Kubernetes cluster using aad-pod-identity", func() {
 		Expect(ok).To(Equal(true))
 
 		// setup mic and nmi with old releases
-		setupInfraOld("mcr.microsoft.com/k8s/aad-pod-identity", "1.4", "1.3")
+		setupInfraOld("mcr.microsoft.com/k8s/aad-pod-identity", "1.4", "1.3", "")
 
 		setUpIdentityAndDeployment(keyvaultIdentity, "", "1")
 
@@ -603,7 +604,7 @@ var _ = Describe("Kubernetes cluster using aad-pod-identity", func() {
 		Expect(ok).To(Equal(true))
 
 		// update the infra to use latest mic and nmi images
-		setupInfra(cfg.Registry, cfg.NMIVersion, cfg.MICVersion, cfg.EnableScaleFeatures)
+		setupInfra(cfg.Registry, cfg.NMIVersion, cfg.MICVersion, cfg.EnableScaleFeatures, cfg.ImmutableUserMSIs)
 
 		ok, err = daemonset.WaitOnReady(nmiDaemonSet)
 		Expect(err).NotTo(HaveOccurred())
@@ -632,29 +633,12 @@ var _ = Describe("Kubernetes cluster using aad-pod-identity", func() {
 		nodeList, err := node.GetAll()
 		Expect(err).NotTo(HaveOccurred())
 
-		vmssNodes := make(map[string][]node.Node)
-		for _, n := range nodeList.Nodes {
-			r, _ := cloudprovider.ParseResourceID(n.Spec.ProviderID)
-			if r.ResourceType == cloudprovider.VMSSResourceType {
-				ls := vmssNodes[r.ResourceName]
-				vmssNodes[r.ResourceName] = append(ls, n)
-			}
-		}
-
-		var vmssID string
-		for id, ls := range vmssNodes {
-			if len(ls) > 1 {
-				vmssID = id
-				break
-			}
-		}
-
+		vmss, vmssID := GetVMSS(nodeList)
 		if vmssID == "" {
 			Skip("Skipping test since there is no vmss with more than 1 node")
 			return
 		}
 
-		vmss := vmssNodes[vmssID]
 
 		setUpIdentityAndDeployment(keyvaultIdentity, "", "1", func(d *infra.IdentityValidatorTemplateData) {
 			d.NodeName = vmss[0].Name
@@ -692,6 +676,45 @@ var _ = Describe("Kubernetes cluster using aad-pod-identity", func() {
 		Expect(exists).To(Equal(true))
 	})
 
+	It("should not delete the Immutable Identity from vmss when the deployment is deleted", func() {
+		nodeList, err := node.GetAll()
+		Expect(err).NotTo(HaveOccurred())
+
+		vmss, vmssID := GetVMSS(nodeList)
+		if vmssID == "" {
+			Skip("Skipping test since there is no vmss with more than 1 node")
+			return
+		}
+
+		// Explicitly assign identity to the underlying VMSS:
+		enableUserAssignedIdentityOnCluster(nodeList, immutableIdentity)
+
+		setUpIdentityAndDeployment(immutableIdentity, "", "1", func(d *infra.IdentityValidatorTemplateData) {
+			d.NodeName = vmss[0].Name
+		})
+
+		ok, err := azureassignedidentity.WaitOnLengthMatched(1)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(ok).To(Equal(true))
+
+		azureAssignedIdentity, err := azureassignedidentity.GetByPrefix(identityValidator)
+		Expect(err).NotTo(HaveOccurred())
+
+		// Ensure that the identity validator is able to get the token
+		validateAzureAssignedIdentity(azureAssignedIdentity, immutableIdentity)
+
+		waitForDeployDeletion(identityValidator)
+
+		ok, err = azureassignedidentity.WaitOnLengthMatched(0)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(ok).To(Equal(true))
+
+		exists, err := azure.UserIdentityAssignedToVMSS(cfg.ResourceGroup, vmssID, immutableIdentity)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(exists).To(Equal(true))
+
+	})
+
 	It("should pass liveness probe test", func() {
 		pods, err := pod.GetAllNameByPrefix("mic")
 		Expect(err).NotTo(HaveOccurred())
@@ -944,6 +967,29 @@ var _ = Describe("Kubernetes cluster using aad-pod-identity", func() {
 	})
 })
 
+func GetVMSS(nodeList *node.List) ([]node.Node, string) {
+	vmssNodes := make(map[string][]node.Node)
+	for _, n := range nodeList.Nodes {
+		r, _ := cloudprovider.ParseResourceID(n.Spec.ProviderID)
+		if r.ResourceType == cloudprovider.VMSSResourceType {
+			ls := vmssNodes[r.ResourceName]
+			vmssNodes[r.ResourceName] = append(ls, n)
+		}
+	}
+	var vmssID string
+	for id, ls := range vmssNodes {
+		if len(ls) > 1 {
+			vmssID = id
+			break
+		}
+	}
+	if vmssID == "" {
+		return nil, ""
+	}
+	vmss := vmssNodes[vmssID]
+	return vmss, vmssID
+}
+
 func runValidatorTest(iterations int) {
 	defer GinkgoRecover()
 	replicas := "1"
@@ -1165,17 +1211,17 @@ func checkInfra() {
 }
 
 // setupInfra creates the crds, mic, nmi and blocks until iptable entries exist
-func setupInfraOld(registry, nmiVersion, micVersion string) {
+func setupInfraOld(registry, nmiVersion, micVersion string, immutableUserMSIs string) {
 	// Install CRDs and deploy MIC and NMI
-	err := infra.CreateInfra("default", registry, nmiVersion, micVersion, templateOutputPath, true, false)
+	err := infra.CreateInfra("default", registry, nmiVersion, micVersion, templateOutputPath, true, false, immutableUserMSIs)
 	Expect(err).NotTo(HaveOccurred())
 	checkInfra()
 }
 
 // setupInfra creates the crds, mic, nmi and blocks until iptable entries exist
-func setupInfra(registry, nmiVersion, micVersion string, enableScaleFeatures bool) {
+func setupInfra(registry, nmiVersion, micVersion string, enableScaleFeatures bool, immutableUserMSIs string) {
 	// Install CRDs and deploy MIC and NMI
-	err := infra.CreateInfra("default", registry, nmiVersion, micVersion, templateOutputPath, false, enableScaleFeatures)
+	err := infra.CreateInfra("default", registry, nmiVersion, micVersion, templateOutputPath, false, enableScaleFeatures, immutableUserMSIs)
 	Expect(err).NotTo(HaveOccurred())
 	checkInfra()
 }
diff --git a/test/e2e/config/config.go b/test/e2e/config/config.go
index 7c024eb8..5272abc6 100644
--- a/test/e2e/config/config.go
+++ b/test/e2e/config/config.go
@@ -16,6 +16,7 @@ type Config struct {
 	IdentityValidatorVersion string `envconfig:"IDENTITY_VALIDATOR_VERSION" default:"1.5.3"`
 	SystemMSICluster         bool   `envconfig:"SYSTEM_MSI_CLUSTER" default:"false"`
 	EnableScaleFeatures      bool   `envconfig:"ENABLE_SCALE_FEATURES" default:"false"`
+	ImmutableUserMSIs        string `envconfig:"IMMUTABLE_IDENTITY_CLIENT_ID"`
 }
 
 // ParseConfig will parse needed environment variables for running the tests
diff --git a/test/e2e/setup.sh b/test/e2e/setup.sh
index 99f2826e..07f451f4 100755
--- a/test/e2e/setup.sh
+++ b/test/e2e/setup.sh
@@ -97,3 +97,14 @@ az role assignment create --role 'Managed Identity Operator' --assignee "$AZURE_
 retry_cmd "az keyvault set-policy -n "$KEYVAULT_NAME" --secret-permissions list --spn "$IDENTITY_CLIENT_ID"" "set policy failed"
 # The following command might need a couple of retries to succeed
 retry_cmd "az role assignment create --role Reader --assignee "$IDENTITY_PRINCIPAL_ID" --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.KeyVault/vaults/$KEYVAULT_NAME""
+
+# Create immutable identity
+IDENTITY_NAME="immutable-identity"
+echo "Creating $IDENTITY_NAME and assign appropriate roles..."
+az identity create -g "$RESOURCE_GROUP" -n "$IDENTITY_NAME"
+IMMUTABLE_IDENTITY_CLIENT_ID=$(az identity show -g "$RESOURCE_GROUP" -n "$IDENTITY_NAME"  --query 'clientId' -otsv)
+IDENTITY_PRINCIPAL_ID=$(az identity show -g "$RESOURCE_GROUP" -n "$IDENTITY_NAME" --query 'principalId' -otsv)
+az role assignment create --role 'Managed Identity Operator' --assignee "$AZURE_CLIENT_ID" --scope "/subscriptions/$SUBSCRIPTION_ID/resourcegroups/$RESOURCE_GROUP/providers/Microsoft.ManagedIdentity/userAssignedIdentities/$IDENTITY_NAME" || true
+retry_cmd "az keyvault set-policy -n "$KEYVAULT_NAME" --secret-permissions get list --spn "$IMMUTABLE_IDENTITY_CLIENT_ID"" "set policy failed"
+# The following command might need a couple of retries to succeed
+retry_cmd "az role assignment create --role Reader --assignee "$IDENTITY_PRINCIPAL_ID" --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.KeyVault/vaults/$KEYVAULT_NAME""
diff --git a/test/e2e/template/aadpodidentity_test_immutable.yaml b/test/e2e/template/aadpodidentity_test_immutable.yaml
new file mode 100644
index 00000000..6a74af5c
--- /dev/null
+++ b/test/e2e/template/aadpodidentity_test_immutable.yaml
@@ -0,0 +1,8 @@
+apiVersion: "aadpodidentity.k8s.io/v1"
+kind: AzureIdentity
+metadata:
+  name: testidentityimmutable
+spec:
+  type: 0
+  ResourceID: /subscriptions/11111111-1111-1111-1111-111111111111/resourcegroups/myResourceGroup/providers/Microsoft.ManagedIdentity/userAssignedIdentities/testidentity
+  ClientID: 11111111-1111-1111-1111-111111111111
\ No newline at end of file
diff --git a/test/e2e/template/deployment-rbac-old.yaml b/test/e2e/template/deployment-rbac-old.yaml
index 50a55737..57115b79 100644
--- a/test/e2e/template/deployment-rbac-old.yaml
+++ b/test/e2e/template/deployment-rbac-old.yaml
@@ -191,6 +191,7 @@ spec:
         args:
           {{if .MICArg}}- mic {{ end }}
           - "--cloudconfig=/etc/kubernetes/azure.json"
+          {{if .ImmutableUserMSIs}}- "--immutableUserMSIs={{.ImmutableUserMSIs}}" {{end}}
           - "--logtostderr"
         volumeMounts:
           - name: k8s-azure-file
diff --git a/test/e2e/template/deployment-rbac.yaml b/test/e2e/template/deployment-rbac.yaml
index 85170d17..434b3756 100644
--- a/test/e2e/template/deployment-rbac.yaml
+++ b/test/e2e/template/deployment-rbac.yaml
@@ -222,6 +222,7 @@ spec:
         args:
           {{if .MICArg}}- mic {{ end }}
           - "--cloudconfig=/etc/kubernetes/azure.json"
+          {{if .ImmutableUserMSIs}}- "--immutableUserMSIs={{.ImmutableUserMSIs}}" {{end}}
           - "--logtostderr"
           {{if .EnableScaleFeatures}}- "--enableScaleFeatures=true" {{end}}
         resources: