Repository: Azure/ARO-RP
Pull request #2001 (2022-03-10 20:23:47)

Fixing E2E Flake Redeploy VM
### Which issue this PR addresses:

<!--
Please include a link to the ADO work item as well as any GitHub issues.

Usage: `Fixes #<GitHub issue number>`, or `Fixes (paste link of issue)`.
-->
Fixes https://msazure.visualstudio.com/AzureRedHatOpenShift/_workitems/edit/13703198/

Here are the flakes (with logs) I was able to discover that this PR might address:

- [ ] [`500` responses on the redeploy request](https://msazure.visualstudio.com/AzureRedHatOpenShift/_build/results?buildId=52880064&view=logs&j=10f66467-59e5-53bc-0c8a-42cc89770de4&t=3bbe7e73-d181-5aca-4b77-91acc0ef0f4a&l=1179)
    - I believe this is not a test issue, but rather something we'd need to solve through code changes (retries)
- [X] [A successful request, that never redeploys the node based on K8s Events](https://msazure.visualstudio.com/AzureRedHatOpenShift/_build/results?buildId=52713570&view=logs&j=10f66467-59e5-53bc-0c8a-42cc89770de4&t=3bbe7e73-d181-5aca-4b77-91acc0ef0f4a&l=1161)

### What this PR does / why we need it:

<!--
Include a brief summary of what the PR is intended to accomplish and how the PR
does it. (2-3 sentences)
-->

What we _really_ care about is: **if I redeploy (successfully), do my Nodes reboot and come back up as Ready eventually in OpenShift?** The new tests answer this question.

#### History of this test

1. Checking the Azure activity log to ensure the redeploy was successful (past)
    - The Azure activity log is eventually consistent, so the delay between events was too great to rely on it
1. Checking the kubernetes events to ensure the node rebooted (current)
    - Kubernetes events aren't guaranteed to fire so this is inconsistent
1. Checking that the node goes not ready then ready again (current)
    - If the node reboots fast enough, the default grace periods built into OCP out of the box will prevent the node and pods from going NotReady and terminating respectively. This had inconsistent behavior as well.
1. Checking `uptime` of the Node kernel (requested change)
    - Has potential to flake (just like anything else), but if we can't create Pods on our rebooted Node, that probably warrants a failed test.

### E2E Flake "Proof"

Here's a [log of all `/azp run e2e` instances](https://msazure.visualstudio.com/AzureRedHatOpenShift/_build?definitionId=119938&branchFilter=5261894).


SudoBrendan: /azp run e2e
azure-pipelines[bot]: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
SudoBrendan: /azp run e2e
azure-pipelines[bot]: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
SudoBrendan: /azp run e2e
azure-pipelines[bot]: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
SudoBrendan: Having looked at this and tested the resiliency of the azure go sdk - I think this is a K8s event "reliability" issue. K8s Events are not guaranteed, especially over short timeframes (like a vm redeploy that takes <40s - the default heartbeat for Node readiness). I believe we're just seeing a Node go down and come back up between heartbeats, so K8s is none the wiser and doesn't bother populating Events it wouldn't know how to generate anyway.

While it looks strange to remove a test entirely, I think this is proper here - this test does not test the functionality/reliability of _our_ code, but rather, the system underneath (kubelet/Events/Node readiness). What we _really_ care about is: if I redeploy (successfully), do my Nodes come back up eventually? The remaining tests still answer this question and prove that business value without additional complexity.
azure-pipelines[bot]: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
SudoBrendan: /azp run e2e
azure-pipelines[bot]: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
SudoBrendan: /azp run ci
azure-pipelines[bot]: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>

Patch:
diff --git a/.gitignore b/.gitignore
index 9117184c9..f8e72571b 100644
--- a/.gitignore
+++ b/.gitignore
@@ -13,7 +13,6 @@ gomock_reflect_*
 /e2e.test
 /env*
 !/env.example
-!/env-int.example
 /id_rsa
 /proxy
 /pyenv*
diff --git a/Makefile b/Makefile
index d79a24eb0..e57f07af2 100644
--- a/Makefile
+++ b/Makefile
@@ -2,7 +2,6 @@ SHELL = /bin/bash
 TAG ?= $(shell git describe --exact-match 2>/dev/null)
 COMMIT = $(shell git rev-parse --short=7 HEAD)$(shell [[ $$(git status --porcelain) = "" ]] || echo -dirty)
 ARO_IMAGE_BASE = ${RP_IMAGE_ACR}.azurecr.io/aro
-E2E_FLAGS ?= -test.timeout 180m -test.v -ginkgo.v
 
 # fluentbit version must also be updated in RP code, see pkg/util/version/const.go
 FLUENTBIT_VERSION = 1.7.8-1
@@ -142,7 +141,7 @@ e2e.test:
 	go test ./test/e2e -tags e2e,codec.safe -c -ldflags "-X github.com/Azure/ARO-RP/pkg/util/version.GitCommit=$(VERSION)" -o e2e.test
 
 test-e2e: e2e.test
-	./e2e.test $(E2E_FLAGS)
+	./e2e.test -test.timeout 180m -test.v -ginkgo.v
 
 test-go: generate build-all validate-go lint-go unit-test-go
 
diff --git a/cmd/aro/operator.go b/cmd/aro/operator.go
index 49701e892..386916359 100644
--- a/cmd/aro/operator.go
+++ b/cmd/aro/operator.go
@@ -23,7 +23,6 @@ import (
 	pkgoperator "github.com/Azure/ARO-RP/pkg/operator"
 	aroclient "github.com/Azure/ARO-RP/pkg/operator/clientset/versioned"
 	"github.com/Azure/ARO-RP/pkg/operator/controllers/alertwebhook"
-	"github.com/Azure/ARO-RP/pkg/operator/controllers/autosizednodes"
 	"github.com/Azure/ARO-RP/pkg/operator/controllers/banner"
 	"github.com/Azure/ARO-RP/pkg/operator/controllers/checker"
 	"github.com/Azure/ARO-RP/pkg/operator/controllers/clusteroperatoraro"
@@ -211,11 +210,6 @@ func operator(ctx context.Context, log *logrus.Entry) error {
 		if err = (muo.NewReconciler(arocli, kubernetescli, dh)).SetupWithManager(mgr); err != nil {
 			return fmt.Errorf("unable to create controller %s: %v", muo.ControllerName, err)
 		}
-		if err = (autosizednodes.NewReconciler(
-			log.WithField("controller", autosizednodes.ControllerName),
-			mgr)).SetupWithManager(mgr); err != nil {
-			return fmt.Errorf("unable to create controller %s: %v", autosizednodes.ControllerName, err)
-		}
 	}
 
 	if err = (checker.NewReconciler(
diff --git a/docs/deploy-development-rp.md b/docs/deploy-development-rp.md
index aa01e63e0..12f00a2dd 100644
--- a/docs/deploy-development-rp.md
+++ b/docs/deploy-development-rp.md
@@ -150,7 +150,7 @@
   curl -X PATCH -k "https://localhost:8443/subscriptions/$AZURE_SUBSCRIPTION_ID/resourceGroups/$RESOURCEGROUP/providers/Microsoft.RedHatOpenShift/openShiftClusters/$CLUSTER?api-version=admin" --header "Content-Type: application/json" -d "{}"
   ```
 
-* Get Cluster details of a dev cluster
+* Get Cluster detials of a dev cluster
   ```bash
   curl -X GET -k "https://localhost:8443/subscriptions/$AZURE_SUBSCRIPTION_ID/resourceGroups/$RESOURCEGROUP/providers/Microsoft.RedHatOpenShift/openShiftClusters/$CLUSTER?api-version=admin" --header "Content-Type: application/json" -d "{}"
   ```
diff --git a/docs/deploy-full-rp-service-in-dev.md b/docs/deploy-full-rp-service-in-dev.md
index c9f01a799..a2746d0f2 100644
--- a/docs/deploy-full-rp-service-in-dev.md
+++ b/docs/deploy-full-rp-service-in-dev.md
@@ -21,11 +21,24 @@
     make dev-config.yaml
     ```
 
-1. Create a full environment file, which overrides some default `./env` options when sourced
+1. Update and resource your environment file
+    > It should look something like below once completed
     ```bash
-    cp env-int.example env-int
-    vi env-int
-    . ./env-int
+    export LOCATION=eastus
+    export ARO_IMAGE=arointsvc.azurecr.io/aro:latest
+
+    . secrets/env
+
+    export RESOURCEGROUP=$USER-aro-$LOCATION
+    export DATABASE_ACCOUNT_NAME=$USER-aro-$LOCATION
+    export DATABASE_NAME=ARO
+    export KEYVAULT_PREFIX=$USER-aro-$LOCATION
+    export ARO_IMAGE=${USER}aro.azurecr.io/aro:$(git rev-parse --short=7 HEAD)$([[ $(git status --porcelain) = "" ]] || echo -dirty)
+    export FLUENTBIT_IMAGE=${USER}aro.azurecr.io/fluentbit:latest
+    ```
+
+    ```bash
+    . ./env
     ```
 
 1. Run `make deploy`
@@ -91,7 +104,6 @@
         export SRC_AUTH_QUAY=$(echo $USER_PULL_SECRET | jq -r '.auths."quay.io".auth')
         export SRC_AUTH_REDHAT=$(echo $USER_PULL_SECRET | jq -r '.auths."registry.redhat.io".auth')
         export DST_AUTH=$(echo -n '00000000-0000-0000-0000-000000000000:'$(az acr login -n ${DST_ACR_NAME} --expose-token | jq -r .accessToken) | base64 -w0)
-        ```
 
     1. Login to the Azure Container Registry
         ```bash
diff --git a/docs/prepare-your-dev-environment.md b/docs/prepare-your-dev-environment.md
index 29f59eade..a88ad655f 100644
--- a/docs/prepare-your-dev-environment.md
+++ b/docs/prepare-your-dev-environment.md
@@ -18,19 +18,11 @@ This document goes through the development dependencies one requires in order to
 
 1. Install the relevant packages required for your OS defined below.
 
-1. Install [Podman](https://podman.io/getting-started/installation) and [podman-docker](https://developers.redhat.com/blog/2019/02/21/podman-and-buildah-for-docker-users#) if you haven't already, used for building container images.
-
-1. Run for `az acr login` compatability
-```bash
-sudo touch /etc/containers/nodocker
-```
-
 ### Fedora Packages
+
 1. Install the `gpgme-devel`, `libassuan-devel`, and `openssl` packages.
 > `sudo dnf install -y gpgme-devel libassuan-devel openssl`
 
-1. Install [Docker 17.05+](https://docs.docker.com/engine/install/fedora/) or later, used as an alternative to podman.
-
 ### Debian Packages
 1. Install the `libgpgme-dev` package.
 
@@ -79,14 +71,5 @@ sudo pip3 install virtualenv
 1. Clone the repository to your local machine:
     ```bash
     go get -u github.com/Azure/ARO-RP/...
-    ```
-    Alternatively you can also use:
-    ```bash
-    git clone https://github.com/Azure/ARO-RP.git $GOPATH/src/github.com/Azure/ARO-RP
-    ```
-    
-1. Go to project:
-    ```bash
     cd ${GOPATH:-$HOME/go}/src/github.com/Azure/ARO-RP
     ```
-
diff --git a/env-int.example b/env-int.example
deleted file mode 100644
index 08e0d574c..000000000
--- a/env-int.example
+++ /dev/null
@@ -1,9 +0,0 @@
-. ./env
-
-# Overrides for a full int-like local environment
-export RESOURCEGROUP=$USER-aro-$LOCATION
-export DATABASE_ACCOUNT_NAME=$USER-aro-$LOCATION
-export DATABASE_NAME=ARO
-export KEYVAULT_PREFIX=$USER-aro-$LOCATION
-export ARO_IMAGE=${USER}aro.azurecr.io/aro:$(git rev-parse --short=7 HEAD)$([[ $(git status --porcelain) = "" ]] || echo -dirty)
-export FLUENTBIT_IMAGE=${USER}aro.azurecr.io/fluentbit:latest
\ No newline at end of file
diff --git a/go.mod b/go.mod
index 790d9298e..e888d42e9 100644
--- a/go.mod
+++ b/go.mod
@@ -57,7 +57,7 @@ require (
 	github.com/gorilla/securecookie v1.1.1
 	github.com/gorilla/sessions v1.2.1
 	github.com/h2non/filetype v1.1.3 // indirect
-	github.com/jewzaam/go-cosmosdb v0.0.0-20220315232836-282b67c5b234
+	github.com/jewzaam/go-cosmosdb v0.0.0-20211018134417-8d1494e7862f
 	github.com/jstemmer/go-junit-report v0.9.1
 	github.com/klauspost/compress v1.14.2 // indirect
 	github.com/libvirt/libvirt-go v7.4.0+incompatible // indirect
diff --git a/go.sum b/go.sum
index 8e1ae4b2c..dfb8c79b5 100644
--- a/go.sum
+++ b/go.sum
@@ -1529,8 +1529,8 @@ github.com/jedib0t/go-pretty v4.3.0+incompatible/go.mod h1:XemHduiw8R651AF9Pt4Fw
 github.com/jen20/awspolicyequivalence v1.1.0/go.mod h1:PV1fS2xyHhCLp83vbgSMFr2drM4GzG61wkz+k4pOG3E=
 github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
 github.com/jessevdk/go-flags v1.5.0/go.mod h1:Fw0T6WPc1dYxT4mKEZRfG5kJhaTDP9pj1c2EWnYs/m4=
-github.com/jewzaam/go-cosmosdb v0.0.0-20220315232836-282b67c5b234 h1:R0Hokq55Hv3SpbXyfZRh8vIrKFAPZ+SQqyTsRc7J/4E=
-github.com/jewzaam/go-cosmosdb v0.0.0-20220315232836-282b67c5b234/go.mod h1:kZxm8EB19+pd3nT92t0aQblXb7K4sHG8cp59cleqdNc=
+github.com/jewzaam/go-cosmosdb v0.0.0-20211018134417-8d1494e7862f h1:G0wto7wY9OnIuwSPiLfyefcUN4FYEdtvwQMsCQdlDas=
+github.com/jewzaam/go-cosmosdb v0.0.0-20211018134417-8d1494e7862f/go.mod h1:Sxto8tJPhBRIa7jDsuTx4TZAcTYu0NH8E39iWqqMeRs=
 github.com/jewzaam/installer-aro v0.9.0-master.0.20220208140934-766bcf74e25c h1:yg14zwP4cFO98qqFlbkNqcouP/J0+zdCUf11qa/J/PU=
 github.com/jewzaam/installer-aro v0.9.0-master.0.20220208140934-766bcf74e25c/go.mod h1:qjvlt/LNPsGbLg2DtyY0VzxfKo+WVwcxJhektwKU+5E=
 github.com/jhump/protoreflect v1.6.0/go.mod h1:eaTn3RZAmMBcV0fifFvlm6VHNz3wSkYyXYWUh7ymB74=
diff --git a/pkg/api/defaults.go b/pkg/api/defaults.go
index c468f4bff..1e6d88b69 100644
--- a/pkg/api/defaults.go
+++ b/pkg/api/defaults.go
@@ -76,6 +76,5 @@ func DefaultOperatorFlags() OperatorFlags {
 		"aro.routefix.enabled":                     flagTrue,
 		"aro.storageaccounts.enabled":              flagTrue,
 		"aro.workaround.enabled":                   flagTrue,
-		"aro.autosizednodes.enable":                flagFalse,
 	}
 }
diff --git a/pkg/cluster/arooperator.go b/pkg/cluster/arooperator.go
index 879361c09..83bd81973 100644
--- a/pkg/cluster/arooperator.go
+++ b/pkg/cluster/arooperator.go
@@ -5,23 +5,17 @@ package cluster
 
 import (
 	"context"
-)
 
-func (m *manager) isIngressProfileAvailable() bool {
-	// We try to aqcuire the IngressProfiles data at frontend best effort enrichment time only.
-	// When we start deallocated VMs and wait for the API do become available again, we don't pick
-	// the information up, even though it would be available.
-	return len(m.doc.OpenShiftCluster.Properties.IngressProfiles) != 0
-}
+	"github.com/Azure/ARO-RP/pkg/operator/deploy"
+)
 
 func (m *manager) ensureAROOperator(ctx context.Context) error {
-	//ensure the IngressProfile information is available from the cluster which is not the case when the cluster vms were freshly restarted.
-	if !m.isIngressProfileAvailable() {
-		m.log.Error("skip ensureAROOperator")
-		return nil
+	dep, err := deploy.New(m.log, m.env, m.doc.OpenShiftCluster, m.arocli, m.extensionscli, m.kubernetescli)
+	if err != nil {
+		m.log.Errorf("cannot ensureAROOperator.New: %s", err.Error())
+		return err
 	}
-
-	err := m.aroOperatorDeployer.CreateOrUpdate(ctx)
+	err = dep.CreateOrUpdate(ctx)
 	if err != nil {
 		m.log.Errorf("cannot ensureAROOperator.CreateOrUpdate: %s", err.Error())
 	}
@@ -29,10 +23,9 @@ func (m *manager) ensureAROOperator(ctx context.Context) error {
 }
 
 func (m *manager) aroDeploymentReady(ctx context.Context) (bool, error) {
-	if !m.isIngressProfileAvailable() {
-		m.log.Error("skip aroDeploymentReady")
-		// skip and don't retry
-		return true, nil
+	dep, err := deploy.New(m.log, m.env, m.doc.OpenShiftCluster, m.arocli, m.extensionscli, m.kubernetescli)
+	if err != nil {
+		return false, err
 	}
-	return m.aroOperatorDeployer.IsReady(ctx)
+	return dep.IsReady(ctx)
 }
diff --git a/pkg/cluster/cluster.go b/pkg/cluster/cluster.go
index 877a10c71..db9a7cc72 100644
--- a/pkg/cluster/cluster.go
+++ b/pkg/cluster/cluster.go
@@ -23,7 +23,6 @@ import (
 	"github.com/Azure/ARO-RP/pkg/database"
 	"github.com/Azure/ARO-RP/pkg/env"
 	aroclient "github.com/Azure/ARO-RP/pkg/operator/clientset/versioned"
-	"github.com/Azure/ARO-RP/pkg/operator/deploy"
 	"github.com/Azure/ARO-RP/pkg/util/azureclient/graphrbac"
 	"github.com/Azure/ARO-RP/pkg/util/azureclient/mgmt/authorization"
 	"github.com/Azure/ARO-RP/pkg/util/azureclient/mgmt/compute"
@@ -91,8 +90,6 @@ type manager struct {
 	securitycli      securityclient.Interface
 	arocli           aroclient.Interface
 	imageregistrycli imageregistryclient.Interface
-
-	aroOperatorDeployer deploy.Operator
 }
 
 // New returns a cluster manager
diff --git a/pkg/cluster/install.go b/pkg/cluster/install.go
index 8980e7c15..1355a4058 100644
--- a/pkg/cluster/install.go
+++ b/pkg/cluster/install.go
@@ -22,7 +22,6 @@ import (
 
 	"github.com/Azure/ARO-RP/pkg/api"
 	aroclient "github.com/Azure/ARO-RP/pkg/operator/clientset/versioned"
-	"github.com/Azure/ARO-RP/pkg/operator/deploy"
 	"github.com/Azure/ARO-RP/pkg/util/restconfig"
 	"github.com/Azure/ARO-RP/pkg/util/steps"
 	"github.com/Azure/ARO-RP/pkg/util/version"
@@ -43,7 +42,6 @@ func (m *manager) adminUpdate() []steps.Step {
 	// don't require a running cluster
 	toRun := []steps.Step{
 		steps.Action(m.initializeKubernetesClients), // must be first
-		steps.Action(m.initializeOperatorDeployer),  // depends on kube clients
 		steps.Action(m.ensureBillingRecord),         // belt and braces
 		steps.Action(m.ensureDefaults),
 		steps.Action(m.fixupClusterSPObjectID),
@@ -109,7 +107,6 @@ func (m *manager) Update(ctx context.Context) error {
 	steps := []steps.Step{
 		steps.AuthorizationRefreshingAction(m.fpAuthorizer, steps.Action(m.validateResources)),
 		steps.Action(m.initializeKubernetesClients), // All init steps are first
-		steps.Action(m.initializeOperatorDeployer),  // depends on kube clients
 		steps.Action(m.initializeClusterSPClients),
 		steps.Action(m.clusterSPObjectID),
 		// credentials rotation flow steps
@@ -166,14 +163,12 @@ func (m *manager) Install(ctx context.Context) error {
 			steps.Action(m.createAPIServerPrivateEndpoint),
 			steps.Action(m.createCertificates),
 			steps.Action(m.initializeKubernetesClients),
-			steps.Action(m.initializeOperatorDeployer), // depends on kube clients
 			steps.Condition(m.bootstrapConfigMapReady, 30*time.Minute, true),
 			steps.Action(m.ensureAROOperator),
 			steps.Action(m.incrInstallPhase),
 		},
 		api.InstallPhaseRemoveBootstrap: {
 			steps.Action(m.initializeKubernetesClients),
-			steps.Action(m.initializeOperatorDeployer), // depends on kube clients
 			steps.Action(m.removeBootstrap),
 			steps.Action(m.removeBootstrapIgnition),
 			steps.Action(m.configureAPIServerCertificate),
@@ -301,13 +296,6 @@ func (m *manager) initializeKubernetesClients(ctx context.Context) error {
 	return err
 }
 
-// initializeKubernetesClients initializes clients which are used
-// once the cluster is up later on in the install process.
-func (m *manager) initializeOperatorDeployer(ctx context.Context) (err error) {
-	m.aroOperatorDeployer, err = deploy.New(m.log, m.env, m.doc.OpenShiftCluster, m.arocli, m.extensionscli, m.kubernetescli)
-	return
-}
-
 // updateProvisionedBy sets the deploying resource provider version in
 // the cluster document for deployment-tracking purposes.
 func (m *manager) updateProvisionedBy(ctx context.Context) error {
diff --git a/pkg/frontend/adminactions/resources_list.go b/pkg/frontend/adminactions/resources_list.go
index c95856101..8b1b1efb9 100644
--- a/pkg/frontend/adminactions/resources_list.go
+++ b/pkg/frontend/adminactions/resources_list.go
@@ -19,7 +19,7 @@ import (
 func (a *azureActions) ResourcesList(ctx context.Context) ([]byte, error) {
 	clusterRGName := stringutils.LastTokenByte(a.oc.Properties.ClusterProfile.ResourceGroupID, '/')
 
-	resources, err := a.resources.ListByResourceGroup(ctx, clusterRGName, "resourceType ne 'Microsoft.Compute/snapshots'", "", nil)
+	resources, err := a.resources.ListByResourceGroup(ctx, clusterRGName, "", "", nil)
 	if err != nil {
 		return nil, err
 	}
diff --git a/pkg/operator/README.md b/pkg/operator/README.md
index ab11044b0..e44291663 100644
--- a/pkg/operator/README.md
+++ b/pkg/operator/README.md
@@ -70,57 +70,6 @@ make generate
 go run -tags aro ./cmd/aro operator master
 ```
 
-### How to create & publish ARO Operator image to ACR/Quay
-
-1. Login to AZ
-  ```bash
-  az login
-  ```
-
-2. Install Docker according to the steps outlined in [Prepare Your Dev Environment](../../docs/prepare-your-dev-environment.md)
-
-3. Publish Image to ACR
-
-   * Pre-requisite:
-     ```
-     ACR created in Azure Portal with Name ${USER}aro
-     2GB+ of Free RAM
-     ```
-
-    * Setup environment variables
-      ```bash
-      export DST_ACR_NAME=${USER}aro
-      export DST_AUTH=$(echo -n '00000000-0000-0000-0000-000000000000:'$(az acr login -n ${DST_ACR_NAME} --expose-token | jq -r .accessToken) | base64 -w0)
-      ```
-
-    * Login to the Azure Container Registry
-      ```bash
-      docker login -u 00000000-0000-0000-0000-000000000000 -p "$(echo $DST_AUTH | base64 -d | cut -d':' -f2)" "${DST_ACR_NAME}.azurecr.io"
-      ```
-
-4. Publish Image to Quay
-
-    * Pre-requisite:
-      ```
-      Quay account with repository created
-      2GB+ of Free RAM
-      ```
-
-    * Setup mirroring environment variables
-      ```bash
-      export DST_QUAY=<quay-user-name>/<repository-name>
-      ```
-
-    * Login to the Quay Registry
-      ```bash
-      docker login quay.io/${DST_QUAY}
-      ```
-
-5. Build and Push ARO Operator Image
-  ```bash
-  make publish-image-aro-multistage
-  ```
-
 ### How to run a custom operator image
 
 Add the following to your "env" before running the rp
diff --git a/pkg/operator/controllers/autosizednodes/autosizednodes_controller.go b/pkg/operator/controllers/autosizednodes/autosizednodes_controller.go
deleted file mode 100644
index 05a0416d8..000000000
--- a/pkg/operator/controllers/autosizednodes/autosizednodes_controller.go
+++ /dev/null
@@ -1,135 +0,0 @@
-package autosizednodes
-
-// Copyright (c) Microsoft Corporation.
-// Licensed under the Apache License 2.0.
-
-import (
-	"context"
-	"fmt"
-	"strings"
-
-	"github.com/coreos/ignition/v2/config/util"
-	mcv1 "github.com/openshift/machine-config-operator/pkg/apis/machineconfiguration.openshift.io/v1"
-	"github.com/sirupsen/logrus"
-	kerrors "k8s.io/apimachinery/pkg/api/errors"
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/types"
-	ctrl "sigs.k8s.io/controller-runtime"
-	"sigs.k8s.io/controller-runtime/pkg/builder"
-	"sigs.k8s.io/controller-runtime/pkg/client"
-	"sigs.k8s.io/controller-runtime/pkg/predicate"
-
-	arov1alpha1 "github.com/Azure/ARO-RP/pkg/operator/apis/aro.openshift.io/v1alpha1"
-)
-
-type Reconciler struct {
-	client client.Client
-
-	log *logrus.Entry
-}
-
-const (
-	ControllerName = "AutoSizedNodes"
-
-	ControllerEnabled = "aro.autosizednodes.enabled"
-	configName        = "dynamic-node"
-)
-
-func NewReconciler(log *logrus.Entry, mgr ctrl.Manager) *Reconciler {
-	return &Reconciler{
-		client: mgr.GetClient(),
-
-		log: log,
-	}
-}
-
-func (r *Reconciler) Reconcile(ctx context.Context, request ctrl.Request) (ctrl.Result, error) {
-	var aro arov1alpha1.Cluster
-	var err error
-
-	err = r.client.Get(ctx, request.NamespacedName, &aro)
-	if err != nil {
-		err = fmt.Errorf("unable to fetch aro cluster: %w", err)
-		return ctrl.Result{}, client.IgnoreNotFound(err)
-	}
-
-	r.log.Infof("Config changed, autoSize: %t\n", aro.Spec.OperatorFlags.GetSimpleBoolean(ControllerEnabled))
-
-	// key is used to locate the object in the etcd
-	key := types.NamespacedName{
-		Name: configName,
-	}
-
-	if !aro.Spec.OperatorFlags.GetSimpleBoolean(ControllerEnabled) {
-		// defaults to deleting the config
-		config := mcv1.KubeletConfig{
-			ObjectMeta: metav1.ObjectMeta{
-				Name: configName,
-			},
-		}
-		err = r.client.Delete(ctx, &config, &client.DeleteOptions{})
-		if err != nil {
-			err = fmt.Errorf("could not delete KubeletConfig: %w", err)
-		}
-		return ctrl.Result{}, client.IgnoreNotFound(err)
-	}
-
-	defaultConfig := makeConfig()
-
-	var config mcv1.KubeletConfig
-	err = r.client.Get(ctx, key, &config)
-	if kerrors.IsNotFound(err) {
-		// If config doesn't exist, create a new one
-		err := r.client.Create(ctx, &defaultConfig, &client.CreateOptions{})
-		if err != nil {
-			err = fmt.Errorf("could not create KubeletConfig: %w", err)
-		}
-		return ctrl.Result{}, err
-	}
-	if err != nil {
-		// If error, return it (controller-runtime will requeue for a retry)
-		return ctrl.Result{}, fmt.Errorf("could not fetch KubeletConfig: %w", err)
-	}
-
-	// If already exists, update the spec
-	config.Spec = defaultConfig.Spec
-	err = r.client.Update(ctx, &config, &client.UpdateOptions{})
-	if err != nil {
-		err = fmt.Errorf("could not update KubeletConfig: %w", err)
-	}
-	return ctrl.Result{}, err
-}
-
-// SetupWithManager prepares the controller with info who to watch
-func (r *Reconciler) SetupWithManager(mgr ctrl.Manager) error {
-	clusterPredicate := predicate.NewPredicateFuncs(func(o client.Object) bool {
-		name := o.GetName()
-		return strings.EqualFold(arov1alpha1.SingletonClusterName, name)
-	})
-
-	b := ctrl.NewControllerManagedBy(mgr).
-		For(&arov1alpha1.Cluster{}, builder.WithPredicates(clusterPredicate))
-
-		// Controller adds ControllerManagedBy to KubeletConfit created by this controller.
-		// Any changes will trigger reconcile, but only for that config.
-	return b.
-		Named(ControllerName).
-		Owns(&mcv1.KubeletConfig{}).
-		Complete(r)
-}
-
-func makeConfig() mcv1.KubeletConfig {
-	return mcv1.KubeletConfig{
-		ObjectMeta: metav1.ObjectMeta{
-			Name: configName,
-		},
-		Spec: mcv1.KubeletConfigSpec{
-			AutoSizingReserved: util.BoolToPtr(true),
-			MachineConfigPoolSelector: &metav1.LabelSelector{
-				MatchLabels: map[string]string{
-					"pools.operator.machineconfiguration.openshift.io/worker": "",
-				},
-			},
-		},
-	}
-}
diff --git a/pkg/operator/controllers/autosizednodes/doc.go b/pkg/operator/controllers/autosizednodes/doc.go
deleted file mode 100644
index 89f784567..000000000
--- a/pkg/operator/controllers/autosizednodes/doc.go
+++ /dev/null
@@ -1,9 +0,0 @@
-package autosizednodes
-
-// Copyright (c) Microsoft Corporation.
-// Licensed under the Apache License 2.0.
-
-// autosizednodes monitors/creates/removes "dynamic-node" KubeletConfig
-// that tells machine-config-operator to turn on auto sized nodes feature
-// the code that is executed by the mco:
-// - https://github.com/openshift/machine-config-operator/blob/fbc4d8e46a7746442f4de3651113d2181d458b12/templates/common/_base/files/kubelet-auto-sizing.yaml
diff --git a/pkg/operator/controllers/workaround/ifreload.go b/pkg/operator/controllers/workaround/ifreload.go
index e78530161..e1a3e18fe 100644
--- a/pkg/operator/controllers/workaround/ifreload.go
+++ b/pkg/operator/controllers/workaround/ifreload.go
@@ -11,7 +11,6 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/client-go/kubernetes"
 
-	arov1alpha1 "github.com/Azure/ARO-RP/pkg/operator/apis/aro.openshift.io/v1alpha1"
 	"github.com/Azure/ARO-RP/pkg/util/version"
 )
 
@@ -37,7 +36,7 @@ func (*ifReload) Name() string {
 	return "ifReload"
 }
 
-func (i *ifReload) IsRequired(clusterVersion *version.Version, cluster *arov1alpha1.Cluster) bool {
+func (i *ifReload) IsRequired(clusterVersion *version.Version) bool {
 	return clusterVersion.Lt(i.versionFixed)
 }
 
diff --git a/pkg/operator/controllers/workaround/systemreserved.go b/pkg/operator/controllers/workaround/systemreserved.go
index 8fd12f42f..0f4b5d65c 100644
--- a/pkg/operator/controllers/workaround/systemreserved.go
+++ b/pkg/operator/controllers/workaround/systemreserved.go
@@ -14,8 +14,6 @@ import (
 	kruntime "k8s.io/apimachinery/pkg/runtime"
 	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
 
-	arov1alpha1 "github.com/Azure/ARO-RP/pkg/operator/apis/aro.openshift.io/v1alpha1"
-	"github.com/Azure/ARO-RP/pkg/operator/controllers/autosizednodes"
 	"github.com/Azure/ARO-RP/pkg/util/dynamichelper"
 	"github.com/Azure/ARO-RP/pkg/util/version"
 )
@@ -47,10 +45,7 @@ func (sr *systemreserved) Name() string {
 	return "SystemReserved fix for bz-1857446"
 }
 
-func (sr *systemreserved) IsRequired(clusterVersion *version.Version, cluster *arov1alpha1.Cluster) bool {
-	if cluster.Spec.OperatorFlags.GetSimpleBoolean(autosizednodes.ControllerEnabled) {
-		return false
-	}
+func (sr *systemreserved) IsRequired(clusterVersion *version.Version) bool {
 	return clusterVersion.Lt(sr.versionFixed)
 }
 
diff --git a/pkg/operator/controllers/workaround/workaround.go b/pkg/operator/controllers/workaround/workaround.go
index 3d0752664..e3a88db1b 100644
--- a/pkg/operator/controllers/workaround/workaround.go
+++ b/pkg/operator/controllers/workaround/workaround.go
@@ -6,7 +6,6 @@ package workaround
 import (
 	"context"
 
-	arov1alpha1 "github.com/Azure/ARO-RP/pkg/operator/apis/aro.openshift.io/v1alpha1"
 	"github.com/Azure/ARO-RP/pkg/util/version"
 )
 
@@ -15,7 +14,7 @@ type Workaround interface {
 	Name() string
 	// IsRequired returns true when the clusterversion is indicates that the cluster
 	// is effected by the bug that the workaround fixes.
-	IsRequired(clusterVersion *version.Version, cluster *arov1alpha1.Cluster) bool
+	IsRequired(clusterVersion *version.Version) bool
 	// Ensure will apply the workaround to the cluster.
 	Ensure(context.Context) error
 	// Remove will remove the workaround from the cluster
diff --git a/pkg/operator/controllers/workaround/workaround_controller.go b/pkg/operator/controllers/workaround/workaround_controller.go
index 7ff96d571..c538a7456 100644
--- a/pkg/operator/controllers/workaround/workaround_controller.go
+++ b/pkg/operator/controllers/workaround/workaround_controller.go
@@ -76,7 +76,7 @@ func (r *Reconciler) Reconcile(ctx context.Context, request ctrl.Request) (ctrl.
 	}
 
 	for _, wa := range r.workarounds {
-		if wa.IsRequired(clusterVersion, instance) {
+		if wa.IsRequired(clusterVersion) {
 			err = wa.Ensure(ctx)
 		} else {
 			err = wa.Remove(ctx)
diff --git a/pkg/operator/deploy/generate.go b/pkg/operator/deploy/generate.go
deleted file mode 100644
index 43f129fd6..000000000
--- a/pkg/operator/deploy/generate.go
+++ /dev/null
@@ -1,8 +0,0 @@
-package deploy
-
-// Copyright (c) Microsoft Corporation.
-// Licensed under the Apache License 2.0.
-
-//go:generate rm -rf ../../util/mocks/operator/$GOPACKAGE
-//go:generate go run ../../../vendor/github.com/golang/mock/mockgen -destination=../../util/mocks/operator/$GOPACKAGE/$GOPACKAGE.go github.com/Azure/ARO-RP/pkg/operator/$GOPACKAGE Operator
-//go:generate go run ../../../vendor/golang.org/x/tools/cmd/goimports -local=github.com/Azure/ARO-RP -e -w ../../util/mocks/operator/$GOPACKAGE/$GOPACKAGE.go
diff --git a/pkg/util/mocks/operator/controllers/workaround/workaround.go b/pkg/util/mocks/operator/controllers/workaround/workaround.go
index 01186df68..04f21b236 100644
--- a/pkg/util/mocks/operator/controllers/workaround/workaround.go
+++ b/pkg/util/mocks/operator/controllers/workaround/workaround.go
@@ -10,7 +10,6 @@ import (
 
 	gomock "github.com/golang/mock/gomock"
 
-	v1alpha1 "github.com/Azure/ARO-RP/pkg/operator/apis/aro.openshift.io/v1alpha1"
 	version "github.com/Azure/ARO-RP/pkg/util/version"
 )
 
@@ -52,17 +51,17 @@ func (mr *MockWorkaroundMockRecorder) Ensure(arg0 interface{}) *gomock.Call {
 }
 
 // IsRequired mocks base method.
-func (m *MockWorkaround) IsRequired(arg0 *version.Version, arg1 *v1alpha1.Cluster) bool {
+func (m *MockWorkaround) IsRequired(arg0 *version.Version) bool {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "IsRequired", arg0, arg1)
+	ret := m.ctrl.Call(m, "IsRequired", arg0)
 	ret0, _ := ret[0].(bool)
 	return ret0
 }
 
 // IsRequired indicates an expected call of IsRequired.
-func (mr *MockWorkaroundMockRecorder) IsRequired(arg0, arg1 interface{}) *gomock.Call {
+func (mr *MockWorkaroundMockRecorder) IsRequired(arg0 interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "IsRequired", reflect.TypeOf((*MockWorkaround)(nil).IsRequired), arg0, arg1)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "IsRequired", reflect.TypeOf((*MockWorkaround)(nil).IsRequired), arg0)
 }
 
 // Name mocks base method.
diff --git a/pkg/util/mocks/operator/deploy/deploy.go b/pkg/util/mocks/operator/deploy/deploy.go
deleted file mode 100644
index 211c4622c..000000000
--- a/pkg/util/mocks/operator/deploy/deploy.go
+++ /dev/null
@@ -1,64 +0,0 @@
-// Code generated by MockGen. DO NOT EDIT.
-// Source: github.com/Azure/ARO-RP/pkg/operator/deploy (interfaces: Operator)
-
-// Package mock_deploy is a generated GoMock package.
-package mock_deploy
-
-import (
-	context "context"
-	reflect "reflect"
-
-	gomock "github.com/golang/mock/gomock"
-)
-
-// MockOperator is a mock of Operator interface.
-type MockOperator struct {
-	ctrl     *gomock.Controller
-	recorder *MockOperatorMockRecorder
-}
-
-// MockOperatorMockRecorder is the mock recorder for MockOperator.
-type MockOperatorMockRecorder struct {
-	mock *MockOperator
-}
-
-// NewMockOperator creates a new mock instance.
-func NewMockOperator(ctrl *gomock.Controller) *MockOperator {
-	mock := &MockOperator{ctrl: ctrl}
-	mock.recorder = &MockOperatorMockRecorder{mock}
-	return mock
-}
-
-// EXPECT returns an object that allows the caller to indicate expected use.
-func (m *MockOperator) EXPECT() *MockOperatorMockRecorder {
-	return m.recorder
-}
-
-// CreateOrUpdate mocks base method.
-func (m *MockOperator) CreateOrUpdate(arg0 context.Context) error {
-	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "CreateOrUpdate", arg0)
-	ret0, _ := ret[0].(error)
-	return ret0
-}
-
-// CreateOrUpdate indicates an expected call of CreateOrUpdate.
-func (mr *MockOperatorMockRecorder) CreateOrUpdate(arg0 interface{}) *gomock.Call {
-	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CreateOrUpdate", reflect.TypeOf((*MockOperator)(nil).CreateOrUpdate), arg0)
-}
-
-// IsReady mocks base method.
-func (m *MockOperator) IsReady(arg0 context.Context) (bool, error) {
-	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "IsReady", arg0)
-	ret0, _ := ret[0].(bool)
-	ret1, _ := ret[1].(error)
-	return ret0, ret1
-}
-
-// IsReady indicates an expected call of IsReady.
-func (mr *MockOperatorMockRecorder) IsReady(arg0 interface{}) *gomock.Call {
-	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "IsReady", reflect.TypeOf((*MockOperator)(nil).IsReady), arg0)
-}
diff --git a/vendor/github.com/jewzaam/go-cosmosdb/pkg/gencosmosdb/bindata.go b/vendor/github.com/jewzaam/go-cosmosdb/pkg/gencosmosdb/bindata.go
index b3dc1ea67..3d3713931 100644
--- a/vendor/github.com/jewzaam/go-cosmosdb/pkg/gencosmosdb/bindata.go
+++ b/vendor/github.com/jewzaam/go-cosmosdb/pkg/gencosmosdb/bindata.go
@@ -1,5 +1,6 @@
-// Code generated for package gencosmosdb by go-bindata DO NOT EDIT. (@generated)
-// sources:
+// Code generated by go-bindata. (@generated) DO NOT EDIT.
+
+ //Package gencosmosdb generated by go-bindata.// sources:
 // cosmosdb/authorizer.go
 // cosmosdb/collection.go
 // cosmosdb/cosmosdb.go
@@ -27,7 +28,7 @@ import (
 func bindataRead(data []byte, name string) ([]byte, error) {
 	gz, err := gzip.NewReader(bytes.NewBuffer(data))
 	if err != nil {
-		return nil, fmt.Errorf("Read %q: %v", name, err)
+		return nil, fmt.Errorf("read %q: %v", name, err)
 	}
 
 	var buf bytes.Buffer
@@ -35,7 +36,7 @@ func bindataRead(data []byte, name string) ([]byte, error) {
 	clErr := gz.Close()
 
 	if err != nil {
-		return nil, fmt.Errorf("Read %q: %v", name, err)
+		return nil, fmt.Errorf("read %q: %v", name, err)
 	}
 	if clErr != nil {
 		return nil, err
@@ -71,7 +72,7 @@ func (fi bindataFileInfo) Mode() os.FileMode {
 	return fi.mode
 }
 
-// Mode return file modify time
+// ModTime return file modify time
 func (fi bindataFileInfo) ModTime() time.Time {
 	return fi.modTime
 }
diff --git a/vendor/github.com/jewzaam/go-cosmosdb/pkg/gencosmosdb/generate.go b/vendor/github.com/jewzaam/go-cosmosdb/pkg/gencosmosdb/generate.go
index 309f689a7..905d3cde6 100644
--- a/vendor/github.com/jewzaam/go-cosmosdb/pkg/gencosmosdb/generate.go
+++ b/vendor/github.com/jewzaam/go-cosmosdb/pkg/gencosmosdb/generate.go
@@ -1,3 +1,3 @@
 package gencosmosdb
 
-//go:generate go run ../../vendor/github.com/go-bindata/go-bindata/go-bindata -pkg gencosmosdb -prefix cosmosdb -nometadata cosmosdb
+//go:generate go-bindata -pkg gencosmosdb -prefix cosmosdb -nometadata cosmosdb
diff --git a/vendor/modules.txt b/vendor/modules.txt
index 59ef9cc88..175507630 100644
--- a/vendor/modules.txt
+++ b/vendor/modules.txt
@@ -612,7 +612,7 @@ github.com/hashicorp/hcl/json/token
 github.com/imdario/mergo
 # github.com/inconshreveable/mousetrap v1.0.0
 github.com/inconshreveable/mousetrap
-# github.com/jewzaam/go-cosmosdb v0.0.0-20220315232836-282b67c5b234
+# github.com/jewzaam/go-cosmosdb v0.0.0-20211018134417-8d1494e7862f
 ## explicit
 github.com/jewzaam/go-cosmosdb/cmd/gencosmosdb
 github.com/jewzaam/go-cosmosdb/pkg/gencosmosdb
@@ -1928,7 +1928,6 @@ sigs.k8s.io/controller-runtime/pkg/certwatcher
 sigs.k8s.io/controller-runtime/pkg/client
 sigs.k8s.io/controller-runtime/pkg/client/apiutil
 sigs.k8s.io/controller-runtime/pkg/client/config
-sigs.k8s.io/controller-runtime/pkg/client/fake/
 sigs.k8s.io/controller-runtime/pkg/cluster
 sigs.k8s.io/controller-runtime/pkg/config
 sigs.k8s.io/controller-runtime/pkg/config/v1alpha1
diff --git a/vendor/sigs.k8s.io/controller-runtime/pkg/client/fake/client.go b/vendor/sigs.k8s.io/controller-runtime/pkg/client/fake/client.go
deleted file mode 100644
index ded8a60d3..000000000
--- a/vendor/sigs.k8s.io/controller-runtime/pkg/client/fake/client.go
+++ /dev/null
@@ -1,675 +0,0 @@
-/*
-Copyright 2018 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package fake
-
-import (
-	"context"
-	"encoding/json"
-	"errors"
-	"fmt"
-	"strconv"
-	"strings"
-	"sync"
-
-	apierrors "k8s.io/apimachinery/pkg/api/errors"
-	"k8s.io/apimachinery/pkg/api/meta"
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
-	"k8s.io/apimachinery/pkg/runtime"
-	"k8s.io/apimachinery/pkg/runtime/schema"
-	utilrand "k8s.io/apimachinery/pkg/util/rand"
-	"k8s.io/apimachinery/pkg/util/validation/field"
-	"k8s.io/apimachinery/pkg/watch"
-	"k8s.io/client-go/kubernetes/scheme"
-	"k8s.io/client-go/testing"
-
-	"sigs.k8s.io/controller-runtime/pkg/client"
-	"sigs.k8s.io/controller-runtime/pkg/client/apiutil"
-	"sigs.k8s.io/controller-runtime/pkg/internal/objectutil"
-)
-
-type versionedTracker struct {
-	testing.ObjectTracker
-	scheme *runtime.Scheme
-}
-
-type fakeClient struct {
-	tracker         versionedTracker
-	scheme          *runtime.Scheme
-	schemeWriteLock sync.Mutex
-}
-
-var _ client.WithWatch = &fakeClient{}
-
-const (
-	maxNameLength          = 63
-	randomLength           = 5
-	maxGeneratedNameLength = maxNameLength - randomLength
-)
-
-// NewFakeClient creates a new fake client for testing.
-// You can choose to initialize it with a slice of runtime.Object.
-//
-// Deprecated: Please use NewClientBuilder instead.
-func NewFakeClient(initObjs ...runtime.Object) client.WithWatch {
-	return NewClientBuilder().WithRuntimeObjects(initObjs...).Build()
-}
-
-// NewFakeClientWithScheme creates a new fake client with the given scheme
-// for testing.
-// You can choose to initialize it with a slice of runtime.Object.
-//
-// Deprecated: Please use NewClientBuilder instead.
-func NewFakeClientWithScheme(clientScheme *runtime.Scheme, initObjs ...runtime.Object) client.WithWatch {
-	return NewClientBuilder().WithScheme(clientScheme).WithRuntimeObjects(initObjs...).Build()
-}
-
-// NewClientBuilder returns a new builder to create a fake client.
-func NewClientBuilder() *ClientBuilder {
-	return &ClientBuilder{}
-}
-
-// ClientBuilder builds a fake client.
-type ClientBuilder struct {
-	scheme             *runtime.Scheme
-	initObject         []client.Object
-	initLists          []client.ObjectList
-	initRuntimeObjects []runtime.Object
-}
-
-// WithScheme sets this builder's internal scheme.
-// If not set, defaults to client-go's global scheme.Scheme.
-func (f *ClientBuilder) WithScheme(scheme *runtime.Scheme) *ClientBuilder {
-	f.scheme = scheme
-	return f
-}
-
-// WithObjects can be optionally used to initialize this fake client with client.Object(s).
-func (f *ClientBuilder) WithObjects(initObjs ...client.Object) *ClientBuilder {
-	f.initObject = append(f.initObject, initObjs...)
-	return f
-}
-
-// WithLists can be optionally used to initialize this fake client with client.ObjectList(s).
-func (f *ClientBuilder) WithLists(initLists ...client.ObjectList) *ClientBuilder {
-	f.initLists = append(f.initLists, initLists...)
-	return f
-}
-
-// WithRuntimeObjects can be optionally used to initialize this fake client with runtime.Object(s).
-func (f *ClientBuilder) WithRuntimeObjects(initRuntimeObjs ...runtime.Object) *ClientBuilder {
-	f.initRuntimeObjects = append(f.initRuntimeObjects, initRuntimeObjs...)
-	return f
-}
-
-// Build builds and returns a new fake client.
-func (f *ClientBuilder) Build() client.WithWatch {
-	if f.scheme == nil {
-		f.scheme = scheme.Scheme
-	}
-
-	tracker := versionedTracker{ObjectTracker: testing.NewObjectTracker(f.scheme, scheme.Codecs.UniversalDecoder()), scheme: f.scheme}
-	for _, obj := range f.initObject {
-		if err := tracker.Add(obj); err != nil {
-			panic(fmt.Errorf("failed to add object %v to fake client: %w", obj, err))
-		}
-	}
-	for _, obj := range f.initLists {
-		if err := tracker.Add(obj); err != nil {
-			panic(fmt.Errorf("failed to add list %v to fake client: %w", obj, err))
-		}
-	}
-	for _, obj := range f.initRuntimeObjects {
-		if err := tracker.Add(obj); err != nil {
-			panic(fmt.Errorf("failed to add runtime object %v to fake client: %w", obj, err))
-		}
-	}
-	return &fakeClient{
-		tracker: tracker,
-		scheme:  f.scheme,
-	}
-}
-
-const trackerAddResourceVersion = "999"
-
-func (t versionedTracker) Add(obj runtime.Object) error {
-	var objects []runtime.Object
-	if meta.IsListType(obj) {
-		var err error
-		objects, err = meta.ExtractList(obj)
-		if err != nil {
-			return err
-		}
-	} else {
-		objects = []runtime.Object{obj}
-	}
-	for _, obj := range objects {
-		accessor, err := meta.Accessor(obj)
-		if err != nil {
-			return fmt.Errorf("failed to get accessor for object: %w", err)
-		}
-		if accessor.GetResourceVersion() == "" {
-			// We use a "magic" value of 999 here because this field
-			// is parsed as uint and and 0 is already used in Update.
-			// As we can't go lower, go very high instead so this can
-			// be recognized
-			accessor.SetResourceVersion(trackerAddResourceVersion)
-		}
-		if err := t.ObjectTracker.Add(obj); err != nil {
-			return err
-		}
-	}
-
-	return nil
-}
-
-func (t versionedTracker) Create(gvr schema.GroupVersionResource, obj runtime.Object, ns string) error {
-	accessor, err := meta.Accessor(obj)
-	if err != nil {
-		return fmt.Errorf("failed to get accessor for object: %v", err)
-	}
-	if accessor.GetName() == "" {
-		return apierrors.NewInvalid(
-			obj.GetObjectKind().GroupVersionKind().GroupKind(),
-			accessor.GetName(),
-			field.ErrorList{field.Required(field.NewPath("metadata.name"), "name is required")})
-	}
-	if accessor.GetResourceVersion() != "" {
-		return apierrors.NewBadRequest("resourceVersion can not be set for Create requests")
-	}
-	accessor.SetResourceVersion("1")
-	if err := t.ObjectTracker.Create(gvr, obj, ns); err != nil {
-		accessor.SetResourceVersion("")
-		return err
-	}
-	return nil
-}
-
-func (t versionedTracker) Update(gvr schema.GroupVersionResource, obj runtime.Object, ns string) error {
-	accessor, err := meta.Accessor(obj)
-	if err != nil {
-		return fmt.Errorf("failed to get accessor for object: %v", err)
-	}
-
-	if accessor.GetName() == "" {
-		return apierrors.NewInvalid(
-			obj.GetObjectKind().GroupVersionKind().GroupKind(),
-			accessor.GetName(),
-			field.ErrorList{field.Required(field.NewPath("metadata.name"), "name is required")})
-	}
-
-	gvk := obj.GetObjectKind().GroupVersionKind()
-	if gvk.Empty() {
-		gvk, err = apiutil.GVKForObject(obj, t.scheme)
-		if err != nil {
-			return err
-		}
-	}
-
-	oldObject, err := t.ObjectTracker.Get(gvr, ns, accessor.GetName())
-	if err != nil {
-		// If the resource is not found and the resource allows create on update, issue a
-		// create instead.
-		if apierrors.IsNotFound(err) && allowsCreateOnUpdate(gvk) {
-			return t.Create(gvr, obj, ns)
-		}
-		return err
-	}
-
-	oldAccessor, err := meta.Accessor(oldObject)
-	if err != nil {
-		return err
-	}
-
-	// If the new object does not have the resource version set and it allows unconditional update,
-	// default it to the resource version of the existing resource
-	if accessor.GetResourceVersion() == "" && allowsUnconditionalUpdate(gvk) {
-		accessor.SetResourceVersion(oldAccessor.GetResourceVersion())
-	}
-	if accessor.GetResourceVersion() != oldAccessor.GetResourceVersion() {
-		return apierrors.NewConflict(gvr.GroupResource(), accessor.GetName(), errors.New("object was modified"))
-	}
-	if oldAccessor.GetResourceVersion() == "" {
-		oldAccessor.SetResourceVersion("0")
-	}
-	intResourceVersion, err := strconv.ParseUint(oldAccessor.GetResourceVersion(), 10, 64)
-	if err != nil {
-		return fmt.Errorf("can not convert resourceVersion %q to int: %v", oldAccessor.GetResourceVersion(), err)
-	}
-	intResourceVersion++
-	accessor.SetResourceVersion(strconv.FormatUint(intResourceVersion, 10))
-	if !accessor.GetDeletionTimestamp().IsZero() && len(accessor.GetFinalizers()) == 0 {
-		return t.ObjectTracker.Delete(gvr, accessor.GetNamespace(), accessor.GetName())
-	}
-	return t.ObjectTracker.Update(gvr, obj, ns)
-}
-
-func (c *fakeClient) Get(ctx context.Context, key client.ObjectKey, obj client.Object) error {
-	gvr, err := getGVRFromObject(obj, c.scheme)
-	if err != nil {
-		return err
-	}
-	o, err := c.tracker.Get(gvr, key.Namespace, key.Name)
-	if err != nil {
-		return err
-	}
-
-	gvk, err := apiutil.GVKForObject(obj, c.scheme)
-	if err != nil {
-		return err
-	}
-	ta, err := meta.TypeAccessor(o)
-	if err != nil {
-		return err
-	}
-	ta.SetKind(gvk.Kind)
-	ta.SetAPIVersion(gvk.GroupVersion().String())
-
-	j, err := json.Marshal(o)
-	if err != nil {
-		return err
-	}
-	decoder := scheme.Codecs.UniversalDecoder()
-	_, _, err = decoder.Decode(j, nil, obj)
-	return err
-}
-
-func (c *fakeClient) Watch(ctx context.Context, list client.ObjectList, opts ...client.ListOption) (watch.Interface, error) {
-	gvk, err := apiutil.GVKForObject(list, c.scheme)
-	if err != nil {
-		return nil, err
-	}
-
-	if strings.HasSuffix(gvk.Kind, "List") {
-		gvk.Kind = gvk.Kind[:len(gvk.Kind)-4]
-	}
-
-	listOpts := client.ListOptions{}
-	listOpts.ApplyOptions(opts)
-
-	gvr, _ := meta.UnsafeGuessKindToResource(gvk)
-	return c.tracker.Watch(gvr, listOpts.Namespace)
-}
-
-func (c *fakeClient) List(ctx context.Context, obj client.ObjectList, opts ...client.ListOption) error {
-	gvk, err := apiutil.GVKForObject(obj, c.scheme)
-	if err != nil {
-		return err
-	}
-
-	originalKind := gvk.Kind
-
-	if strings.HasSuffix(gvk.Kind, "List") {
-		gvk.Kind = gvk.Kind[:len(gvk.Kind)-4]
-	}
-
-	if _, isUnstructuredList := obj.(*unstructured.UnstructuredList); isUnstructuredList && !c.scheme.Recognizes(gvk) {
-		// We need tor register the ListKind with UnstructuredList:
-		// https://github.com/kubernetes/kubernetes/blob/7b2776b89fb1be28d4e9203bdeec079be903c103/staging/src/k8s.io/client-go/dynamic/fake/simple.go#L44-L51
-		c.schemeWriteLock.Lock()
-		c.scheme.AddKnownTypeWithName(gvk.GroupVersion().WithKind(gvk.Kind+"List"), &unstructured.UnstructuredList{})
-		c.schemeWriteLock.Unlock()
-	}
-
-	listOpts := client.ListOptions{}
-	listOpts.ApplyOptions(opts)
-
-	gvr, _ := meta.UnsafeGuessKindToResource(gvk)
-	o, err := c.tracker.List(gvr, gvk, listOpts.Namespace)
-	if err != nil {
-		return err
-	}
-
-	ta, err := meta.TypeAccessor(o)
-	if err != nil {
-		return err
-	}
-	ta.SetKind(originalKind)
-	ta.SetAPIVersion(gvk.GroupVersion().String())
-
-	j, err := json.Marshal(o)
-	if err != nil {
-		return err
-	}
-	decoder := scheme.Codecs.UniversalDecoder()
-	_, _, err = decoder.Decode(j, nil, obj)
-	if err != nil {
-		return err
-	}
-
-	if listOpts.LabelSelector != nil {
-		objs, err := meta.ExtractList(obj)
-		if err != nil {
-			return err
-		}
-		filteredObjs, err := objectutil.FilterWithLabels(objs, listOpts.LabelSelector)
-		if err != nil {
-			return err
-		}
-		err = meta.SetList(obj, filteredObjs)
-		if err != nil {
-			return err
-		}
-	}
-	return nil
-}
-
-func (c *fakeClient) Scheme() *runtime.Scheme {
-	return c.scheme
-}
-
-func (c *fakeClient) RESTMapper() meta.RESTMapper {
-	// TODO: Implement a fake RESTMapper.
-	return nil
-}
-
-func (c *fakeClient) Create(ctx context.Context, obj client.Object, opts ...client.CreateOption) error {
-	createOptions := &client.CreateOptions{}
-	createOptions.ApplyOptions(opts)
-
-	for _, dryRunOpt := range createOptions.DryRun {
-		if dryRunOpt == metav1.DryRunAll {
-			return nil
-		}
-	}
-
-	gvr, err := getGVRFromObject(obj, c.scheme)
-	if err != nil {
-		return err
-	}
-	accessor, err := meta.Accessor(obj)
-	if err != nil {
-		return err
-	}
-
-	if accessor.GetName() == "" && accessor.GetGenerateName() != "" {
-		base := accessor.GetGenerateName()
-		if len(base) > maxGeneratedNameLength {
-			base = base[:maxGeneratedNameLength]
-		}
-		accessor.SetName(fmt.Sprintf("%s%s", base, utilrand.String(randomLength)))
-	}
-
-	return c.tracker.Create(gvr, obj, accessor.GetNamespace())
-}
-
-func (c *fakeClient) Delete(ctx context.Context, obj client.Object, opts ...client.DeleteOption) error {
-	gvr, err := getGVRFromObject(obj, c.scheme)
-	if err != nil {
-		return err
-	}
-	accessor, err := meta.Accessor(obj)
-	if err != nil {
-		return err
-	}
-	delOptions := client.DeleteOptions{}
-	delOptions.ApplyOptions(opts)
-
-	return c.deleteObject(gvr, accessor)
-}
-
-func (c *fakeClient) DeleteAllOf(ctx context.Context, obj client.Object, opts ...client.DeleteAllOfOption) error {
-	gvk, err := apiutil.GVKForObject(obj, c.scheme)
-	if err != nil {
-		return err
-	}
-
-	dcOptions := client.DeleteAllOfOptions{}
-	dcOptions.ApplyOptions(opts)
-
-	gvr, _ := meta.UnsafeGuessKindToResource(gvk)
-	o, err := c.tracker.List(gvr, gvk, dcOptions.Namespace)
-	if err != nil {
-		return err
-	}
-
-	objs, err := meta.ExtractList(o)
-	if err != nil {
-		return err
-	}
-	filteredObjs, err := objectutil.FilterWithLabels(objs, dcOptions.LabelSelector)
-	if err != nil {
-		return err
-	}
-	for _, o := range filteredObjs {
-		accessor, err := meta.Accessor(o)
-		if err != nil {
-			return err
-		}
-		err = c.deleteObject(gvr, accessor)
-		if err != nil {
-			return err
-		}
-	}
-	return nil
-}
-
-func (c *fakeClient) Update(ctx context.Context, obj client.Object, opts ...client.UpdateOption) error {
-	updateOptions := &client.UpdateOptions{}
-	updateOptions.ApplyOptions(opts)
-
-	for _, dryRunOpt := range updateOptions.DryRun {
-		if dryRunOpt == metav1.DryRunAll {
-			return nil
-		}
-	}
-
-	gvr, err := getGVRFromObject(obj, c.scheme)
-	if err != nil {
-		return err
-	}
-	accessor, err := meta.Accessor(obj)
-	if err != nil {
-		return err
-	}
-	return c.tracker.Update(gvr, obj, accessor.GetNamespace())
-}
-
-func (c *fakeClient) Patch(ctx context.Context, obj client.Object, patch client.Patch, opts ...client.PatchOption) error {
-	patchOptions := &client.PatchOptions{}
-	patchOptions.ApplyOptions(opts)
-
-	for _, dryRunOpt := range patchOptions.DryRun {
-		if dryRunOpt == metav1.DryRunAll {
-			return nil
-		}
-	}
-
-	gvr, err := getGVRFromObject(obj, c.scheme)
-	if err != nil {
-		return err
-	}
-	accessor, err := meta.Accessor(obj)
-	if err != nil {
-		return err
-	}
-	data, err := patch.Data(obj)
-	if err != nil {
-		return err
-	}
-
-	reaction := testing.ObjectReaction(c.tracker)
-	handled, o, err := reaction(testing.NewPatchAction(gvr, accessor.GetNamespace(), accessor.GetName(), patch.Type(), data))
-	if err != nil {
-		return err
-	}
-	if !handled {
-		panic("tracker could not handle patch method")
-	}
-
-	gvk, err := apiutil.GVKForObject(obj, c.scheme)
-	if err != nil {
-		return err
-	}
-	ta, err := meta.TypeAccessor(o)
-	if err != nil {
-		return err
-	}
-	ta.SetKind(gvk.Kind)
-	ta.SetAPIVersion(gvk.GroupVersion().String())
-
-	j, err := json.Marshal(o)
-	if err != nil {
-		return err
-	}
-	decoder := scheme.Codecs.UniversalDecoder()
-	_, _, err = decoder.Decode(j, nil, obj)
-	return err
-}
-
-func (c *fakeClient) Status() client.StatusWriter {
-	return &fakeStatusWriter{client: c}
-}
-
-func (c *fakeClient) deleteObject(gvr schema.GroupVersionResource, accessor metav1.Object) error {
-	old, err := c.tracker.Get(gvr, accessor.GetNamespace(), accessor.GetName())
-	if err == nil {
-		oldAccessor, err := meta.Accessor(old)
-		if err == nil {
-			if len(oldAccessor.GetFinalizers()) > 0 {
-				now := metav1.Now()
-				oldAccessor.SetDeletionTimestamp(&now)
-				return c.tracker.Update(gvr, old, accessor.GetNamespace())
-			}
-		}
-	}
-
-	//TODO: implement propagation
-	return c.tracker.Delete(gvr, accessor.GetNamespace(), accessor.GetName())
-}
-
-func getGVRFromObject(obj runtime.Object, scheme *runtime.Scheme) (schema.GroupVersionResource, error) {
-	gvk, err := apiutil.GVKForObject(obj, scheme)
-	if err != nil {
-		return schema.GroupVersionResource{}, err
-	}
-	gvr, _ := meta.UnsafeGuessKindToResource(gvk)
-	return gvr, nil
-}
-
-type fakeStatusWriter struct {
-	client *fakeClient
-}
-
-func (sw *fakeStatusWriter) Update(ctx context.Context, obj client.Object, opts ...client.UpdateOption) error {
-	// TODO(droot): This results in full update of the obj (spec + status). Need
-	// a way to update status field only.
-	return sw.client.Update(ctx, obj, opts...)
-}
-
-func (sw *fakeStatusWriter) Patch(ctx context.Context, obj client.Object, patch client.Patch, opts ...client.PatchOption) error {
-	// TODO(droot): This results in full update of the obj (spec + status). Need
-	// a way to update status field only.
-	return sw.client.Patch(ctx, obj, patch, opts...)
-}
-
-func allowsUnconditionalUpdate(gvk schema.GroupVersionKind) bool {
-	switch gvk.Group {
-	case "apps":
-		switch gvk.Kind {
-		case "ControllerRevision", "DaemonSet", "Deployment", "ReplicaSet", "StatefulSet":
-			return true
-		}
-	case "autoscaling":
-		switch gvk.Kind {
-		case "HorizontalPodAutoscaler":
-			return true
-		}
-	case "batch":
-		switch gvk.Kind {
-		case "CronJob", "Job":
-			return true
-		}
-	case "certificates":
-		switch gvk.Kind {
-		case "Certificates":
-			return true
-		}
-	case "flowcontrol":
-		switch gvk.Kind {
-		case "FlowSchema", "PriorityLevelConfiguration":
-			return true
-		}
-	case "networking":
-		switch gvk.Kind {
-		case "Ingress", "IngressClass", "NetworkPolicy":
-			return true
-		}
-	case "policy":
-		switch gvk.Kind {
-		case "PodSecurityPolicy":
-			return true
-		}
-	case "rbac":
-		switch gvk.Kind {
-		case "ClusterRole", "ClusterRoleBinding", "Role", "RoleBinding":
-			return true
-		}
-	case "scheduling":
-		switch gvk.Kind {
-		case "PriorityClass":
-			return true
-		}
-	case "settings":
-		switch gvk.Kind {
-		case "PodPreset":
-			return true
-		}
-	case "storage":
-		switch gvk.Kind {
-		case "StorageClass":
-			return true
-		}
-	case "":
-		switch gvk.Kind {
-		case "ConfigMap", "Endpoint", "Event", "LimitRange", "Namespace", "Node",
-			"PersistentVolume", "PersistentVolumeClaim", "Pod", "PodTemplate",
-			"ReplicationController", "ResourceQuota", "Secret", "Service",
-			"ServiceAccount", "EndpointSlice":
-			return true
-		}
-	}
-
-	return false
-}
-
-func allowsCreateOnUpdate(gvk schema.GroupVersionKind) bool {
-	switch gvk.Group {
-	case "coordination":
-		switch gvk.Kind {
-		case "Lease":
-			return true
-		}
-	case "node":
-		switch gvk.Kind {
-		case "RuntimeClass":
-			return true
-		}
-	case "rbac":
-		switch gvk.Kind {
-		case "ClusterRole", "ClusterRoleBinding", "Role", "RoleBinding":
-			return true
-		}
-	case "":
-		switch gvk.Kind {
-		case "Endpoint", "Event", "LimitRange", "Service":
-			return true
-		}
-	}
-
-	return false
-}
diff --git a/vendor/sigs.k8s.io/controller-runtime/pkg/client/fake/doc.go b/vendor/sigs.k8s.io/controller-runtime/pkg/client/fake/doc.go
deleted file mode 100644
index 7d680690d..000000000
--- a/vendor/sigs.k8s.io/controller-runtime/pkg/client/fake/doc.go
+++ /dev/null
@@ -1,39 +0,0 @@
-/*
-Copyright 2018 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-/*
-Package fake provides a fake client for testing.
-
-A fake client is backed by its simple object store indexed by GroupVersionResource.
-You can create a fake client with optional objects.
-
-	client := NewFakeClientWithScheme(scheme, initObjs...) // initObjs is a slice of runtime.Object
-
-You can invoke the methods defined in the Client interface.
-
-When in doubt, it's almost always better not to use this package and instead use
-envtest.Environment with a real client and API server.
-
-WARNING: ⚠️ Current Limitations / Known Issues with the fake Client ⚠️
-- This client does not have a way to inject specific errors to test handled vs. unhandled errors.
-- There is some support for sub resources which can cause issues with tests if you're trying to update
-  e.g. metadata and status in the same reconcile.
-- No OpeanAPI validation is performed when creating or updating objects.
-- ObjectMeta's `Generation` and `ResourceVersion` don't behave properly, Patch or Update
-operations that rely on these fields will fail, or give false positives.
-
-*/
-package fake


Test patch:
diff --git a/docs/testing.md b/docs/testing.md
index 9f0717662..cac756424 100644
--- a/docs/testing.md
+++ b/docs/testing.md
@@ -42,8 +42,6 @@ E2e tests can also be run locally as follows:
 - Run the `make test-e2e` target
 - Delete the cosmos database, if applicable
 
-You can also modify the flags passed to the e2e.test run by setting the E2E_FLAGS environment variable before running `make test-e2e`.
-
 These steps can be acheived using commands below.  Look at the [e2e helper
 file](../hack/e2e/run-rp-and-e2e.sh) to understand each of the bash functions
 below.
diff --git a/pkg/cluster/adminupdate_test.go b/pkg/cluster/adminupdate_test.go
index 52fc01893..fa1ea56b3 100644
--- a/pkg/cluster/adminupdate_test.go
+++ b/pkg/cluster/adminupdate_test.go
@@ -41,7 +41,6 @@ func TestAdminUpdateSteps(t *testing.T) {
 			},
 			shouldRunSteps: []string{
 				"[Action initializeKubernetesClients-fm]",
-				"[Action initializeOperatorDeployer-fm]",
 				"[Action ensureBillingRecord-fm]",
 				"[Action ensureDefaults-fm]",
 				"[Action fixupClusterSPObjectID-fm]",
@@ -62,7 +61,6 @@ func TestAdminUpdateSteps(t *testing.T) {
 			},
 			shouldRunSteps: []string{
 				"[Action initializeKubernetesClients-fm]",
-				"[Action initializeOperatorDeployer-fm]",
 				"[Action ensureBillingRecord-fm]",
 				"[Action ensureDefaults-fm]",
 				"[Action fixupClusterSPObjectID-fm]",
@@ -101,7 +99,6 @@ func TestAdminUpdateSteps(t *testing.T) {
 			},
 			shouldRunSteps: []string{
 				"[Action initializeKubernetesClients-fm]",
-				"[Action initializeOperatorDeployer-fm]",
 				"[Action ensureBillingRecord-fm]",
 				"[Action ensureDefaults-fm]",
 				"[Action fixupClusterSPObjectID-fm]",
diff --git a/pkg/cluster/arooperator_test.go b/pkg/cluster/arooperator_test.go
deleted file mode 100644
index 5832d2f41..000000000
--- a/pkg/cluster/arooperator_test.go
+++ /dev/null
@@ -1,204 +0,0 @@
-package cluster
-
-// Copyright (c) Microsoft Corporation.
-// Licensed under the Apache License 2.0.
-
-import (
-	"context"
-	"errors"
-	"strings"
-	"testing"
-
-	"github.com/golang/mock/gomock"
-	"github.com/sirupsen/logrus"
-
-	"github.com/Azure/ARO-RP/pkg/api"
-	mock_deploy "github.com/Azure/ARO-RP/pkg/util/mocks/operator/deploy"
-)
-
-func TestEnsureAROOperator(t *testing.T) {
-	ctx := context.Background()
-
-	const (
-		key = "/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/resourceGroup/providers/Microsoft.RedHatOpenShift/openShiftClusters/resourceName1"
-	)
-
-	for _, tt := range []struct {
-		name    string
-		doc     *api.OpenShiftClusterDocument
-		mocks   func(*mock_deploy.MockOperator)
-		wantErr string
-	}{
-		{
-			name: "create/update success",
-			doc: &api.OpenShiftClusterDocument{
-				Key: strings.ToLower(key),
-				OpenShiftCluster: &api.OpenShiftCluster{
-					ID: key,
-					Properties: api.OpenShiftClusterProperties{
-						IngressProfiles: []api.IngressProfile{
-							{
-								Visibility: api.VisibilityPublic,
-								Name:       "default",
-							},
-						},
-						ProvisioningState: api.ProvisioningStateAdminUpdating,
-						ClusterProfile: api.ClusterProfile{
-							Version: "4.8.18",
-						},
-						NetworkProfile: api.NetworkProfile{
-							APIServerPrivateEndpointIP: "1.2.3.4",
-						},
-					},
-				},
-			},
-			mocks: func(dep *mock_deploy.MockOperator) {
-				dep.EXPECT().
-					CreateOrUpdate(gomock.Any()).
-					Return(nil)
-			},
-		},
-		{
-			name: "create/update failure",
-			doc: &api.OpenShiftClusterDocument{
-				Key: strings.ToLower(key),
-				OpenShiftCluster: &api.OpenShiftCluster{
-					ID: key,
-					Properties: api.OpenShiftClusterProperties{
-						IngressProfiles: []api.IngressProfile{
-							{
-								Visibility: api.VisibilityPublic,
-								Name:       "default",
-							},
-						},
-						ProvisioningState: api.ProvisioningStateAdminUpdating,
-						ClusterProfile: api.ClusterProfile{
-							Version: "4.8.18",
-						},
-						NetworkProfile: api.NetworkProfile{
-							APIServerPrivateEndpointIP: "1.2.3.4",
-						},
-					},
-				},
-			},
-			mocks: func(dep *mock_deploy.MockOperator) {
-				dep.EXPECT().
-					CreateOrUpdate(gomock.Any()).
-					Return(errors.New("Mock return: CreateFailed"))
-			},
-
-			wantErr: "Mock return: CreateFailed",
-		},
-		{
-			name: "enriched data not available - skip",
-
-			doc: &api.OpenShiftClusterDocument{
-				Key: strings.ToLower(key),
-				OpenShiftCluster: &api.OpenShiftCluster{
-					ID: key,
-					Properties: api.OpenShiftClusterProperties{
-						IngressProfiles:   nil,
-						ProvisioningState: api.ProvisioningStateAdminUpdating,
-					},
-				},
-			},
-		},
-	} {
-		t.Run(tt.name, func(t *testing.T) {
-			controller := gomock.NewController(t)
-			defer controller.Finish()
-
-			dep := mock_deploy.NewMockOperator(controller)
-			if tt.mocks != nil {
-				tt.mocks(dep)
-			}
-
-			m := &manager{
-				log: logrus.NewEntry(logrus.StandardLogger()),
-				doc: tt.doc,
-
-				aroOperatorDeployer: dep,
-			}
-
-			err := m.ensureAROOperator(ctx)
-			if err != nil && err.Error() != tt.wantErr ||
-				err == nil && tt.wantErr != "" {
-				t.Error(err)
-			}
-		})
-	}
-}
-
-func TestAroDeploymentReady(t *testing.T) {
-	ctx := context.Background()
-
-	const (
-		key = "/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/resourceGroup/providers/Microsoft.RedHatOpenShift/openShiftClusters/resourceName1"
-	)
-
-	for _, tt := range []struct {
-		name    string
-		doc     *api.OpenShiftClusterDocument
-		mocks   func(*mock_deploy.MockOperator)
-		wantRes bool
-	}{
-		{
-			name: "create/update success",
-			doc: &api.OpenShiftClusterDocument{
-				Key: strings.ToLower(key),
-				OpenShiftCluster: &api.OpenShiftCluster{
-					ID: key,
-					Properties: api.OpenShiftClusterProperties{
-						IngressProfiles: []api.IngressProfile{
-							{
-								Visibility: api.VisibilityPublic,
-								Name:       "default",
-							},
-						},
-					},
-				},
-			},
-			mocks: func(dep *mock_deploy.MockOperator) {
-				dep.EXPECT().
-					IsReady(gomock.Any()).
-					Return(true, nil)
-			},
-			wantRes: true,
-		},
-		{
-			name: "enriched data not available - skip",
-			doc: &api.OpenShiftClusterDocument{
-				Key: strings.ToLower(key),
-				OpenShiftCluster: &api.OpenShiftCluster{
-					ID: key,
-					Properties: api.OpenShiftClusterProperties{
-						IngressProfiles: nil,
-					},
-				},
-			},
-			wantRes: true,
-		},
-	} {
-		t.Run(tt.name, func(t *testing.T) {
-			controller := gomock.NewController(t)
-			defer controller.Finish()
-
-			dep := mock_deploy.NewMockOperator(controller)
-			if tt.mocks != nil {
-				tt.mocks(dep)
-			}
-
-			m := &manager{
-				log:                 logrus.NewEntry(logrus.StandardLogger()),
-				doc:                 tt.doc,
-				aroOperatorDeployer: dep,
-			}
-
-			ok, err := m.aroDeploymentReady(ctx)
-			if err != nil || ok != tt.wantRes {
-				t.Error(err)
-			}
-
-		})
-	}
-}
diff --git a/pkg/frontend/adminactions/resources_list_test.go b/pkg/frontend/adminactions/resources_list_test.go
index 521cd14d3..7dc34aac9 100644
--- a/pkg/frontend/adminactions/resources_list_test.go
+++ b/pkg/frontend/adminactions/resources_list_test.go
@@ -25,7 +25,7 @@ import (
 )
 
 func validListByResourceGroupMock(resources *mock_features.MockResourcesClient) {
-	resources.EXPECT().ListByResourceGroup(gomock.Any(), "test-cluster", "resourceType ne 'Microsoft.Compute/snapshots'", "", nil).Return([]mgmtfeatures.GenericResourceExpanded{
+	resources.EXPECT().ListByResourceGroup(gomock.Any(), "test-cluster", "", "", nil).Return([]mgmtfeatures.GenericResourceExpanded{
 		{
 			Name: to.StringPtr("vm-1"),
 			ID:   to.StringPtr("/subscriptions/id"),
diff --git a/pkg/operator/controllers/autosizednodes/autosizednodes_controller_test.go b/pkg/operator/controllers/autosizednodes/autosizednodes_controller_test.go
deleted file mode 100644
index 7deb7c62f..000000000
--- a/pkg/operator/controllers/autosizednodes/autosizednodes_controller_test.go
+++ /dev/null
@@ -1,129 +0,0 @@
-package autosizednodes
-
-// Copyright (c) Microsoft Corporation.
-// Licensed under the Apache License 2.0.
-
-import (
-	"context"
-	"reflect"
-	"strconv"
-	"testing"
-
-	"github.com/coreos/ignition/v2/config/util"
-	"github.com/google/go-cmp/cmp"
-	mcv1 "github.com/openshift/machine-config-operator/pkg/apis/machineconfiguration.openshift.io/v1"
-	"github.com/sirupsen/logrus"
-	kerrors "k8s.io/apimachinery/pkg/api/errors"
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/types"
-	ctrl "sigs.k8s.io/controller-runtime"
-	"sigs.k8s.io/controller-runtime/pkg/client"
-	"sigs.k8s.io/controller-runtime/pkg/client/fake"
-
-	// This "_" import is counterintuitive but is required to initialize the scheme
-	// ARO unfortunately relies on implicit import and its side effect for this
-	arov1alpha1 "github.com/Azure/ARO-RP/pkg/operator/apis/aro.openshift.io/v1alpha1"
-	_ "github.com/Azure/ARO-RP/pkg/util/scheme"
-)
-
-func TestAutosizednodesReconciler(t *testing.T) {
-	aro := func(autoSizeEnabled bool) *arov1alpha1.Cluster {
-		return &arov1alpha1.Cluster{
-			ObjectMeta: metav1.ObjectMeta{
-				Name:      "aro",
-				Namespace: "openshift-azure-operator",
-			},
-			Spec: arov1alpha1.ClusterSpec{
-				OperatorFlags: arov1alpha1.OperatorFlags{
-					ControllerEnabled: strconv.FormatBool(autoSizeEnabled),
-				},
-			},
-		}
-	}
-
-	emptyConfig := mcv1.KubeletConfig{}
-	config := makeConfig()
-
-	tests := []struct {
-		name       string
-		wantGetErr error
-		client     client.Client
-		wantConfig *mcv1.KubeletConfig
-	}{
-		{
-			name:       "is not needed",
-			client:     fake.NewClientBuilder().WithRuntimeObjects(aro(false)).Build(),
-			wantConfig: &emptyConfig,
-			wantGetErr: kerrors.NewNotFound(mcv1.Resource("kubeletconfigs"), "dynamic-node"),
-		},
-		{
-			name:       "is needed and not present already",
-			client:     fake.NewClientBuilder().WithRuntimeObjects(aro(true)).Build(),
-			wantConfig: &config,
-			wantGetErr: nil,
-		},
-		{
-			name:       "is needed and present already",
-			client:     fake.NewClientBuilder().WithRuntimeObjects(aro(true), &config).Build(),
-			wantConfig: &config,
-		},
-		{
-			name:       "is not needed and is present",
-			client:     fake.NewClientBuilder().WithRuntimeObjects(aro(false), &config).Build(),
-			wantConfig: &emptyConfig,
-			wantGetErr: kerrors.NewNotFound(mcv1.Resource("kubeletconfigs"), "dynamic-node"),
-		},
-		{
-			name: "is needed and config got modified",
-			client: fake.NewClientBuilder().WithRuntimeObjects(
-				aro(true),
-				&mcv1.KubeletConfig{
-					ObjectMeta: metav1.ObjectMeta{
-						Name: configName,
-					},
-					Spec: mcv1.KubeletConfigSpec{
-						AutoSizingReserved: util.BoolToPtr(false),
-						MachineConfigPoolSelector: &metav1.LabelSelector{
-							MatchLabels: map[string]string{
-								"pools.operator.machineconfiguration.openshift.io/worker": "",
-							},
-						},
-					},
-				}).Build(),
-			wantConfig: &config,
-			wantGetErr: nil,
-		},
-	}
-
-	for _, test := range tests {
-		t.Run(test.name, func(t *testing.T) {
-			ctx := context.Background()
-			r := Reconciler{
-				client: test.client,
-				log:    logrus.NewEntry(logrus.StandardLogger()),
-			}
-			result, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: types.NamespacedName{Namespace: "openshift-azure-operator", Name: "aro"}})
-			if err != nil {
-				t.Error(err)
-			}
-
-			key := types.NamespacedName{
-				Name: configName,
-			}
-			var c mcv1.KubeletConfig
-
-			err = r.client.Get(ctx, key, &c)
-			if err != nil && err.Error() != test.wantGetErr.Error() || err == nil && test.wantGetErr != nil {
-				t.Error(err)
-			}
-
-			if !reflect.DeepEqual(test.wantConfig.Spec, c.Spec) {
-				t.Error(cmp.Diff(test.wantConfig.Spec, c.Spec))
-			}
-
-			if result != (ctrl.Result{}) {
-				t.Error("reconcile returned an unexpected result")
-			}
-		})
-	}
-}
diff --git a/pkg/operator/controllers/workaround/workaround_controller_test.go b/pkg/operator/controllers/workaround/workaround_controller_test.go
index c88f780a5..96990accf 100644
--- a/pkg/operator/controllers/workaround/workaround_controller_test.go
+++ b/pkg/operator/controllers/workaround/workaround_controller_test.go
@@ -65,7 +65,7 @@ func TestWorkaroundReconciler(t *testing.T) {
 		{
 			name: "is required",
 			mocker: func(mw *mock_workaround.MockWorkaround) {
-				c := mw.EXPECT().IsRequired(gomock.Any(), gomock.Any()).Return(true)
+				c := mw.EXPECT().IsRequired(gomock.Any()).Return(true)
 				mw.EXPECT().Ensure(gomock.Any()).After(c).Return(nil)
 			},
 			want: ctrl.Result{RequeueAfter: time.Hour},
@@ -73,7 +73,7 @@ func TestWorkaroundReconciler(t *testing.T) {
 		{
 			name: "is not required",
 			mocker: func(mw *mock_workaround.MockWorkaround) {
-				c := mw.EXPECT().IsRequired(gomock.Any(), gomock.Any()).Return(false)
+				c := mw.EXPECT().IsRequired(gomock.Any()).Return(false)
 				mw.EXPECT().Remove(gomock.Any()).After(c).Return(nil)
 			},
 			want: ctrl.Result{RequeueAfter: time.Hour},
@@ -81,7 +81,7 @@ func TestWorkaroundReconciler(t *testing.T) {
 		{
 			name: "has error",
 			mocker: func(mw *mock_workaround.MockWorkaround) {
-				mw.EXPECT().IsRequired(gomock.Any(), gomock.Any()).Return(true)
+				mw.EXPECT().IsRequired(gomock.Any()).Return(true)
 				mw.EXPECT().Name().Return("test").AnyTimes()
 				mw.EXPECT().Ensure(gomock.Any()).Return(fmt.Errorf("oops"))
 			},
diff --git a/test/e2e/adminapi_redeployvm.go b/test/e2e/adminapi_redeployvm.go
index df35a2f24..fb1d55826 100644
--- a/test/e2e/adminapi_redeployvm.go
+++ b/test/e2e/adminapi_redeployvm.go
@@ -4,14 +4,19 @@ package e2e
 // Licensed under the Apache License 2.0.
 
 import (
+	"bufio"
 	"context"
+	"fmt"
 	"net/http"
 	"net/url"
+	"strings"
 	"time"
 
 	. "github.com/onsi/ginkgo"
 	. "github.com/onsi/gomega"
 
+	mgmtcompute "github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2020-06-01/compute"
+	corev1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/util/wait"
 
@@ -19,6 +24,10 @@ import (
 	"github.com/Azure/ARO-RP/pkg/util/stringutils"
 )
 
+const (
+	uptimeStrFmt = "2006-01-02 15:04:05" // https://go.dev/src/time/format.go
+)
+
 var _ = Describe("[Admin API] VM redeploy action", func() {
 	BeforeEach(skipIfNotInDevelopmentEnv)
 
@@ -38,73 +47,128 @@ var _ = Describe("[Admin API] VM redeploy action", func() {
 		vm := vms[0]
 		log.Infof("selected vm: %s", *vm.Name)
 
-		By("triggering the redeploy action")
-		clockDrift := -1 * time.Minute
-		startTime := time.Now().Add(clockDrift)
+		By("saving the current uptime")
+		oldUptime, err := getNodeUptime(*vm.Name)
+		Expect(err).NotTo(HaveOccurred())
+
+		By("verifying redeploy action completes without error")
 		resp, err := adminRequest(ctx, http.MethodPost, "/admin"+resourceID+"/redeployvm", url.Values{"vmName": []string{*vm.Name}}, nil, nil)
 		Expect(err).NotTo(HaveOccurred())
 		Expect(resp.StatusCode).To(Equal(http.StatusOK))
 
-		By("verifying through cluster events that the redeployment happened")
-		err = wait.PollImmediate(10*time.Second, 10*time.Minute, func() (bool, error) {
-			events, err := clients.Kubernetes.EventsV1().Events("default").List(ctx, metav1.ListOptions{})
-
+		By("verifying node power state is eventually Running in Azure")
+		// we can pollimmediate without fear of false positive because we have
+		// already waited on the redeploy future
+		err = wait.PollImmediate(1*time.Minute, 10*time.Minute, func() (bool, error) {
+			restartedVm, err := clients.VirtualMachines.Get(ctx, clusterResourceGroup, *vm.Name, mgmtcompute.InstanceView)
 			if err != nil {
-				return false, err
+				log.Info(fmt.Sprintf("Failed to get restarted vm: %v", err))
+				return false, nil // swallow err, retry
 			}
-
-			var nodeKillTime metav1.MicroTime
-
-			for _, event := range events.Items {
-				if nodeKillTime.IsZero() &&
-					event.Reason == "TerminationStart" &&
-					!event.CreationTimestamp.After(startTime) {
-					nodeKillTime = metav1.MicroTime(event.CreationTimestamp)
-					break
-				}
-
-			}
-
-			var nodeNotReady, rebooted, nodeReady bool
-
-			for _, event := range events.Items {
-				if event.CreationTimestamp.After(nodeKillTime.Time) && event.Regarding.Name == *vm.Name {
-					if !nodeNotReady && event.Reason == "NodeNotReady" {
-						nodeNotReady = true
-						log.Info("node entered not ready state")
-					}
-					if !rebooted && event.Reason == "Rebooted" {
-						rebooted = true
-						log.Info("node reboooted")
-					}
-					if !nodeReady && event.Reason == "NodeReady" {
-						nodeReady = true
-						log.Info("node returned to ready state")
-						break
-					}
+			for _, status := range *restartedVm.InstanceView.Statuses {
+				if *status.Code == "PowerState/running" {
+					return true, nil
 				}
 			}
-
-			return nodeNotReady && rebooted && nodeReady, nil
+			return false, nil
 		})
 		Expect(err).NotTo(HaveOccurred())
 
-		By("waiting for all nodes to be ready")
-		err = wait.PollImmediate(10*time.Second, 10*time.Minute, func() (bool, error) {
-			nodes, err := clients.Kubernetes.CoreV1().Nodes().List(context.Background(), metav1.ListOptions{})
+		By("verifying redeployed node is eventually Ready in OpenShift")
+		// wait 1 minute - this will guarantee we pass the minimum (default) threshold of Node heartbeats (40 seconds)
+		err = wait.Poll(1*time.Minute, 10*time.Minute, func() (bool, error) {
+			node, err := clients.Kubernetes.CoreV1().Nodes().Get(ctx, *vm.Name, metav1.GetOptions{})
 			if err != nil {
 				log.Warn(err)
-				return false, nil // swallow error
+				return false, nil // swallow error, retry
 			}
-
-			for _, node := range nodes.Items {
-				if !ready.NodeIsReady(&node) {
-					return false, nil
-				}
+			if ready.NodeIsReady(node) {
+				return true, nil
 			}
-
-			return true, nil
+			return false, nil
 		})
 		Expect(err).NotTo(HaveOccurred())
+
+		By("getting system uptime again and making sure it is newer")
+		newUptime, err := getNodeUptime(*vm.Name)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(oldUptime.Before(newUptime)).To(BeTrue())
 	})
 })
+
+func getNodeUptime(node string) (time.Time, error) {
+	// container kernel = node kernel = `uptime` in a Pod reflects the Node as well
+	ctx := context.Background()
+	namespace := "default"
+	name := node
+	pod := &corev1.Pod{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      name,
+			Namespace: namespace,
+		},
+		Spec: corev1.PodSpec{
+			Containers: []corev1.Container{
+				{
+					Name:  "cli",
+					Image: "image-registry.openshift-image-registry.svc:5000/openshift/cli",
+					Command: []string{
+						"/bin/sh",
+						"-c",
+						"uptime -s",
+					},
+				},
+			},
+			RestartPolicy: "Never",
+			NodeName:      node,
+		},
+	}
+
+	// Create
+	_, err := clients.Kubernetes.CoreV1().Pods(namespace).Create(ctx, pod, metav1.CreateOptions{})
+	if err != nil {
+		return time.Time{}, err
+	}
+
+	// Defer Delete
+	defer func() {
+		err := clients.Kubernetes.CoreV1().Pods(namespace).Delete(ctx, name, metav1.DeleteOptions{})
+		if err != nil {
+			log.Error("Could not delete test Pod")
+		}
+	}()
+
+	// Wait for Completion
+	err = wait.PollImmediate(5*time.Second, 3*time.Minute, func() (bool, error) {
+		p, err := clients.Kubernetes.CoreV1().Pods(namespace).Get(ctx, name, metav1.GetOptions{})
+		if err != nil {
+			return false, err
+		}
+		if p.Status.Phase == "Succeeded" {
+			return true, nil
+		}
+		return false, nil // retry
+	})
+	if err != nil {
+		return time.Time{}, err
+	}
+
+	// Logs (uptime)
+	req := clients.Kubernetes.CoreV1().Pods(namespace).GetLogs(name, &corev1.PodLogOptions{})
+	stream, err := req.Stream(ctx)
+	if err != nil {
+		return time.Time{}, err
+	}
+	defer stream.Close()
+	message := ""
+	reader := bufio.NewScanner(stream)
+	for reader.Scan() {
+		select {
+		case <-ctx.Done():
+			break
+		default:
+			line := reader.Text()
+			message += line
+		}
+	}
+	return time.Parse(uptimeStrFmt, strings.TrimSpace(message))
+}