Repository: Azure/ARO-RP
Pull request #2189 (2022-06-16 14:58:01)

operator deployment improvement
### Which issue this PR addresses:
Improved the operator deployment process so we can use proper templating instead of custom  (needed for ocm pull secret validation). (needed for https://msazure.visualstudio.com/AzureRedHatOpenShift/_workitems/edit/13933226/)
<!--
Please include a link to the ADO work item as well as any GitHub issues.
https://msazure.visualstudio.com/AzureRedHatOpenShift/_workitems/edit/13933226/
Usage: `Fixes #<GitHub issue number>`, or `Fixes (paste link of issue)`.
-->
Fixes

### What this PR does / why we need it:

<!--
Include a brief summary of what the PR is intended to accomplish and how the PR
does it. (2-3 sentences)
-->

### Test plan for issue:
already covered by existing tests
<!--
How did you test that this PR works?

- Are there unit tests?
- Are there integration/e2e tests?
- If it is not possible to write automated tests, explain why and document how
  to manually test and verify the feature.
-->

### Is there any documentation that needs to be updated for this PR?

<!--
- If yes and the docs are in GitHub, include doc updates in the PR.
- If yes and the docs are not in GitHub (i.e. ADO wiki), include a link to the
  docs.
- If no, explain why (e.g. "tech debt cleanup, N/A").
-->


facchettos: > These changes look OK to me, thank you for the work here! I just have a few questions about the added labels/envs on the deployments - why are those added here? Can you add that to the description of the PR?

@SudoBrendan  These used to be added inside the code: 
```go
	// set the image for the deployments
		if d, ok := obj.(*appsv1.Deployment); ok {
			if d.Labels == nil {
				d.Labels = map[string]string{}
			}
			var image string

			if o.oc.Properties.OperatorVersion != "" {
				image = fmt.Sprintf("%s/aro:%s", o.env.ACRDomain(), o.oc.Properties.OperatorVersion)
				d.Labels["version"] = o.oc.Properties.OperatorVersion
			} else {
				image = o.env.AROOperatorImage()
				d.Labels["version"] = version.GitCommit
			}
		for i := range d.Spec.Template.Spec.Containers {
			d.Spec.Template.Spec.Containers[i].Image = image
			if o.env.IsLocalDevelopmentMode() {
					d.Spec.Template.Spec.Containers[i].Env = append(d.Spec.Template.Spec.Containers[i].Env, corev1.EnvVar{
						Name:  "RP_MODE",
						Value: "development",
					})
				}
               }
```

So basically they always were there, it just was not obvious since we had to understand the code to know that they were added to the deployments. 

Patch:
diff --git a/.gitignore b/.gitignore
index 1ca2eb098..b7c35b9eb 100644
--- a/.gitignore
+++ b/.gitignore
@@ -30,5 +30,3 @@ gomock_reflect_*
 /portal/v1/node_modules/
 /portal/v2/node_modules/
 .idea*
-/hack/hive-config/crds
-/hack/hive-config/hive-deployment.yaml
diff --git a/Makefile b/Makefile
index 0d3c3fe9f..c278dd1bb 100644
--- a/Makefile
+++ b/Makefile
@@ -155,7 +155,7 @@ validate-go:
 	@[ -z "$$(ls pkg/util/*.go 2>/dev/null)" ] || (echo error: go files are not allowed in pkg/util, use a subpackage; exit 1)
 	@[ -z "$$(find -name "*:*")" ] || (echo error: filenames with colons are not allowed on Windows, please rename; exit 1)
 	@sha256sum --quiet -c .sha256sum || (echo error: client library is stale, please run make client; exit 1)
-	go vet -tags containers_image_openpgp ./...
+	go vet ./...
 	go test -tags e2e -run ^$$ ./test/e2e/...
 
 validate-go-action:
diff --git a/docs/hive.md b/docs/hive.md
deleted file mode 100644
index f1dad9654..000000000
--- a/docs/hive.md
+++ /dev/null
@@ -1,37 +0,0 @@
-# Hive
-
-## Version
-
-Update the HIVE_IMAGE_COMMIT_HASH in `hack/hive-generate-config.sh` with the latest commit sha of the Hive image you are deploying. The commit sha is used to specify the image tag and also used during config generation to checkout the correct version of the config files. The config files are subsequently used by the `hack/hive-dev-install.sh` script during iunstallation or during config updates.
-
-## Generating config
-
-In order to generate config for a dev environment you need to ensure you have the correct `LOCATION` is set in your env file. Once this is done you can simply run the config generation script.
-
-```bash
-# source your environment file
-. ./env
-# run the config generation
-./hack/hive-generate-config.sh
-```
-
-This will download the latest source, reset to the hash specified in HIVE_IMAGE_COMMIT_HASH, and build the config using kustomise.
-
-## Installing
-
-Ensure you have the latest AKS kubeconfig:
-```bash
-# get the AKS kubeconfig
-make aks.kubeconfig
-```
-
-Set KUBECONFIG to the aks.kubeconfig file, for example:
-```bash
-export KUBECONFIG="$PWD/aks.kubeconfig"
-```
-
-Installing then simply requires the running of the install script.
-
-```bash
-./hack/hive-dev-install.sh
-```
diff --git a/hack/hive-config/cluster-image-sets/4.10.15.yaml b/hack/hive-config/cluster-image-sets/4.10.15.yaml
deleted file mode 100644
index 5dbff8143..000000000
--- a/hack/hive-config/cluster-image-sets/4.10.15.yaml
+++ /dev/null
@@ -1,6 +0,0 @@
-apiVersion: hive.openshift.io/v1
-kind: ClusterImageSet
-metadata:
-  name: openshift-v4.10.15
-spec:
-  releaseImage: quay.io/openshift-release-dev/ocp-release@sha256:ddcb70ce04a01ce487c0f4ad769e9e36a10c8c832a34307c1b1eb8e03a5b7ddb
diff --git a/hack/hive-config/hive-config.yaml b/hack/hive-config/hive-config.yaml
deleted file mode 100644
index 6c2bad6d8..000000000
--- a/hack/hive-config/hive-config.yaml
+++ /dev/null
@@ -1,15 +0,0 @@
-apiVersion: hive.openshift.io/v1
-kind: HiveConfig
-metadata:
-  name: hive
-spec:
-  logLevel: debug
-  targetNamespace: HIVE_OPERATOR_NS
-  deleteProtection: enabled
-  disabledControllers:
-  - remoteingress
-  failedProvisionConfig:
-    retryReasons:
-    - UnknownError
-  globalPullSecretRef:
-    name: hive-global-pull-secret
diff --git a/hack/hive-dev-install.sh b/hack/hive-dev-install.sh
deleted file mode 100755
index 313f66b34..000000000
--- a/hack/hive-dev-install.sh
+++ /dev/null
@@ -1,80 +0,0 @@
-#!/bin/bash
-
-HIVE_OPERATOR_NS="hive"
-KUBECTL=$( which kubectl 2> /dev/null || which oc 2> /dev/null)
-
-function cleanup {
-	[ -f "$(pwd)/kubectl" ] && rm -f "$(pwd)/kubectl"
-}
-
-function download_tmp_kubectl {
-	curl -sLO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
-	if [ $? -ne 0 ]; then
-		echo ": error downloading kubectl"
-		exit 1
-	fi
-	chmod 755 kubectl
-	KUBECTL="$(pwd)/kubectl"
-}
-
-function verify_tools {
-	if [ ! -z "$KUBECTL" ]; then
-		return
-	fi
-	echo -n "kubectl or oc not detected, downloading"
-	download_tmp_kubectl
-	echo ", done."
-
-	if [ $( $KUBECTL get nodes 2>/dev/null | wc -l ) -eq 0 ]; then
-		echo "unable to connect to the cluster"
-		exit 1
-	fi
-}
-
-set -e
-trap cleanup EXIT
-
-if [ ! -f go.mod ] || [ ! -d ".git" ]; then
-	echo "this script must by run from the repo's root directory"
-	exit 1
-fi
-
-if [ ! -f "./hack/hive-config/hive-deployment.yaml" ] || [ ! -d "./hack/hive-config/crds" ] ; then
-	echo "hive config is missing, generating config, please rerun this script afterwards"
-	./hack/hive-generate-config.sh
-	if [ $? -ne 0 ]; then
-		echo "error generating the hive configs"
-		exit 1
-	fi
-fi
-
-if [ -z "$PULL_SECRET" ]; then
-	echo "global pull secret variable required, please source ./env"
-	exit
-fi
-
-verify_tools
-
-if [ $( $KUBECTL get namespace $HIVE_OPERATOR_NS -o yaml 2>/dev/null | wc -l ) -ne 0 ]; then
-	echo "hive is already installed in the namespace"
-	echo -n "would you like to reapply the configs? (y/N): "
-	read answer
-	if [[ "$answer" != "y" ]]; then
-		exit
-	fi
-else
-	$KUBECTL create namespace $HIVE_OPERATOR_NS
-fi
-
-$KUBECTL apply -f ./hack/hive-config/crds
-$KUBECTL apply -f ./hack/hive-config/hive-deployment.yaml
-$KUBECTL apply -f ./hack/hive-config/cluster-image-sets
-
-echo "$PULL_SECRET" > /tmp/.tmp-secret
-# Using dry-run allows updates to work seamlessly
-$KUBECTL create secret generic hive-global-pull-secret --from-file=.dockerconfigjson=/tmp/.tmp-secret --type=kubernetes.io/dockerconfigjson --namespace $HIVE_OPERATOR_NS -o yaml --dry-run=client | $KUBECTL apply -f - 2>/dev/null
-rm -f /tmp/.tmp-secret
-
-sed "s/HIVE_OPERATOR_NS/$HIVE_OPERATOR_NS/g" hack/hive-config/hive-config.yaml | $KUBECTL apply -f -
-
-echo -e "\nHive is installed."
diff --git a/hack/hive-generate-config.sh b/hack/hive-generate-config.sh
deleted file mode 100755
index ce4b9d6e0..000000000
--- a/hack/hive-generate-config.sh
+++ /dev/null
@@ -1,94 +0,0 @@
-#!/bin/bash
-
-# This is the commit sha that the image was built from and ensures we use the correct configs for the release
-HIVE_IMAGE_COMMIT_HASH="2383a88"
-
-# For now we'll use the quay hive image, but this will change to an ACR once the quay.io -> ACR mirroring is setup
-# Note: semi-scientific way to get the latest image: `podman search --list-tags --limit 10000 quay.io/app-sre/hive | tail -n1`
-HIVE_IMAGE="quay.io/app-sre/hive:${HIVE_IMAGE_COMMIT_HASH}"
-
-HIVE_OPERATOR_NS="hive"
-
-# This version is specified in the hive repo and is the only hard dependency for this script
-# https://github.com/openshift/hive/blob/master/vendor/github.com/openshift/build-machinery-go/make/targets/openshift/kustomize.mk#L7
-KUSTOMIZE_VERSION=4.1.3
-KUSTOMIZE=$( which kustomize 2>/dev/null )
-TMPDIR=$( mktemp -d )
-
-function cleanup {
-	popd >& /dev/null
-	[ -d "$TMPDIR" ] && rm -fr "$TMPDIR"
-}
-
-function verify_kustomize {
-	if [ ! -z "$KUSTOMIZE" ]; then
-		return
-	fi
-	echo -n "kustomize not detected, downloading "
-	curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/kustomize/v${KUSTOMIZE_VERSION}/hack/install_kustomize.sh" | bash -s "$KUSTOMIZE_VERSION" "$TMPDIR"
-	if [ $? -ne 0 ]; then
-		echo "error downloading kustomize"
-		exit 1
-	fi
-	KUSTOMIZE="${TMPDIR}/kustomize"
-}
-
-function hive_repo_clone {
-	echo -n "Cloning hive repo into tmp for config generation"
-	CLONE_ERROR=$(git clone https://github.com/openshift/hive.git "$TMPDIR" 2>/dev/null )
-	if [ $? -ne 0 ]; then
-		echo ": error cloning the hive repo: ${CLONE_ERROR}"
-		exit 1
-	fi
-	echo ", done."
-}
-
-function hive_repo_hash_checkout {
-	# go into $TMPDIR and checkout the commit the image was built with
-	pushd $TMPDIR >& /dev/null
-	git reset --hard $HIVE_IMAGE_COMMIT_HASH
-	if [ $? -ne 0 ] || [[ $( git rev-parse --short=${#HIVE_IMAGE_COMMIT_HASH} HEAD ) != ${HIVE_IMAGE_COMMIT_HASH} ]]; then
-		echo "error resetting the hive repo to the correct git hash '${HIVE_IMAGE_COMMIT_HASH}'"
-		exit 1
-	fi
-}
-
-function generate_hive_config {
-	# Create the hive operator install config using kustomize
-	mkdir -p overlays/deploy
-	cp overlays/template/kustomization.yaml overlays/deploy
-	pushd overlays/deploy >& /dev/null
-	$KUSTOMIZE edit set image registry.ci.openshift.org/openshift/hive-v4.0:hive=$HIVE_IMAGE
-	$KUSTOMIZE edit set namespace $HIVE_OPERATOR_NS
-	popd >& /dev/null
-
-	$KUSTOMIZE build overlays/deploy > hive-deployment.yaml
-
-	# return to the repo directory to copy the generated config from $TMPDIR
-	popd >& /dev/null
-	mv "$TMPDIR/hive-deployment.yaml" ./hack/hive-config/
-
-	if [ -d ./hack/hive-config/crds ]; then
-		rm -fr ./hack/hive-config/crds
-	fi
-	cp -R "$TMPDIR/config/crds" ./hack/hive-config/
-}
-
-set -e
-trap cleanup EXIT
-
-if [ ! -f go.mod ] || [ ! -d ".git" ]; then
-	echo "this script must by run from the repo's root directory"
-	exit 1
-fi
-if [[ ! "$TMPDIR" || ! -d "$TMPDIR" ]]; then
-	echo "could not create temp working dir"
-	exit 1
-fi
-
-hive_repo_clone
-hive_repo_hash_checkout
-verify_kustomize
-generate_hive_config
-
-echo -e "\nHive config generated."
diff --git a/pkg/installer/deployresources.go b/pkg/cluster/deployresources.go
similarity index 99%
rename from pkg/installer/deployresources.go
rename to pkg/cluster/deployresources.go
index f07538bf0..da9a5e948 100644
--- a/pkg/installer/deployresources.go
+++ b/pkg/cluster/deployresources.go
@@ -1,4 +1,4 @@
-package installer
+package cluster
 
 // Copyright (c) Microsoft Corporation.
 // Licensed under the Apache License 2.0.
diff --git a/pkg/installer/deployresources_resources.go b/pkg/cluster/deployresources_resources.go
similarity index 99%
rename from pkg/installer/deployresources_resources.go
rename to pkg/cluster/deployresources_resources.go
index 71864133c..12db8fd0e 100644
--- a/pkg/installer/deployresources_resources.go
+++ b/pkg/cluster/deployresources_resources.go
@@ -1,4 +1,4 @@
-package installer
+package cluster
 
 // Copyright (c) Microsoft Corporation.
 // Licensed under the Apache License 2.0.
diff --git a/pkg/cluster/deploystorage.go b/pkg/cluster/deploystorage.go
index e6fb43ccc..dabe55aaf 100644
--- a/pkg/cluster/deploystorage.go
+++ b/pkg/cluster/deploystorage.go
@@ -5,6 +5,8 @@ package cluster
 
 import (
 	"context"
+	"crypto/rand"
+	"encoding/hex"
 	"encoding/json"
 	"fmt"
 	"net/http"
@@ -16,9 +18,15 @@ import (
 	"github.com/Azure/go-autorest/autorest"
 	"github.com/Azure/go-autorest/autorest/azure"
 	"github.com/Azure/go-autorest/autorest/to"
+	"github.com/openshift/installer/pkg/asset/installconfig"
+	"github.com/openshift/installer/pkg/asset/releaseimage"
+	"github.com/openshift/installer/pkg/asset/targets"
+	"github.com/openshift/installer/pkg/asset/templates/content/bootkube"
 	utilrand "k8s.io/apimachinery/pkg/util/rand"
 
 	"github.com/Azure/ARO-RP/pkg/api"
+	"github.com/Azure/ARO-RP/pkg/bootstraplogging"
+	"github.com/Azure/ARO-RP/pkg/cluster/graph"
 	"github.com/Azure/ARO-RP/pkg/env"
 	"github.com/Azure/ARO-RP/pkg/util/arm"
 	"github.com/Azure/ARO-RP/pkg/util/stringutils"
@@ -148,6 +156,77 @@ func (m *manager) deployStorageTemplate(ctx context.Context) error {
 	return arm.DeployTemplate(ctx, m.log, m.deployments, resourceGroup, "storage", t, nil)
 }
 
+// applyInstallConfigCustomisations modifies the InstallConfig and creates
+// parent assets, then regenerates the InstallConfig for use for Ignition
+// generation, etc.
+func (m *manager) applyInstallConfigCustomisations(ctx context.Context, installConfig *installconfig.InstallConfig, image *releaseimage.Image) (graph.Graph, error) {
+	clusterID := &installconfig.ClusterID{
+		UUID:    m.doc.ID,
+		InfraID: m.doc.OpenShiftCluster.Properties.InfraID,
+	}
+
+	bootstrapLoggingConfig, err := bootstraplogging.GetConfig(m.env, m.doc)
+	if err != nil {
+		return nil, err
+	}
+
+	httpSecret := make([]byte, 64)
+	_, err = rand.Read(httpSecret)
+	if err != nil {
+		return nil, err
+	}
+
+	imageRegistryConfig := &bootkube.AROImageRegistryConfig{
+		AccountName:   m.doc.OpenShiftCluster.Properties.ImageRegistryStorageAccountName,
+		ContainerName: "image-registry",
+		HTTPSecret:    hex.EncodeToString(httpSecret),
+	}
+
+	dnsConfig := &bootkube.ARODNSConfig{
+		APIIntIP:  m.doc.OpenShiftCluster.Properties.APIServerProfile.IntIP,
+		IngressIP: m.doc.OpenShiftCluster.Properties.IngressProfiles[0].IP,
+	}
+
+	if m.doc.OpenShiftCluster.Properties.NetworkProfile.GatewayPrivateEndpointIP != "" {
+		dnsConfig.GatewayPrivateEndpointIP = m.doc.OpenShiftCluster.Properties.NetworkProfile.GatewayPrivateEndpointIP
+		dnsConfig.GatewayDomains = append(m.env.GatewayDomains(), m.doc.OpenShiftCluster.Properties.ImageRegistryStorageAccountName+".blob."+m.env.Environment().StorageEndpointSuffix)
+	}
+
+	g := graph.Graph{}
+	g.Set(installConfig, image, clusterID, bootstrapLoggingConfig, dnsConfig, imageRegistryConfig)
+
+	m.log.Print("resolving graph")
+	for _, a := range targets.Cluster {
+		err = g.Resolve(a)
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	// Handle MTU3900 feature flag
+	if m.doc.OpenShiftCluster.Properties.NetworkProfile.MTUSize == api.MTU3900 {
+		m.log.Printf("applying feature flag %s", api.FeatureFlagMTU3900)
+		if err = m.overrideEthernetMTU(g); err != nil {
+			return nil, err
+		}
+	}
+
+	return g, nil
+}
+
+func (m *manager) persistGraph(ctx context.Context, g graph.Graph) error {
+	resourceGroup := stringutils.LastTokenByte(m.doc.OpenShiftCluster.Properties.ClusterProfile.ResourceGroupID, '/')
+	clusterStorageAccountName := "cluster" + m.doc.OpenShiftCluster.Properties.StorageSuffix
+
+	exists, err := m.graph.Exists(ctx, resourceGroup, clusterStorageAccountName)
+	if err != nil || exists {
+		return err
+	}
+
+	// the graph is quite big, so we store it in a storage account instead of in cosmosdb
+	return m.graph.Save(ctx, resourceGroup, clusterStorageAccountName, g)
+}
+
 func (m *manager) attachNSGs(ctx context.Context) error {
 	for _, subnetID := range []string{
 		m.doc.OpenShiftCluster.Properties.MasterProfile.SubnetID,
diff --git a/pkg/installer/generateconfig.go b/pkg/cluster/generateconfig.go
similarity index 99%
rename from pkg/installer/generateconfig.go
rename to pkg/cluster/generateconfig.go
index 39aaf6da6..bf3f0612b 100644
--- a/pkg/installer/generateconfig.go
+++ b/pkg/cluster/generateconfig.go
@@ -1,4 +1,4 @@
-package installer
+package cluster
 
 // Copyright (c) Microsoft Corporation.
 // Licensed under the Apache License 2.0.
diff --git a/pkg/cluster/install.go b/pkg/cluster/install.go
index f8e3ca3eb..1bb5caa4a 100644
--- a/pkg/cluster/install.go
+++ b/pkg/cluster/install.go
@@ -14,12 +14,14 @@ import (
 	operatorclient "github.com/openshift/client-go/operator/clientset/versioned"
 	samplesclient "github.com/openshift/client-go/samples/clientset/versioned"
 	securityclient "github.com/openshift/client-go/security/clientset/versioned"
+	"github.com/openshift/installer/pkg/asset/installconfig"
+	"github.com/openshift/installer/pkg/asset/releaseimage"
 	mcoclient "github.com/openshift/machine-config-operator/pkg/generated/clientset/versioned"
 	extensionsclient "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"
 	"k8s.io/client-go/kubernetes"
 
 	"github.com/Azure/ARO-RP/pkg/api"
-	"github.com/Azure/ARO-RP/pkg/installer"
+	"github.com/Azure/ARO-RP/pkg/cluster/graph"
 	aroclient "github.com/Azure/ARO-RP/pkg/operator/clientset/versioned"
 	"github.com/Azure/ARO-RP/pkg/operator/deploy"
 	"github.com/Azure/ARO-RP/pkg/util/restconfig"
@@ -120,23 +122,26 @@ func (m *manager) Update(ctx context.Context) error {
 	return m.runSteps(ctx, steps)
 }
 
-// callInstaller initialises and calls the Installer code. This will later be replaced with a call into Hive.
-func (m *manager) callInstaller(ctx context.Context) error {
-	i := installer.NewInstaller(m.log, m.env, m.doc, m.subscriptionDoc, m.fpAuthorizer, m.deployments, m.graph)
-	return i.Install(ctx)
-}
-
 // Install installs an ARO cluster
 func (m *manager) Install(ctx context.Context) error {
+	var (
+		installConfig *installconfig.InstallConfig
+		image         *releaseimage.Image
+		g             graph.Graph
+	)
+
 	steps := map[api.InstallPhase][]steps.Step{
 		api.InstallPhaseBootstrap: {
 			steps.AuthorizationRefreshingAction(m.fpAuthorizer, steps.Action(m.validateResources)),
 			steps.Action(m.ensureACRToken),
 			steps.Action(m.ensureInfraID),
-			steps.Action(m.ensureSSHKey),
-			steps.Action(m.ensureStorageSuffix),
+			steps.Action(m.generateSSHKey),
 			steps.Action(m.populateMTUSize),
-
+			steps.Action(func(ctx context.Context) error {
+				var err error
+				installConfig, image, err = m.generateInstallConfig(ctx)
+				return err
+			}),
 			steps.Action(m.createDNS),
 			steps.Action(m.initializeClusterSPClients), // must run before clusterSPObjectID
 			steps.Action(m.clusterSPObjectID),
@@ -144,22 +149,28 @@ func (m *manager) Install(ctx context.Context) error {
 			steps.AuthorizationRefreshingAction(m.fpAuthorizer, steps.Action(m.enableServiceEndpoints)),
 			steps.AuthorizationRefreshingAction(m.fpAuthorizer, steps.Action(m.setMasterSubnetPolicies)),
 			steps.AuthorizationRefreshingAction(m.fpAuthorizer, steps.Action(m.deployStorageTemplate)),
-			steps.AuthorizationRefreshingAction(m.fpAuthorizer, steps.Action(m.attachNSGs)),
 			steps.AuthorizationRefreshingAction(m.fpAuthorizer, steps.Action(m.updateAPIIPEarly)),
 			steps.AuthorizationRefreshingAction(m.fpAuthorizer, steps.Action(m.createOrUpdateRouterIPEarly)),
 			steps.AuthorizationRefreshingAction(m.fpAuthorizer, steps.Action(m.ensureGatewayCreate)),
-			steps.Action(m.createAPIServerPrivateEndpoint),
-			steps.Action(m.createCertificates),
-
-			// Run installer. For M5/M6 we will persist the graph inside the
-			// installer code since it's easier, but in the future, this data
-			// should be collected from Hive's outputs where needed.
-			steps.Action(m.callInstaller),
-
+			steps.Action(func(ctx context.Context) error {
+				var err error
+				// Applies ARO-specific customisations to the InstallConfig
+				g, err = m.applyInstallConfigCustomisations(ctx, installConfig, image)
+				return err
+			}),
+			steps.Action(func(ctx context.Context) error {
+				// saves the graph to storage account
+				return m.persistGraph(ctx, g)
+			}),
+			steps.AuthorizationRefreshingAction(m.fpAuthorizer, steps.Action(m.attachNSGs)),
 			steps.AuthorizationRefreshingAction(m.fpAuthorizer, steps.Action(m.generateKubeconfigs)),
 			steps.Action(m.ensureBillingRecord),
+			steps.AuthorizationRefreshingAction(m.fpAuthorizer, steps.Action(m.deployResourceTemplate)),
+			steps.Action(m.createAPIServerPrivateEndpoint),
+			steps.Action(m.createCertificates),
 			steps.Action(m.initializeKubernetesClients),
 			steps.Action(m.initializeOperatorDeployer), // depends on kube clients
+			steps.Condition(m.bootstrapConfigMapReady, 30*time.Minute, true),
 			steps.Condition(m.apiServersReady, 30*time.Minute, true),
 			steps.Action(m.ensureAROOperator),
 			steps.Action(m.incrInstallPhase),
diff --git a/pkg/installer/overridemtu.go b/pkg/cluster/overridemtu.go
similarity index 99%
rename from pkg/installer/overridemtu.go
rename to pkg/cluster/overridemtu.go
index 7eed9d9e2..7fa017997 100644
--- a/pkg/installer/overridemtu.go
+++ b/pkg/cluster/overridemtu.go
@@ -1,4 +1,4 @@
-package installer
+package cluster
 
 // Copyright (c) Microsoft Corporation.
 // Licensed under the Apache License 2.0.
diff --git a/pkg/cluster/sshkeys.go b/pkg/cluster/sshkeys.go
index 8d4df03cb..ffc0b23bc 100644
--- a/pkg/cluster/sshkeys.go
+++ b/pkg/cluster/sshkeys.go
@@ -13,25 +13,31 @@ import (
 	"github.com/Azure/ARO-RP/pkg/api"
 )
 
-func mutateSSHKey(doc *api.OpenShiftClusterDocument) error {
-	if doc.OpenShiftCluster.Properties.SSHKey != nil {
-		return nil
-	}
-
-	sshKey, err := rsa.GenerateKey(rand.Reader, 2048)
-	if err != nil {
-		return err
-	}
+func (m *manager) generateSSHKey(ctx context.Context) error {
+	var err error
+	m.doc, err = m.db.PatchWithLease(ctx, m.doc.Key, func(doc *api.OpenShiftClusterDocument) error {
+		if doc.OpenShiftCluster.Properties.SSHKey == nil {
+			sshKey, err := rsa.GenerateKey(rand.Reader, 2048)
+			if err != nil {
+				return err
+			}
+
+			doc.OpenShiftCluster.Properties.SSHKey = x509.MarshalPKCS1PrivateKey(sshKey)
+		}
 
-	doc.OpenShiftCluster.Properties.SSHKey = x509.MarshalPKCS1PrivateKey(sshKey)
+		if doc.OpenShiftCluster.Properties.StorageSuffix == "" {
+			doc.OpenShiftCluster.Properties.StorageSuffix, err = randomLowerCaseAlphanumericStringWithNoVowels(5)
+			if err != nil {
+				return err
+			}
 
-	return nil
-}
+			doc.OpenShiftCluster.Properties.ImageRegistryStorageAccountName = "imageregistry" + doc.OpenShiftCluster.Properties.StorageSuffix
+		}
 
-func (m *manager) ensureSSHKey(ctx context.Context) error {
-	updatedDoc, err := m.db.PatchWithLease(ctx, m.doc.Key, mutateSSHKey)
-	m.doc = updatedDoc
+		doc.OpenShiftCluster.Properties.ImageRegistryStorageAccountName = "imageregistry" + doc.OpenShiftCluster.Properties.StorageSuffix
 
+		return nil
+	})
 	return err
 }
 
diff --git a/pkg/cluster/storageSuffix.go b/pkg/cluster/storageSuffix.go
deleted file mode 100644
index ddb091ac7..000000000
--- a/pkg/cluster/storageSuffix.go
+++ /dev/null
@@ -1,40 +0,0 @@
-package cluster
-
-// Copyright (c) Microsoft Corporation.
-// Licensed under the Apache License 2.0.
-
-import (
-	"context"
-
-	"github.com/Azure/ARO-RP/pkg/api"
-)
-
-func setDocStorageSuffix(doc *api.OpenShiftClusterDocument) error {
-	isDocStorageSuffixValid := doc.OpenShiftCluster.Properties.StorageSuffix != ""
-	if isDocStorageSuffixValid {
-		return nil
-	}
-
-	storageSuffix, err := randomLowerCaseAlphanumericStringWithNoVowels(5)
-	doc.OpenShiftCluster.Properties.StorageSuffix = storageSuffix
-
-	return err
-}
-
-func mutateStorageSuffix(doc *api.OpenShiftClusterDocument) error {
-	err := setDocStorageSuffix(doc)
-	if err != nil {
-		return err
-	}
-
-	doc.OpenShiftCluster.Properties.ImageRegistryStorageAccountName = "imageregistry" + doc.OpenShiftCluster.Properties.StorageSuffix
-
-	return nil
-}
-
-func (m *manager) ensureStorageSuffix(ctx context.Context) error {
-	updatedDoc, err := m.db.PatchWithLease(ctx, m.doc.Key, mutateStorageSuffix)
-	m.doc = updatedDoc
-
-	return err
-}
diff --git a/pkg/database/openshiftclusters.go b/pkg/database/openshiftclusters.go
index 930a2ca78..c91371a17 100644
--- a/pkg/database/openshiftclusters.go
+++ b/pkg/database/openshiftclusters.go
@@ -25,8 +25,6 @@ const (
 	OpenshiftClustersResourceGroupQuery = `SELECT * FROM OpenShiftClusters doc WHERE doc.clusterResourceGroupIdKey = @resourceGroupID`
 )
 
-type OpenShiftClusterDocumentMutator func(*api.OpenShiftClusterDocument) error
-
 type openShiftClusters struct {
 	c     cosmosdb.OpenShiftClusterDocumentClient
 	collc cosmosdb.CollectionClient
@@ -38,8 +36,8 @@ type OpenShiftClusters interface {
 	Create(context.Context, *api.OpenShiftClusterDocument) (*api.OpenShiftClusterDocument, error)
 	Get(context.Context, string) (*api.OpenShiftClusterDocument, error)
 	QueueLength(context.Context, string) (int, error)
-	Patch(context.Context, string, OpenShiftClusterDocumentMutator) (*api.OpenShiftClusterDocument, error)
-	PatchWithLease(context.Context, string, OpenShiftClusterDocumentMutator) (*api.OpenShiftClusterDocument, error)
+	Patch(context.Context, string, func(*api.OpenShiftClusterDocument) error) (*api.OpenShiftClusterDocument, error)
+	PatchWithLease(context.Context, string, func(*api.OpenShiftClusterDocument) error) (*api.OpenShiftClusterDocument, error)
 	Update(context.Context, *api.OpenShiftClusterDocument) (*api.OpenShiftClusterDocument, error)
 	Delete(context.Context, *api.OpenShiftClusterDocument) error
 	ChangeFeed() cosmosdb.OpenShiftClusterDocumentIterator
@@ -181,11 +179,11 @@ func (c *openShiftClusters) QueueLength(ctx context.Context, collid string) (int
 	return countTotal, nil
 }
 
-func (c *openShiftClusters) Patch(ctx context.Context, key string, f OpenShiftClusterDocumentMutator) (*api.OpenShiftClusterDocument, error) {
+func (c *openShiftClusters) Patch(ctx context.Context, key string, f func(*api.OpenShiftClusterDocument) error) (*api.OpenShiftClusterDocument, error) {
 	return c.patch(ctx, key, f, nil)
 }
 
-func (c *openShiftClusters) patch(ctx context.Context, key string, f OpenShiftClusterDocumentMutator, options *cosmosdb.Options) (*api.OpenShiftClusterDocument, error) {
+func (c *openShiftClusters) patch(ctx context.Context, key string, f func(*api.OpenShiftClusterDocument) error, options *cosmosdb.Options) (*api.OpenShiftClusterDocument, error) {
 	var doc *api.OpenShiftClusterDocument
 
 	err := cosmosdb.RetryOnPreconditionFailed(func() (err error) {
@@ -206,11 +204,11 @@ func (c *openShiftClusters) patch(ctx context.Context, key string, f OpenShiftCl
 	return doc, err
 }
 
-func (c *openShiftClusters) PatchWithLease(ctx context.Context, key string, f OpenShiftClusterDocumentMutator) (*api.OpenShiftClusterDocument, error) {
+func (c *openShiftClusters) PatchWithLease(ctx context.Context, key string, f func(*api.OpenShiftClusterDocument) error) (*api.OpenShiftClusterDocument, error) {
 	return c.patchWithLease(ctx, key, f, nil)
 }
 
-func (c *openShiftClusters) patchWithLease(ctx context.Context, key string, f OpenShiftClusterDocumentMutator, options *cosmosdb.Options) (*api.OpenShiftClusterDocument, error) {
+func (c *openShiftClusters) patchWithLease(ctx context.Context, key string, f func(*api.OpenShiftClusterDocument) error, options *cosmosdb.Options) (*api.OpenShiftClusterDocument, error) {
 	return c.patch(ctx, key, func(doc *api.OpenShiftClusterDocument) error {
 		if doc.LeaseOwner != c.uuid {
 			return fmt.Errorf("lost lease")
diff --git a/pkg/installer/condition.go b/pkg/installer/condition.go
deleted file mode 100644
index 0fb147a5b..000000000
--- a/pkg/installer/condition.go
+++ /dev/null
@@ -1,21 +0,0 @@
-package installer
-
-// Copyright (c) Microsoft Corporation.
-// Licensed under the Apache License 2.0.
-
-import (
-	"context"
-
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-)
-
-// condition functions should return an error only if it's not retryable
-// if a condition function encounters a retryable error it should return false, nil.
-
-func (m *manager) bootstrapConfigMapReady(ctx context.Context) (bool, error) {
-	cm, err := m.kubernetescli.CoreV1().ConfigMaps("kube-system").Get(ctx, "bootstrap", metav1.GetOptions{})
-	if err != nil && m.env.IsLocalDevelopmentMode() {
-		m.log.Printf("bootstrapConfigMapReady condition error %s", err)
-	}
-	return err == nil && cm.Data["status"] == "complete", nil
-}
diff --git a/pkg/installer/custominstallconfig.go b/pkg/installer/custominstallconfig.go
deleted file mode 100644
index f038dda24..000000000
--- a/pkg/installer/custominstallconfig.go
+++ /dev/null
@@ -1,77 +0,0 @@
-package installer
-
-// Copyright (c) Microsoft Corporation.
-// Licensed under the Apache License 2.0.
-
-import (
-	"context"
-	"crypto/rand"
-	"encoding/hex"
-
-	"github.com/openshift/installer/pkg/asset/installconfig"
-	"github.com/openshift/installer/pkg/asset/releaseimage"
-	"github.com/openshift/installer/pkg/asset/targets"
-	"github.com/openshift/installer/pkg/asset/templates/content/bootkube"
-
-	"github.com/Azure/ARO-RP/pkg/api"
-	"github.com/Azure/ARO-RP/pkg/bootstraplogging"
-	"github.com/Azure/ARO-RP/pkg/cluster/graph"
-)
-
-// applyInstallConfigCustomisations modifies the InstallConfig and creates
-// parent assets, then regenerates the InstallConfig for use for Ignition
-// generation, etc.
-func (m *manager) applyInstallConfigCustomisations(ctx context.Context, installConfig *installconfig.InstallConfig, image *releaseimage.Image) (graph.Graph, error) {
-	clusterID := &installconfig.ClusterID{
-		UUID:    m.doc.ID,
-		InfraID: m.doc.OpenShiftCluster.Properties.InfraID,
-	}
-
-	bootstrapLoggingConfig, err := bootstraplogging.GetConfig(m.env, m.doc)
-	if err != nil {
-		return nil, err
-	}
-
-	httpSecret := make([]byte, 64)
-	_, err = rand.Read(httpSecret)
-	if err != nil {
-		return nil, err
-	}
-
-	imageRegistryConfig := &bootkube.AROImageRegistryConfig{
-		AccountName:   m.doc.OpenShiftCluster.Properties.ImageRegistryStorageAccountName,
-		ContainerName: "image-registry",
-		HTTPSecret:    hex.EncodeToString(httpSecret),
-	}
-
-	dnsConfig := &bootkube.ARODNSConfig{
-		APIIntIP:  m.doc.OpenShiftCluster.Properties.APIServerProfile.IntIP,
-		IngressIP: m.doc.OpenShiftCluster.Properties.IngressProfiles[0].IP,
-	}
-
-	if m.doc.OpenShiftCluster.Properties.NetworkProfile.GatewayPrivateEndpointIP != "" {
-		dnsConfig.GatewayPrivateEndpointIP = m.doc.OpenShiftCluster.Properties.NetworkProfile.GatewayPrivateEndpointIP
-		dnsConfig.GatewayDomains = append(m.env.GatewayDomains(), m.doc.OpenShiftCluster.Properties.ImageRegistryStorageAccountName+".blob."+m.env.Environment().StorageEndpointSuffix)
-	}
-
-	g := graph.Graph{}
-	g.Set(installConfig, image, clusterID, bootstrapLoggingConfig, dnsConfig, imageRegistryConfig)
-
-	m.log.Print("resolving graph")
-	for _, a := range targets.Cluster {
-		err = g.Resolve(a)
-		if err != nil {
-			return nil, err
-		}
-	}
-
-	// Handle MTU3900 feature flag
-	if m.doc.OpenShiftCluster.Properties.NetworkProfile.MTUSize == api.MTU3900 {
-		m.log.Printf("applying feature flag %s", api.FeatureFlagMTU3900)
-		if err = m.overrideEthernetMTU(g); err != nil {
-			return nil, err
-		}
-	}
-
-	return g, nil
-}
diff --git a/pkg/installer/graph.go b/pkg/installer/graph.go
deleted file mode 100644
index e49f21c2e..000000000
--- a/pkg/installer/graph.go
+++ /dev/null
@@ -1,24 +0,0 @@
-package installer
-
-// Copyright (c) Microsoft Corporation.
-// Licensed under the Apache License 2.0.
-
-import (
-	"context"
-
-	"github.com/Azure/ARO-RP/pkg/cluster/graph"
-	"github.com/Azure/ARO-RP/pkg/util/stringutils"
-)
-
-func (m *manager) persistGraph(ctx context.Context, g graph.Graph) error {
-	resourceGroup := stringutils.LastTokenByte(m.doc.OpenShiftCluster.Properties.ClusterProfile.ResourceGroupID, '/')
-	clusterStorageAccountName := "cluster" + m.doc.OpenShiftCluster.Properties.StorageSuffix
-
-	exists, err := m.graph.Exists(ctx, resourceGroup, clusterStorageAccountName)
-	if err != nil || exists {
-		return err
-	}
-
-	// the graph is quite big, so we store it in a storage account instead of in cosmosdb
-	return m.graph.Save(ctx, resourceGroup, clusterStorageAccountName, g)
-}
diff --git a/pkg/installer/install.go b/pkg/installer/install.go
deleted file mode 100644
index fb11fb9d3..000000000
--- a/pkg/installer/install.go
+++ /dev/null
@@ -1,94 +0,0 @@
-package installer
-
-// Copyright (c) Microsoft Corporation.
-// Licensed under the Apache License 2.0.
-
-import (
-	"context"
-	"time"
-
-	"github.com/openshift/installer/pkg/asset/installconfig"
-	"github.com/openshift/installer/pkg/asset/kubeconfig"
-	"github.com/openshift/installer/pkg/asset/releaseimage"
-	"github.com/pkg/errors"
-	"k8s.io/client-go/kubernetes"
-	"k8s.io/client-go/tools/clientcmd"
-
-	"github.com/Azure/ARO-RP/pkg/cluster/graph"
-	"github.com/Azure/ARO-RP/pkg/util/restconfig"
-	"github.com/Azure/ARO-RP/pkg/util/steps"
-	"github.com/Azure/ARO-RP/pkg/util/stringutils"
-)
-
-func (m *manager) Install(ctx context.Context) error {
-	var (
-		installConfig *installconfig.InstallConfig
-		image         *releaseimage.Image
-		g             graph.Graph
-	)
-
-	s := []steps.Step{
-		steps.Action(func(ctx context.Context) error {
-			var err error
-			installConfig, image, err = m.generateInstallConfig(ctx)
-			return err
-		}),
-
-		steps.Action(func(ctx context.Context) error {
-			var err error
-			// Applies ARO-specific customisations to the InstallConfig
-			g, err = m.applyInstallConfigCustomisations(ctx, installConfig, image)
-			return err
-		}),
-		steps.Action(func(ctx context.Context) error {
-			return m.persistGraph(ctx, g)
-		}),
-		steps.AuthorizationRefreshingAction(m.fpAuthorizer, steps.Action(m.deployResourceTemplate)),
-		steps.Action(m.initializeKubernetesClients),
-		steps.Condition(m.bootstrapConfigMapReady, 30*time.Minute, true),
-	}
-
-	err := steps.Run(ctx, m.log, 10*time.Second, s)
-	return err
-}
-
-// initializeKubernetesClients initializes clients using the Installer-generated
-// kubeconfig.
-func (m *manager) initializeKubernetesClients(ctx context.Context) error {
-	resourceGroup := stringutils.LastTokenByte(m.doc.OpenShiftCluster.Properties.ClusterProfile.ResourceGroupID, '/')
-	account := "cluster" + m.doc.OpenShiftCluster.Properties.StorageSuffix
-
-	// Load the installer's generated kubeconfig from the graph
-	pg, err := m.graph.LoadPersisted(ctx, resourceGroup, account)
-	if err != nil {
-		return err
-	}
-
-	var adminInternalClient *kubeconfig.AdminInternalClient
-	err = pg.Get(&adminInternalClient)
-	if err != nil {
-		return err
-	}
-
-	// must not proceed if PrivateEndpointIP is not set.  In
-	// k8s.io/client-go/transport/cache.go, k8s caches our transport, and it
-	// can't tell if data in the restconfig.Dial closure has changed.  We don't
-	// want it to cache a transport that can never work.
-	if m.doc.OpenShiftCluster.Properties.NetworkProfile.APIServerPrivateEndpointIP == "" {
-		return errors.New("privateEndpointIP is empty")
-	}
-
-	config, err := clientcmd.Load(adminInternalClient.File.Data)
-	if err != nil {
-		return err
-	}
-
-	r, err := clientcmd.NewDefaultClientConfig(*config, &clientcmd.ConfigOverrides{}).ClientConfig()
-	if err != nil {
-		return err
-	}
-	r.Dial = restconfig.DialContext(m.env, m.doc.OpenShiftCluster)
-
-	m.kubernetescli, err = kubernetes.NewForConfig(r)
-	return err
-}
diff --git a/pkg/installer/manager.go b/pkg/installer/manager.go
deleted file mode 100644
index 2eb63d137..000000000
--- a/pkg/installer/manager.go
+++ /dev/null
@@ -1,48 +0,0 @@
-package installer
-
-// Copyright (c) Microsoft Corporation.
-// Licensed under the Apache License 2.0.
-
-import (
-	"context"
-
-	"github.com/sirupsen/logrus"
-	"k8s.io/client-go/kubernetes"
-
-	"github.com/Azure/ARO-RP/pkg/api"
-	"github.com/Azure/ARO-RP/pkg/cluster/graph"
-	"github.com/Azure/ARO-RP/pkg/env"
-	"github.com/Azure/ARO-RP/pkg/util/azureclient/mgmt/features"
-	"github.com/Azure/ARO-RP/pkg/util/refreshable"
-)
-
-type manager struct {
-	log *logrus.Entry
-	env env.Interface
-
-	doc             *api.OpenShiftClusterDocument
-	subscriptionDoc *api.SubscriptionDocument
-	fpAuthorizer    refreshable.Authorizer
-
-	deployments features.DeploymentsClient
-
-	graph graph.Manager
-
-	kubernetescli kubernetes.Interface
-}
-
-type Interface interface {
-	Install(ctx context.Context) error
-}
-
-func NewInstaller(log *logrus.Entry, _env env.Interface, doc *api.OpenShiftClusterDocument, subscriptionDoc *api.SubscriptionDocument, fpAuthorizer refreshable.Authorizer, deployments features.DeploymentsClient, g graph.Manager) Interface {
-	return &manager{
-		log:             log,
-		env:             _env,
-		doc:             doc,
-		subscriptionDoc: subscriptionDoc,
-		fpAuthorizer:    fpAuthorizer,
-		deployments:     deployments,
-		graph:           g,
-	}
-}
diff --git a/pkg/operator/deploy/deploy.go b/pkg/operator/deploy/deploy.go
index bafed828a..5f0f60718 100644
--- a/pkg/operator/deploy/deploy.go
+++ b/pkg/operator/deploy/deploy.go
@@ -4,15 +4,16 @@ package deploy
 // Licensed under the Apache License 2.0.
 
 import (
+	"bytes"
 	"context"
 	"embed"
 	"errors"
 	"fmt"
 	"strings"
+	"text/template"
 	"time"
 
 	"github.com/sirupsen/logrus"
-	appsv1 "k8s.io/api/apps/v1"
 	corev1 "k8s.io/api/core/v1"
 	extensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"
 	extensionsclient "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"
@@ -33,7 +34,6 @@ import (
 	aroclient "github.com/Azure/ARO-RP/pkg/operator/clientset/versioned"
 	"github.com/Azure/ARO-RP/pkg/operator/controllers/genevalogging"
 	"github.com/Azure/ARO-RP/pkg/util/dynamichelper"
-	utilembed "github.com/Azure/ARO-RP/pkg/util/embed"
 	"github.com/Azure/ARO-RP/pkg/util/pullsecret"
 	"github.com/Azure/ARO-RP/pkg/util/ready"
 	"github.com/Azure/ARO-RP/pkg/util/restconfig"
@@ -83,49 +83,82 @@ func New(log *logrus.Entry, env env.Interface, oc *api.OpenShiftCluster, arocli
 	}, nil
 }
 
-func (o *operator) staticResources() ([]kruntime.Object, error) {
-	results := []kruntime.Object{}
-	for _, fileBytes := range utilembed.ReadDirRecursive(embeddedFiles, "staticresources") {
-		obj, _, err := scheme.Codecs.UniversalDeserializer().Decode(fileBytes, nil, nil)
-		if err != nil {
-			return nil, err
-		}
+type deploymentData struct {
+	Image              string
+	Version            string
+	GitCommit          string
+	IsLocalDevelopment bool
+	HasVersion         bool
+}
 
-		// set the image for the deployments
-		if d, ok := obj.(*appsv1.Deployment); ok {
-			if d.Labels == nil {
-				d.Labels = map[string]string{}
-			}
-			var image string
-
-			if o.oc.Properties.OperatorVersion != "" {
-				image = fmt.Sprintf("%s/aro:%s", o.env.ACRDomain(), o.oc.Properties.OperatorVersion)
-				d.Labels["version"] = o.oc.Properties.OperatorVersion
-			} else {
-				image = o.env.AROOperatorImage()
-				d.Labels["version"] = version.GitCommit
-			}
+func templateManifests(data deploymentData) ([][]byte, error) {
+	templatesRoot, err := template.ParseFS(embeddedFiles, "staticresources/*.yaml")
+	if err != nil {
+		return nil, err
+	}
+	templatesMaster, err := template.ParseFS(embeddedFiles, "staticresources/master/*")
+	if err != nil {
+		return nil, err
+	}
+	templatesWorker, err := template.ParseFS(embeddedFiles, "staticresources/worker/*")
+	if err != nil {
+		return nil, err
+	}
 
-			for i := range d.Spec.Template.Spec.Containers {
-				d.Spec.Template.Spec.Containers[i].Image = image
+	templatedFiles := make([][]byte, 0)
+	templatesArray := []*template.Template{templatesMaster, templatesRoot, templatesWorker}
 
-				if o.env.IsLocalDevelopmentMode() {
-					d.Spec.Template.Spec.Containers[i].Env = append(d.Spec.Template.Spec.Containers[i].Env, corev1.EnvVar{
-						Name:  "RP_MODE",
-						Value: "development",
-					})
-				}
+	for _, templates := range templatesArray {
+		for _, templ := range templates.Templates() {
+			buff := &bytes.Buffer{}
+			if err := templ.Execute(buff, data); err != nil {
+				return nil, err
 			}
+			templatedFiles = append(templatedFiles, buff.Bytes())
 		}
+	}
+	return templatedFiles, nil
+}
+
+func (o *operator) createDeploymentData() deploymentData {
+	image := o.env.AROOperatorImage()
+	hasVersion := false
+	if o.oc.Properties.OperatorVersion != "" {
+		image = fmt.Sprintf("%s/aro", o.env.ACRDomain())
+		hasVersion = true
+	}
 
-		results = append(results, obj)
+	return deploymentData{
+		IsLocalDevelopment: o.env.IsLocalDevelopmentMode(),
+		Image:              image,
+		Version:            o.oc.Properties.OperatorVersion,
+		GitCommit:          version.GitCommit,
+		HasVersion:         hasVersion,
 	}
-	return results, nil
+}
+
+func (o *operator) createObjects() ([]kruntime.Object, error) {
+	deploymentData := o.createDeploymentData()
+	templated, err := templateManifests(deploymentData)
+	if err != nil {
+		return nil, err
+	}
+	objects := make([]kruntime.Object, 0, len(templated))
+	for _, v := range templated {
+		obj, _, err := scheme.Codecs.UniversalDeserializer().Decode(v, nil, nil)
+		if err != nil {
+			return nil, err
+		}
+		objects = append(objects, obj)
+	}
+
+	return objects, nil
 }
 
 func (o *operator) resources() ([]kruntime.Object, error) {
 	// first static resources from Assets
-	results, err := o.staticResources()
+
+	results, err := o.createObjects()
 	if err != nil {
 		return nil, err
 	}
diff --git a/pkg/operator/deploy/staticresources/master/deployment.yaml b/pkg/operator/deploy/staticresources/master/deployment.yaml.tmpl
similarity index 71%
rename from pkg/operator/deploy/staticresources/master/deployment.yaml
rename to pkg/operator/deploy/staticresources/master/deployment.yaml.tmpl
index 20dcffcc0..844adf473 100644
--- a/pkg/operator/deploy/staticresources/master/deployment.yaml
+++ b/pkg/operator/deploy/staticresources/master/deployment.yaml.tmpl
@@ -2,7 +2,12 @@ apiVersion: apps/v1
 kind: Deployment
 metadata:
   labels:
-    app: aro-operator-master
+    app: aro-operator-master 
+    {{ if .HasVersion}}
+    version: {{ .Version }} 
+    {{ else }}
+    version: unknown
+    {{end}}
   name: aro-operator-master
   namespace: openshift-azure-operator
 spec:
@@ -25,8 +30,13 @@ spec:
         args:
         - operator
         - master
-        image: GENERATED
+        image: "{{ .Image }}{{ if .HasVersion}}:{{ if .IsLocalDevelopment }}{{ .GitCommit }}{{ else }}{{ .Version }}{{end}}{{end}}"
         name: aro-operator
+        {{ if .IsLocalDevelopment}}
+        env: 
+        - name: "RP_MODE"
+          value: "development" 
+        {{ end }}
         ports:
         - containerPort: 8080
           name: http
diff --git a/pkg/operator/deploy/staticresources/worker/deployment.yaml b/pkg/operator/deploy/staticresources/worker/deployment.yaml.tmpl
similarity index 66%
rename from pkg/operator/deploy/staticresources/worker/deployment.yaml
rename to pkg/operator/deploy/staticresources/worker/deployment.yaml.tmpl
index a1809ef30..ada7882f4 100644
--- a/pkg/operator/deploy/staticresources/worker/deployment.yaml
+++ b/pkg/operator/deploy/staticresources/worker/deployment.yaml.tmpl
@@ -2,7 +2,12 @@ apiVersion: apps/v1
 kind: Deployment
 metadata:
   labels:
-    app: aro-operator-worker
+    app: aro-operator-worker 
+    {{ if .HasVersion}}
+    version: {{ .Version }} 
+    {{ else }}
+    version: unknown
+    {{end}}
   name: aro-operator-worker
   namespace: openshift-azure-operator
 spec:
@@ -25,7 +30,12 @@ spec:
         args:
         - operator
         - worker
-        image: GENERATED
+        image: "{{ .Image }}{{ if .HasVersion}}:{{ if .IsLocalDevelopment }}{{ .GitCommit }}{{ else }}{{ .Version }}{{end}}{{end}}"
+        {{ if .IsLocalDevelopment}}
+        env: 
+        - name: "RP_MODE"
+          value: "development" 
+        {{ end }}
         name: aro-operator
         livenessProbe:
           httpGet:


Test patch:
diff --git a/pkg/installer/deployresources_test.go b/pkg/cluster/deployresources_test.go
similarity index 99%
rename from pkg/installer/deployresources_test.go
rename to pkg/cluster/deployresources_test.go
index 0b7224d25..ba26eac02 100644
--- a/pkg/installer/deployresources_test.go
+++ b/pkg/cluster/deployresources_test.go
@@ -1,4 +1,4 @@
-package installer
+package cluster
 
 // Copyright (c) Microsoft Corporation.
 // Licensed under the Apache License 2.0.
diff --git a/pkg/operator/deploy/deploy_test.go b/pkg/operator/deploy/deploy_test.go
index 59bcfeeb5..c4d494c19 100644
--- a/pkg/operator/deploy/deploy_test.go
+++ b/pkg/operator/deploy/deploy_test.go
@@ -174,7 +174,7 @@ func TestOperatorVersion(t *testing.T) {
 				env: _env,
 			}
 
-			staticResources, err := o.staticResources()
+			staticResources, err := o.createObjects()
 			if err != nil {
 				t.Error(err)
 			}
@@ -192,7 +192,7 @@ func TestOperatorVersion(t *testing.T) {
 
 			for _, d := range deployments {
 				if d.Labels["version"] != tt.wantVersion {
-					t.Errorf("Got %q, not %q", d.Labels["version"], tt.wantVersion)
+					t.Errorf("Got %q, not %q for label \"version\"", d.Labels["version"], tt.wantVersion)
 				}
 
 				if len(d.Spec.Template.Spec.Containers) != 1 {
@@ -201,7 +201,7 @@ func TestOperatorVersion(t *testing.T) {
 
 				image := d.Spec.Template.Spec.Containers[0].Image
 				if image != tt.wantPullspec {
-					t.Errorf("Got %q, not %q", image, tt.wantPullspec)
+					t.Errorf("Got %q, not %q for the image", image, tt.wantPullspec)
 				}
 			}
 		})