Repository: EnMasseProject/enmasse
Pull request #3446 (2019-11-08 11:52:58)

Scale down iot-operator before beginning upgrade
### Type of change

<!--

_Select the type of your PR_

-->

- Refactoring


### Description

* Split upgrader into separate module
* Run upgrader before starting controller loops
* Refactor scaling code into separate function.
* Stop iot-operator if it exists
* When scaling up, wait until readiness probe has passed


<!--

_Please describe your pull request_

-->

### Checklist

<!--

_Please go through this checklist and make sure all applicable tasks have been done_

-->

- [ ] Update/write design documentation in `./documentation/design`
- [ ] Write tests and make sure they pass
- [ ] Update documentation
- [ ] Check RBAC rights for Kubernetes / OpenShift roles
- [ ] Try your changes from Pod inside your Kubernetes and OpenShift cluster, not just locally
- [ ] Reference relevant issue(s) and close them after merging
- [ ] Update CHANGELOG.md


lulf: @enmasse-ci run tests profile=upgrade upgrade_from=0.29.2
lulf: @enmasse-ci run tests profile=upgrade upgrade_from=0.29.2
enmasse-ci: ### :x: Test Summary :x:
**TEST_PROFILE**: upgrade
**TEST_CASE:** 
**TOTAL:** 2
**PASS:** 0
**FAIL:** 2
**SKIP:** 0
### :heavy_exclamation_mark: Test Failures :heavy_exclamation_mark:
- testUpgradeBundle in io.enmasse.systemtest.isolated.upgrade.UpgradeTest
- testUpgradeAnsible in io.enmasse.systemtest.isolated.upgrade.UpgradeTest
lulf: @enmasse-ci run tests profile=upgrade upgrade_from=0.29.2
enmasse-ci: ### :heavy_check_mark: Test Summary :heavy_check_mark:
**TEST_PROFILE**: upgrade
**TEST_CASE:** 
**TOTAL:** 2
**PASS:** 2
**FAIL:** 0
**SKIP:** 0

lulf: @enmasse-ci run tests profile=upgrade upgrade_from=0.29.2
enmasse-ci: ### :heavy_check_mark: Test Summary :heavy_check_mark:
**TEST_PROFILE**: upgrade
**TEST_CASE:** 
**TOTAL:** 2
**PASS:** 2
**FAIL:** 0
**SKIP:** 0

lulf: @enmasse-ci run tests profile=iot
lulf: @enmasse-ci run tests profile=shared
enmasse-ci: ### :x: Test Summary :x:
**TEST_PROFILE**: iot
**TEST_CASE:** 
**TOTAL:** 34
**PASS:** 0
**FAIL:** 30
**SKIP:** 4
### :heavy_exclamation_mark: Test Failures :heavy_exclamation_mark:
- testDeviceCredentials in io.enmasse.systemtest.iot.isolated.registry.InfinispanDeviceRegistryTest
- testDeviceCredentialsPlainPassword in io.enmasse.systemtest.iot.isolated.registry.InfinispanDeviceRegistryTest
- testSetExpiryForCredentials in io.enmasse.systemtest.iot.isolated.registry.InfinispanDeviceRegistryTest
- testCreateDuplicateDeviceFails in io.enmasse.systemtest.iot.isolated.registry.InfinispanDeviceRegistryTest
- testCorrectTypeDeployed in io.enmasse.systemtest.iot.isolated.registry.InfinispanDeviceRegistryTest
- testDisableDevice in io.enmasse.systemtest.iot.isolated.registry.InfinispanDeviceRegistryTest
- testCreateForNonExistingTenantFails in io.enmasse.systemtest.iot.isolated.registry.InfinispanDeviceRegistryTest
- testRegisterDevice in io.enmasse.systemtest.iot.isolated.registry.InfinispanDeviceRegistryTest
- testDeviceCredentials in io.enmasse.systemtest.iot.isolated.registry.FileDeviceRegistryTest
- testDeviceCredentialsPlainPassword in io.enmasse.systemtest.iot.isolated.registry.FileDeviceRegistryTest
- testCorrectTypeDeployed in io.enmasse.systemtest.iot.isolated.registry.FileDeviceRegistryTest
- testDisableDevice in io.enmasse.systemtest.iot.isolated.registry.FileDeviceRegistryTest
- testRegisterDevice in io.enmasse.systemtest.iot.isolated.registry.FileDeviceRegistryTest
- testCreate in io.enmasse.systemtest.iot.isolated.IoTProjectManagedTest
- testMultipleProjects in io.enmasse.systemtest.iot.isolated.MultipleProjectsTest
- testChangeAddressSpace in io.enmasse.systemtest.iot.isolated.project.ManagedTest
- testEventSingle in io.enmasse.systemtest.iot.shared.mqtt.MqttAdapterTest
- testEventBatch5Before in io.enmasse.systemtest.iot.shared.mqtt.MqttAdapterTest
- testTelemetrySingleQos1 in io.enmasse.systemtest.iot.shared.mqtt.MqttAdapterTest
- testTelemetryBatch50Qos0 in io.enmasse.systemtest.iot.shared.mqtt.MqttAdapterTest
- testTelemetryBatch50Qos1 in io.enmasse.systemtest.iot.shared.mqtt.MqttAdapterTest
- testEventBatch5After in io.enmasse.systemtest.iot.shared.mqtt.MqttAdapterTest
- testOneShotCommand in io.enmasse.systemtest.iot.shared.control.CommandAndControlTest
- testRequestResponseCommand in io.enmasse.systemtest.iot.shared.control.CommandAndControlTest
- testEventSingle in io.enmasse.systemtest.iot.shared.http.HttpAdapterTest
- testEventBatch5Before in io.enmasse.systemtest.iot.shared.http.HttpAdapterTest
- testTelemetryBatch50 in io.enmasse.systemtest.iot.shared.http.HttpAdapterTest
- testTelemetrySingle in io.enmasse.systemtest.iot.shared.http.HttpAdapterTest
- testEventBatch5After in io.enmasse.systemtest.iot.shared.http.HttpAdapterTest
enmasse-ci: ### :x: Test Summary :x:
**TEST_PROFILE**: shared
**TEST_CASE:** 
**TOTAL:** 243
**PASS:** 179
**FAIL:** 6
**SKIP:** 58
### :heavy_exclamation_mark: Test Failures :heavy_exclamation_mark:
- testCreateDeleteQueue in io.enmasse.systemtest.shared.brokered.web.FirefoxWebConsoleTest
- testCreateUserWithWrongPayloadCRD in io.enmasse.systemtest.shared.brokered.api.UserApiTest
- previousPersistentSessionDiscarded in io.enmasse.systemtest.shared.standard.mqtt.SessionTest
- testWebSocketSendReceiveTopic in io.enmasse.systemtest.shared.standard.web.FirefoxWebSocketBrowserTest
- testVerylongAddressName in io.enmasse.systemtest.shared.standard.web.FirefoxWebConsoleTest
- testLargeMessages{JmsProvider} in io.enmasse.systemtest.shared.standard.TopicTest
lulf: @enmasse-ci run tests profile=upgrade
enmasse-ci: ### :heavy_check_mark: Test Summary :heavy_check_mark:
**TEST_PROFILE**: upgrade
**TEST_CASE:** 
**TOTAL:** 2
**PASS:** 2
**FAIL:** 0
**SKIP:** 0


Patch:
diff --git a/amqp-utils/pom.xml b/amqp-utils/pom.xml
index 340bf181d..0bed35c65 100644
--- a/amqp-utils/pom.xml
+++ b/amqp-utils/pom.xml
@@ -40,7 +40,7 @@
     </dependency>
     <dependency>
       <groupId>org.hamcrest</groupId>
-      <artifactId>hamcrest-all</artifactId>
+      <artifactId>hamcrest</artifactId>
       <scope>test</scope>
     </dependency>
     <dependency>
diff --git a/api-model/pom.xml b/api-model/pom.xml
index afe0db7f9..fc4b3fdb7 100644
--- a/api-model/pom.xml
+++ b/api-model/pom.xml
@@ -88,7 +88,7 @@
     </dependency>
     <dependency>
       <groupId>org.hamcrest</groupId>
-      <artifactId>hamcrest-all</artifactId>
+      <artifactId>hamcrest</artifactId>
       <scope>test</scope>
     </dependency>
   </dependencies>
diff --git a/cmd/controller-manager/main.go b/cmd/controller-manager/main.go
index 58f76f758..a0516e91a 100644
--- a/cmd/controller-manager/main.go
+++ b/cmd/controller-manager/main.go
@@ -62,6 +62,11 @@ func main() {
 		os.Exit(1)
 	}
 
+	if err := controller.CheckUpgrade(mgr); err != nil {
+		log.Error(err, "Failed to upgrade")
+		os.Exit(1)
+	}
+
 	// register controller
 
 	if err := controller.AddToManager(mgr); err != nil {
diff --git a/iot/iot-device-registry-infinispan/pom.xml b/iot/iot-device-registry-infinispan/pom.xml
index 453ae362b..2695bd656 100644
--- a/iot/iot-device-registry-infinispan/pom.xml
+++ b/iot/iot-device-registry-infinispan/pom.xml
@@ -76,7 +76,7 @@
     </dependency>
     <dependency>
       <groupId>org.hamcrest</groupId>
-      <artifactId>hamcrest-all</artifactId>
+      <artifactId>hamcrest</artifactId>
     </dependency>
     <dependency>
       <groupId>io.vertx</groupId>
diff --git a/k8s-api/pom.xml b/k8s-api/pom.xml
index 82935e995..9dba4ac80 100644
--- a/k8s-api/pom.xml
+++ b/k8s-api/pom.xml
@@ -45,7 +45,7 @@
     </dependency>
     <dependency>
       <groupId>org.hamcrest</groupId>
-      <artifactId>hamcrest-all</artifactId>
+      <artifactId>hamcrest</artifactId>
       <scope>test</scope>
     </dependency>
     <dependency>
diff --git a/pkg/controller/add_address_space_controller_deployer.go b/pkg/controller/add_address_space_controller.go
similarity index 55%
rename from pkg/controller/add_address_space_controller_deployer.go
rename to pkg/controller/add_address_space_controller.go
index 70388433f..eee7c7369 100644
--- a/pkg/controller/add_address_space_controller_deployer.go
+++ b/pkg/controller/add_address_space_controller.go
@@ -6,7 +6,7 @@
 package controller
 
 import (
-	deployer "github.com/enmasseproject/enmasse/pkg/controller/address_space_controller_deployer"
+	address_space_controller "github.com/enmasseproject/enmasse/pkg/controller/address_space_controller"
 	"github.com/enmasseproject/enmasse/pkg/util"
 )
 
@@ -14,7 +14,7 @@ func init() {
 
 	// add ourselves to the list of controllers
 
-	if util.IsModuleEnabled("ADDRESS_SPACE_CONTROLLER_DEPLOYER") {
-		AddToManagerFuncs = append(AddToManagerFuncs, deployer.Add)
+	if util.IsModuleEnabled("ADDRESS_SPACE_CONTROLLER") {
+		AddToManagerFuncs = append(AddToManagerFuncs, address_space_controller.Add)
 	}
 }
diff --git a/pkg/controller/address_space_controller/controller.go b/pkg/controller/address_space_controller/controller.go
new file mode 100644
index 000000000..1fea44a4c
--- /dev/null
+++ b/pkg/controller/address_space_controller/controller.go
@@ -0,0 +1,261 @@
+/*
+ * Copyright 2019, EnMasse authors.
+ * License: Apache License 2.0 (see the file LICENSE or http://apache.org/licenses/LICENSE-2.0.html).
+ */
+
+package address_space_controller
+
+import (
+	"context"
+	"os"
+	"time"
+
+	"github.com/enmasseproject/enmasse/pkg/util"
+	"github.com/enmasseproject/enmasse/pkg/util/images"
+	"github.com/enmasseproject/enmasse/pkg/util/install"
+
+	"github.com/go-logr/logr"
+
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/controller"
+	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
+	"sigs.k8s.io/controller-runtime/pkg/handler"
+	"sigs.k8s.io/controller-runtime/pkg/manager"
+	"sigs.k8s.io/controller-runtime/pkg/reconcile"
+	logf "sigs.k8s.io/controller-runtime/pkg/runtime/log"
+	"sigs.k8s.io/controller-runtime/pkg/source"
+
+	appsv1 "k8s.io/api/apps/v1"
+	corev1 "k8s.io/api/core/v1"
+	k8errors "k8s.io/apimachinery/pkg/api/errors"
+	resource "k8s.io/apimachinery/pkg/api/resource"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	runtime "k8s.io/apimachinery/pkg/runtime"
+	schema "k8s.io/apimachinery/pkg/runtime/schema"
+	controllertypes "k8s.io/apimachinery/pkg/types"
+	intstr "k8s.io/apimachinery/pkg/util/intstr"
+	kubernetes "k8s.io/client-go/kubernetes"
+)
+
+var log = logf.Log.WithName("controller_address_space_controller")
+
+const ADDRESS_SPACE_CONTROLLER_NAME = "address-space-controller"
+const ANNOTATION_VERSION = "enmasse.io/version"
+const ENV_VERSION = "VERSION"
+
+var _ reconcile.Reconciler = &ReconcileAddressSpaceController{}
+
+type ReconcileAddressSpaceController struct {
+	client     client.Client
+	kubeClient *kubernetes.Clientset
+	scheme     *runtime.Scheme
+	namespace  string
+}
+
+// Gets called by parent "init", adding as to the manager
+func Add(mgr manager.Manager) error {
+	reconciler, err := newReconciler(mgr)
+	if err != nil {
+		return err
+	}
+	return add(mgr, reconciler)
+}
+
+func newReconciler(mgr manager.Manager) (*ReconcileAddressSpaceController, error) {
+	// Create initial deployment if it does not exist
+	client, err := kubernetes.NewForConfig(mgr.GetConfig())
+	if err != nil {
+		log.Error(err, "Error creating kubernetes client")
+		return nil, err
+	}
+	return &ReconcileAddressSpaceController{
+		client:     mgr.GetClient(),
+		kubeClient: client,
+		scheme:     mgr.GetScheme(),
+		namespace:  util.GetEnvOrDefault("NAMESPACE", "enmasse-infra"),
+	}, nil
+}
+
+func add(mgr manager.Manager, r *ReconcileAddressSpaceController) error {
+
+	// Create initial deployment if it does not exist. We cannot yet rely on
+	// the controller-runtime client, as the runtime has not yet started.
+	deploymentClient := r.kubeClient.AppsV1().Deployments(r.namespace)
+	_, err := deploymentClient.Get(ADDRESS_SPACE_CONTROLLER_NAME, metav1.GetOptions{})
+	if err != nil {
+		if k8errors.IsNotFound(err) {
+			_, err = r.Reconcile(reconcile.Request{NamespacedName: controllertypes.NamespacedName{
+				Namespace: r.namespace,
+				Name:      ADDRESS_SPACE_CONTROLLER_NAME,
+			}})
+		} else {
+			return err
+		}
+	}
+
+	// Start reconciler for address-space-controller deployment
+	c, err := controller.New("address-space-controller-controller", mgr, controller.Options{Reconciler: r})
+	if err != nil {
+		return err
+	}
+
+	return c.Watch(&source.Kind{Type: &appsv1.Deployment{}}, &handler.EnqueueRequestForObject{})
+}
+
+func (r *ReconcileAddressSpaceController) Reconcile(request reconcile.Request) (reconcile.Result, error) {
+
+	expectedName := controllertypes.NamespacedName{
+		Namespace: r.namespace,
+		Name:      ADDRESS_SPACE_CONTROLLER_NAME,
+	}
+
+	if expectedName == request.NamespacedName {
+		reqLogger := log.WithValues("Request.Namespace", request.Namespace, "Request.Name", request.Name)
+		reqLogger.Info("Reconciling Address Space Controller")
+
+		ctx := context.TODO()
+
+		_, err := r.ensureDeployment(ctx, request, reqLogger)
+		if err != nil {
+			reqLogger.Error(err, "Error creating address space controller deployment")
+			return reconcile.Result{}, err
+		}
+
+		_, err = r.ensureService(ctx, request, reqLogger)
+		if err != nil {
+			reqLogger.Error(err, "Error creating address space controller service")
+			return reconcile.Result{}, err
+		}
+	}
+	return reconcile.Result{RequeueAfter: 30 * time.Second}, nil
+}
+
+func (r *ReconcileAddressSpaceController) ensureDeployment(ctx context.Context, request reconcile.Request, reqLogger logr.Logger) (reconcile.Result, error) {
+
+	deployment := &appsv1.Deployment{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      request.NamespacedName.Name,
+			Namespace: request.NamespacedName.Namespace,
+		},
+	}
+
+	deployment.SetGroupVersionKind(schema.GroupVersionKind{
+		Group:   "apps",
+		Kind:    "Deployment",
+		Version: "v1",
+	})
+
+	_, err := controllerutil.CreateOrUpdate(ctx, r.client, deployment, func(existing runtime.Object) error {
+		dep := existing.(*appsv1.Deployment)
+		return ApplyDeployment(dep)
+	})
+
+	return reconcile.Result{}, err
+}
+
+func ApplyDeployment(deployment *appsv1.Deployment) error {
+	install.ApplyDeploymentDefaults(deployment, "address-space-controller", deployment.Name)
+	err := install.ApplyContainerWithError(deployment, "address-space-controller", func(container *corev1.Container) error {
+		install.ApplyContainerImage(container, "address-space-controller", nil)
+		install.ApplyHttpProbe(container.LivenessProbe, 30, "/healthz", 8080)
+		install.ApplyHttpProbe(container.ReadinessProbe, 30, "/healthz", 8080)
+		container.Ports = []corev1.ContainerPort{
+			{Name: "http", ContainerPort: 8080, Protocol: corev1.ProtocolTCP},
+		}
+		install.ApplyEnvSimple(container, "JAVA_OPTS", "-verbose:gc")
+		install.ApplyEnvSimple(container, "ENABLE_EVENT_LOGGER", "true")
+		install.ApplyEnvSimple(container, "TEMPLATE_DIR", "/opt/templates")
+		install.ApplyEnvSimple(container, "RESOURCES_DIR", "/opt")
+		t := true
+		install.ApplyEnvConfigMap(container, "WILDCARD_ENDPOINT_CERT_SECRET", "wildcardEndpointCertSecret", "address-space-controller-config", &t)
+		install.ApplyEnvConfigMap(container, "RESYNC_INTERVAL", "resyncInterval", "address-space-controller-config", &t)
+		install.ApplyEnvConfigMap(container, "RECHECK_INTERVAL", "recheckInterval", "address-space-controller-config", &t)
+
+		install.ApplyEnvSimple(container, "IMAGE_PULL_POLICY", string(images.PullPolicyFromImageName(container.Image)))
+		applyImageEnv(container, "ROUTER_IMAGE", "qdrouterd-base")
+		applyImageEnv(container, "STANDARD_CONTROLLER_IMAGE", "standard-controller")
+		applyImageEnv(container, "AGENT_IMAGE", "agent")
+		applyImageEnv(container, "BROKER_IMAGE", "broker")
+		applyImageEnv(container, "BROKER_PLUGIN_IMAGE", "broker-plugin")
+		applyImageEnv(container, "TOPIC_FORWARDER_IMAGE", "topic-forwarder")
+		applyImageEnv(container, "MQTT_GATEWAY_IMAGE", "mqtt-gateway")
+		applyImageEnv(container, "MQTT_LWT_IMAGE", "mqtt-lwt")
+
+		memoryEnv := util.GetEnvOrDefault("ADDRSS_SPACE_CONTROLLER_MEMORY_LIMIT", "512Mi")
+		memoryLimit, err := resource.ParseQuantity(memoryEnv)
+		if err != nil {
+			return err
+		}
+
+		container.Resources = corev1.ResourceRequirements{
+			Requests: corev1.ResourceList{
+				corev1.ResourceMemory: memoryLimit,
+			},
+			Limits: corev1.ResourceList{
+				corev1.ResourceMemory: memoryLimit,
+			},
+		}
+		return nil
+	})
+	if err != nil {
+		return err
+	}
+	// This means we are running
+	version, ok := os.LookupEnv(ENV_VERSION)
+	if ok {
+		deployment.Annotations[ANNOTATION_VERSION] = version
+	}
+
+	var one int32 = 1
+	deployment.Spec.Template.Spec.ServiceAccountName = "address-space-controller"
+	deployment.Spec.Replicas = &one
+	install.ApplyNodeAffinity(&deployment.Spec.Template, "node-role.enmasse.io/operator-infra")
+	install.ApplyFsGroupOverride(deployment)
+
+	deployment.SetGroupVersionKind(schema.GroupVersionKind{
+		Group:   "apps",
+		Kind:    "Deployment",
+		Version: "v1",
+	})
+
+	return nil
+}
+
+func applyImageEnv(container *corev1.Container, env string, imageName string) error {
+	image, err := images.GetImage(imageName)
+	if err != nil {
+		return err
+	}
+	install.ApplyEnvSimple(container, env, image)
+	return nil
+}
+
+func (r *ReconcileAddressSpaceController) ensureService(ctx context.Context, request reconcile.Request, reqLogger logr.Logger) (reconcile.Result, error) {
+
+	service := &corev1.Service{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      request.NamespacedName.Name,
+			Namespace: request.NamespacedName.Namespace,
+		},
+	}
+
+	_, err := controllerutil.CreateOrUpdate(ctx, r.client, service, func(existing runtime.Object) error {
+		dep := existing.(*corev1.Service)
+		return applyService(dep)
+	})
+
+	return reconcile.Result{}, err
+}
+
+func applyService(service *corev1.Service) error {
+	install.ApplyServiceDefaults(service, service.Name, service.Name)
+	service.Spec.Ports = []corev1.ServicePort{
+		{
+			Port:       8080,
+			Protocol:   corev1.ProtocolTCP,
+			TargetPort: intstr.FromString("http"),
+			Name:       "health",
+		},
+	}
+	return nil
+}
diff --git a/pkg/controller/address_space_controller_deployer/deployer.go b/pkg/controller/address_space_controller_deployer/deployer.go
deleted file mode 100644
index a64cbdbd4..000000000
--- a/pkg/controller/address_space_controller_deployer/deployer.go
+++ /dev/null
@@ -1,395 +0,0 @@
-/*
- * Copyright 2019, EnMasse authors.
- * License: Apache License 2.0 (see the file LICENSE or http://apache.org/licenses/LICENSE-2.0.html).
- */
-
-package address_space_controller_deployment
-
-import (
-	"errors"
-	"fmt"
-	"github.com/enmasseproject/enmasse/pkg/util"
-	"github.com/enmasseproject/enmasse/pkg/util/images"
-	"github.com/enmasseproject/enmasse/pkg/util/install"
-	"os"
-	"strconv"
-	"time"
-
-	appsv1 "k8s.io/api/apps/v1"
-	corev1 "k8s.io/api/core/v1"
-	k8errors "k8s.io/apimachinery/pkg/api/errors"
-	resource "k8s.io/apimachinery/pkg/api/resource"
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	labels "k8s.io/apimachinery/pkg/labels"
-	"k8s.io/apimachinery/pkg/runtime"
-	intstr "k8s.io/apimachinery/pkg/util/intstr"
-	kubernetes "k8s.io/client-go/kubernetes"
-	corev1client "k8s.io/client-go/kubernetes/typed/core/v1"
-	rest "k8s.io/client-go/rest"
-	aggregatorclient "k8s.io/kube-aggregator/pkg/client/clientset_generated/clientset"
-	apiserviceclient "k8s.io/kube-aggregator/pkg/client/clientset_generated/clientset/typed/apiregistration/v1"
-	"sigs.k8s.io/controller-runtime/pkg/manager"
-	logf "sigs.k8s.io/controller-runtime/pkg/runtime/log"
-)
-
-var log = logf.Log.WithName("controller_address_space_controller_deployer")
-
-const ADDRESS_SPACE_CONTROLLER_NAME = "address-space-controller"
-const ANNOTATION_VERSION = "enmasse.io/version"
-const ENV_VERSION = "VERSION"
-
-type AddressSpaceControllerDeployment struct {
-	client    *kubernetes.Clientset
-	config    *rest.Config
-	scheme    *runtime.Scheme
-	namespace string
-}
-
-// Gets called by parent "init", adding as to the manager
-func Add(mgr manager.Manager) error {
-	dep, err := newDeployment(mgr)
-	if err != nil {
-		return err
-	}
-	return add(mgr, dep)
-}
-
-func newDeployment(mgr manager.Manager) (*AddressSpaceControllerDeployment, error) {
-	client, err := kubernetes.NewForConfig(mgr.GetConfig())
-	if err != nil {
-		log.Error(err, "Error creating kubernetes client")
-		return nil, err
-	}
-	return &AddressSpaceControllerDeployment{client: client, config: mgr.GetConfig(), scheme: mgr.GetScheme(), namespace: util.GetEnvOrDefault("NAMESPACE", "enmasse-infra")}, nil
-}
-
-func add(mgr manager.Manager, d *AddressSpaceControllerDeployment) error {
-
-	err := d.ensureDeployment()
-	if err != nil {
-		log.Error(err, "Error creating address space controller deployment")
-		return err
-	}
-
-	err = d.ensureService()
-	if err != nil {
-		log.Error(err, "Error creating address space controller service")
-		return err
-	}
-
-	return nil
-}
-
-func (d *AddressSpaceControllerDeployment) ensureDeployment() error {
-
-	deploymentClient := d.client.AppsV1().Deployments(d.namespace)
-
-	var deployment *appsv1.Deployment
-	deployment, err := deploymentClient.Get(ADDRESS_SPACE_CONTROLLER_NAME, metav1.GetOptions{})
-	if err != nil && k8errors.IsNotFound(err) {
-		deployment := &appsv1.Deployment{
-			ObjectMeta: metav1.ObjectMeta{Namespace: d.namespace, Name: ADDRESS_SPACE_CONTROLLER_NAME},
-		}
-		err = applyDeployment(deployment)
-		if err != nil {
-			return err
-		}
-		_, err := deploymentClient.Create(deployment)
-		return err
-	} else {
-		if _, ok := deployment.Annotations[ANNOTATION_VERSION]; !ok {
-			log.Info("address-space-controller is missing annotation, initiating upgrade")
-			err = d.performUpgrade(deployment)
-			if err != nil {
-				return err
-			}
-		}
-
-		err = applyDeployment(deployment)
-		if err != nil {
-			return err
-		}
-		log.Info("Updating address-space-controller deployment")
-		_, err := deploymentClient.Update(deployment)
-		return err
-	}
-}
-
-func (d *AddressSpaceControllerDeployment) performUpgrade(deployment *appsv1.Deployment) error {
-
-	podClient := d.client.CoreV1().Pods(d.namespace)
-	deploymentClient := d.client.AppsV1().Deployments(d.namespace)
-	var zero int32 = 0
-
-	// Scale down api-server
-	apiServerDeployment, err := deploymentClient.Get("api-server", metav1.GetOptions{})
-	if err != nil {
-		return err
-	}
-	log.Info("Downscaling api-server", "deployment", apiServerDeployment.Name)
-	apiServerDeployment.Spec.Replicas = &zero
-	_, err = deploymentClient.Update(apiServerDeployment)
-	if err != nil {
-		return err
-	}
-	err = waitForReplicas(podClient, apiServerDeployment, 0)
-	if err != nil {
-		return err
-	}
-	log.Info("Api-server scaled down", "deployment", apiServerDeployment.Name)
-
-	// Scale down address-space-controller
-	log.Info("Downscaling address-space-controller", "deployment", deployment.Name)
-	deployment.Spec.Replicas = &zero
-	_, err = deploymentClient.Update(deployment)
-	if err != nil {
-		return err
-	}
-	err = waitForReplicas(podClient, deployment, 0)
-	if err != nil {
-		return err
-	}
-	log.Info("Address-space-controller scaled down", "deployment", deployment.Name)
-
-	// Scale down admin pods
-	admins, err := deploymentClient.List(metav1.ListOptions{
-		LabelSelector: "name=admin",
-	})
-	if err != nil {
-		return err
-	}
-	for _, adminDeployment := range admins.Items {
-		adminDeployment.Spec.Replicas = &zero
-		log.Info("Downscaling admin", "deployment", adminDeployment.Name)
-		_, err = deploymentClient.Update(&adminDeployment)
-		if err != nil {
-			return err
-		}
-		err = waitForReplicas(podClient, &adminDeployment, 0)
-		if err != nil {
-			return err
-		}
-		log.Info("Admin scaled down", "deployment", adminDeployment.Name)
-	}
-
-	// Scale down agent pods
-	agents, err := deploymentClient.List(metav1.ListOptions{
-		LabelSelector: "role=agent",
-	})
-	if err != nil {
-		return err
-	}
-	for _, agentDeployment := range agents.Items {
-		agentDeployment.Spec.Replicas = &zero
-		log.Info("Downscaling agent", "deployment", agentDeployment.Name)
-		_, err = deploymentClient.Update(&agentDeployment)
-		if err != nil {
-			return err
-		}
-		err = waitForReplicas(podClient, &agentDeployment, 0)
-		if err != nil {
-			return err
-		}
-		log.Info("Agent scaled down", "deployment", agentDeployment.Name)
-	}
-
-	// Get rid of API service so that it doesn't prevent upgrade
-	aggClient, err := aggregatorclient.NewForConfig(d.config)
-	if err != nil {
-		return err
-	}
-	apiServiceClient := aggClient.ApiregistrationV1().APIServices()
-
-	log.Info("Deleting v1beta1.enmasse.io api service")
-	err = deleteApiServiceIfPresent(apiServiceClient, "v1beta1.enmasse.io")
-	if err != nil {
-		return err
-	}
-
-	log.Info("Deleting v1alpha1.enmasse.io api service")
-	err = deleteApiServiceIfPresent(apiServiceClient, "v1alpha1.enmasse.io")
-	if err != nil {
-		return err
-	}
-
-	// Scale api-server back up
-	log.Info("Scaling up api-server", "deployment", apiServerDeployment.Name)
-	var one int32 = 1
-	apiServerDeployment.Spec.Replicas = &one
-	_, err = deploymentClient.Update(apiServerDeployment)
-	if err != nil {
-		return err
-	}
-	err = waitForReplicas(podClient, apiServerDeployment, int(one))
-	if err != nil {
-		return err
-	}
-	log.Info("Api-server scaled up", "deployment", apiServerDeployment.Name)
-	return nil
-}
-
-func deleteApiServiceIfPresent(client apiserviceclient.APIServiceInterface, name string) error {
-	_, err := client.Get(name, metav1.GetOptions{})
-	if err != nil {
-		if k8errors.IsNotFound(err) {
-			return nil
-		}
-		return err
-	}
-
-	dopts := metav1.DeleteOptions{}
-	return client.Delete(name, &dopts)
-}
-
-func applyDeployment(deployment *appsv1.Deployment) error {
-	install.ApplyDeploymentDefaults(deployment, "address-space-controller", deployment.Name)
-	err := install.ApplyContainerWithError(deployment, "address-space-controller", func(container *corev1.Container) error {
-		install.ApplyContainerImage(container, "address-space-controller", nil)
-		install.ApplyHttpProbe(container.LivenessProbe, 30, "/healthz", 8080)
-		install.ApplyHttpProbe(container.ReadinessProbe, 30, "/healthz", 8080)
-		container.Ports = []corev1.ContainerPort{
-			{Name: "http", ContainerPort: 8080, Protocol: corev1.ProtocolTCP},
-		}
-		install.ApplyEnvSimple(container, "JAVA_OPTS", "-verbose:gc")
-		install.ApplyEnvSimple(container, "ENABLE_EVENT_LOGGER", "true")
-		install.ApplyEnvSimple(container, "TEMPLATE_DIR", "/opt/templates")
-		install.ApplyEnvSimple(container, "RESOURCES_DIR", "/opt")
-		t := true
-		install.ApplyEnvConfigMap(container, "WILDCARD_ENDPOINT_CERT_SECRET", "wildcardEndpointCertSecret", "address-space-controller-config", &t)
-		install.ApplyEnvConfigMap(container, "RESYNC_INTERVAL", "resyncInterval", "address-space-controller-config", &t)
-		install.ApplyEnvConfigMap(container, "RECHECK_INTERVAL", "recheckInterval", "address-space-controller-config", &t)
-
-		install.ApplyEnvSimple(container, "IMAGE_PULL_POLICY", string(images.PullPolicyFromImageName(container.Image)))
-		applyImageEnv(container, "ROUTER_IMAGE", "qdrouterd-base")
-		applyImageEnv(container, "STANDARD_CONTROLLER_IMAGE", "standard-controller")
-		applyImageEnv(container, "AGENT_IMAGE", "agent")
-		applyImageEnv(container, "BROKER_IMAGE", "broker")
-		applyImageEnv(container, "BROKER_PLUGIN_IMAGE", "broker-plugin")
-		applyImageEnv(container, "TOPIC_FORWARDER_IMAGE", "topic-forwarder")
-		applyImageEnv(container, "MQTT_GATEWAY_IMAGE", "mqtt-gateway")
-		applyImageEnv(container, "MQTT_LWT_IMAGE", "mqtt-lwt")
-
-		memoryEnv := util.GetEnvOrDefault("ADDRSS_SPACE_CONTROLLER_MEMORY_LIMIT", "512Mi")
-		memoryLimit, err := resource.ParseQuantity(memoryEnv)
-		if err != nil {
-			return err
-		}
-
-		container.Resources = corev1.ResourceRequirements{
-			Requests: corev1.ResourceList{
-				corev1.ResourceMemory: memoryLimit,
-			},
-			Limits: corev1.ResourceList{
-				corev1.ResourceMemory: memoryLimit,
-			},
-		}
-		return nil
-	})
-	if err != nil {
-		return err
-	}
-	// This means we are running
-	version, ok := os.LookupEnv(ENV_VERSION)
-	if ok {
-		deployment.Annotations[ANNOTATION_VERSION] = version
-	}
-
-	var one int32 = 1
-	deployment.Spec.Template.Spec.ServiceAccountName = "address-space-controller"
-	deployment.Spec.Replicas = &one
-	install.ApplyNodeAffinity(&deployment.Spec.Template, "node-role.enmasse.io/operator-infra")
-	install.ApplyFsGroupOverride(deployment)
-
-	return nil
-}
-
-func getBooleanAnnotationValue(Annotations map[string]string, name string) bool {
-	if Annotations != nil {
-		if sval, ok := Annotations[name]; ok {
-			if bval, ok := strconv.ParseBool(sval); ok == nil && bval {
-				return bval
-			}
-		}
-	}
-	return false
-}
-
-func applyImageEnv(container *corev1.Container, env string, imageName string) error {
-	image, err := images.GetImage(imageName)
-	if err != nil {
-		return err
-	}
-	install.ApplyEnvSimple(container, env, image)
-	return nil
-}
-
-func (d *AddressSpaceControllerDeployment) ensureService() error {
-
-	serviceClient := d.client.CoreV1().Services(d.namespace)
-
-	var service *corev1.Service
-	service, err := serviceClient.Get(ADDRESS_SPACE_CONTROLLER_NAME, metav1.GetOptions{})
-	if err != nil && k8errors.IsNotFound(err) {
-		service := &corev1.Service{
-			ObjectMeta: metav1.ObjectMeta{Namespace: d.namespace, Name: ADDRESS_SPACE_CONTROLLER_NAME},
-		}
-		applyService(service)
-		_, err := serviceClient.Create(service)
-		return err
-	} else {
-		applyService(service)
-		_, err := serviceClient.Update(service)
-		return err
-	}
-}
-
-func applyService(service *corev1.Service) error {
-	install.ApplyServiceDefaults(service, service.Name, service.Name)
-	service.Spec.Ports = []corev1.ServicePort{
-		{
-			Port:       8080,
-			Protocol:   corev1.ProtocolTCP,
-			TargetPort: intstr.FromString("http"),
-			Name:       "health",
-		},
-	}
-	return nil
-}
-
-func waitForReplicas(podClient corev1client.PodInterface, deployment *appsv1.Deployment, expectedReplicas int) error {
-
-	labelMap, err := metav1.LabelSelectorAsMap(deployment.Spec.Selector)
-	if err != nil {
-		return err
-	}
-	selector := labels.SelectorFromSet(labelMap).String()
-
-	log.Info(fmt.Sprintf("Looking for pods with label selector %s", selector))
-	opts := metav1.ListOptions{
-		LabelSelector: selector,
-	}
-
-	// Set a long timeout like 15 minutes to prevent from looping forever in case of errors
-	var timeout time.Duration = 15 * time.Minute
-	now := time.Now()
-	end := now.Add(timeout)
-
-	for now.Before(end) {
-		list, err := podClient.List(opts)
-		if err == nil && len(list.Items) == expectedReplicas {
-			return nil
-		}
-		log.Info(fmt.Sprintf("Found %d pods (expected %d)", len(list.Items), expectedReplicas))
-		time.Sleep(5 * time.Second)
-		now = time.Now()
-	}
-
-	list, err := podClient.List(opts)
-	if err != nil {
-		return err
-	}
-	if len(list.Items) != expectedReplicas {
-		return errors.New(fmt.Sprintf("Deployment %s has %d replicas. Expected: %d", deployment.Name, len(list.Items), expectedReplicas))
-	}
-	return nil
-}
diff --git a/pkg/controller/controller.go b/pkg/controller/controller.go
index 608acdf83..f2ce388b9 100644
--- a/pkg/controller/controller.go
+++ b/pkg/controller/controller.go
@@ -6,6 +6,7 @@
 package controller
 
 import (
+	"github.com/enmasseproject/enmasse/pkg/controller/upgrader"
 	"sigs.k8s.io/controller-runtime/pkg/manager"
 )
 
@@ -19,3 +20,11 @@ func AddToManager(m manager.Manager) error {
 	}
 	return nil
 }
+
+func CheckUpgrade(m manager.Manager) error {
+	upgrader, err := upgrader.New(m)
+	if err != nil {
+		return err
+	}
+	return upgrader.Upgrade()
+}
diff --git a/pkg/controller/upgrader/upgrader.go b/pkg/controller/upgrader/upgrader.go
new file mode 100644
index 000000000..70b19a7d9
--- /dev/null
+++ b/pkg/controller/upgrader/upgrader.go
@@ -0,0 +1,284 @@
+/*
+ * Copyright 2019, EnMasse authors.
+ * License: Apache License 2.0 (see the file LICENSE or http://apache.org/licenses/LICENSE-2.0.html).
+ */
+
+package upgrader
+
+import (
+	"errors"
+	"fmt"
+	deployer "github.com/enmasseproject/enmasse/pkg/controller/address_space_controller"
+	"github.com/enmasseproject/enmasse/pkg/util"
+	"strconv"
+	"time"
+
+	appsv1 "k8s.io/api/apps/v1"
+	corev1 "k8s.io/api/core/v1"
+	k8errors "k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	labels "k8s.io/apimachinery/pkg/labels"
+	"k8s.io/apimachinery/pkg/runtime"
+	kubernetes "k8s.io/client-go/kubernetes"
+	corev1client "k8s.io/client-go/kubernetes/typed/core/v1"
+	rest "k8s.io/client-go/rest"
+	"k8s.io/client-go/util/retry"
+	aggregatorclient "k8s.io/kube-aggregator/pkg/client/clientset_generated/clientset"
+	apiserviceclient "k8s.io/kube-aggregator/pkg/client/clientset_generated/clientset/typed/apiregistration/v1"
+	"sigs.k8s.io/controller-runtime/pkg/manager"
+	logf "sigs.k8s.io/controller-runtime/pkg/runtime/log"
+)
+
+var log = logf.Log.WithName("upgrader")
+
+const ADDRESS_SPACE_CONTROLLER_NAME = "address-space-controller"
+const ANNOTATION_VERSION = "enmasse.io/version"
+const ENV_VERSION = "VERSION"
+
+type Upgrader struct {
+	client    *kubernetes.Clientset
+	config    *rest.Config
+	scheme    *runtime.Scheme
+	namespace string
+}
+
+func New(mgr manager.Manager) (*Upgrader, error) {
+	client, err := kubernetes.NewForConfig(mgr.GetConfig())
+	if err != nil {
+		log.Error(err, "Error creating kubernetes client")
+		return nil, err
+	}
+	return &Upgrader{
+		client:    client,
+		config:    mgr.GetConfig(),
+		scheme:    mgr.GetScheme(),
+		namespace: util.GetEnvOrDefault("NAMESPACE", "enmasse-infra"),
+	}, nil
+}
+
+func (u *Upgrader) Upgrade() error {
+
+	deploymentClient := u.client.AppsV1().Deployments(u.namespace)
+
+	var deployment *appsv1.Deployment
+	deployment, err := deploymentClient.Get(ADDRESS_SPACE_CONTROLLER_NAME, metav1.GetOptions{})
+	if err != nil && k8errors.IsNotFound(err) {
+		return nil
+	} else if err != nil {
+		return err
+	} else {
+		if _, ok := deployment.Annotations[ANNOTATION_VERSION]; !ok {
+			log.Info("address-space-controller is missing annotation, initiating upgrade")
+			err = u.performUpgrade(deployment)
+			if err != nil {
+				return err
+			}
+		}
+	}
+	return nil
+}
+
+func (u *Upgrader) performUpgrade(addressSpaceControllerDeployment *appsv1.Deployment) error {
+	deploymentClient := u.client.AppsV1().Deployments(u.namespace)
+
+	// Scale down iot-operator
+	err := u.scale("iot-operator", 0)
+	if err != nil {
+		return err
+	}
+
+	// Scale down api-server
+	err = u.scale("api-server", 0)
+	if err != nil {
+		return err
+	}
+
+	// Scale down address-space-controller
+	err = u.scaleDeployment(addressSpaceControllerDeployment, 0)
+	if err != nil {
+		return err
+	}
+
+	// Scale down admin pods
+	admins, err := deploymentClient.List(metav1.ListOptions{
+		LabelSelector: "name=admin",
+	})
+	if err != nil {
+		return err
+	}
+	for _, adminDeployment := range admins.Items {
+		err = u.scaleDeployment(&adminDeployment, 0)
+		if err != nil {
+			return err
+		}
+	}
+
+	// Scale down agent pods
+	agents, err := deploymentClient.List(metav1.ListOptions{
+		LabelSelector: "role=agent",
+	})
+	if err != nil {
+		return err
+	}
+	for _, agentDeployment := range agents.Items {
+		err = u.scaleDeployment(&agentDeployment, 0)
+		if err != nil {
+			return err
+		}
+	}
+
+	// Get rid of API service so that it doesn't prevent upgrade
+	aggClient, err := aggregatorclient.NewForConfig(u.config)
+	if err != nil {
+		return err
+	}
+	apiServiceClient := aggClient.ApiregistrationV1().APIServices()
+
+	log.Info("Deleting v1beta1.enmasse.io api service")
+	err = deleteApiServiceIfPresent(apiServiceClient, "v1beta1.enmasse.io")
+	if err != nil {
+		return err
+	}
+
+	log.Info("Deleting v1alpha1.enmasse.io api service")
+	err = deleteApiServiceIfPresent(apiServiceClient, "v1alpha1.enmasse.io")
+	if err != nil {
+		return err
+	}
+
+	// Scale api-server back up
+	err = u.scale("api-server", 1)
+	if err != nil {
+		return err
+	}
+
+	// Scale address-space-controller back up. Retry in case the deployment got modified while we were scaling down
+	err = retry.RetryOnConflict(retry.DefaultRetry, func() error {
+		// Update address-space-controller deployment object to new version
+		err = deployer.ApplyDeployment(addressSpaceControllerDeployment)
+		if err != nil {
+			return err
+		}
+
+		// Apply new configuration and wait for it to scale up
+		return u.scaleDeployment(addressSpaceControllerDeployment, 1)
+	})
+	if err != nil {
+		return err
+	}
+
+	// Scale up iot-operator if deployed
+	err = u.scale("iot-operator", 1)
+	if err != nil {
+		return err
+	}
+	return nil
+}
+
+func (u *Upgrader) scale(deploymentName string, replicas int32) error {
+	deploymentClient := u.client.AppsV1().Deployments(u.namespace)
+
+	deployment, err := deploymentClient.Get(deploymentName, metav1.GetOptions{})
+	if err != nil && k8errors.IsNotFound(err) {
+		return nil
+	} else if err != nil {
+		return err
+	} else {
+		return u.scaleDeployment(deployment, replicas)
+	}
+}
+
+func (u *Upgrader) scaleDeployment(deployment *appsv1.Deployment, replicas int32) error {
+	podClient := u.client.CoreV1().Pods(u.namespace)
+	deploymentClient := u.client.AppsV1().Deployments(u.namespace)
+
+	log.Info("Downscaling", "deployment", deployment.Name)
+	deployment.Spec.Replicas = &replicas
+	_, err := deploymentClient.Update(deployment)
+	if err != nil {
+		return err
+	}
+	err = waitForReplicas(podClient, deployment, int(replicas))
+	if err != nil {
+		return err
+	}
+	log.Info("Scaled down", "deployment", deployment.Name)
+	return nil
+}
+
+func deleteApiServiceIfPresent(client apiserviceclient.APIServiceInterface, name string) error {
+	_, err := client.Get(name, metav1.GetOptions{})
+	if err != nil {
+		if k8errors.IsNotFound(err) {
+			return nil
+		}
+		return err
+	}
+
+	dopts := metav1.DeleteOptions{}
+	return client.Delete(name, &dopts)
+}
+
+func getBooleanAnnotationValue(Annotations map[string]string, name string) bool {
+	if Annotations != nil {
+		if sval, ok := Annotations[name]; ok {
+			if bval, ok := strconv.ParseBool(sval); ok == nil && bval {
+				return bval
+			}
+		}
+	}
+	return false
+}
+
+func waitForReplicas(podClient corev1client.PodInterface, deployment *appsv1.Deployment, expectedReplicas int) error {
+
+	labelMap, err := metav1.LabelSelectorAsMap(deployment.Spec.Selector)
+	if err != nil {
+		return err
+	}
+	selector := labels.SelectorFromSet(labelMap).String()
+
+	log.Info(fmt.Sprintf("Looking for pods with label selector %s", selector))
+	opts := metav1.ListOptions{
+		LabelSelector: selector,
+	}
+
+	// Set a long timeout like 15 minutes to prevent from looping forever in case of errors
+	var timeout time.Duration = 15 * time.Minute
+	now := time.Now()
+	end := now.Add(timeout)
+
+	for now.Before(end) {
+		list, err := podClient.List(opts)
+		numReady := 0
+		if err == nil && len(list.Items) == expectedReplicas {
+			numReady := 0
+			for _, pod := range list.Items {
+				if pod.Status.Phase == corev1.PodRunning {
+					for _, condition := range pod.Status.Conditions {
+						if condition.Type == corev1.PodReady &&
+							condition.Status == corev1.ConditionTrue {
+							numReady += 1
+							break
+						}
+					}
+				}
+			}
+			if numReady == expectedReplicas {
+				return nil
+			}
+		}
+		log.Info(fmt.Sprintf("Found %d ready pods (expected %d)", numReady, expectedReplicas))
+		time.Sleep(5 * time.Second)
+		now = time.Now()
+	}
+
+	list, err := podClient.List(opts)
+	if err != nil {
+		return err
+	}
+	if len(list.Items) != expectedReplicas {
+		return errors.New(fmt.Sprintf("Deployment %s has %d replicas. Expected: %d", deployment.Name, len(list.Items), expectedReplicas))
+	}
+	return nil
+}
diff --git a/pom.xml b/pom.xml
index bfdbe1b17..796bcf773 100644
--- a/pom.xml
+++ b/pom.xml
@@ -62,7 +62,7 @@
         <mockito.version>2.21.0</mockito.version>
         <commons-collections.version>3.2.2</commons-collections.version>
         <guava.version>25.0-jre</guava.version>
-        <hamcrest.version>1.3</hamcrest.version>
+        <hamcrest.version>2.2</hamcrest.version>
         <junit.jupiter.version>5.5.0</junit.jupiter.version>
         <junit.platform.runner.version>1.5.0</junit.platform.runner.version>
         <junit.platform.launcher.version>1.5.0</junit.platform.launcher.version>
@@ -622,7 +622,7 @@
 
             <dependency>
                 <groupId>org.hamcrest</groupId>
-                <artifactId>hamcrest-all</artifactId>
+                <artifactId>hamcrest</artifactId>
                 <version>${hamcrest.version}</version>
             </dependency>
 
diff --git a/standard-controller/pom.xml b/standard-controller/pom.xml
index 045c582ef..ee895b062 100644
--- a/standard-controller/pom.xml
+++ b/standard-controller/pom.xml
@@ -65,7 +65,7 @@
     </dependency>
     <dependency>
       <groupId>org.hamcrest</groupId>
-      <artifactId>hamcrest-all</artifactId>
+      <artifactId>hamcrest</artifactId>
       <scope>test</scope>
     </dependency>
     <dependency>
diff --git a/templates/csv/enmasse.clusterserviceversion.yaml b/templates/csv/enmasse.clusterserviceversion.yaml
index 51c7ba376..dfe9715c0 100644
--- a/templates/csv/enmasse.clusterserviceversion.yaml
+++ b/templates/csv/enmasse.clusterserviceversion.yaml
@@ -631,7 +631,7 @@ spec:
                   value: "true"
                 - name: CONTROLLER_ENABLE_AUTHENTICATION_SERVICE
                   value: "true"
-                - name: CONTROLLER_ENABLE_ADDRESS_SPACE_CONTROLLER_DEPLOYER
+                - name: CONTROLLER_ENABLE_ADDRESS_SPACE_CONTROLLER
                   value: "true"
                 - name: ADDRESS_SPACE_CONTROLLER_IMAGE
                   value: ${ADDRESS_SPACE_CONTROLLER_IMAGE}
diff --git a/templates/enmasse-operator/050-Deployment-enmasse-operator.yaml b/templates/enmasse-operator/050-Deployment-enmasse-operator.yaml
index c9897007f..b4c2f2d61 100644
--- a/templates/enmasse-operator/050-Deployment-enmasse-operator.yaml
+++ b/templates/enmasse-operator/050-Deployment-enmasse-operator.yaml
@@ -51,7 +51,7 @@ spec:
           value: "true"
         - name: CONTROLLER_ENABLE_AUTHENTICATION_SERVICE
           value: "true"
-        - name: CONTROLLER_ENABLE_ADDRESS_SPACE_CONTROLLER_DEPLOYER
+        - name: CONTROLLER_ENABLE_ADDRESS_SPACE_CONTROLLER
           value: "true"
         - name: ADDRESS_SPACE_CONTROLLER_IMAGE
           value: ${ADDRESS_SPACE_CONTROLLER_IMAGE}
diff --git a/topic-forwarder/pom.xml b/topic-forwarder/pom.xml
index 1ab8e2ae6..0643ce21d 100644
--- a/topic-forwarder/pom.xml
+++ b/topic-forwarder/pom.xml
@@ -61,7 +61,7 @@
     </dependency>
     <dependency>
       <groupId>org.hamcrest</groupId>
-      <artifactId>hamcrest-all</artifactId>
+      <artifactId>hamcrest</artifactId>
       <scope>test</scope>
     </dependency>
   </dependencies>


Test patch:
diff --git a/systemtests/pom.xml b/systemtests/pom.xml
index 95d2a7150..5f89e4b83 100644
--- a/systemtests/pom.xml
+++ b/systemtests/pom.xml
@@ -120,7 +120,7 @@
         </dependency>
         <dependency>
             <groupId>org.hamcrest</groupId>
-            <artifactId>hamcrest-all</artifactId>
+            <artifactId>hamcrest</artifactId>
         </dependency>
         <dependency>
             <groupId>com.google.code.gson</groupId>