Repository: linkerd/linkerd2
Pull request #3942 (2020-01-17 10:56:34)

cli: Update 'check' command to validate HA configuration
This PR addresses https://github.com/linkerd/linkerd2/issues/3505

Checks are added to validate the following properties of HA:
- [x] The control plane components (except, for web, grafana and prometheus) should have more than one replicas.
- [x] ~~The replicated pods should be scheduled on different hosts.~~ (Not required anymore)
- [x] ~~The MWC and VWC's FailurePolicy should be set to Fail to prevent uninjected workloads from entering the service mesh.~~ (Not required anymore)
- [x] ~~The control plane components are assigned CPU and memory resource requirements.~~ (Not required anymore)

Signed-off-by: Mayank Shah <mayankshah1614@gmail.com>

<!--  Thanks for sending a pull request!

If you already have a well-structured git commit message, chances are GitHub
set the title and description of this PR to the git commit message subject and
body, respectively. If so, you may delete these instructions and submit your PR.

If this is your first time, please read our contributor guide:
https://github.com/linkerd/linkerd2/blob/master/CONTRIBUTING.md

The title and description of your Pull Request should match the git commit
subject and body, respectively. Git commit messages are structured as follows:

```
Subject

Problem

Solution

Validation

Fixes #[GitHub issue ID]

DCO Sign off
```

Example git commit message:

```
Introduce Pull Request Template

GitHub's community guidelines recommend a pull request template, the repo was
lacking one.

Introduce a `PULL_REQUEST_TEMPLATE.md` file.

Once merged, the
[Community profile checklist](https://github.com/linkerd/linkerd2/community)
should indicate the repo now provides a pull request template.

Fixes #3321

Signed-off-by: Jane Smith <jane.smith@example.com>
```

Note the git commit message subject becomes the pull request title.

For more details around git commits, see the section on Committing in our
contributor guide:
https://github.com/linkerd/linkerd2/blob/master/CONTRIBUTING.md#committing
-->


adleong: Thanks for working on this, @mayankshah1607!  I'm going to wait until this is complete before giving it a full review, but if you have any questions or would like early feedback on some specific aspect of it, please let me know.
mayankshah1607: Done! Apologies for the delay. I have added the above mentioned checks. Would you mind taking a look? @adleong @zaharidichev @ihcsim 
adleong: Thanks @mayankshah1607, this is really good work.

As part of reviewing this I've gone back and read #3505 and #3305 in more detail and I think most of these checks are not actually necessary any more.  Since the issue was filed, we added hard host anti-affinity rules to the HA components that require that Kubernetes schedule them on different hosts.  Similarly, the resource requirements and the webhook failure policies are part of the HA manifests and are automatically configured when Linkerd is installed with the --HA flag.  

With all that said, I still think the check for at least 1 available replica is still valuable.

I apologize for the poorly scoped issue and I'm sorry that I didn't notice this earlier.  Would you mind keeping the available replica check and removing the others?

As a side note, it would be interesting to have a more general purpose check that verifies that the installed resources match the manifest yamls.  This would encompass the anti-affinity configuration, the webhook failure policy, the resource requirements, and much more.  However, I suspect that this is a much more complex and involved change.
mayankshah1607: @adleong no problem at all! :smile: 
Mind taking a look now? I have removed all the checks except the multiple replica one.
How would you feel about opening a separate issue for the general purpose check you mentioned? I could take that up with a little more detail provided :smile: 
mayankshah1607: @adleong @zaharidichev mind taking a look now?
- pulled changes from master
- renamed the func used in the check
- made the check retryable
zaharidichev: @mayankshah1607 This looks great to me as well. Will run the tests tomorrow to make sure it all passes :) 
zaharidichev: @mayankshah1607 I agree with @cpretzer. Wrt to adding tests you would like to look at `healthcheck_test.go`. There are facilities there to create "fake" resources and exercise this piece of code. Let me know if you have any questions
mayankshah1607: @zaharidichev thanks! I've made the changes. Mind taking a look now?

Patch:
diff --git a/CHANGES.md b/CHANGES.md
index e234ca567..0900b5a66 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -1,3 +1,168 @@
+## stable-2.7.0
+
+This release adds support for integrating Linkerd's PKI with an external
+certificate issuer such as [`cert-manager`] as well as streamlining the
+certificate rotation process in general. For more details about cert-manager
+and certificate rotation, see the
+[docs](https://linkerd.io/2/tasks/use_external_certs/). This release also
+includes performance improvements to the dashboard, reduced memory usage of the
+proxy, various improvements to the Helm chart, and much much more.
+
+To install this release, run: `curl https://run.linkerd.io/install | sh`
+
+**Upgrade notes**: This release includes breaking changes to our Helm charts.
+Please see the [upgrade instructions](https://linkerd.io/2/tasks/upgrade/#upgrade-notice-stable-270).
+
+**Special thanks to**: @alenkacz, @bmcstdio, @daxmc99, @droidnoob, @ereslibre,
+@javaducky, @joakimr-axis, @JohannesEH, @KIVagant, @mayankshah1607,
+@Pothulapati, and @StupidScience!
+
+**Full release notes**:
+
+* CLI
+  * Updated the mTLS trust anchor checks to eliminate false positives caused by
+    extra trailing spaces
+  * Reduced the severity level of the Linkerd version checks, so that they
+    don't fail when the external version endpoint is unreachable
+    (thanks @mayankshah1607!)
+  * Added a new `tap` APIService check to aid with uncovering Kubernetes API
+    aggregatation layer issues (thanks @droidnoob!)
+  * Introduced CNI checks to confirm the CNI plugin is installed and ready;
+    this is done through `linkerd check --pre --linkerd-cni-enabled` before
+    installation and `linkerd check` after installation if the CNI plugin is
+    present
+  * Added support for the `--as-group` flag so that users can impersonate
+    groups for Kubernetes operations (thanks @mayankshah1607!)
+  * Added HA specific checks to `linkerd check` to ensure that the `kube-system`
+    namespace has the `config.linkerd.io/admission-webhooks:disabled`
+    label set
+  * Fixed a problem causing the presence of unnecessary empty fields in
+    generated resource definitions (thanks @mayankshah1607)
+  * Added the ability to pass both port numbers and port ranges to
+    `--skip-inbound-ports` and `--skip-outbound-ports` (thanks to @javaducky!)
+  * Increased the comprehensiveness of `linkerd check --pre`
+  * Added TLS certificate validation to `check` and `upgrade` commands
+  * Added support for injecting CronJobs and ReplicaSets, as well as the ability
+    to use them as targets in the CLI subcommands
+  * Introduced the new flags `--identity-issuer-certificate-file`,
+    `--identity-issuer-key-file` and `identity-trust-anchors-file` to `linkerd
+    upgrade` to support trust anchor and issuer certificate rotation
+  * Added a check that ensures using `--namespace` and `--all-namespaces`
+    results in an error as they are mutually exclusive
+  * Added a `Dashboard.Replicas` parameter to the Linkerd Helm chart to allow
+    configuring the number of dashboard replicas (thanks @KIVagant!)
+  * Removed redundant service profile check (thanks @alenkacz!)
+  * Updated `uninject` command to work with namespace resources
+    (thanks @mayankshah1607!)
+  * Added a new `--identity-external-issuer` flag to `linkerd install` that
+    configures Linkerd to use certificates issued by an external certificate
+    issuer (such as `cert-manager`)
+  * Added support for injecting a namespace to `linkerd inject` (thanks
+    @mayankshah1607!)
+  * Added checks to `linkerd check --preinstall` ensuring Kubernetes Secrets
+    can be created and accessed
+  * Fixed `linkerd tap` sometimes displaying incorrect pod names for unmeshed
+    IPs that match multiple running pods
+  * Made `linkerd install --ignore-cluster` and `--skip-checks` faster
+  * Fixed a bug causing `linkerd upgrade` to fail when used with
+  `--from-manifest`
+  * Made `--cluster-domain` an install-only flag (thanks @bmcstdio!)
+  * Updated `check` to ensure that proxy trust anchors match configuration
+       (thanks @ereslibre!)
+  * Added condition to the `linkerd stat` command that requires a window size
+    of at least 15 seconds to work properly with Prometheus
+* Controller
+  * Fixed an issue where an override of the Docker registry was not being
+    applied to debug containers (thanks @javaducky!)
+  * Added check for the Subject Alternate Name attributes to the API server
+    when access restrictions have been enabled (thanks @javaducky!)
+  * Added support for arbitrary pod labels so that users can leverage the
+    Linkerd provided Prometheus instance to scrape for their own labels
+    (thanks @daxmc99!)
+  * Fixed an issue with CNI config parsing
+  * Fixed a race condition in the `linkerd-web` service
+  * Updated Prometheus to 2.15.2 (thanks @Pothulapati)
+  * Increased minimum kubernetes version to 1.13.0
+  * Added support for pod ip and service cluster ip lookups in the destination 
+    service
+  * Added recommended kubernetes labels to control-plane
+  * Added the `--wait-before-exit-seconds` flag to linkerd inject for the proxy 
+    sidecar to delay the start of its shutdown process (a huge commit from 
+    @KIVagant, thanks!)
+  * Added a pre-sign check to the identity service 
+  * Fixed inject failures for pods with security context capabilities
+  * Added `conntrack` to the `debug` container to help with connection tracking
+    debugging
+  * Fixed a bug in `tap` where mismatch cluster domain and trust domain caused
+    `tap` to hang
+  * Fixed an issue in the `identity` RBAC resource which caused start up errors
+    in k8s 1.6 (thanks @Pothulapati!)
+  * Added support for using trust anchors from an external certificate issuer
+    (such as `cert-mananger`) to the `linkerd-identity` service
+  * Added support for headless services (thanks @JohannesEH!)
+* Helm
+  * **Breaking change**: Renamed `noInitContainer` parameter to `cniEnabled`
+  * **Breaking Change** Updated Helm charts to follow best practices (thanks
+    @Pothulapati and @javaducky!)
+  * Fixed an issue with `helm install` where the lists of ignored inbound and
+    outbound ports would not be reflected
+  * Fixed the `linkerd-cni` Helm chart not setting proper namespace annotations
+    and labels
+  * Fixed certificate issuance lifetime not being set when installing through
+    Helm
+  * Updated the helm build to retain previous releases
+  * Moved CNI template into its own Helm chart
+* Proxy
+  * Fixed an issue that could cause the OpenCensus exporter to stall
+  * Improved error classification and error responses for gRPC services
+  * Fixed a bug where the proxy could stop receiving service discovery updates,
+    resulting in 503 errors
+  * Improved debug/error logging to include detailed contextual information
+  * Fixed a bug in the proxy's logging subsystem that could cause the proxy to
+    consume memory until the process is OOM killed, especially when the proxy was
+    configured to log diagnostic information
+  * Updated proxy dependencies to address RUSTSEC-2019-0033, RUSTSEC-2019-0034,
+    and RUSTSEC-2020-02
+* Web UI
+  * Fixed an error when refreshing an already open dashboard when the Linkerd
+    version has changed
+  * Increased the speed of the dashboard by pausing network activity when the 
+    dashboard is not visible to the user
+  * Added support for CronJobs and ReplicaSets, including new Grafana dashboards
+    for them
+  * Added `linkerd check` to the dashboard in the `/controlplane` view
+  * Added request and response headers to the `tap` expanded view in the
+    dashboard
+  * Added filter to namespace select button
+  * Improved how empty tables are displayed
+  * Added `Host:` header validation to the `linkerd-web` service, to protect
+    against DNS rebinding attacks
+  * Made the dashboard sidebar component responsive
+  * Changed the navigation bar color to the one used on the [Linkerd](https://linkerd.io/) website
+* Internal
+  * Added validation to incoming sidecar injection requests that ensures
+    the value of `linkerd.io/inject` is either `enabled` or `disabled`
+    (thanks @mayankshah1607)
+  * Upgraded the Prometheus Go client library to v1.2.1 (thanks @daxmc99!)
+  * Fixed an issue causing `tap`, `injector` and `sp-validator` to use 
+    old certificates after `helm upgrade` due to not being restarted
+  * Fixed incomplete Swagger definition of the tap api, causing benign
+    error logging in the kube-apiserver
+  * Removed the destination container from the linkerd-controller deployment as
+    it now runs in the linkerd-destination deployment
+  * Allowed the control plane to be injected with the `debug` container
+  * Updated proxy image build script to support HTTP proxy options
+    (thanks @joakimr-axis!)
+  * Updated the CLI `doc` command to auto-generate documentation for the proxy
+    configuration annotations (thanks @StupidScience!)
+  * Added new `--trace-collector` and `--trace-collector-svc-account` flags to
+    `linkerd inject` that configures the OpenCensus trace collector used by
+    proxies in the injected workload (thanks @Pothulapati!)
+  * Added a new `--control-plane-tracing` flag to `linkerd install` that enables
+    distributed tracing in the control plane (thanks @Pothulapati!)
+  * Added distributed tracing support to the control plane (thanks
+    @Pothulapati!)
+
 ## edge-20.2.1
 
 This edge release is a release candidate for `stable-2.7` and fixes an issue
diff --git a/Dockerfile-proxy b/Dockerfile-proxy
index 0ae139182..7d0da11f4 100644
--- a/Dockerfile-proxy
+++ b/Dockerfile-proxy
@@ -9,7 +9,7 @@ RUN (proxy=$(bin/fetch-proxy $(cat proxy-version)) && \
     mv "$proxy" linkerd2-proxy)
 
 ## compile proxy-identity agent
-FROM gcr.io/linkerd-io/go-deps:8d2ce4bf as golang
+FROM gcr.io/linkerd-io/go-deps:93ea34a8 as golang
 WORKDIR /linkerd-build
 COPY pkg/flags pkg/flags
 COPY pkg/tls pkg/tls
diff --git a/charts/linkerd2-cni/values.yaml b/charts/linkerd2-cni/values.yaml
index 610e5079f..3a3c502f9 100644
--- a/charts/linkerd2-cni/values.yaml
+++ b/charts/linkerd2-cni/values.yaml
@@ -6,7 +6,7 @@ ignoreInboundPorts: ""
 ignoreOutboundPorts: ""
 createdByAnnotation: linkerd.io/created-by
 cniPluginImage:   "gcr.io/linkerd-io/cni-plugin"
-cniPluginVersion: edge-20.2.1
+cniPluginVersion: stable-2.7.0
 logLevel:         info
 portsToRedirect:  ""
 proxyUID:         2102
diff --git a/charts/linkerd2/values.yaml b/charts/linkerd2/values.yaml
index 7123590b7..f8b609205 100644
--- a/charts/linkerd2/values.yaml
+++ b/charts/linkerd2/values.yaml
@@ -11,7 +11,7 @@ global:
   controlPlaneTracing: false
 
   # control plane version. See Proxy section for proxy version
-  linkerdVersion: &linkerd_version edge-20.2.1
+  linkerdVersion: &linkerd_version stable-2.7.0
 
   namespace: linkerd
 
diff --git a/cli/Dockerfile-bin b/cli/Dockerfile-bin
index ced183955..c465d05df 100644
--- a/cli/Dockerfile-bin
+++ b/cli/Dockerfile-bin
@@ -1,5 +1,5 @@
 ## compile binaries
-FROM gcr.io/linkerd-io/go-deps:8d2ce4bf as golang
+FROM gcr.io/linkerd-io/go-deps:93ea34a8 as golang
 WORKDIR /linkerd-build
 COPY cli cli
 COPY charts charts
diff --git a/cni-plugin/Dockerfile b/cni-plugin/Dockerfile
index b229039b6..9ae99bcb2 100644
--- a/cni-plugin/Dockerfile
+++ b/cni-plugin/Dockerfile
@@ -1,5 +1,5 @@
 ## compile cni-plugin utility
-FROM gcr.io/linkerd-io/go-deps:8d2ce4bf as golang
+FROM gcr.io/linkerd-io/go-deps:93ea34a8 as golang
 WORKDIR /linkerd-build
 COPY pkg pkg
 COPY controller controller
diff --git a/controller/Dockerfile b/controller/Dockerfile
index 1c9dd7f65..b4cc08afc 100644
--- a/controller/Dockerfile
+++ b/controller/Dockerfile
@@ -1,5 +1,5 @@
 ## compile controller service
-FROM gcr.io/linkerd-io/go-deps:8d2ce4bf as golang
+FROM gcr.io/linkerd-io/go-deps:93ea34a8 as golang
 WORKDIR /linkerd-build
 COPY controller/gen controller/gen
 COPY pkg pkg
diff --git a/go.mod b/go.mod
index 3d5d222ee..dd73d025b 100644
--- a/go.mod
+++ b/go.mod
@@ -1,6 +1,6 @@
 module github.com/linkerd/linkerd2
 
-go 1.13.4
+go 1.13
 
 require (
 	contrib.go.opencensus.io/exporter/ocagent v0.6.0
diff --git a/go.sum b/go.sum
index 48a0f60aa..518fec565 100644
--- a/go.sum
+++ b/go.sum
@@ -211,10 +211,6 @@ github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
 github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
 github.com/linkerd/linkerd2-proxy-api v0.1.10 h1:zGwz2Fek7fdfCxyZW7jq8IE6sexeVFW5UFBQFL6MrP4=
 github.com/linkerd/linkerd2-proxy-api v0.1.10/go.mod h1:2WJHEYXoww5ALM6c1QspRFiROGZg08tGxCO1js0tTVA=
-github.com/linkerd/linkerd2-proxy-init v1.3.0 h1:kyrMpaNKGPfd7FA8O/6VGxsYXx++TxxZVgRL6knWSa8=
-github.com/linkerd/linkerd2-proxy-init v1.3.0/go.mod h1:lK/sORPZDP6aFhPZps+AHMKphX5v9bKVDNESllz18C0=
-github.com/linkerd/linkerd2-proxy-init v1.3.1-0.20200117170322-a7441ab79b25 h1:RVz/okrTQugk9GKUDY9mJbl8mKUh90jfxTu97IvBvjg=
-github.com/linkerd/linkerd2-proxy-init v1.3.1-0.20200117170322-a7441ab79b25/go.mod h1:lK/sORPZDP6aFhPZps+AHMKphX5v9bKVDNESllz18C0=
 github.com/linkerd/linkerd2-proxy-init v1.3.1 h1:KTvUvPlSKUkpzZM0L/EIk+EJaKk7dwygu0CP95fOyGI=
 github.com/linkerd/linkerd2-proxy-init v1.3.1/go.mod h1:lK/sORPZDP6aFhPZps+AHMKphX5v9bKVDNESllz18C0=
 github.com/linkerd/stern v0.0.0-20190907020106-201e8ccdff9c h1:v8RmXOGO65HHLqcDpVNGGe3krA8FptfxcYS3wSS8bJU=
diff --git a/pkg/healthcheck/healthcheck.go b/pkg/healthcheck/healthcheck.go
index 2bba7aa45..27cdd4d38 100644
--- a/pkg/healthcheck/healthcheck.go
+++ b/pkg/healthcheck/healthcheck.go
@@ -166,6 +166,15 @@ const HintBaseURL = "https://linkerd.io/checks/#"
 // TODO: Make this default value overridiable, e.g. by CLI flag
 const AllowedClockSkew = time.Minute + tls.DefaultClockSkewAllowance
 
+var linkerdHAControlPlaneComponents = []string{
+	"linkerd-controller",
+	"linkerd-destination",
+	"linkerd-identity",
+	"linkerd-proxy-injector",
+	"linkerd-sp-validator",
+	"linkerd-tap",
+}
+
 var (
 	retryWindow    = 5 * time.Second
 	requestTimeout = 30 * time.Second
@@ -1167,11 +1176,43 @@ func (hc *HealthChecker) allCategories() []category {
 						return &SkipError{Reason: "not run for non HA installs"}
 					},
 				},
+				{
+					description:   "multiple replicas of control plane pods",
+					hintAnchor:    "l5d-control-plane-replicas",
+					retryDeadline: hc.RetryDeadline,
+					warning:       true,
+					check: func(ctx context.Context) error {
+						if hc.isHA() {
+							return hc.checkMinReplicasAvailable()
+						}
+						return &SkipError{Reason: "not run for non HA installs"}
+					},
+				},
 			},
 		},
 	}
 }
 
+func (hc *HealthChecker) checkMinReplicasAvailable() error {
+	faulty := []string{}
+
+	for _, component := range linkerdHAControlPlaneComponents {
+		conf, err := hc.kubeAPI.AppsV1().Deployments(hc.ControlPlaneNamespace).Get(component, metav1.GetOptions{})
+		if err != nil {
+			return err
+		}
+
+		if conf.Status.AvailableReplicas <= 1 {
+			faulty = append(faulty, component)
+		}
+	}
+
+	if len(faulty) > 0 {
+		return fmt.Errorf("not enough replicas available for %v", faulty)
+	}
+	return nil
+}
+
 func (hc *HealthChecker) issuerIdentity() string {
 	return fmt.Sprintf("identity.%s.%s", hc.ControlPlaneNamespace, hc.linkerdConfig.Global.IdentityContext.TrustDomain)
 }
diff --git a/web/Dockerfile b/web/Dockerfile
index 6de090f67..439831140 100644
--- a/web/Dockerfile
+++ b/web/Dockerfile
@@ -21,7 +21,7 @@ COPY web/app ./web/app
 RUN ./bin/web build
 
 ## compile go server
-FROM gcr.io/linkerd-io/go-deps:8d2ce4bf as golang
+FROM gcr.io/linkerd-io/go-deps:93ea34a8 as golang
 WORKDIR /linkerd-build
 RUN mkdir -p web
 COPY web/main.go web


Test patch:
diff --git a/pkg/healthcheck/healthcheck_test.go b/pkg/healthcheck/healthcheck_test.go
index 1f45532c9..0ce06776e 100644
--- a/pkg/healthcheck/healthcheck_test.go
+++ b/pkg/healthcheck/healthcheck_test.go
@@ -3029,3 +3029,125 @@ func TestCniChecks(t *testing.T) {
 	}
 
 }
+
+func TestMinReplicaCheck(t *testing.T) {
+	hc := NewHealthChecker(
+		[]CategoryID{LinkerdHAChecks},
+		&Options{
+			ControlPlaneNamespace: "linkerd",
+		},
+	)
+
+	var err error
+
+	testCases := []struct {
+		controlPlaneResourceDefs []string
+		expected                 error
+	}{
+		{
+			controlPlaneResourceDefs: generateAllControlPlaneDef(&controlPlaneReplicaOptions{
+				controller:    1,
+				destination:   3,
+				identity:      3,
+				proxyInjector: 3,
+				spValidator:   1,
+				tap:           3,
+			}, t),
+			expected: fmt.Errorf("not enough replicas available for [linkerd-controller linkerd-sp-validator]"),
+		},
+		{
+			controlPlaneResourceDefs: generateAllControlPlaneDef(&controlPlaneReplicaOptions{
+				controller:    3,
+				destination:   2,
+				identity:      1,
+				proxyInjector: 1,
+				spValidator:   0,
+				tap:           3,
+			}, t),
+			expected: fmt.Errorf("not enough replicas available for [linkerd-identity linkerd-proxy-injector linkerd-sp-validator]"),
+		},
+		{
+			controlPlaneResourceDefs: generateAllControlPlaneDef(&controlPlaneReplicaOptions{
+				controller:    3,
+				destination:   2,
+				identity:      2,
+				proxyInjector: 3,
+				spValidator:   2,
+				tap:           3,
+			}, t),
+			expected: nil,
+		},
+	}
+
+	for i, tc := range testCases {
+		tc := tc //pin
+		t.Run(fmt.Sprintf("%d", i), func(t *testing.T) {
+			hc.kubeAPI, err = k8s.NewFakeAPI(tc.controlPlaneResourceDefs...)
+			if err != nil {
+				t.Fatal(err)
+			}
+			err = hc.checkMinReplicasAvailable()
+			if err == nil && tc.expected != nil {
+				t.Log("Expected error: nil")
+				t.Logf("Received error: %s\n", err)
+				t.Fatal("test case failed")
+			}
+			if err != nil {
+				if err.Error() != tc.expected.Error() {
+					t.Logf("Expected error: %s\n", tc.expected)
+					t.Logf("Received error: %s\n", err)
+					t.Fatal("test case failed")
+				}
+			}
+		})
+	}
+}
+
+type controlPlaneReplicaOptions struct {
+	controller    int
+	destination   int
+	identity      int
+	proxyInjector int
+	spValidator   int
+	tap           int
+}
+
+func getSingleControlPlaneDef(component string, availableReplicas int) string {
+	return fmt.Sprintf(`
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: %s
+  namespace: linkerd
+spec:
+  template:
+    spec:
+      containers:
+        - image: "hello-world"
+          name: test
+status:
+  availableReplicas: %d`, component, availableReplicas)
+}
+
+func generateAllControlPlaneDef(replicaOptions *controlPlaneReplicaOptions, t *testing.T) []string {
+	resourceDefs := []string{}
+	for _, component := range linkerdHAControlPlaneComponents {
+		switch component {
+		case "linkerd-controller":
+			resourceDefs = append(resourceDefs, getSingleControlPlaneDef(component, replicaOptions.controller))
+		case "linkerd-destination":
+			resourceDefs = append(resourceDefs, getSingleControlPlaneDef(component, replicaOptions.destination))
+		case "linkerd-identity":
+			resourceDefs = append(resourceDefs, getSingleControlPlaneDef(component, replicaOptions.identity))
+		case "linkerd-sp-validator":
+			resourceDefs = append(resourceDefs, getSingleControlPlaneDef(component, replicaOptions.spValidator))
+		case "linkerd-proxy-injector":
+			resourceDefs = append(resourceDefs, getSingleControlPlaneDef(component, replicaOptions.proxyInjector))
+		case "linkerd-tap":
+			resourceDefs = append(resourceDefs, getSingleControlPlaneDef(component, replicaOptions.tap))
+		default:
+			t.Fatal("Could not find the resource")
+		}
+	}
+	return resourceDefs
+}
diff --git a/test/install_test.go b/test/install_test.go
index a049475f0..2dcfdee30 100644
--- a/test/install_test.go
+++ b/test/install_test.go
@@ -334,15 +334,14 @@ func TestInstallOrUpgradeCli(t *testing.T) {
 // These need to be updated (if there are changes) once a new stable is released
 func helmOverridesStable(root *tls.CA) []string {
 	return []string{
-		"--set", "Namespace=" + TestHelper.GetLinkerdNamespace(),
-		"--set", "ControllerLogLevel=debug",
-		"--set", "LinkerdVersion=" + TestHelper.UpgradeHelmFromVersion(),
-		"--set", "Proxy.Image.Version=" + TestHelper.UpgradeHelmFromVersion(),
-		"--set", "Identity.TrustDomain=cluster.local",
-		"--set", "Identity.TrustAnchorsPEM=" + root.Cred.Crt.EncodeCertificatePEM(),
-		"--set", "Identity.Issuer.TLS.CrtPEM=" + root.Cred.Crt.EncodeCertificatePEM(),
-		"--set", "Identity.Issuer.TLS.KeyPEM=" + root.Cred.EncodePrivateKeyPEM(),
-		"--set", "Identity.Issuer.CrtExpiry=" + root.Cred.Crt.Certificate.NotAfter.Format(time.RFC3339),
+		"--set", "controllerLogLevel=debug",
+		"--set", "global.linkerdVersion=" + TestHelper.UpgradeHelmFromVersion(),
+		"--set", "global.proxy.image.version=" + TestHelper.UpgradeHelmFromVersion(),
+		"--set", "global.identityTrustDomain=cluster.local",
+		"--set", "global.identityTrustAnchorsPEM=" + root.Cred.Crt.EncodeCertificatePEM(),
+		"--set", "identity.issuer.tls.crtPEM=" + root.Cred.Crt.EncodeCertificatePEM(),
+		"--set", "identity.issuer.tls.keyPEM=" + root.Cred.EncodePrivateKeyPEM(),
+		"--set", "identity.issuer.crtExpiry=" + root.Cred.Crt.Certificate.NotAfter.Format(time.RFC3339),
 	}
 }