Repository: linkerd/linkerd2 (go)

Issue:
linkerd upgrade --ha fails with "failed to read CA: not PEM-encoded"
## Bug Report

### What is the issue?

`linkerd upgrade --ha` fails on `edge-20.12.3`

### How can it be reproduced?

Command succeeds prior to this commit, fails with it: https://github.com/linkerd/linkerd2/commit/35612ae26823a9dfec51facc927f1b68051cd255

```bash
# works
$ git checkout 377b38f0bfb83421a3b74434aea43188810a0d01
$ go run cli/main.go upgrade --ha
---
###
### Linkerd Namespace
...

# breaks on the next commit
$ git checkout 35612ae26823a9dfec51facc927f1b68051cd255
$ go run cli/main.go upgrade --ha
Error: failed to validate issuer credentials: failed to read CA: not PEM-encoded
Usage:
...
```

#### `linkerd check` output

```bash
$ linkerd check --context aks-dive-staging
kubernetes-api
--------------
√ can initialize the client
...
√ issuer cert is within its validity period
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2021-01-27T08:02:19Z
    see https://linkerd.io/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
√ issuer cert is issued by the trust anchor
...
control-plane-version
---------------------
‼ control plane is up-to-date
    is running version 20.12.1 but the latest edge version is 20.12.3
    see https://linkerd.io/checks/#l5d-version-control for hints
‼ control plane and cli versions match
    control plane running edge-20.12.1 but cli running edge-20.12.3
    see https://linkerd.io/checks/#l5d-version-control for hints
...
Status check results are √
```

### Environment

- Kubernetes Version: v1.18.2
- Cluster Environment: AKS
- Host OS: Linux
- Linkerd version: edge-20.12.3

### Additional context

We use cert-manager.


Pull request description:
Fix CLI install/upgrade overriding settings in HA
Fixes #5385

## The problems

- `linkerd install --ha` isn't honoring flags
- `linkerd upgrade --ha` is overridding existing configs silently or failing with an error
- **Upgrading HA instances from before 2.9 to version 2.9.1 results in configs being overridden silently, or the upgrade fails with an error**

## The cause

The change in #5358 attempted to fix `linkerd install --ha` that was only applying some of the `values-ha.yaml` defaults, by calling `charts.NewValues(true)` and merging that with the values built from `values.yaml` overriden by the flags. It turns out the `charts.NewValues()` implementation was by itself merging against `values.yaml` and as a result any flag was getting overridden by its default.

This also happened when doing `linkerd upgrade --ha` on an existing instance, which could result in silently overriding settings, or it could also fail loudly like for example when upgrading set up that has an external issuer (in this case the issuer cert won't be able to be read during upgrade and an error would occur as described in #5385).

Finally, when doing `linkerd upgrade` (no --ha flag) on an HA install from before 2.9 results in configs getting overridden as well (silently or with an error) because in order to generate the `linkerd-config-overrides` secret, the original install flags are retrieved from `linkerd-config` via the `loadStoredValuesLegacy()` function which then effectively ends up performing a `linkerd upgrade` with all the flags used for `linkerd install` and falls into the same trap as above.

## The fix

In `values.go` the faulting merging logic is not used anymore, so now `NewValues()` only returns the default values from `values.yaml` and doesn't require an argument anymore. It calls `readDefaults()` which now only returns the appropriate values depending on whether we're on HA or not.
There's a new function `MergeHAValues()` that merges `values-ha.yaml` into the current values (it doesn't look into `values.yaml` anymore), which is only used when processing the `--ha` flag in `options.go`.

## How to test

To replicate the issue try setting a custom setting and check it's not applied:
```bash
linkerd install --ha --controller-log level debug | grep log.level
- -log-level=info
```

## Followup

This wasn't caught because we don't have HA integration tests. Now that our test infra is based on k3d, it should be easy to make such a test using a cluster with multiple nodes. Either that or issuing `linkerd install --ha` with additional configs and compare against a golden file.


Patch:
diff --git a/charts/linkerd2-cni/templates/cni-plugin.yaml b/charts/linkerd2-cni/templates/cni-plugin.yaml
index 68e15a254..f899797c9 100644
--- a/charts/linkerd2-cni/templates/cni-plugin.yaml
+++ b/charts/linkerd2-cni/templates/cni-plugin.yaml
@@ -184,6 +184,9 @@ spec:
       annotations:
         {{.Values.createdByAnnotation}}: {{default (printf "linkerd/helm %s" .Values.cniPluginVersion) .Values.cliVersion}}
     spec:
+      {{- if .Values.tolerations }}
+      {{- include "linkerd.tolerations" . | nindent 6 }}
+      {{- end }}
       nodeSelector:
         beta.kubernetes.io/os: linux
       hostNetwork: true
diff --git a/charts/linkerd2-cni/values.yaml b/charts/linkerd2-cni/values.yaml
index c6112ca05..4dfb13ac8 100644
--- a/charts/linkerd2-cni/values.yaml
+++ b/charts/linkerd2-cni/values.yaml
@@ -41,6 +41,11 @@ priorityClassName: ""
 proxyInjectAnnotation: linkerd.io/inject
 proxyInjectDisabled: disabled
 
+# -|- Tolerations section, See the 
+# [K8S documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/)
+# for more information
+#tolerations:
+
 #
 ## For Private docker registries, authentication is needed.
 # If the control plane service images are pulled from a
diff --git a/cli/cmd/check.go b/cli/cmd/check.go
index 50aab13e1..2ea0c6476 100644
--- a/cli/cmd/check.go
+++ b/cli/cmd/check.go
@@ -221,7 +221,7 @@ func configureAndRunChecks(ctx context.Context, wout io.Writer, werr io.Writer,
 }
 
 func renderInstallManifest(ctx context.Context) (string, error) {
-	values, err := charts.NewValues(false)
+	values, err := charts.NewValues()
 	if err != nil {
 		return "", err
 	}
diff --git a/cli/cmd/inject.go b/cli/cmd/inject.go
index c1cfcfd19..603e857e6 100644
--- a/cli/cmd/inject.go
+++ b/cli/cmd/inject.go
@@ -48,7 +48,7 @@ func runInjectCmd(inputs []io.Reader, errWriter, outWriter io.Writer, transforme
 }
 
 func newCmdInject() *cobra.Command {
-	defaults, err := charts.NewValues(false)
+	defaults, err := charts.NewValues()
 	if err != nil {
 		fmt.Fprint(os.Stderr, err.Error())
 		os.Exit(1)
diff --git a/cli/cmd/install.go b/cli/cmd/install.go
index d0337867f..3639e1a46 100644
--- a/cli/cmd/install.go
+++ b/cli/cmd/install.go
@@ -211,7 +211,7 @@ control plane. It should be run after "linkerd install config".`,
 }
 
 func newCmdInstall() *cobra.Command {
-	values, err := l5dcharts.NewValues(false)
+	values, err := l5dcharts.NewValues()
 
 	allStageFlags, allStageFlagSet := makeAllStageFlags(values)
 	installOnlyFlags, installOnlyFlagSet := makeInstallFlags(values)
@@ -412,7 +412,7 @@ func render(w io.Writer, values *l5dcharts.Values, stage string) error {
 // resource is not part of the Helm chart and will not be present when installing
 // with Helm.
 func renderOverrides(values *l5dcharts.Values, namespace string) ([]byte, error) {
-	defaults, err := l5dcharts.NewValues(false)
+	defaults, err := l5dcharts.NewValues()
 	if err != nil {
 		return nil, err
 	}
diff --git a/cli/cmd/options.go b/cli/cmd/options.go
index 05801b941..bad985a5d 100644
--- a/cli/cmd/options.go
+++ b/cli/cmd/options.go
@@ -54,12 +54,7 @@ func makeInstallUpgradeFlags(defaults *l5dcharts.Values) ([]flag.Flag, *pflag.Fl
 			func(values *l5dcharts.Values, value bool) error {
 				values.GetGlobal().HighAvailability = value
 				if value {
-					haValues, err := l5dcharts.NewValues(true)
-					if err != nil {
-						return err
-					}
-					*values, err = values.Merge(*haValues)
-					if err != nil {
+					if err := l5dcharts.MergeHAValues(values); err != nil {
 						return err
 					}
 				}
diff --git a/cli/cmd/upgrade.go b/cli/cmd/upgrade.go
index fdca7e314..23f3cd2c6 100644
--- a/cli/cmd/upgrade.go
+++ b/cli/cmd/upgrade.go
@@ -114,7 +114,7 @@ install command. It should be run after "linkerd upgrade config".`,
 }
 
 func newCmdUpgrade() *cobra.Command {
-	values, err := l5dcharts.NewValues(false)
+	values, err := l5dcharts.NewValues()
 	if err != nil {
 		fmt.Fprint(os.Stderr, err.Error())
 		os.Exit(1)
@@ -290,7 +290,7 @@ func upgrade(ctx context.Context, k *k8s.KubernetesAPI, flags []flag.Flag, stage
 
 func loadStoredValues(ctx context.Context, k *k8s.KubernetesAPI) (*charts.Values, error) {
 	// Load the default values from the chart.
-	values, err := charts.NewValues(false)
+	values, err := charts.NewValues()
 	if err != nil {
 		return nil, err
 	}
@@ -361,7 +361,7 @@ func ensureIssuerCertWorksWithAllProxies(ctx context.Context, k *k8s.KubernetesA
 }
 
 func clearAddonOverrides(values *l5dcharts.Values) error {
-	defaults, err := l5dcharts.NewValues(false)
+	defaults, err := l5dcharts.NewValues()
 	if err != nil {
 		return err
 	}
diff --git a/cli/cmd/upgrade_legacy.go b/cli/cmd/upgrade_legacy.go
index 759cbdb1e..bb71e909c 100644
--- a/cli/cmd/upgrade_legacy.go
+++ b/cli/cmd/upgrade_legacy.go
@@ -32,7 +32,7 @@ func loadStoredValuesLegacy(ctx context.Context, k *k8s.KubernetesAPI) (*charts.
 	}
 	repairConfigs(configs)
 
-	values, err := charts.NewValues(false)
+	values, err := charts.NewValues()
 	if err != nil {
 		return nil, err
 	}
diff --git a/controller/cmd/tap/main.go b/controller/cmd/tap/main.go
index 67d74c467..dae386674 100644
--- a/controller/cmd/tap/main.go
+++ b/controller/cmd/tap/main.go
@@ -2,7 +2,6 @@ package tap
 
 import (
 	"context"
-	"crypto/tls"
 	"flag"
 	"os"
 	"os/signal"
@@ -12,7 +11,6 @@ import (
 	"github.com/linkerd/linkerd2/controller/tap"
 	"github.com/linkerd/linkerd2/pkg/admin"
 	"github.com/linkerd/linkerd2/pkg/flags"
-	pkgK8s "github.com/linkerd/linkerd2/pkg/k8s"
 	"github.com/linkerd/linkerd2/pkg/trace"
 	log "github.com/sirupsen/logrus"
 )
@@ -28,8 +26,6 @@ func Main(args []string) {
 	kubeConfigPath := cmd.String("kubeconfig", "", "path to kube config")
 	controllerNamespace := cmd.String("controller-namespace", "linkerd", "namespace in which Linkerd is installed")
 	tapPort := cmd.Uint("tap-port", 4190, "proxy tap port to connect to")
-	tlsCertPath := cmd.String("tls-cert", pkgK8s.MountPathTLSCrtPEM, "path to TLS Cert PEM")
-	tlsKeyPath := cmd.String("tls-key", pkgK8s.MountPathTLSKeyPEM, "path to TLS Key PEM")
 	disableCommonNames := cmd.Bool("disable-common-names", false, "disable checks for Common Names (for development)")
 	trustDomain := cmd.String("identity-trust-domain", defaultDomain, "configures the name suffix used for identities")
 
@@ -70,24 +66,14 @@ func Main(args []string) {
 	}
 	grpcTapServer := tap.NewGrpcTapServer(*tapPort, *controllerNamespace, *trustDomain, k8sAPI)
 
-	// TODO: make this configurable for local development
-	cert, err := tls.LoadX509KeyPair(*tlsCertPath, *tlsKeyPath)
-	if err != nil {
-		log.Fatal(err.Error())
-	}
-
-	apiServer, apiLis, err := tap.NewAPIServer(ctx, *apiServerAddr, cert, k8sAPI, grpcTapServer, *disableCommonNames)
+	apiServer, err := tap.NewAPIServer(ctx, *apiServerAddr, k8sAPI, grpcTapServer, *disableCommonNames)
 	if err != nil {
 		log.Fatal(err.Error())
 	}
 
 	k8sAPI.Sync(nil) // blocks until caches are synced
 
-	go func() {
-		log.Infof("starting APIServer on %s", *apiServerAddr)
-		apiServer.ServeTLS(apiLis, "", "")
-	}()
-
+	go apiServer.Start(ctx)
 	go admin.StartServer(*metricsAddr)
 
 	<-stop
diff --git a/controller/tap/apiserver.go b/controller/tap/apiserver.go
index 0b4daf5f5..d9c329cca 100644
--- a/controller/tap/apiserver.go
+++ b/controller/tap/apiserver.go
@@ -8,19 +8,27 @@ import (
 	"fmt"
 	"net"
 	"net/http"
+	"sync/atomic"
 
 	"github.com/julienschmidt/httprouter"
 	"github.com/linkerd/linkerd2/controller/gen/controller/tap"
 	"github.com/linkerd/linkerd2/controller/k8s"
 	k8sutils "github.com/linkerd/linkerd2/pkg/k8s"
+	pkgk8s "github.com/linkerd/linkerd2/pkg/k8s"
 	"github.com/linkerd/linkerd2/pkg/prometheus"
+	pkgTls "github.com/linkerd/linkerd2/pkg/tls"
+	"github.com/prometheus/common/log"
 	"github.com/sirupsen/logrus"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 )
 
-type apiServer struct {
+// APIServer holds the underlying http server and its config
+type APIServer struct {
+	*http.Server
+	listener     net.Listener
 	router       *httprouter.Router
 	allowedNames []string
+	certValue    *atomic.Value
 	log          *logrus.Entry
 }
 
@@ -28,14 +36,23 @@ type apiServer struct {
 func NewAPIServer(
 	ctx context.Context,
 	addr string,
-	cert tls.Certificate,
 	k8sAPI *k8s.API,
 	grpcTapServer tap.TapServer,
 	disableCommonNames bool,
-) (*http.Server, net.Listener, error) {
+) (*APIServer, error) {
+	updateEvent := make(chan struct{})
+	errEvent := make(chan error)
+	watcher := pkgTls.NewFsCredsWatcher(pkgk8s.MountPathTLSBase, updateEvent, errEvent).
+		WithFilePaths(pkgk8s.MountPathTLSCrtPEM, pkgk8s.MountPathTLSKeyPEM)
+	go func() {
+		if err := watcher.StartWatching(ctx); err != nil {
+			log.Fatalf("Failed to start creds watcher: %s", err)
+		}
+	}()
+
 	clientCAPem, allowedNames, usernameHeader, groupHeader, err := apiServerAuth(ctx, k8sAPI)
 	if err != nil {
-		return nil, nil, err
+		return nil, err
 	}
 
 	// for development
@@ -48,6 +65,18 @@ func NewAPIServer(
 		"addr":      addr,
 	})
 
+	clientCertPool := x509.NewCertPool()
+	clientCertPool.AppendCertsFromPEM([]byte(clientCAPem))
+
+	httpServer := &http.Server{
+		Addr: addr,
+		TLSConfig: &tls.Config{
+			ClientAuth: tls.VerifyClientCertIfGiven,
+			ClientCAs:  clientCertPool,
+		},
+	}
+
+	var emptyCert atomic.Value
 	h := &handler{
 		k8sAPI:         k8sAPI,
 		usernameHeader: usernameHeader,
@@ -56,39 +85,48 @@ func NewAPIServer(
 		log:            log,
 	}
 
-	router := initRouter(h)
+	lis, err := net.Listen("tcp", addr)
+	if err != nil {
+		return nil, fmt.Errorf("net.Listen failed with: %s", err)
+	}
 
-	server := &apiServer{
-		router:       router,
+	s := &APIServer{
+		Server:       httpServer,
+		listener:     lis,
+		router:       initRouter(h),
 		allowedNames: allowedNames,
+		certValue:    &emptyCert,
 		log:          log,
 	}
+	s.Handler = prometheus.WithTelemetry(s)
+	httpServer.TLSConfig.GetCertificate = s.getCertificate
 
-	clientCertPool := x509.NewCertPool()
-	clientCertPool.AppendCertsFromPEM([]byte(clientCAPem))
+	if err := watcher.UpdateCert(s.certValue); err != nil {
+		return nil, fmt.Errorf("Failed to initialized certificate: %s", err)
+	}
 
-	wrappedServer := prometheus.WithTelemetry(server)
+	go watcher.ProcessEvents(log, s.certValue, updateEvent, errEvent)
 
-	s := &http.Server{
-		Addr:    addr,
-		Handler: wrappedServer,
-		TLSConfig: &tls.Config{
-			Certificates: []tls.Certificate{cert},
-			ClientAuth:   tls.VerifyClientCertIfGiven,
-			ClientCAs:    clientCertPool,
-		},
-	}
+	return s, nil
+}
 
-	lis, err := net.Listen("tcp", addr)
-	if err != nil {
-		log.Fatalf("net.Listen failed with: %s", err)
+// Start starts the https server
+func (a *APIServer) Start(ctx context.Context) {
+	a.log.Infof("starting APIServer on %s", a.Server.Addr)
+	if err := a.ServeTLS(a.listener, "", ""); err != nil {
+		if err == http.ErrServerClosed {
+			return
+		}
+		a.log.Fatal(err)
 	}
+}
 
-	return s, lis, nil
+func (a *APIServer) getCertificate(_ *tls.ClientHelloInfo) (*tls.Certificate, error) {
+	return a.certValue.Load().(*tls.Certificate), nil
 }
 
 // ServeHTTP handles all routes for the APIServer.
-func (a *apiServer) ServeHTTP(w http.ResponseWriter, req *http.Request) {
+func (a *APIServer) ServeHTTP(w http.ResponseWriter, req *http.Request) {
 	a.log.Debugf("ServeHTTP(): %+v", req)
 	if err := a.validate(req); err != nil {
 		a.log.Debug(err)
@@ -99,7 +137,7 @@ func (a *apiServer) ServeHTTP(w http.ResponseWriter, req *http.Request) {
 }
 
 // validate ensures that the request should be honored returning an error otherwise.
-func (a *apiServer) validate(req *http.Request) error {
+func (a *APIServer) validate(req *http.Request) error {
 	// if `requestheader-allowed-names` was empty, allow any CN
 	if len(a.allowedNames) > 0 {
 		for _, cn := range a.allowedNames {
diff --git a/controller/webhook/server.go b/controller/webhook/server.go
index b1588b0c1..542ff9f55 100644
--- a/controller/webhook/server.go
+++ b/controller/webhook/server.go
@@ -4,7 +4,6 @@ import (
 	"context"
 	"crypto/tls"
 	"encoding/json"
-	"fmt"
 	"io/ioutil"
 	"net/http"
 	"sync/atomic"
@@ -12,6 +11,7 @@ import (
 	"github.com/linkerd/linkerd2/controller/k8s"
 	pkgk8s "github.com/linkerd/linkerd2/pkg/k8s"
 	pkgTls "github.com/linkerd/linkerd2/pkg/tls"
+	"github.com/sirupsen/logrus"
 	log "github.com/sirupsen/logrus"
 	admissionv1beta1 "k8s.io/api/admission/v1beta1"
 	v1 "k8s.io/api/core/v1"
@@ -36,7 +36,7 @@ type Server struct {
 	*http.Server
 	api       *k8s.API
 	handler   Handler
-	certValue atomic.Value
+	certValue *atomic.Value
 	recorder  record.EventRecorder
 }
 
@@ -50,7 +50,8 @@ func NewServer(
 ) (*Server, error) {
 	updateEvent := make(chan struct{})
 	errEvent := make(chan error)
-	watcher := pkgTls.NewFsCredsWatcher(certPath, updateEvent, errEvent)
+	watcher := pkgTls.NewFsCredsWatcher(certPath, updateEvent, errEvent).
+		WithFilePaths(pkgk8s.MountPathTLSCrtPEM, pkgk8s.MountPathTLSKeyPEM)
 	go func() {
 		if err := watcher.StartWatching(ctx); err != nil {
 			log.Fatalf("Failed to start creds watcher: %s", err)
@@ -71,13 +72,16 @@ func NewServer(
 	recorder := eventBroadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: component})
 
 	s := getConfiguredServer(server, api, handler, recorder)
-	if err := s.updateCert(); err != nil {
+	if err := watcher.UpdateCert(s.certValue); err != nil {
 		log.Fatalf("Failed to initialized certificate: %s", err)
 	}
 
-	go func() {
-		s.run(updateEvent, errEvent)
-	}()
+	log := logrus.WithFields(logrus.Fields{
+		"component": "proxy-injector",
+		"addr":      addr,
+	})
+
+	go watcher.ProcessEvents(log, s.certValue, updateEvent, errEvent)
 
 	return s, nil
 }
@@ -89,32 +93,12 @@ func getConfiguredServer(
 	recorder record.EventRecorder,
 ) *Server {
 	var emptyCert atomic.Value
-	s := &Server{httpServer, api, handler, emptyCert, recorder}
+	s := &Server{httpServer, api, handler, &emptyCert, recorder}
 	s.Handler = http.HandlerFunc(s.serve)
 	httpServer.TLSConfig.GetCertificate = s.getCertificate
 	return s
 }
 
-func (s *Server) updateCert() error {
-	creds, err := pkgTls.ReadPEMCreds(
-		pkgk8s.MountPathTLSKeyPEM,
-		pkgk8s.MountPathTLSCrtPEM,
-	)
-	if err != nil {
-		return fmt.Errorf("failed to read cert from disk: %s", err)
-	}
-
-	certPEM := creds.EncodePEM()
-	keyPEM := creds.EncodePrivateKeyPEM()
-	cert, err := tls.X509KeyPair([]byte(certPEM), []byte(keyPEM))
-	if err != nil {
-		return err
-	}
-	s.certValue.Store(&cert)
-	log.Debug("Certificate has been updated")
-	return nil
-}
-
 // Start starts the https server
 func (s *Server) Start() {
 	log.Infof("listening at %s", s.Server.Addr)
@@ -131,22 +115,6 @@ func (s *Server) getCertificate(_ *tls.ClientHelloInfo) (*tls.Certificate, error
 	return s.certValue.Load().(*tls.Certificate), nil
 }
 
-// run reads from the update and error channels and reloads the certs when necessary
-func (s *Server) run(updateEvent <-chan struct{}, errEvent <-chan error) {
-	for {
-		select {
-		case <-updateEvent:
-			if err := s.updateCert(); err != nil {
-				log.Warnf("Skipping update as cert could not be read from disk: %s", err)
-			} else {
-				log.Infof("Updated certificate")
-			}
-		case err := <-errEvent:
-			log.Warnf("Received error from fs watcher: %s", err)
-		}
-	}
-}
-
 func (s *Server) serve(res http.ResponseWriter, req *http.Request) {
 	var (
 		data []byte
diff --git a/jaeger/charts/jaeger/Chart.yaml b/jaeger/charts/jaeger/Chart.yaml
index f17e45bff..d487720d0 100644
--- a/jaeger/charts/jaeger/Chart.yaml
+++ b/jaeger/charts/jaeger/Chart.yaml
@@ -3,6 +3,8 @@ appVersion: "1.0"
 description: A Helm chart for the jaeger add-on in Linkerd
 name: jaeger
 version: 0.1.0
+kubeVersion: ">=1.15.0-0"
+icon: https://linkerd.io/images/logo-only-200h.png
 maintainers:
   - name: Linkerd authors
     email: cncf-linkerd-dev@lists.cncf.io
diff --git a/jaeger/charts/jaeger/README.md b/jaeger/charts/jaeger/README.md
index b5629d786..de6bab171 100644
--- a/jaeger/charts/jaeger/README.md
+++ b/jaeger/charts/jaeger/README.md
@@ -8,6 +8,8 @@ A Helm chart for the jaeger add-on in Linkerd
 
 ## Requirements
 
+Kubernetes: `>=1.15.0-0`
+
 | Repository | Name | Version |
 |------------|------|---------|
 | file://../../../charts/partials | partials | 0.1.0 |
diff --git a/pkg/charts/linkerd2/values.go b/pkg/charts/linkerd2/values.go
index 2f9a76b3e..2b9270fc1 100644
--- a/pkg/charts/linkerd2/values.go
+++ b/pkg/charts/linkerd2/values.go
@@ -250,9 +250,8 @@ type (
 )
 
 // NewValues returns a new instance of the Values type.
-func NewValues(ha bool) (*Values, error) {
-	chartDir := fmt.Sprintf("%s/", helmDefaultChartDir)
-	v, err := readDefaults(chartDir, ha)
+func NewValues() (*Values, error) {
+	v, err := readDefaults(false)
 	if err != nil {
 		return nil, err
 	}
@@ -269,38 +268,34 @@ func NewValues(ha bool) (*Values, error) {
 	return v, nil
 }
 
-// readDefaults read all the default variables from the values.yaml file.
-// chartDir is the root directory of the Helm chart where values.yaml is.
-func readDefaults(chartDir string, ha bool) (*Values, error) {
-	valuesFiles := []*loader.BufferedFile{
-		{Name: chartutil.ValuesfileName},
+// MergeHAValues retrieves the default HA values and merges them into the received values
+func MergeHAValues(values *Values) error {
+	haValues, err := readDefaults(true)
+	if err != nil {
+		return err
 	}
+	*values, err = values.Merge(*haValues)
+	return err
+}
 
+// readDefaults read all the default variables from the values.yaml file.
+func readDefaults(ha bool) (*Values, error) {
+	var valuesFile *loader.BufferedFile
 	if ha {
-		valuesFiles = append(valuesFiles, &loader.BufferedFile{
-			Name: helmDefaultHAValuesFile,
-		})
+		valuesFile = &loader.BufferedFile{Name: helmDefaultHAValuesFile}
+	} else {
+		valuesFile = &loader.BufferedFile{Name: chartutil.ValuesfileName}
 	}
 
-	if err := charts.FilesReader(static.Templates, chartDir, valuesFiles); err != nil {
+	chartDir := fmt.Sprintf("%s/", helmDefaultChartDir)
+	if err := charts.ReadFile(static.Templates, chartDir, valuesFile); err != nil {
 		return nil, err
 	}
 
-	values := Values{}
-	for _, valuesFile := range valuesFiles {
-		var v Values
-		if err := yaml.Unmarshal(charts.InsertVersion(valuesFile.Data), &v); err != nil {
-			return nil, err
-		}
-
-		var err error
-		values, err = values.Merge(v)
-		if err != nil {
-			return nil, err
-		}
-	}
+	var values Values
+	err := yaml.Unmarshal(charts.InsertVersion(valuesFile.Data), &values)
 
-	return &values, nil
+	return &values, err
 }
 
 // Merge merges the non-empty properties of src into v.
diff --git a/pkg/tls/creds_watcher.go b/pkg/tls/creds_watcher.go
index d519f0afd..53bfeef58 100644
--- a/pkg/tls/creds_watcher.go
+++ b/pkg/tls/creds_watcher.go
@@ -2,9 +2,13 @@ package tls
 
 import (
 	"context"
+	"crypto/tls"
+	"fmt"
 	"path/filepath"
+	"sync/atomic"
 
 	"github.com/fsnotify/fsnotify"
+	"github.com/sirupsen/logrus"
 	log "github.com/sirupsen/logrus"
 )
 
@@ -12,14 +16,23 @@ const dataDirectoryLnName = "..data"
 
 // FsCredsWatcher is used to monitor tls credentials on the filesystem
 type FsCredsWatcher struct {
-	certPath  string
-	EventChan chan<- struct{}
-	ErrorChan chan<- error
+	certRootPath string
+	certFilePath string
+	keyFilePath  string
+	EventChan    chan<- struct{}
+	ErrorChan    chan<- error
 }
 
 // NewFsCredsWatcher constructs a FsCredsWatcher instance
-func NewFsCredsWatcher(certPath string, updateEvent chan<- struct{}, errEvent chan<- error) *FsCredsWatcher {
-	return &FsCredsWatcher{certPath, updateEvent, errEvent}
+func NewFsCredsWatcher(certRootPath string, updateEvent chan<- struct{}, errEvent chan<- error) *FsCredsWatcher {
+	return &FsCredsWatcher{certRootPath, "", "", updateEvent, errEvent}
+}
+
+// WithFilePaths completes the FsCredsWatcher instance with the cert and key files locations
+func (fscw *FsCredsWatcher) WithFilePaths(certFilePath, keyFilePath string) *FsCredsWatcher {
+	fscw.certFilePath = certFilePath
+	fscw.keyFilePath = keyFilePath
+	return fscw
 }
 
 // StartWatching starts watching the filesystem for cert updates
@@ -31,7 +44,7 @@ func (fscw *FsCredsWatcher) StartWatching(ctx context.Context) error {
 	defer watcher.Close()
 
 	// no point of proceeding if we fail to watch this
-	if err := watcher.Add(fscw.certPath); err != nil {
+	if err := watcher.Add(fscw.certRootPath); err != nil {
 		return err
 	}
 
@@ -43,12 +56,12 @@ LOOP:
 			// Watching the folder for create events as this indicates
 			// that the secret has been updated.
 			if event.Op&fsnotify.Create == fsnotify.Create &&
-				event.Name == filepath.Join(fscw.certPath, dataDirectoryLnName) {
+				event.Name == filepath.Join(fscw.certRootPath, dataDirectoryLnName) {
 				fscw.EventChan <- struct{}{}
 			}
 		case err := <-watcher.Errors:
 			fscw.ErrorChan <- err
-			log.Warnf("Error while watching %s: %s", fscw.certPath, err)
+			log.Warnf("Error while watching %s: %s", fscw.certRootPath, err)
 			break LOOP
 		case <-ctx.Done():
 			if err := ctx.Err(); err != nil {
@@ -60,3 +73,41 @@ LOOP:
 
 	return nil
 }
+
+// UpdateCert reads the cert and key files and stores the key pair in certVal
+func (fscw *FsCredsWatcher) UpdateCert(certVal *atomic.Value) error {
+	creds, err := ReadPEMCreds(fscw.keyFilePath, fscw.certFilePath)
+	if err != nil {
+		return fmt.Errorf("failed to read cert from disk: %s", err)
+	}
+
+	certPEM := creds.EncodePEM()
+	keyPEM := creds.EncodePrivateKeyPEM()
+	cert, err := tls.X509KeyPair([]byte(certPEM), []byte(keyPEM))
+	if err != nil {
+		return err
+	}
+	certVal.Store(&cert)
+	return nil
+}
+
+// ProcessEvents reads from the update and error channels and reloads the certs when necessary
+func (fscw *FsCredsWatcher) ProcessEvents(
+	log *logrus.Entry,
+	certVal *atomic.Value,
+	updateEvent <-chan struct{},
+	errEvent <-chan error,
+) {
+	for {
+		select {
+		case <-updateEvent:
+			if err := fscw.UpdateCert(certVal); err != nil {
+				log.Warnf("Skipping update as cert could not be read from disk: %s", err)
+			} else {
+				log.Infof("Updated certificate")
+			}
+		case err := <-errEvent:
+			log.Warnf("Received error from fs watcher: %s", err)
+		}
+	}
+}


Test patch:
diff --git a/cli/cmd/inject_test.go b/cli/cmd/inject_test.go
index 0f328b3e6..9668b143a 100644
--- a/cli/cmd/inject_test.go
+++ b/cli/cmd/inject_test.go
@@ -575,7 +575,7 @@ func TestWalk(t *testing.T) {
 }
 
 func TestProxyConfigurationAnnotations(t *testing.T) {
-	baseValues, err := linkerd2.NewValues(false)
+	baseValues, err := linkerd2.NewValues()
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -627,7 +627,7 @@ func TestProxyConfigurationAnnotations(t *testing.T) {
 }
 
 func TestProxyImageAnnotations(t *testing.T) {
-	baseValues, err := linkerd2.NewValues(false)
+	baseValues, err := linkerd2.NewValues()
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -653,7 +653,7 @@ func TestProxyImageAnnotations(t *testing.T) {
 }
 
 func TestProxyInitImageAnnotations(t *testing.T) {
-	baseValues, err := linkerd2.NewValues(false)
+	baseValues, err := linkerd2.NewValues()
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -677,7 +677,7 @@ func TestProxyInitImageAnnotations(t *testing.T) {
 }
 
 func TestNoAnnotations(t *testing.T) {
-	baseValues, err := linkerd2.NewValues(false)
+	baseValues, err := linkerd2.NewValues()
 	if err != nil {
 		t.Fatal(err)
 	}
diff --git a/cli/cmd/install_helm_test.go b/cli/cmd/install_helm_test.go
index be57750db..ccf21ad53 100644
--- a/cli/cmd/install_helm_test.go
+++ b/cli/cmd/install_helm_test.go
@@ -332,10 +332,15 @@ func chartPartials(t *testing.T, paths []string) *chart.Chart {
 }
 
 func readTestValues(ha bool, ignoreOutboundPorts string, ignoreInboundPorts string) (*l5dcharts.Values, error) {
-	values, err := l5dcharts.NewValues(ha)
+	values, err := l5dcharts.NewValues()
 	if err != nil {
 		return nil, err
 	}
+	if ha {
+		if err = l5dcharts.MergeHAValues(values); err != nil {
+			return nil, err
+		}
+	}
 	values.GetGlobal().ProxyInit.IgnoreOutboundPorts = ignoreOutboundPorts
 	values.GetGlobal().ProxyInit.IgnoreInboundPorts = ignoreInboundPorts
 
diff --git a/cli/cmd/install_test.go b/cli/cmd/install_test.go
index 08cbf8c5e..10fa5dbd9 100644
--- a/cli/cmd/install_test.go
+++ b/cli/cmd/install_test.go
@@ -307,10 +307,15 @@ func testInstallOptionsHA(ha bool) (*charts.Values, error) {
 }
 
 func testInstallOptionsNoCerts(ha bool) (*charts.Values, error) {
-	values, err := charts.NewValues(ha)
+	values, err := charts.NewValues()
 	if err != nil {
 		return nil, err
 	}
+	if ha {
+		if err = charts.MergeHAValues(values); err != nil {
+			return nil, err
+		}
+	}
 
 	values.GetGlobal().Proxy.Image.Version = installProxyVersion
 	values.DebugContainer.Image.Version = installDebugVersion
@@ -321,7 +326,7 @@ func testInstallOptionsNoCerts(ha bool) (*charts.Values, error) {
 }
 
 func testInstallValues() (*charts.Values, error) {
-	values, err := charts.NewValues(false)
+	values, err := charts.NewValues()
 	if err != nil {
 		return nil, err
 	}
diff --git a/cli/cmd/uninject_test.go b/cli/cmd/uninject_test.go
index bb957894f..ae1312dfc 100644
--- a/cli/cmd/uninject_test.go
+++ b/cli/cmd/uninject_test.go
@@ -93,7 +93,7 @@ func TestUninjectYAML(t *testing.T) {
 		},
 	}
 
-	values, err := charts.NewValues(false)
+	values, err := charts.NewValues()
 	if err != nil {
 		t.Fatal(err)
 	}
diff --git a/cli/cmd/upgrade_test.go b/cli/cmd/upgrade_test.go
index fc64476cf..6640a53c3 100644
--- a/cli/cmd/upgrade_test.go
+++ b/cli/cmd/upgrade_test.go
@@ -544,7 +544,7 @@ func TestUpgradeOverwriteRemoveAddonKeys(t *testing.T) {
 /* Helpers */
 
 func testUpgradeOptions() ([]flag.Flag, *pflag.FlagSet, error) {
-	defaults, err := charts.NewValues(false)
+	defaults, err := charts.NewValues()
 	if err != nil {
 		return nil, nil, err
 	}
diff --git a/controller/proxy-injector/webhook_test.go b/controller/proxy-injector/webhook_test.go
index 20e803632..e6a9e3298 100644
--- a/controller/proxy-injector/webhook_test.go
+++ b/controller/proxy-injector/webhook_test.go
@@ -21,7 +21,7 @@ import (
 type unmarshalledPatch []map[string]interface{}
 
 var (
-	values, _ = linkerd2.NewValues(false)
+	values, _ = linkerd2.NewValues()
 )
 
 func confNsEnabled() *inject.ResourceConfig {
diff --git a/controller/tap/apiserver_test.go b/controller/tap/apiserver_test.go
index 52f53dc2e..1a007a696 100644
--- a/controller/tap/apiserver_test.go
+++ b/controller/tap/apiserver_test.go
@@ -16,54 +16,6 @@ import (
 	k8sutils "github.com/linkerd/linkerd2/pkg/k8s"
 )
 
-func TestNewAPIServer(t *testing.T) {
-	expectations := []struct {
-		k8sRes []string
-		err    error
-	}{
-		{
-			err: fmt.Errorf("failed to load [%s] config: configmaps %q not found", k8sutils.ExtensionAPIServerAuthenticationConfigMapName, k8sutils.ExtensionAPIServerAuthenticationConfigMapName),
-		},
-		{
-			err: nil,
-			k8sRes: []string{`
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: extension-apiserver-authentication
-  namespace: kube-system
-data:
-  client-ca-file: 'client-ca-file'
-  requestheader-allowed-names: '["name1", "name2"]'
-  requestheader-client-ca-file: 'requestheader-client-ca-file'
-  requestheader-extra-headers-prefix: '["X-Remote-Extra-"]'
-  requestheader-group-headers: '["X-Remote-Group"]'
-  requestheader-username-headers: '["X-Remote-User"]'
-`,
-			},
-		},
-	}
-
-	ctx := context.Background()
-	for i, exp := range expectations {
-		exp := exp // pin
-
-		t.Run(fmt.Sprintf("%d returns a configured API Server", i), func(t *testing.T) {
-			k8sAPI, err := k8s.NewFakeAPI(exp.k8sRes...)
-			if err != nil {
-				t.Fatalf("NewFakeAPI returned an error: %s", err)
-			}
-
-			fakeGrpcServer := newGRPCTapServer(4190, "controller-ns", "cluster.local", k8sAPI)
-
-			_, _, err = NewAPIServer(ctx, "localhost:0", tls.Certificate{}, k8sAPI, fakeGrpcServer, false)
-			if !reflect.DeepEqual(err, exp.err) {
-				t.Errorf("NewAPIServer returned unexpected error: %s, expected: %s", err, exp.err)
-			}
-		})
-	}
-}
-
 func TestAPIServerAuth(t *testing.T) {
 	expectations := []struct {
 		k8sRes         []string
@@ -138,7 +90,7 @@ func TestValidate(t *testing.T) {
 
 	req := http.Request{TLS: &tls}
 
-	server := apiServer{}
+	server := APIServer{}
 	if err := server.validate(&req); err != nil {
 		t.Fatalf("No error expected for %q but encountered %q", cert.Subject.CommonName, err.Error())
 	}
@@ -152,7 +104,7 @@ func TestValidate_ClientAllowed(t *testing.T) {
 
 	req := http.Request{TLS: &tls}
 
-	server := apiServer{allowedNames: []string{"name-trusted"}}
+	server := APIServer{allowedNames: []string{"name-trusted"}}
 	if err := server.validate(&req); err != nil {
 		t.Fatalf("No error expected for %q but encountered %q", cert.Subject.CommonName, err.Error())
 	}
@@ -166,7 +118,7 @@ func TestValidate_ClientAllowedViaSAN(t *testing.T) {
 
 	req := http.Request{TLS: &tls}
 
-	server := apiServer{allowedNames: []string{"linkerd.io"}}
+	server := APIServer{allowedNames: []string{"linkerd.io"}}
 	if err := server.validate(&req); err != nil {
 		t.Fatalf("No error expected for %q but encountered %q", cert.Subject.CommonName, err.Error())
 	}
@@ -180,7 +132,7 @@ func TestValidate_ClientNotAllowed(t *testing.T) {
 
 	req := http.Request{TLS: &tls}
 
-	server := apiServer{allowedNames: []string{"name-trusted"}}
+	server := APIServer{allowedNames: []string{"name-trusted"}}
 	if err := server.validate(&req); err == nil {
 		t.Fatalf("Expected request to be rejected for %q", cert.Subject.CommonName)
 	}
diff --git a/pkg/charts/linkerd2/values_test.go b/pkg/charts/linkerd2/values_test.go
index 44fc56c7f..a110d41cb 100644
--- a/pkg/charts/linkerd2/values_test.go
+++ b/pkg/charts/linkerd2/values_test.go
@@ -10,7 +10,7 @@ import (
 )
 
 func TestNewValues(t *testing.T) {
-	actual, err := NewValues(false)
+	actual, err := NewValues()
 	if err != nil {
 		t.Fatalf("Unexpected error: %v\n", err)
 	}
@@ -168,7 +168,7 @@ func TestNewValues(t *testing.T) {
 	}
 
 	t.Run("HA", func(t *testing.T) {
-		actual, err := NewValues(true)
+		err := MergeHAValues(actual)
 
 		if err != nil {
 			t.Fatalf("Unexpected error: %v\n", err)
diff --git a/pkg/healthcheck/healthcheck_test.go b/pkg/healthcheck/healthcheck_test.go
index 501c1bfae..015799c11 100644
--- a/pkg/healthcheck/healthcheck_test.go
+++ b/pkg/healthcheck/healthcheck_test.go
@@ -2539,7 +2539,7 @@ data:
 }
 
 func TestFetchCurrentConfiguration(t *testing.T) {
-	defaultValues, err := linkerd2.NewValues(false)
+	defaultValues, err := linkerd2.NewValues()
 
 	if err != nil {
 		t.Fatalf("Unexpected error validating options: %v", err)
diff --git a/pkg/inject/inject_test.go b/pkg/inject/inject_test.go
index 27e440bcf..aab01fd97 100644
--- a/pkg/inject/inject_test.go
+++ b/pkg/inject/inject_test.go
@@ -23,7 +23,7 @@ func TestGetOverriddenValues(t *testing.T) {
 		pullPolicy           = "Always"
 	)
 
-	testConfig, err := l5dcharts.NewValues(false)
+	testConfig, err := l5dcharts.NewValues()
 	if err != nil {
 		t.Fatalf("Unexpected error: %v", err)
 	}
@@ -70,7 +70,7 @@ func TestGetOverriddenValues(t *testing.T) {
 				},
 			},
 			expected: func() *l5dcharts.Values {
-				values, _ := l5dcharts.NewValues(false)
+				values, _ := l5dcharts.NewValues()
 
 				values.GetGlobal().Proxy.Cores = 2
 				values.GetGlobal().Proxy.DisableIdentity = true
@@ -116,7 +116,7 @@ func TestGetOverriddenValues(t *testing.T) {
 				},
 			},
 			expected: func() *l5dcharts.Values {
-				values, _ := l5dcharts.NewValues(false)
+				values, _ := l5dcharts.NewValues()
 				return values
 			},
 		},
@@ -152,7 +152,7 @@ func TestGetOverriddenValues(t *testing.T) {
 				},
 			},
 			expected: func() *l5dcharts.Values {
-				values, _ := l5dcharts.NewValues(false)
+				values, _ := l5dcharts.NewValues()
 
 				values.GetGlobal().Proxy.Cores = 2
 				values.GetGlobal().Proxy.DisableIdentity = true
@@ -200,7 +200,7 @@ func TestGetOverriddenValues(t *testing.T) {
 				},
 			},
 			expected: func() *l5dcharts.Values {
-				values, _ := l5dcharts.NewValues(false)
+				values, _ := l5dcharts.NewValues()
 				return values
 			},
 		},
@@ -217,7 +217,7 @@ func TestGetOverriddenValues(t *testing.T) {
 				},
 			},
 			expected: func() *l5dcharts.Values {
-				values, _ := l5dcharts.NewValues(false)
+				values, _ := l5dcharts.NewValues()
 				values.GetGlobal().Proxy.OutboundConnectTimeout = "6005ms"
 				values.GetGlobal().Proxy.InboundConnectTimeout = "2005ms"
 				return values
@@ -247,7 +247,7 @@ func TestGetOverriddenValues(t *testing.T) {
 				},
 			},
 			expected: func() *l5dcharts.Values {
-				values, _ := l5dcharts.NewValues(false)
+				values, _ := l5dcharts.NewValues()
 				values.GetGlobal().Proxy.OpaquePorts = "3306"
 				return values
 			},