Repository: linkerd/linkerd2
Pull request #4220 (2020-03-31 22:40:51)

Prometheus Alertmanager + extra args + rules
This PR allow users to use linkerd metrics directly from linkerd prometheus to setup alerts, furthermore it provides flexibility by means of extra cmd line arguments.

**Add prometheus rules configurable values.**

These are set as empty by default, but allow the user to configure recording and alerting rules on the linkerd prometheus.

**Add prometheusAlertmanagers value.**

This allows configuring a list of statically targetted alertmanager instances.

**Allow addition of extra arguements.**

This allows end user flexibility for options such as log format. Rather than bubbling up such possible config options into helm values, extra arguments provides more flexibility.

alpeb: Thanks @naseemkullah !

Regarding the `alerting_rules.yaml` and `recording_rules.yaml` parts, have you considered loading them through separate ConfigMaps? Instead of loading them through the main `linkerd-prometheus-config` CM, we could allow users to define as many separate CMs as they want, and Linkerd would mount them under `/etc/prometheus` for Prometheus to load them. 
So instead of declaring in `values.yaml` the content of those extra configs, we would just declare the list of extra CMs to load, e.g. in a `prometheusExtraConfigs` array. And then we'd modify `prometheus.yaml` so that it mounts them all. That might imply mounting them in subpaths and modifying the `rule_files` directive so it crawls down those subpaths.

I think decoupling these configs from the main Linkerd `values.yaml` might be more flexible. LMKWYT.
naseemkullah: Thanks for the feedback @alpeb and @zaharidichev, I'll go with that approach then.
naseemkullah: Just pushed a change, but I have not had a chance to test it yet.
naseemkullah: Just removed the leftover values and fixed indentations @alpeb , pardon the force-push! 
Pothulapati: Hey Naseem,

Thanks for taking this up! I've been trying to fix this problem in a more broader sense (and using the current add-on model), and have a [rfc up](https://github.com/linkerd/rfc/pull/16) for the same. It would be really awesome, to have your reviews/suggestions there as a end user/contributor.

I have the following question:

-  Any particular reasoning for dividing configuration into separate configmaps other than to have the state stored during upgrades? In my RFC, as we already have `linkerd-values` for storing add-on configuration details I was planning to use the same and not separate them into configmaps. WDYT?

Also, The RFC model ain't much different from this PR, especially that we get better separation of configuration there i.e `prometheus.*`, and it's also trying to address the Bringing your own prometheus use-case. If the rfc isn't missing something that you want, should we directly work on that? 

naseemkullah: > Hey Naseem,
> 
> Thanks for taking this up! I've been trying to fix this problem in a more broader sense (and using the current add-on model), and have a [rfc up](https://github.com/linkerd/rfc/pull/16) for the same. It would be really awesome, to have your reviews/suggestions there as a end user/contributor.
> 
> I have the following question:
> 
> * Any particular reasoning for dividing configuration into separate configmaps other than to have the state stored during upgrades? In my RFC, as we already have `linkerd-values` for storing add-on configuration details I was planning to use the same and not separate them into configmaps. WDYT?
> 


Hi @Pothulapati, thanks for bringing up these questions.

The original PR had a `prometheusRules` value but @alpeb and @zaharidichev advised the configmap approach as per their comments https://github.com/linkerd/linkerd2/pull/4220#issuecomment-609947357 and https://github.com/linkerd/linkerd2/pull/4220#pullrequestreview-389858587, either of these approaches would work to allow a user to define alerting and recording rules, the latter just requiring that the configmaps pre exist. 

> Also, The RFC model ain't much different from this PR, especially that we get better separation of configuration there i.e `prometheus.*`, 

The flat values (e.g. `prometheusSomeValue`) was already in place and so the pattern was followed, I'm not sure what is meant by better separation of configuration though?

> and it's also trying to address the Bringing your own prometheus use-case. If the rfc isn't missing something that you want, should we directly work on that?

Seeing that this PR was opened quite a while ago and is ready to be merged I'd rather this go in. The RFC can fill in the gaps from there perhaps?

Thanks,
Naseem
naseemkullah: cc @alpeb Can we merge this please? Thanks.
alpeb: > Any particular reasoning for dividing configuration into separate configmaps other than to have the state stored during upgrades? In my RFC, as we already have linkerd-values for storing add-on configuration details I was planning to use the same and not separate them into configmaps.

@Pothulapati Just for decoupling those configs from the rest, in generic configmaps instead of somehing linkerd-specific like `linkerd-values` which also deals with the other add-on configs.

@naseemkullah I was waiting for the `pkg/charts/linkerd2/values.go` change to be addressed before approving this PR (mentioned in my last comment). 

Having Prometheus become an add-on, its config items will change to become something similar to what we have for `tracing` in `values.yaml`. If we merge the changes from this PR, things will have to change again when the RFC changes get merged. So I propose we do merge this PR, but not announce the changes, and then the RFC should pick on these changes when the config migrates to be an add-on. Sounds good to you @Pothulapati ?
Pothulapati: > instead of somehing linkerd-specific like linkerd-values which also deals with the other add-on configs.

Yep, that makes sense. Also, okay with merging this as the rfc implementation to get merged could take some time! 
naseemkullah: > @naseemkullah I was waiting for the `pkg/charts/linkerd2/values.go` change to be addressed before approving this PR (mentioned in my last comment).

Oh right, I'll have that in shortly! Thanks.
naseemkullah: Hello @alpeb I tried adding the value structs in values.go via https://github.com/linkerd/linkerd2/pull/4220/commits/ab2887e2561fd8e2139e789098a99f9bde650ebe?diff=unified&w=1 ... not sure if I am doing this right? AlertManager is quite complex (https://prometheus.io/docs/prometheus/latest/configuration/configuration/#alertmanager_config) so I used `interface{}` as it's struct, would it be possible to import the alert manager struct from https://github.com/prometheus/prometheus/blob/fc3fb3265a0f0061d07d0bf5fb4103e2144565b2/config/config.go#L519 ?
alpeb: Thanks @naseemkullah. I tried importing that struct from the Prometheus project, but it does carry along a ton of other dependencies and then there are some build conflicts arising. So I'd say it's not worth the effort, and the best thing we can do in this case is
```
PrometheusAlertmanagers       []interface{}                  `json:"prometheusAlertmanagers"`
```
On the other hand, the `linkerd-prometheus` pod is failing with this:
```
rror: failed to create containerd task: OCI runtime create failed: container_linux.go:346: starting container process caused "process_linux.go:449: container init caus
ed \"rootfs_linux.go:58: mounting \\\"/var/lib/kubelet/pods/e6c9c0e5-0196-44d4-8751-4ae055d6a6a1/volume-subpaths/prometheus-config/prometheus/1\\\" to rootfs \\\"/run/containerd/io.containerd.runtime.v2.task/k8s.io/prometheus/rootfs\\\" a
t \\\"/run/containerd/io.containerd.runtime.v2.task/k8s.io/prometheus/rootfs/etc/prometheus\\\" caused \\\"not a directory\\\"\"": unknown
```
Maybe `mountPath` does require including the subPath? (`/etc/prometheus/{{ .subPath }}`)
naseemkullah: > Thanks @naseemkullah. I tried importing that struct from the Prometheus project, but it does carry along a ton of other dependencies and then there are some build conflicts arising. So I'd say it's not worth the effort, and the best thing we can do in this case is
> 
> ```
> PrometheusAlertmanagers       []interface{}                  `json:"prometheusAlertmanagers"`
> ```
> 
> On the other hand, the `linkerd-prometheus` pod is failing with this:
> 
> ```
> rror: failed to create containerd task: OCI runtime create failed: container_linux.go:346: starting container process caused "process_linux.go:449: container init caus
> ed \"rootfs_linux.go:58: mounting \\\"/var/lib/kubelet/pods/e6c9c0e5-0196-44d4-8751-4ae055d6a6a1/volume-subpaths/prometheus-config/prometheus/1\\\" to rootfs \\\"/run/containerd/io.containerd.runtime.v2.task/k8s.io/prometheus/rootfs\\\" a
> t \\\"/run/containerd/io.containerd.runtime.v2.task/k8s.io/prometheus/rootfs/etc/prometheus\\\" caused \\\"not a directory\\\"\"": unknown
> ```
> 
> Maybe `mountPath` does require including the subPath? (`/etc/prometheus/{{ .subPath }}`)

💯

Thanks for the tips @alpeb, I've done the needful but a failing test remains:

https://github.com/linkerd/linkerd2/pull/4220/checks?check_run_id=603813616#step:4:2075

Can you please advise?
naseemkullah: > ```diff
> expected := &Values{
> ```

Thanks @alpeb , how did you reset the values for `expected := &Values{` ? 

Changes applied, hoping tests will pass now.
naseemkullah: Hi @alpeb it now fails, i noticed the generate data command fails as well now:


```
go test ./cli/cmd/... --update
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/emoji"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/emoji"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/controller"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/not-controller"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/controller"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/not-controller"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: statefulset/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: statefulset/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: cronjob/hello"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: cronjob/hello"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: cronjob/hello"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: cronjob/hello"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: pod/vote-bot"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: pod/vote-bot"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: pod/vote-bot"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: pod/vote-bot"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web1"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web2"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web3"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web4"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web1"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web2"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web3"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web4"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/contour"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/contour"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/emoji"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/emoji"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="inject debug container"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="inject debug container"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="inject debug container"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/linkerd-tap"
time="2020-04-23T09:12:19-04:00" level=info msg="inject debug container"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/linkerd-tap"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: pod/vote-bot"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: pod/vote-bot"
time="2020-04-23T09:12:19-04:00" level=info msg="tracing enabled: remote service=linkerd-collector, service account=linkerd-collector.linkerd."
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="tracing enabled: remote service=linkerd-collector, service account=linkerd-collector.linkerd."
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/web"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/"
time="2020-04-23T09:12:19-04:00" level=info msg="patch generated for: deployment/get-test-deploy-injected-1"
time="2020-04-23T09:12:20-04:00" level=info msg="patch generated for: deployment/get-test-deploy-injected-2"
time="2020-04-23T09:12:20-04:00" level=info msg="patch generated for: deployment/get-test-deploy-injected-1"
time="2020-04-23T09:12:20-04:00" level=info msg="patch generated for: deployment/get-test-deploy-injected-2"
time="2020-04-23T09:12:20-04:00" level=info msg="patch generated for: deployment/nginx"
time="2020-04-23T09:12:20-04:00" level=info msg="patch generated for: deployment/nginx"
time="2020-04-23T09:12:20-04:00" level=info msg="patch generated for: deployment/redis"
time="2020-04-23T09:12:20-04:00" level=info msg="patch generated for: deployment/redis"
time="2020-04-23T09:12:20-04:00" level=info msg="patch generated for: deployment/redis"
time="2020-04-23T09:12:20-04:00" level=info msg="patch generated for: deployment/nginx"
time="2020-04-23T09:12:20-04:00" level=info msg="patch generated for: deployment/redis"
time="2020-04-23T09:12:20-04:00" level=info msg="patch generated for: deployment/nginx"
--- FAIL: TestRender (6.68s)
    --- FAIL: TestRender/2:_install_control-plane.golden (0.02s)
        main_test.go:67: mismatch: install_control-plane.golden
            @@ -34215,24 +34215,25 @@
             heus.yml: %7C-
            +%0A
             %0A    global:
error reading byte stream header: error while reading message length: EOF
error reading byte stream header: error while reading message length: EOF
error reading byte stream header: error while reading message length: EOF
error reading byte stream header: error while reading message length: EOF
--- FAIL: TestRenderUpgrade (4.27s)
    --- FAIL: TestRenderUpgrade/0 (0.92s)
        main_test.go:67: mismatch: upgrade_default.golden
            @@ -70459,24 +70459,25 @@
             eus.yml: %7C-%0A
            +%0A
                 global:%0A
            @@ -70611,16 +70611,51 @@
             les.yml%0A
            +    - /etc/prometheus/*_rules.yaml%0A
             %0A    scr
            @@ -76932,32 +76932,47 @@
              /etc/prometheus
            +/prometheus.yml
             %0A          name:
            @@ -76981,32 +76981,66 @@
             rometheus-config
            +%0A          subPath: prometheus.yml
             %0A          readO
    --- FAIL: TestRenderUpgrade/1 (0.56s)
        main_test.go:67: mismatch: upgrade_ha.golden
            @@ -74056,24 +74056,25 @@
             eus.yml: %7C-%0A
            +%0A
                 global:%0A
            @@ -74208,16 +74208,51 @@
             les.yml%0A
            +    - /etc/prometheus/*_rules.yaml%0A
             %0A    scr
            @@ -80688,32 +80688,47 @@
              /etc/prometheus
            +/prometheus.yml
             %0A          name:
            @@ -80737,32 +80737,66 @@
             rometheus-config
            +%0A          subPath: prometheus.yml
             %0A          readO
    --- FAIL: TestRenderUpgrade/4 (0.53s)
        main_test.go:67: mismatch: upgrade_external_issuer.golden
            @@ -68752,24 +68752,25 @@
             eus.yml: %7C-%0A
            +%0A
                 global:%0A
            @@ -68904,16 +68904,51 @@
             les.yml%0A
            +    - /etc/prometheus/*_rules.yaml%0A
             %0A    scr
            @@ -75225,32 +75225,47 @@
              /etc/prometheus
            +/prometheus.yml
             %0A          name:
            @@ -75274,32 +75274,66 @@
             rometheus-config
            +%0A          subPath: prometheus.yml
             %0A          readO
    --- FAIL: TestRenderUpgrade/6 (0.62s)
        main_test.go:67: mismatch: upgrade_overwrite_issuer.golden
            @@ -69405,24 +69405,25 @@
             eus.yml: %7C-%0A
            +%0A
                 global:%0A
            @@ -69557,16 +69557,51 @@
             les.yml%0A
            +    - /etc/prometheus/*_rules.yaml%0A
             %0A    scr
            @@ -75878,32 +75878,47 @@
              /etc/prometheus
            +/prometheus.yml
             %0A          name:
            @@ -75927,32 +75927,66 @@
             rometheus-config
            +%0A          subPath: prometheus.yml
             %0A          readO
    --- FAIL: TestRenderUpgrade/7 (0.47s)
        main_test.go:67: mismatch: upgrade_overwrite_trust_anchors.golden
            @@ -69405,24 +69405,25 @@
             eus.yml: %7C-%0A
            +%0A
                 global:%0A
            @@ -69557,16 +69557,51 @@
             les.yml%0A
            +    - /etc/prometheus/*_rules.yaml%0A
             %0A    scr
            @@ -75878,32 +75878,47 @@
              /etc/prometheus
            +/prometheus.yml
             %0A          name:
            @@ -75927,32 +75927,66 @@
             rometheus-config
            +%0A          subPath: prometheus.yml
             %0A          readO
    --- FAIL: TestRenderUpgrade/11 (0.55s)
        main_test.go:67: mismatch: upgrade_overwrite_trust_anchors-external-issuer.golden
            @@ -68004,24 +68004,25 @@
             eus.yml: %7C-%0A
            +%0A
                 global:%0A
            @@ -68156,16 +68156,51 @@
             les.yml%0A
            +    - /etc/prometheus/*_rules.yaml%0A
             %0A    scr
            @@ -74477,32 +74477,47 @@
              /etc/prometheus
            +/prometheus.yml
             %0A          name:
            @@ -74526,32 +74526,66 @@
             rometheus-config
            +%0A          subPath: prometheus.yml
             %0A          readO
FAIL
FAIL    github.com/linkerd/linkerd2/cli/cmd     14.803s
FAIL
```
naseemkullah: Worth noting: After applying the patch `go test ./cli/cmd/... --update` had to be run twice to succeed 🤔 ... failed on first try but succeeded on second 🤷‍♂️ 
naseemkullah: 😭 
Still failing.
```
--- FAIL: TestNewValues (0.02s)
    --- FAIL: TestNewValues/HA (0.01s)
        values_test.go:241: Mismatch Helm HA defaults.
            Expected: &{Stage: ControllerImage:gcr.io/linkerd-io/controller ControllerImageVersion:linkerd-dev WebImage:gcr.io/linkerd-io/web PrometheusImage:prom/prometheus:v2.15.2 GrafanaImage:gcr.io/linkerd-io/grafana ControllerReplicas:3 ControllerLogLevel:info PrometheusLogLevel:info PrometheusExtraArgs:map[] PrometheusAlertmanagers:[] PrometheusRuleConfigMapMounts:[] ControllerUID:2103 EnableH2Upgrade:true EnablePodAntiAffinity:true WebhookFailurePolicy:Fail OmitWebhookSideEffects:false RestrictDashboardPrivileges:false DisableHeartBeat:false HeartbeatSchedule:0 0 * * * InstallNamespace:true Configs:{Global: Proxy: Install:} Global:0xc0000eed00 Identity:0xc000010890 Dashboard:0xc0004c8d40 DebugContainer:0xc0000108a0 ProxyInjector:0xc0000108b0 ProfileValidator:0xc0000108c0 Tap:0xc0000108d0 NodeSelector:map[beta.kubernetes.io/os:linux] SMIMetrics:0xc00000f720 DestinationResources:0xc00003d2c0 GrafanaResources:0xc00003d340 HeartbeatResources:0xc00003d2c0 IdentityResources:0xc00003d380 PrometheusResources:0xc00003d3c0 ProxyInjectorResources:0xc00003d2c0 PublicAPIResources:0xc00003d2c0 SPValidatorResources:0xc00003d2c0 TapResources:0xc00003d2c0 WebResources:0xc00003d2c0 Tracing:map[]}
            Actual: &{Stage: ControllerImage:gcr.io/linkerd-io/controller ControllerImageVersion:linkerd-dev WebImage:gcr.io/linkerd-io/web PrometheusImage:prom/prometheus:v2.15.2 GrafanaImage:gcr.io/linkerd-io/grafana ControllerReplicas:3 ControllerLogLevel:info PrometheusLogLevel:info PrometheusExtraArgs:map[] PrometheusAlertmanagers:[] PrometheusRuleConfigMapMounts:[] ControllerUID:2103 EnableH2Upgrade:true EnablePodAntiAffinity:true WebhookFailurePolicy:Fail OmitWebhookSideEffects:false RestrictDashboardPrivileges:false DisableHeartBeat:false HeartbeatSchedule:0 0 * * * InstallNamespace:true Configs:{Global: Proxy: Install:} Global:0xc0000ef300 Identity:0xc000010910 Dashboard:0xc0004c9750 DebugContainer:0xc000010920 ProxyInjector:0xc000010980 ProfileValidator:0xc000010970 Tap:0xc000010990 NodeSelector:map[beta.kubernetes.io/os:linux] SMIMetrics:0xc0001641c0 DestinationResources:0xc00003d000 GrafanaResources:0xc00003d080 HeartbeatResources:0xc00003d140 IdentityResources:0xc00003ce80 PrometheusResources:0xc00003cec0 ProxyInjectorResources:0xc00003cf40 PublicAPIResources:0xc00003d040 SPValidatorResources:0xc00003cf80 TapResources:0xc00003d1c0 WebResources:0xc00003d200 Tracing:map[]}
FAIL
```
alpeb: @naseemkullah my fault; the mergo workaround should have gone inside `t.Run("HA", func(t *testing.T) {` , right after calling `NewValues(true)`
naseemkullah: > NewValues(true)

testing ain't easy! :)
Trying again.
naseemkullah: > {{- toYaml .Values.prometheusAlertmanagers | trim | nindent 8 }}

Hi @alpeb thank for your guidance thus far.

The current rendering of

```yaml
prometheusAlertmanagers:
  - scheme: http
    static_configs:
    - targets:
      - "alertmanager.linkerd.svc:9093"
  - scheme: https
    static_configs:
    - targets:
      - alertmanager.mycomain.com
```

is

```yaml
  prometheus.yml: |-
    alerting:
      alertmanagers:
        - scheme: http
          static_configs:
          - targets:
            - alertmanager.linkerd.svc:9093
        - scheme: https
          static_configs:
          - targets:
            - alertmanager.mycomain.com
```

which seems fine, in what case does it render funny?
alpeb: It renders fine locally, e.g. using `helm template`, but when actually installing with `helm install` if you ask for it through `kubectl -n linkerd get cm linkerd-prometheus-config -oyaml` you'll see there are issues.
naseemkullah: Thanks @Pothulapati, my absolute pleasure!
alpeb: Hey @naseemkullah , please fill out this [form](https://docs.google.com/forms/d/e/1FAIpQLSdXA7kAl6oyUQ4DFRjiDc2biWlvkhNirrMwsXaQ8V3Uh8Yj5A/viewform) if you'd like to receive some well-earned Linkerd swag! :wink: 

Patch:
diff --git a/charts/linkerd2/README.md b/charts/linkerd2/README.md
index 24d158d3b..8ffa8a658 100644
--- a/charts/linkerd2/README.md
+++ b/charts/linkerd2/README.md
@@ -82,80 +82,83 @@ helm install \
 
 The following table lists the configurable parameters of the Linkerd2 chart and their default values.
 
-| Parameter                             | Description                                                                                                                                                                           | Default                              |
-|:--------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------|
-| `controllerImage`                     | Docker image for the controller, tap and identity components                                                                                                                          | `gcr.io/linkerd-io/controller`       |
-| `controllerImageVersion`              | Tag for the controller container docker image                                                                                                                                         | latest version                       |
-| `controllerLogLevel`                  | Log level for the control plane components                                                                                                                                            | `info`                               |
-| `controllerReplicas`                  | Number of replicas for each control plane pod                                                                                                                                         | `1`                                  |
-| `controllerUID`                       | User ID for the control plane components                                                                                                                                              | `2103`                               |
-| `dashboard.replicas`                  | Number of replicas of dashboard                                                                                                                                                       | `1`                                  |
-| `debugContainer.image.name`           | Docker image for the debug container                                                                                                                                                  | `gcr.io/linkerd-io/debug`            |
-| `debugContainer.image.pullPolicy`     | Pull policy for the debug container Docker image                                                                                                                                      | `IfNotPresent`                       |
-| `debugContainer.image.version`        | Tag for the debug container Docker image                                                                                                                                              | latest version                       |
-| `disableHeartBeat`                    | Set to true to not start the heartbeat cronjob                                                                                                                                        | `false`                              |
-| `enableH2Upgrade`                     | Allow proxies to perform transparent HTTP/2 upgrading                                                                                                                                 | `true`                               |
-| `global.clusterDomain`                       | Kubernetes DNS Domain name to use                                                                                                                                                     | `cluster.local`                      |
-| `global.cniEnabled`                          | Omit the NET_ADMIN capability in the PSP and the proxy-init container when injecting the proxy; requires the linkerd-cni plugin to already be installed                               | `false`                      |
-| `global.controllerComponentLabel`            | Control plane label. Do not edit                                                                                                                                                      | `linkerd.io/control-plane-component` |
-| `global.controllerNamespaceLabel`            | Control plane label. Do not edit                                                                                                                                                      | `linkerd.io/control-plane-component` |
-| `global.createdByAnnotation`                 | Annotation label for the proxy create. Do not edit.                                                                                                                                   | `linkerd.io/created-by`              |
-| `global.identityTrustAnchorsPEM`            | Trust root certificate (ECDSA). It must be provided during install.                                                                                                                                                         ||
+| Parameter                                   | Description                                                                                                                                                                           | Default                              |
+|:--------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------|
+| `controllerImage`                           | Docker image for the controller, tap and identity components                                                                                                                          | `gcr.io/linkerd-io/controller`       |
+| `controllerImageVersion`                    | Tag for the controller container docker image                                                                                                                                         | latest version                       |
+| `controllerLogLevel`                        | Log level for the control plane components                                                                                                                                            | `info`                               |
+| `controllerReplicas`                        | Number of replicas for each control plane pod                                                                                                                                         | `1`                                  |
+| `controllerUID`                             | User ID for the control plane components                                                                                                                                              | `2103`                               |
+| `dashboard.replicas`                        | Number of replicas of dashboard                                                                                                                                                       | `1`                                  |
+| `debugContainer.image.name`                 | Docker image for the debug container                                                                                                                                                  | `gcr.io/linkerd-io/debug`            |
+| `debugContainer.image.pullPolicy`           | Pull policy for the debug container Docker image                                                                                                                                      | `IfNotPresent`                       |
+| `debugContainer.image.version`              | Tag for the debug container Docker image                                                                                                                                              | latest version                       |
+| `disableHeartBeat`                          | Set to true to not start the heartbeat cronjob                                                                                                                                        | `false`                              |
+| `enableH2Upgrade`                           | Allow proxies to perform transparent HTTP/2 upgrading                                                                                                                                 | `true`                               |
+| `global.clusterDomain`                      | Kubernetes DNS Domain name to use                                                                                                                                                     | `cluster.local`                      |
+| `global.cniEnabled`                         | Omit the NET_ADMIN capability in the PSP and the proxy-init container when injecting the proxy; requires the linkerd-cni plugin to already be installed                               | `false`                              |
+| `global.controllerComponentLabel`           | Control plane label. Do not edit                                                                                                                                                      | `linkerd.io/control-plane-component` |
+| `global.controllerNamespaceLabel`           | Control plane label. Do not edit                                                                                                                                                      | `linkerd.io/control-plane-component` |
+| `global.createdByAnnotation`                | Annotation label for the proxy create. Do not edit.                                                                                                                                   | `linkerd.io/created-by`              |
+| `global.identityTrustAnchorsPEM`            | Trust root certificate (ECDSA). It must be provided during install.                                                                                                                   |                                      |
 | `global.identityTrustDomain`                | Trust domain used for identity                                                                                                                                                        | `cluster.local`                      |
-| `global.imagePullPolicy`                     | Docker image pull policy                                                                                                                                                              | `IfNotPresent`                       |
-| `global.linkerdNamespaceLabel`               | Control plane label. Do not edit                                                                                                                                                      | `linkerd.io/control-plane-component` |
-| `global.linkerdVersion`                      | Control plane version                                                                                                                                                                 | latest version                       |
-| `global.namespace`                           | Control plane namespace                                                                                                                                                               | `linkerd`                            |
-| `global.proxy.enableExternalProfiles`        | Enable service profiles for non-Kubernetes services                                                                                                                                   | `false`                              |
-| `global.proxy.image.name`                    | Docker image for the proxy                                                                                                                                                            | `gcr.io/linkerd-io/proxy`            |
-| `global.proxy.image.pullPolicy`              | Pull policy for the proxy container Docker image                                                                                                                                      | `IfNotPresent`                       |
-| `global.proxy.image.version`                 | Tag for the proxy container Docker image                                                                                                                                              | latest version                       |
-| `global.proxy.logLevel`                      | Log level for the proxy                                                                                                                                                               | `warn,linkerd=info`           |
-| `global.proxy.ports.admin`                   | Admin port for the proxy container                                                                                                                                                    | `4191`                               |
-| `global.proxy.ports.control`                 | Control port for the proxy container                                                                                                                                                  | `4190`                               |
-| `global.proxy.ports.inbound`                 | Inbound port for the proxy container                                                                                                                                                  | `4143`                               |
-| `global.proxy.ports.outbound`                | Outbound port for the proxy container                                                                                                                                                 | `4140`                               |
-| `global.proxy.resources.cpu.limit`           | Maximum amount of CPU units that the proxy can use                                                                                                                                                                          ||
-| `global.proxy.resources.cpu.request`         | Amount of CPU units that the proxy requests                                                                                                                                                                                 ||
-| `global.proxy.resources.memory.limit`        | Maximum amount of memory that the proxy can use                                                                                                                                                                             ||
-| `global.proxy.resources.memory.request`      | Amount of memory that the proxy requests                                                                                                                                                                                    ||
-| `global.proxy.trace.collectorSvcAccount`     | Service account associated with the Trace collector instance                                                                                                                                                                ||
-| `global.proxy.trace.collectorSvcAddr`        | Collector Service address for the proxies to send Trace Data                                                                                                                                                                ||
-| `global.proxy.uid`                           | User id under which the proxy runs                                                                                                                                                    | `2102`                               |
-| `global.proxy.waitBeforeExitSeconds`         | The proxy sidecar will stay alive for at least the given period before receiving SIGTERM signal from Kubernetes but no longer than pod's `terminationGracePeriodSeconds`.             | `0`                                  |
-| `global.proxyInit.ignoreInboundPorts`        | Inbound ports the proxy should ignore                                                                                                                                                                                       ||
-| `global.proxyInit.ignoreOutboundPorts`       | Outbound ports the proxy should ignore                                                                                                                                                                                      ||
-| `global.proxyInit.image.name`                | Docker image for the proxy-init container                                                                                                                                             | `gcr.io/linkerd-io/proxy-init`       |
-| `global.proxyInit.image.pullPolicy`          | Pull policy for the proxy-init container Docker image                                                                                                                                 | `IfNotPresent`                       |
-| `global.proxyInit.image.version`             | Tag for the proxy-init container Docker image                                                                                                                                         | latest version                       |
-| `global.proxyInit.resources.cpu.limit`       | Maximum amount of CPU units that the proxy-init container can use                                                                                                                     | `100m`                               |
-| `global.proxyInit.resources.cpu.request`     | Amount of CPU units that the proxy-init container requests                                                                                                                            | `10m`                                |
-| `global.ProxyInit.resources.memory.limit`    | Maximum amount of memory that the proxy-init container can use                                                                                                                        | `50Mi`                               |
-| `global.proxyInit.resources.memory.request`  | Amount of memory that the proxy-init container requests                                                                                                                               | `10Mi`                               |
-| `global.proxyInjectAnnotation`               | Annotation label to signal injection. Do not edit.                                                                                                                                                                          ||
-| `global.proxyInjectDisabled`                 | Annotation value to disable injection. Do not edit.                                                                                                                                   | `disabled`                           |
-| `grafanaImage`                        | Docker image for the Grafana container                                                                                                                                                | `gcr.io/linkerd-io/grafana`          |
-| `heartbeatSchedule`                   | Config for the heartbeat cronjob                                                                                                                                                      | `0 0 * * *`                          |
-| `identity.issuer.clockSkewAllowance`  | Amount of time to allow for clock skew within a Linkerd cluster                                                                                                                       | `20s`                                |
-| `identity.issuer.crtExpiry`           | Expiration timestamp for the issuer certificate. It must be provided during install                                                                                                                                         ||
-| `identity.issuer.crtExpiryAnnotation` | Annotation used to identity the issuer certificate expiration timestamp. Do not edit.                                                                                                 | `linkerd.io/identity-issuer-expiry`  |
-| `identity.issuer.issuanceLifetime`    | Amount of time for which the Identity issuer should certify identity                                                                                                                  | `86400s`                             |
-| `identity.issuer.scheme`              | Which scheme is used for the identity issuer secret format                                                                                                                            | `linkerd.io/tls`                     |
-| `identity.issuer.tls.crtPEM`          | Issuer certificate (ECDSA). It must be provided during install.                                                                                                                                                             ||
-| `identity.issuer.tls.keyPEM`          | Key for the issuer certificate (ECDSA). It must be provided during install.                                                                                                                                                 ||
-| `installNamespace`                    | Set to false when installing Linkerd in a custom namespace. See the [Linkerd documentation](https://linkerd.io/2/tasks/install-helm/#customizing-the-namespace) for more information. | `true`                               |
-| `omitWebhookSideEffects`              | Omit the `sideEffects` flag in the webhook manifests                                                                                                                                  | `false`                              |
-| `prometheusImage`                     | Docker image for the Prometheus container                                                                                                                                             | `prom/prometheus:v2.15.2`            |
-| `prometheusLogLevel`                  | Log level for Prometheus                                                                                                                                                              | `info`                               |
-| `proxyInjector.crtPEM`                | Certificate for the proxy injector. If not provided then Helm will generate one.                                                                                                                                            ||
-| `proxyInjector.keyPEM`                | Certificate key for the proxy injector. If not provided then Helm will generate one.                                                                                                                                        ||
-| `profileValidator.crtPEM`             | Certificate for the service profile validator. If not provided then Helm will generate one.                                                                                                                                 ||
-| `profileValidator.keyPEM`             | Certificate key for the service profile validator. If not provided then Helm will generate one.                                                                                                                             ||
-| `tap.crtPEM`                          | Certificate for the Tap component. If not provided then Helm will generate one.                                                                                                                                             ||
-| `tap.keyPEM`                          | Certificate key for Tap component. If not provided then Helm will generate one.                                                                                                                                             ||
-| `webhookFailurePolicy`                | Failure policy for the proxy injector                                                                                                                                                 | `Ignore`                             |
-| `webImage`                            | Docker image for the web container                                                                                                                                                    | `gcr.io/linkerd-io/web`              |
-| `enforcedHostRegexp`                  | Host header validation regex for the dashboard. See the [Linkerd documentation](https://linkerd.io/2/tasks/exposing-dashboard) for more information                                   | `""`              |
+| `global.imagePullPolicy`                    | Docker image pull policy                                                                                                                                                              | `IfNotPresent`                       |
+| `global.linkerdNamespaceLabel`              | Control plane label. Do not edit                                                                                                                                                      | `linkerd.io/control-plane-component` |
+| `global.linkerdVersion`                     | Control plane version                                                                                                                                                                 | latest version                       |
+| `global.namespace`                          | Control plane namespace                                                                                                                                                               | `linkerd`                            |
+| `global.proxy.enableExternalProfiles`       | Enable service profiles for non-Kubernetes services                                                                                                                                   | `false`                              |
+| `global.proxy.image.name`                   | Docker image for the proxy                                                                                                                                                            | `gcr.io/linkerd-io/proxy`            |
+| `global.proxy.image.pullPolicy`             | Pull policy for the proxy container Docker image                                                                                                                                      | `IfNotPresent`                       |
+| `global.proxy.image.version`                | Tag for the proxy container Docker image                                                                                                                                              | latest version                       |
+| `global.proxy.logLevel`                     | Log level for the proxy                                                                                                                                                               | `warn,linkerd=info`                  |
+| `global.proxy.ports.admin`                  | Admin port for the proxy container                                                                                                                                                    | `4191`                               |
+| `global.proxy.ports.control`                | Control port for the proxy container                                                                                                                                                  | `4190`                               |
+| `global.proxy.ports.inbound`                | Inbound port for the proxy container                                                                                                                                                  | `4143`                               |
+| `global.proxy.ports.outbound`               | Outbound port for the proxy container                                                                                                                                                 | `4140`                               |
+| `global.proxy.resources.cpu.limit`          | Maximum amount of CPU units that the proxy can use                                                                                                                                    |                                      |
+| `global.proxy.resources.cpu.request`        | Amount of CPU units that the proxy requests                                                                                                                                           |                                      |
+| `global.proxy.resources.memory.limit`       | Maximum amount of memory that the proxy can use                                                                                                                                       |                                      |
+| `global.proxy.resources.memory.request`     | Amount of memory that the proxy requests                                                                                                                                              |                                      |
+| `global.proxy.trace.collectorSvcAccount`    | Service account associated with the Trace collector instance                                                                                                                          |                                      |
+| `global.proxy.trace.collectorSvcAddr`       | Collector Service address for the proxies to send Trace Data                                                                                                                          |                                      |
+| `global.proxy.uid`                          | User id under which the proxy runs                                                                                                                                                    | `2102`                               |
+| `global.proxy.waitBeforeExitSeconds`        | The proxy sidecar will stay alive for at least the given period before receiving SIGTERM signal from Kubernetes but no longer than pod's `terminationGracePeriodSeconds`.             | `0`                                  |
+| `global.proxyInit.ignoreInboundPorts`       | Inbound ports the proxy should ignore                                                                                                                                                 |                                      |
+| `global.proxyInit.ignoreOutboundPorts`      | Outbound ports the proxy should ignore                                                                                                                                                |                                      |
+| `global.proxyInit.image.name`               | Docker image for the proxy-init container                                                                                                                                             | `gcr.io/linkerd-io/proxy-init`       |
+| `global.proxyInit.image.pullPolicy`         | Pull policy for the proxy-init container Docker image                                                                                                                                 | `IfNotPresent`                       |
+| `global.proxyInit.image.version`            | Tag for the proxy-init container Docker image                                                                                                                                         | latest version                       |
+| `global.proxyInit.resources.cpu.limit`      | Maximum amount of CPU units that the proxy-init container can use                                                                                                                     | `100m`                               |
+| `global.proxyInit.resources.cpu.request`    | Amount of CPU units that the proxy-init container requests                                                                                                                            | `10m`                                |
+| `global.ProxyInit.resources.memory.limit`   | Maximum amount of memory that the proxy-init container can use                                                                                                                        | `50Mi`                               |
+| `global.proxyInit.resources.memory.request` | Amount of memory that the proxy-init container requests                                                                                                                               | `10Mi`                               |
+| `global.proxyInjectAnnotation`              | Annotation label to signal injection. Do not edit.                                                                                                                                    |                                      |
+| `global.proxyInjectDisabled`                | Annotation value to disable injection. Do not edit.                                                                                                                                   | `disabled`                           |
+| `grafanaImage`                              | Docker image for the Grafana container                                                                                                                                                | `gcr.io/linkerd-io/grafana`          |
+| `heartbeatSchedule`                         | Config for the heartbeat cronjob                                                                                                                                                      | `0 0 * * *`                          |
+| `identity.issuer.clockSkewAllowance`        | Amount of time to allow for clock skew within a Linkerd cluster                                                                                                                       | `20s`                                |
+| `identity.issuer.crtExpiry`                 | Expiration timestamp for the issuer certificate. It must be provided during install                                                                                                   |                                      |
+| `identity.issuer.crtExpiryAnnotation`       | Annotation used to identity the issuer certificate expiration timestamp. Do not edit.                                                                                                 | `linkerd.io/identity-issuer-expiry`  |
+| `identity.issuer.issuanceLifetime`          | Amount of time for which the Identity issuer should certify identity                                                                                                                  | `86400s`                             |
+| `identity.issuer.scheme`                    | Which scheme is used for the identity issuer secret format                                                                                                                            | `linkerd.io/tls`                     |
+| `identity.issuer.tls.crtPEM`                | Issuer certificate (ECDSA). It must be provided during install.                                                                                                                       |                                      |
+| `identity.issuer.tls.keyPEM`                | Key for the issuer certificate (ECDSA). It must be provided during install.                                                                                                           |                                      |
+| `installNamespace`                          | Set to false when installing Linkerd in a custom namespace. See the [Linkerd documentation](https://linkerd.io/2/tasks/install-helm/#customizing-the-namespace) for more information. | `true`                               |
+| `omitWebhookSideEffects`                    | Omit the `sideEffects` flag in the webhook manifests                                                                                                                                  | `false`                              |
+| `prometheusAlertmanagers`                   | Alertmanager instances the Prometheus server sends alerts to configured via the static_configs parameter.                                                                             | `[]`                                 |
+| `prometheusExtraArgs`                       | Extra command line options for Prometheus                                                                                                                                             | `{}`                                 |
+| `prometheusImage`                           | Docker image for the Prometheus container                                                                                                                                             | `prom/prometheus:v2.15.2`            |
+| `prometheusLogLevel`                        | Log level for Prometheus                                                                                                                                                              | `info`                               |
+| `prometheusRuleConfigMapMounts`             | Alerting/recording rule ConfigMap mounts (sub-path names must end in `_rules.yml` or `_rules.yaml`)                                                                                   | `[]`                                 |
+| `proxyInjector.crtPEM`                      | Certificate for the proxy injector. If not provided then Helm will generate one.                                                                                                      |                                      |
+| `proxyInjector.keyPEM`                      | Certificate key for the proxy injector. If not provided then Helm will generate one.                                                                                                  |                                      |
+| `profileValidator.crtPEM`                   | Certificate for the service profile validator. If not provided then Helm will generate one.                                                                                           |                                      |
+| `profileValidator.keyPEM`                   | Certificate key for the service profile validator. If not provided then Helm will generate one.                                                                                       |                                      |
+| `tap.crtPEM`                                | Certificate for the Tap component. If not provided then Helm will generate one.                                                                                                       |                                      |
+| `tap.keyPEM`                                | Certificate key for Tap component. If not provided then Helm will generate one.                                                                                                       |                                      |
+| `webhookFailurePolicy`                      | Failure policy for the proxy injector                                                                                                                                                 | `Ignore`                             |
+| `webImage`                                  | Docker image for the web container                                                                                                                                                    | `gcr.io/linkerd-io/web`              |
+| `enforcedHostRegexp`                        | Host header validation regex for the dashboard. See the [Linkerd documentation](https://linkerd.io/2/tasks/exposing-dashboard) for more information                                   | `""`                                 |
 
 ## Add-Ons Configuration
 
@@ -163,21 +166,21 @@ The following table lists the configurable parameters of the Linkerd2 chart and
 
 The following table lists the configurable parameters for the Tracing Add-On.
 
-| Parameter                             | Description                                                                                                                                                                           | Default                              |
-|:--------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------|
-| `tracing.enabled`                     | Flag to enable tracing components to be installed                                                                                                                                                | `false`
-| `tracing.collector.name`                | Name of the trace collector Service                                                                                                                                                 | `linkerd-collector`                             |
-| `tracing.collector.image`                | Docker image for the trace collector                                                                                                                                                 | `omnition/opencensus-collector:0.1.10`                             |
-| `tracing.collector.resources.cpu.limit`       | Maximum amount of CPU units that the trace collector container can use                                                                                                                     | `1`                               |
-| `tracing.collector.resources.cpu.request`     | Amount of CPU units that the trace collector container requests                                                                                                                            | `200m`                                |
-| `tracing.collector.resources.memory.limit`    | Maximum amount of memory that the trace collector container can use                                                                                                                        | `2Gi`                               |
-| `tracing.collector.resources.memory.request`  | Amount of memory that the trace collector container requests                                                                                                                               | `400Mi`                               |
-| `tracing.jaeger.name`                | Name of the jaeger instance                                                                                                                                                 | `linkerd-jaeger`                             |
-| `tracing.jaeger.image`                | Docker image for the jaeger instance                                                                                                                                                 | `jaegertracing/all-in-one:1.8`                             |
-| `tracing.jaeger.resources.cpu.limit`       | Maximum amount of CPU units that the jaeger container can use                                                                                                                     ||
-| `tracing.jaeger.resources.cpu.request`     | Amount of CPU units that the jaeger container requests                                                                                                                            ||
-| `tracing.jaeger.resources.memory.limit`    | Maximum amount of memory that the jaeger container can use                                                                                                                        ||
-| `tracing.jaeger.resources.memory.request`  | Amount of memory that the jaeger container requests                                                                                                                               ||
+| Parameter                                    | Description                                                            | Default                                |
+|:---------------------------------------------|:-----------------------------------------------------------------------|:---------------------------------------|
+| `tracing.enabled`                            | Flag to enable tracing components to be installed                      | `false`                                |
+| `tracing.collector.name`                     | Name of the trace collector Service                                    | `linkerd-collector`                    |
+| `tracing.collector.image`                    | Docker image for the trace collector                                   | `omnition/opencensus-collector:0.1.10` |
+| `tracing.collector.resources.cpu.limit`      | Maximum amount of CPU units that the trace collector container can use | `1`                                    |
+| `tracing.collector.resources.cpu.request`    | Amount of CPU units that the trace collector container requests        | `200m`                                 |
+| `tracing.collector.resources.memory.limit`   | Maximum amount of memory that the trace collector container can use    | `2Gi`                                  |
+| `tracing.collector.resources.memory.request` | Amount of memory that the trace collector container requests           | `400Mi`                                |
+| `tracing.jaeger.name`                        | Name of the jaeger instance                                            | `linkerd-jaeger`                       |
+| `tracing.jaeger.image`                       | Docker image for the jaeger instance                                   | `jaegertracing/all-in-one:1.8`         |
+| `tracing.jaeger.resources.cpu.limit`         | Maximum amount of CPU units that the jaeger container can use          |                                        |
+| `tracing.jaeger.resources.cpu.request`       | Amount of CPU units that the jaeger container requests                 |                                        |
+| `tracing.jaeger.resources.memory.limit`      | Maximum amount of memory that the jaeger container can use             |                                        |
+| `tracing.jaeger.resources.memory.request`    | Amount of memory that the jaeger container requests                    |                                        |
 
 ## Get involved
 
diff --git a/charts/linkerd2/templates/prometheus.yaml b/charts/linkerd2/templates/prometheus.yaml
index 59176fe67..d190d62e1 100644
--- a/charts/linkerd2/templates/prometheus.yaml
+++ b/charts/linkerd2/templates/prometheus.yaml
@@ -15,6 +15,11 @@ metadata:
     {{.Values.global.createdByAnnotation}}: {{default (printf "linkerd/helm %s" .Values.global.linkerdVersion) .Values.global.cliVersion}}
 data:
   prometheus.yml: |-
+    {{- if .Values.prometheusAlertmanagers }}
+    alerting:
+      alertmanagers:
+        {{- toYaml .Values.prometheusAlertmanagers | trim | nindent 8 }}
+    {{- end }}
     global:
       scrape_interval: 10s
       scrape_timeout: 10s
@@ -22,6 +27,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -209,6 +215,9 @@ spec:
         - --storage.tsdb.retention.time=6h
         - --config.file=/etc/prometheus/prometheus.yml
         - --log.level={{lower .Values.prometheusLogLevel}}
+        {{- range $key, $value := .Values.prometheusExtraArgs}}
+        - --{{ $key }}{{ if $value }}={{ $value }}{{ end }}
+        {{- end }}
         image: {{.Values.prometheusImage}}
         imagePullPolicy: {{.Values.global.imagePullPolicy}}
         livenessProbe:
@@ -233,10 +242,17 @@ spec:
         securityContext:
           runAsUser: 65534
         volumeMounts:
+      {{- range .Values.prometheusRuleConfigMapMounts }}
+        - name: {{ .name }}
+          mountPath: /etc/prometheus/{{ .subPath }}
+          subPath: {{ .subPath }}
+          readOnly: true
+      {{- end }}
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - {{- include "partials.proxy" . | indent 8 | trimPrefix (repeat 7 " ") }}
       {{ if not .Values.global.cniEnabled -}}
@@ -245,6 +261,11 @@ spec:
       {{ end -}}
       serviceAccountName: linkerd-prometheus
       volumes:
+    {{- range .Values.prometheusRuleConfigMapMounts }}
+      - name: {{ .name }}
+        configMap:
+          name: {{ .configMap }}
+    {{- end }}
       - emptyDir: {}
         name: data
       - configMap:
diff --git a/charts/linkerd2/values.yaml b/charts/linkerd2/values.yaml
index ad6cb6c69..92cc7ffce 100644
--- a/charts/linkerd2/values.yaml
+++ b/charts/linkerd2/values.yaml
@@ -7,7 +7,7 @@ global:
   clusterDomain: &cluster_domain cluster.local
   imagePullPolicy: &image_pull_policy IfNotPresent
 
- # control plane trace configuration
+  # control plane trace configuration
   controlPlaneTracing: false
 
   # control plane version. See Proxy section for proxy version
@@ -133,6 +133,20 @@ heartbeatSchedule: "0 0 * * *"
 # prometheus configuration
 prometheusImage: prom/prometheus:v2.15.2
 prometheusLogLevel: *controller_log_level
+prometheusExtraArgs: {}
+  # log.format: json
+prometheusAlertmanagers: []
+  # - scheme: http
+  #   static_configs:
+  #   - targets:
+  #     - "alertmanager.linkerd.svc:9093"
+prometheusRuleConfigMapMounts: []
+  # - name: alerting-rules
+  #   subPath: alerting_rules.yml
+  #   configMap: linkerd-prometheus-rules
+  # - name: recording-rules
+  #   subPath: recording_rules.yml
+  #   configMap: linkerd-prometheus-rules
 
 # proxy injector configuration
 proxyInjector:
diff --git a/pkg/charts/linkerd2/values.go b/pkg/charts/linkerd2/values.go
index 01e9b119a..7ef3ed525 100644
--- a/pkg/charts/linkerd2/values.go
+++ b/pkg/charts/linkerd2/values.go
@@ -19,34 +19,37 @@ const (
 type (
 	// Values contains the top-level elements in the Helm charts
 	Values struct {
-		Stage                       string            `json:"stage"`
-		ControllerImage             string            `json:"controllerImage"`
-		ControllerImageVersion      string            `json:"controllerImageVersion"`
-		WebImage                    string            `json:"webImage"`
-		PrometheusImage             string            `json:"prometheusImage"`
-		GrafanaImage                string            `json:"grafanaImage"`
-		ControllerReplicas          uint              `json:"controllerReplicas"`
-		ControllerLogLevel          string            `json:"controllerLogLevel"`
-		PrometheusLogLevel          string            `json:"prometheusLogLevel"`
-		ControllerUID               int64             `json:"controllerUID"`
-		EnableH2Upgrade             bool              `json:"enableH2Upgrade"`
-		EnablePodAntiAffinity       bool              `json:"enablePodAntiAffinity"`
-		WebhookFailurePolicy        string            `json:"webhookFailurePolicy"`
-		OmitWebhookSideEffects      bool              `json:"omitWebhookSideEffects"`
-		RestrictDashboardPrivileges bool              `json:"restrictDashboardPrivileges"`
-		DisableHeartBeat            bool              `json:"disableHeartBeat"`
-		HeartbeatSchedule           string            `json:"heartbeatSchedule"`
-		InstallNamespace            bool              `json:"installNamespace"`
-		Configs                     ConfigJSONs       `json:"configs"`
-		Global                      *Global           `json:"global"`
-		Identity                    *Identity         `json:"identity"`
-		Dashboard                   *Dashboard        `json:"dashboard"`
-		DebugContainer              *DebugContainer   `json:"debugContainer"`
-		ProxyInjector               *ProxyInjector    `json:"proxyInjector"`
-		ProfileValidator            *ProfileValidator `json:"profileValidator"`
-		Tap                         *Tap              `json:"tap"`
-		NodeSelector                map[string]string `json:"nodeSelector"`
-		SMIMetrics                  *SMIMetrics       `json:"smiMetrics"`
+		Stage                         string                         `json:"stage"`
+		ControllerImage               string                         `json:"controllerImage"`
+		ControllerImageVersion        string                         `json:"controllerImageVersion"`
+		WebImage                      string                         `json:"webImage"`
+		PrometheusImage               string                         `json:"prometheusImage"`
+		GrafanaImage                  string                         `json:"grafanaImage"`
+		ControllerReplicas            uint                           `json:"controllerReplicas"`
+		ControllerLogLevel            string                         `json:"controllerLogLevel"`
+		PrometheusLogLevel            string                         `json:"prometheusLogLevel"`
+		PrometheusExtraArgs           map[string]string              `json:"prometheusExtraArgs"`
+		PrometheusAlertmanagers       []interface{}                  `json:"prometheusAlertmanagers"`
+		PrometheusRuleConfigMapMounts []PrometheusRuleConfigMapMount `json:"prometheusRuleConfigMapMounts"`
+		ControllerUID                 int64                          `json:"controllerUID"`
+		EnableH2Upgrade               bool                           `json:"enableH2Upgrade"`
+		EnablePodAntiAffinity         bool                           `json:"enablePodAntiAffinity"`
+		WebhookFailurePolicy          string                         `json:"webhookFailurePolicy"`
+		OmitWebhookSideEffects        bool                           `json:"omitWebhookSideEffects"`
+		RestrictDashboardPrivileges   bool                           `json:"restrictDashboardPrivileges"`
+		DisableHeartBeat              bool                           `json:"disableHeartBeat"`
+		HeartbeatSchedule             string                         `json:"heartbeatSchedule"`
+		InstallNamespace              bool                           `json:"installNamespace"`
+		Configs                       ConfigJSONs                    `json:"configs"`
+		Global                        *Global                        `json:"global"`
+		Identity                      *Identity                      `json:"identity"`
+		Dashboard                     *Dashboard                     `json:"dashboard"`
+		DebugContainer                *DebugContainer                `json:"debugContainer"`
+		ProxyInjector                 *ProxyInjector                 `json:"proxyInjector"`
+		ProfileValidator              *ProfileValidator              `json:"profileValidator"`
+		Tap                           *Tap                           `json:"tap"`
+		NodeSelector                  map[string]string              `json:"nodeSelector"`
+		SMIMetrics                    *SMIMetrics                    `json:"smiMetrics"`
 
 		DestinationResources   *Resources `json:"destinationResources"`
 		GrafanaResources       *Resources `json:"grafanaResources"`
@@ -188,6 +191,13 @@ type (
 		TLS                 *TLS      `json:"tls"`
 	}
 
+	// PrometheusRuleConfigMapMount is a user supplied prometheus rule config maps.
+	PrometheusRuleConfigMapMount struct {
+		Name      string `json:"name"`
+		SubPath   string `json:"subPath"`
+		ConfigMap string `json:"configMap"`
+	}
+
 	// ProxyInjector has all the proxy injector's Helm variables
 	ProxyInjector struct {
 		*TLS


Test patch:
diff --git a/cli/cmd/testdata/install_addon_control-plane.golden b/cli/cmd/testdata/install_addon_control-plane.golden
index 8e3792739..fd3f58a58 100644
--- a/cli/cmd/testdata/install_addon_control-plane.golden
+++ b/cli/cmd/testdata/install_addon_control-plane.golden
@@ -982,6 +982,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -1188,8 +1189,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/cli/cmd/testdata/install_control-plane.golden b/cli/cmd/testdata/install_control-plane.golden
index 6bae28cb2..44930c42c 100644
--- a/cli/cmd/testdata/install_control-plane.golden
+++ b/cli/cmd/testdata/install_control-plane.golden
@@ -982,6 +982,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -1188,8 +1189,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/cli/cmd/testdata/install_controlplane_tracing_output.golden b/cli/cmd/testdata/install_controlplane_tracing_output.golden
index db075eab6..98dc53854 100644
--- a/cli/cmd/testdata/install_controlplane_tracing_output.golden
+++ b/cli/cmd/testdata/install_controlplane_tracing_output.golden
@@ -1882,6 +1882,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -2088,8 +2089,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/cli/cmd/testdata/install_custom_registry.golden b/cli/cmd/testdata/install_custom_registry.golden
index c4ea1464d..8e1c1ea67 100644
--- a/cli/cmd/testdata/install_custom_registry.golden
+++ b/cli/cmd/testdata/install_custom_registry.golden
@@ -1822,6 +1822,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -2028,8 +2029,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/cli/cmd/testdata/install_default.golden b/cli/cmd/testdata/install_default.golden
index e84b34e0f..c5c19db03 100644
--- a/cli/cmd/testdata/install_default.golden
+++ b/cli/cmd/testdata/install_default.golden
@@ -1822,6 +1822,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -2028,8 +2029,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/cli/cmd/testdata/install_ha_output.golden b/cli/cmd/testdata/install_ha_output.golden
index f49bd513c..876d7f58b 100644
--- a/cli/cmd/testdata/install_ha_output.golden
+++ b/cli/cmd/testdata/install_ha_output.golden
@@ -1950,6 +1950,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -2163,8 +2164,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/cli/cmd/testdata/install_ha_with_overrides_output.golden b/cli/cmd/testdata/install_ha_with_overrides_output.golden
index f6cc45d27..01b85ad6c 100644
--- a/cli/cmd/testdata/install_ha_with_overrides_output.golden
+++ b/cli/cmd/testdata/install_ha_with_overrides_output.golden
@@ -1950,6 +1950,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -2163,8 +2164,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/cli/cmd/testdata/install_heartbeat_disabled_output.golden b/cli/cmd/testdata/install_heartbeat_disabled_output.golden
index 3f56776a4..87c82ac26 100644
--- a/cli/cmd/testdata/install_heartbeat_disabled_output.golden
+++ b/cli/cmd/testdata/install_heartbeat_disabled_output.golden
@@ -1733,6 +1733,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -1939,8 +1940,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/cli/cmd/testdata/install_helm_output.golden b/cli/cmd/testdata/install_helm_output.golden
index 50d5b36cf..f7fd527ba 100644
--- a/cli/cmd/testdata/install_helm_output.golden
+++ b/cli/cmd/testdata/install_helm_output.golden
@@ -1888,6 +1888,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -2094,8 +2095,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/cli/cmd/testdata/install_helm_output_addons.golden b/cli/cmd/testdata/install_helm_output_addons.golden
index 42ee47dfb..7703a7b19 100644
--- a/cli/cmd/testdata/install_helm_output_addons.golden
+++ b/cli/cmd/testdata/install_helm_output_addons.golden
@@ -1888,6 +1888,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -2094,8 +2095,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/cli/cmd/testdata/install_helm_output_ha.golden b/cli/cmd/testdata/install_helm_output_ha.golden
index bef79fc42..0179cc6ec 100644
--- a/cli/cmd/testdata/install_helm_output_ha.golden
+++ b/cli/cmd/testdata/install_helm_output_ha.golden
@@ -2016,6 +2016,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -2229,8 +2230,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/cli/cmd/testdata/install_no_init_container.golden b/cli/cmd/testdata/install_no_init_container.golden
index 1c70b2af7..87db5d455 100644
--- a/cli/cmd/testdata/install_no_init_container.golden
+++ b/cli/cmd/testdata/install_no_init_container.golden
@@ -1687,6 +1687,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -1893,8 +1894,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/cli/cmd/testdata/install_output.golden b/cli/cmd/testdata/install_output.golden
index 797c59fae..f9286fa2e 100644
--- a/cli/cmd/testdata/install_output.golden
+++ b/cli/cmd/testdata/install_output.golden
@@ -1818,6 +1818,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -2024,8 +2025,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/cli/cmd/testdata/install_proxy_ignores.golden b/cli/cmd/testdata/install_proxy_ignores.golden
index 23a2b5c0f..9442de79f 100644
--- a/cli/cmd/testdata/install_proxy_ignores.golden
+++ b/cli/cmd/testdata/install_proxy_ignores.golden
@@ -1822,6 +1822,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -2028,8 +2029,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/cli/cmd/testdata/install_restricted_dashboard.golden b/cli/cmd/testdata/install_restricted_dashboard.golden
index b387a4057..2c98a783d 100644
--- a/cli/cmd/testdata/install_restricted_dashboard.golden
+++ b/cli/cmd/testdata/install_restricted_dashboard.golden
@@ -1757,6 +1757,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -1963,8 +1964,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/cli/cmd/testdata/install_tracing.golden b/cli/cmd/testdata/install_tracing.golden
index 04373e024..579532484 100644
--- a/cli/cmd/testdata/install_tracing.golden
+++ b/cli/cmd/testdata/install_tracing.golden
@@ -1822,6 +1822,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -2028,8 +2029,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/cli/cmd/testdata/upgrade_add-on_controlplane.golden b/cli/cmd/testdata/upgrade_add-on_controlplane.golden
index 690543ad0..1a927e0ba 100644
--- a/cli/cmd/testdata/upgrade_add-on_controlplane.golden
+++ b/cli/cmd/testdata/upgrade_add-on_controlplane.golden
@@ -990,6 +990,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -1196,8 +1197,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/cli/cmd/testdata/upgrade_add-on_overwrite.golden b/cli/cmd/testdata/upgrade_add-on_overwrite.golden
index af4e14047..495bc419e 100644
--- a/cli/cmd/testdata/upgrade_add-on_overwrite.golden
+++ b/cli/cmd/testdata/upgrade_add-on_overwrite.golden
@@ -1830,6 +1830,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -2036,8 +2037,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/cli/cmd/testdata/upgrade_add_add-on.golden b/cli/cmd/testdata/upgrade_add_add-on.golden
index 86351d233..95bf1e3bd 100644
--- a/cli/cmd/testdata/upgrade_add_add-on.golden
+++ b/cli/cmd/testdata/upgrade_add_add-on.golden
@@ -1830,6 +1830,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -2036,8 +2037,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/cli/cmd/testdata/upgrade_default.golden b/cli/cmd/testdata/upgrade_default.golden
index 1ceba9206..9a349a398 100644
--- a/cli/cmd/testdata/upgrade_default.golden
+++ b/cli/cmd/testdata/upgrade_default.golden
@@ -1830,6 +1830,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -2036,8 +2037,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/cli/cmd/testdata/upgrade_external_issuer.golden b/cli/cmd/testdata/upgrade_external_issuer.golden
index b06b5a8e2..ff6e58684 100644
--- a/cli/cmd/testdata/upgrade_external_issuer.golden
+++ b/cli/cmd/testdata/upgrade_external_issuer.golden
@@ -1816,6 +1816,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -2022,8 +2023,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/cli/cmd/testdata/upgrade_ha.golden b/cli/cmd/testdata/upgrade_ha.golden
index f844525a5..23aa6e441 100644
--- a/cli/cmd/testdata/upgrade_ha.golden
+++ b/cli/cmd/testdata/upgrade_ha.golden
@@ -1958,6 +1958,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -2171,8 +2172,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/cli/cmd/testdata/upgrade_overwrite_issuer.golden b/cli/cmd/testdata/upgrade_overwrite_issuer.golden
index 3ec11fce8..d99c2cd70 100644
--- a/cli/cmd/testdata/upgrade_overwrite_issuer.golden
+++ b/cli/cmd/testdata/upgrade_overwrite_issuer.golden
@@ -1822,6 +1822,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -2028,8 +2029,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/cli/cmd/testdata/upgrade_overwrite_trust_anchors-external-issuer.golden b/cli/cmd/testdata/upgrade_overwrite_trust_anchors-external-issuer.golden
index 138cb8485..f2eda01b1 100644
--- a/cli/cmd/testdata/upgrade_overwrite_trust_anchors-external-issuer.golden
+++ b/cli/cmd/testdata/upgrade_overwrite_trust_anchors-external-issuer.golden
@@ -1808,6 +1808,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -2014,8 +2015,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/cli/cmd/testdata/upgrade_overwrite_trust_anchors.golden b/cli/cmd/testdata/upgrade_overwrite_trust_anchors.golden
index 3ec11fce8..d99c2cd70 100644
--- a/cli/cmd/testdata/upgrade_overwrite_trust_anchors.golden
+++ b/cli/cmd/testdata/upgrade_overwrite_trust_anchors.golden
@@ -1822,6 +1822,7 @@ data:
 
     rule_files:
     - /etc/prometheus/*_rules.yml
+    - /etc/prometheus/*_rules.yaml
 
     scrape_configs:
     - job_name: 'prometheus'
@@ -2028,8 +2029,9 @@ spec:
         volumeMounts:
         - mountPath: /data
           name: data
-        - mountPath: /etc/prometheus
+        - mountPath: /etc/prometheus/prometheus.yml
           name: prometheus-config
+          subPath: prometheus.yml
           readOnly: true
       - env:
         - name: LINKERD2_PROXY_LOG
diff --git a/pkg/charts/linkerd2/values_test.go b/pkg/charts/linkerd2/values_test.go
index 2805270a1..0c65178e8 100644
--- a/pkg/charts/linkerd2/values_test.go
+++ b/pkg/charts/linkerd2/values_test.go
@@ -14,24 +14,27 @@ func TestNewValues(t *testing.T) {
 	testVersion := "linkerd-dev"
 
 	expected := &Values{
-		Stage:                       "",
-		ControllerImage:             "gcr.io/linkerd-io/controller",
-		ControllerImageVersion:      testVersion,
-		WebImage:                    "gcr.io/linkerd-io/web",
-		PrometheusImage:             "prom/prometheus:v2.15.2",
-		GrafanaImage:                "gcr.io/linkerd-io/grafana",
-		ControllerReplicas:          1,
-		ControllerLogLevel:          "info",
-		PrometheusLogLevel:          "info",
-		ControllerUID:               2103,
-		EnableH2Upgrade:             true,
-		EnablePodAntiAffinity:       false,
-		WebhookFailurePolicy:        "Ignore",
-		OmitWebhookSideEffects:      false,
-		RestrictDashboardPrivileges: false,
-		DisableHeartBeat:            false,
-		HeartbeatSchedule:           "0 0 * * *",
-		InstallNamespace:            true,
+		Stage:                         "",
+		ControllerImage:               "gcr.io/linkerd-io/controller",
+		ControllerImageVersion:        testVersion,
+		WebImage:                      "gcr.io/linkerd-io/web",
+		PrometheusImage:               "prom/prometheus:v2.15.2",
+		GrafanaImage:                  "gcr.io/linkerd-io/grafana",
+		ControllerReplicas:            1,
+		ControllerLogLevel:            "info",
+		PrometheusLogLevel:            "info",
+		PrometheusExtraArgs:           map[string]string{},
+		PrometheusAlertmanagers:       []interface{}{},
+		PrometheusRuleConfigMapMounts: []PrometheusRuleConfigMapMount{},
+		ControllerUID:                 2103,
+		EnableH2Upgrade:               true,
+		EnablePodAntiAffinity:         false,
+		WebhookFailurePolicy:          "Ignore",
+		OmitWebhookSideEffects:        false,
+		RestrictDashboardPrivileges:   false,
+		DisableHeartBeat:              false,
+		HeartbeatSchedule:             "0 0 * * *",
+		InstallNamespace:              true,
 		Global: &Global{
 			Namespace:                "linkerd",
 			ClusterDomain:            "cluster.local",
@@ -148,6 +151,12 @@ func TestNewValues(t *testing.T) {
 
 	t.Run("HA", func(t *testing.T) {
 		actual, err := NewValues(true)
+
+		// workaround for mergo, which resets these to []interface{}(nil)
+		// and []PrometheusRuleConfigMapMount(nil)
+		actual.PrometheusAlertmanagers = []interface{}{}
+		actual.PrometheusRuleConfigMapMounts = []PrometheusRuleConfigMapMount{}
+
 		if err != nil {
 			t.Fatalf("Unexpected error: %v\n", err)
 		}