Repository: linkerd/linkerd2
Pull request #4233 (2020-04-03 15:10:55)

Add liveliness checks and metrics for multicluster gateway
This PR adds liveliness checks for remote gateways. The way its is done is by maintaining probes for each gateway. The probes track whether the gateway is alive by issuing an HTTP request to an endpoint. Additionally the probes publish Prometheus metrics such as number of mirrored services per gateway and histogram latencies. 

Next steps (coming in different PRs): 

- Externalize the port, path and period through  the `get-credentials` command
- Add unit tests
- Enable Prometheus to scrape the `/metrics` endpoint
- Enable users to retreive the stats of the gateways via a CLI command (similar to `stat`)

Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>


zaharidichev: Ahh, som unit tests are locking up. Will address that shortly
zaharidichev: > These metrics seem cool! Can you explain what you mean by liveness checks? I see the code that publishes prometheus metrics based on the probes to each gateway, but I don't see anywhere that acts on that data.

Yes, so the liveliness checks do exactly that, just publish metrics. We do not actively act on these, the same way kubelet acts on the results of its liveliness checks. The idea behind that is to provide more info around the status of the gateways that we are talking to on remote clusters. This is supposed to work with dashboard + a cli command, similar to the stat one. Something like: 

<img width="964" alt="Screenshot 2020-04-07 at 16 17 39" src="https://user-images.githubusercontent.com/4391506/78760034-3848d880-7989-11ea-8414-c12e81c7ba1f.png">

zaharidichev: @adleong I  incorporated your suggestions for additional metrics. Feel free to take another look :) 
zaharidichev: @adleong not that $4265 introduces probe settings per gateway, I will have to think about how to handle this one here a bit better. The problem stems from the fact that we need to keep the data of what these settings are somewhere on the local cluster. The reason for that arises because... the prober tries to recrete its state on startup based on the services that are already mirrored and we want to have the correct probe settings, but we dont get any data about it once we start up. Probably will keep that as data in the local endpoints and take care to update them when something about the gateway changes (which we already do for other data such as gateway identitiy, etc.... )

Patch:
diff --git a/CHANGES.md b/CHANGES.md
index bf186db41..eccbbcc1f 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -1,3 +1,83 @@
+## edge-20.4.4
+
+This edge release fixes a packaging issue in `edge-20.4.3`.
+
+_From `edge.20.4.3` release notes_:
+
+This edge release adds functionality to the CLI to output more detail and
+includes changes which support the multi-cluster functionality. Also, the helm
+support has been expanded to make installation more configurable. Finally, the
+HA reliability is improved by ensuring that control plane pods are restarted
+with a rolling strategy
+
+* CLI
+  * Added output to the `linkerd check --proxy` command to list all data
+    plane pods which are not up-to-date rather than just printing the first
+    one it encounters
+  * Added a `--proxy` flag to the `linkerd version` command which lists all
+    proxy versions running in the cluster and the number of pods running
+    each version
+  * Lifted requirement of using --unmeshed for linkerd stat when querying
+    TrafficSplit resources
+  * Added support for multi-stage installs with Add-Ons
+* Controller
+  * Added a rolling update strategy to Linkerd deployments that have
+    multiple replicas during HA deployments to ensure that at most
+    one pod begins terminating before a new pod ready is ready
+  * Added a new label for the proxy injector to write to the template,
+    `linkerd.io/workload-ns` which indicates the namespace of the workload/pod
+* Internal
+  * Added a [security policy](https://help.github.com/en/github/managing-security-vulnerabilities/adding-a-security-policy-to-your-repository)
+    to facilitate conversations around security
+* Helm
+  * Changed charts to use downwardAPI to mount labels to the proxy container
+    making them easier to identify
+* Proxy
+  * Changed the Linkerd proxy endpoint for liveness to use the new `/live` admin
+    endpoint instead of the `/metrics` endpoint, because the `/live` endpoint
+    returns a smaller payload
+  * Added a per-endpoint authority-override feature to support
+    multi-cluster gateways
+
+## edge-20.4.3
+
+**This release is superseded by `edge-20.4.4`**
+
+This edge release adds functionality to the CLI to output more detail and
+includes changes which support the multi-cluster functionality. Also, the helm
+support has been expanded to make installation more configurable. Finally, the
+HA reliability is improved by ensuring that control plane pods are restarted
+with a rolling strategy
+
+* CLI
+  * Added output to the `linkerd check --proxy` command to list all data
+    plane pods which are not up-to-date rather than just printing the first
+    one it encounters
+  * Added a `--proxy` flag to the `linkerd version` command which lists all
+    proxy versions running in the cluster and the number of pods running
+    each version
+  * Lifted requirement of using --unmeshed for linkerd stat when querying
+    TrafficSplit resources
+  * Added support for multi-stage installs with Add-Ons
+* Controller
+  * Added a rolling update strategy to Linkerd deployments that have
+    multiple replicas during HA deployments to ensure that at most
+    one pod begins terminating before a new pod ready is ready
+  * Added a new label for the proxy injector to write to the template,
+    `linkerd.io/workload-ns` which indicates the namespace of the workload/pod
+* Internal
+  * Added a [security policy](https://help.github.com/en/github/managing-security-vulnerabilities/adding-a-security-policy-to-your-repository)
+    to facilitate conversations around security
+* Helm
+  * Changed charts to use downwardAPI to mount labels to the proxy container
+    making them easier to identify
+* Proxy
+  * Changed the Linkerd proxy endpoint for liveness to use the new `/live` admin
+    endpoint instead of the `/metrics` endpoint, because the `/live` endpoint
+    returns a smaller payload
+  * Added a per-endpoint authority-override feature to support
+    multi-cluster gateways
+
 ## edge-20.4.2
 
 This release brings a number of CLI fixes and Controller improvements.
@@ -19,7 +99,7 @@ This release brings a number of CLI fixes and Controller improvements.
   * Improved endpoints change detection in the `linkerd-destination` service, enabling
     mirrored remote services to change cluster gateways
   * Added `operationID` field to tap OpenAPI response to prevent issues during
-    upgrade from 2.6 to 2.7 
+    upgrade from 2.6 to 2.7
 * Proxy
   * Added a new protocol detection timeout to prevent clients from consuming
     resources indefinitely when not sending any data
@@ -87,7 +167,7 @@ new observability and security functionality.
 
 * CLI
   * Added the `linkerd alpha stat` command, which uses the smi-metrics
-    API; the latter enables access to metrics to be controlled with RBAC 
+    API; the latter enables access to metrics to be controlled with RBAC
 * Controller
   * Added support for configuring service profile timeouts
     `(x-linkerd-timeout)` via OpenAPI spec (thanks @lewiscowper!)
@@ -122,7 +202,7 @@ experimenting with this feature, please join us in
 * Proxy
   * Fixed a bug that could cause the proxy's load balancer to stop processing
     updates from service discovery.
-    
+
 ## edge-20.2.3
 
 This release introduces the first optional add-on `tracing`, added through the
diff --git a/charts/linkerd2-service-mirror/README.md b/charts/linkerd2-service-mirror/README.md
index 4450fdf8c..4fcdd751a 100644
--- a/charts/linkerd2-service-mirror/README.md
+++ b/charts/linkerd2-service-mirror/README.md
@@ -13,6 +13,7 @@ The following table lists the configurable parameters of the linkerd2-service-mi
 
 | Parameter                            | Description                                                                       | Default                       |
 |--------------------------------------|-----------------------------------------------------------------------------------|-------------------------------|
+|`controllerComponentLabel`            | Control plane label. Do not edit                                                  | `linkerd.io/control-plane-component`|
 |`controllerImage`                     | Docker image for the Service mirror component (uses the Linkerd controller image) |`gcr.io/linkerd-io/controller`|
 |`controllerImageVersion`              | Tag for the Service Mirror container Docker image                                 |latest version|
 |`namespace`                           | Service Mirror component namespace                                                |`linkerd-service-mirror`|
diff --git a/charts/linkerd2-service-mirror/templates/service-mirror.yaml b/charts/linkerd2-service-mirror/templates/service-mirror.yaml
index 0f66a81a1..abf10511b 100644
--- a/charts/linkerd2-service-mirror/templates/service-mirror.yaml
+++ b/charts/linkerd2-service-mirror/templates/service-mirror.yaml
@@ -4,7 +4,7 @@ metadata:
   name: linkerd-service-mirror
   namespace: {{.Values.namespace}}
   labels:
-    ControllerComponentLabel: service-mirror
+    {{.Values.controllerComponentLabel}}: linkerd-service-mirror
 rules:
 - apiGroups: [""]
   resources: ["endpoints", "services", "secrets", "namespaces"]
@@ -16,7 +16,7 @@ metadata:
   name: linkerd-service-mirror
   namespace: {{.Values.namespace}}
   labels:
-    ControllerComponentLabel: service-mirror
+    {{.Values.controllerComponentLabel}}: linkerd-service-mirror
 roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
@@ -32,24 +32,24 @@ metadata:
   name: linkerd-service-mirror
   namespace: {{.Values.namespace}}
   labels:
-    ControllerComponentLabel: service-mirror
+    {{.Values.controllerComponentLabel}}: linkerd-service-mirror
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   labels:
-    ControllerComponentLabel: service-mirror
+    {{.Values.controllerComponentLabel}}: linkerd-service-mirror
   name: linkerd-service-mirror
   namespace: {{.Values.namespace}}
 spec:
   replicas: 1
   selector:
     matchLabels:
-      ControllerComponentLabel: service-mirror
+      {{.Values.controllerComponentLabel}}: linkerd-service-mirror
   template:
     metadata:
       labels:
-        ControllerComponentLabel: service-mirror
+        {{.Values.controllerComponentLabel}}: linkerd-service-mirror
     spec:
       containers:
       - args:
@@ -60,4 +60,7 @@ spec:
         name: service-mirror
         securityContext:
           runAsUser: {{.Values.serviceMirrorUID}}
+        ports:
+        - containerPort: 9999
+          name: admin-http
       serviceAccountName: linkerd-service-mirror
diff --git a/charts/linkerd2-service-mirror/values.yaml b/charts/linkerd2-service-mirror/values.yaml
index 6aff9f3fa..652cc785d 100644
--- a/charts/linkerd2-service-mirror/values.yaml
+++ b/charts/linkerd2-service-mirror/values.yaml
@@ -4,3 +4,4 @@ logLevel: info
 eventRequeueLimit: 3
 controllerImage: gcr.io/linkerd-io/controller
 controllerImageVersion: {version}
+controllerComponentLabel: linkerd.io/control-plane-component
diff --git a/charts/linkerd2/templates/prometheus.yaml b/charts/linkerd2/templates/prometheus.yaml
index 060f412fe..59176fe67 100644
--- a/charts/linkerd2/templates/prometheus.yaml
+++ b/charts/linkerd2/templates/prometheus.yaml
@@ -81,6 +81,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/controller/cmd/service-mirror/cluster_watcher.go b/controller/cmd/service-mirror/cluster_watcher.go
index ffedf91c1..f4fc0be70 100644
--- a/controller/cmd/service-mirror/cluster_watcher.go
+++ b/controller/cmd/service-mirror/cluster_watcher.go
@@ -3,6 +3,7 @@ package servicemirror
 import (
 	"errors"
 	"fmt"
+	"strconv"
 	"strings"
 
 	"github.com/linkerd/linkerd2/controller/k8s"
@@ -34,6 +35,26 @@ type (
 		log             *logging.Entry
 		eventsQueue     workqueue.RateLimitingInterface
 		requeueLimit    int
+		probeEventsSink ProbeEventSink
+	}
+
+	// ProbeConfig describes the configured probe on particular gateway (if presents)
+	ProbeConfig struct {
+		path            string
+		port            uint32
+		periodInSeconds uint32
+	}
+
+	// GatewaySpec contains essential data about the gateway
+	GatewaySpec struct {
+		gatewayName      string
+		gatewayNamespace string
+		clusterName      string
+		addresses        []corev1.EndpointAddress
+		incomingPort     uint32
+		resourceVersion  string
+		identity         string
+		*ProbeConfig
 	}
 
 	// RemoteServiceCreated is generated whenever a remote service is created Observing
@@ -51,32 +72,29 @@ type (
 		localService   *corev1.Service
 		localEndpoints *corev1.Endpoints
 		remoteUpdate   *corev1.Service
-		gatewayData    *gatewayMetadata
+		gatewayData    gatewayMetadata
 	}
 
 	// RemoteServiceDeleted when a remote service is going away or it is not
 	// considered mirrored anymore
 	RemoteServiceDeleted struct {
-		Name      string
-		Namespace string
+		Name        string
+		Namespace   string
+		GatewayData gatewayMetadata
 	}
 
 	// RemoteGatewayDeleted is observed when a service that is a gateway to at least
 	// one already mirrored service is deleted
 	RemoteGatewayDeleted struct {
-		gatewayData *gatewayMetadata
+		gatewayData gatewayMetadata
 	}
 
 	// RemoteGatewayUpdated happens when a service that is a gateway to at least
 	// one already mirrored service is updated. This might mean an IP change,
 	// incoming port change, etc...
 	RemoteGatewayUpdated struct {
-		newPort              int32
-		newEndpointAddresses []corev1.EndpointAddress
-		gatewayData          *gatewayMetadata
-		newResourceVersion   string
-		affectedServices     []*corev1.Service
-		identity             string
+		gatewaySpec      GatewaySpec
+		affectedServices []*corev1.Service
 	}
 
 	// ConsiderGatewayUpdateDispatch is issued when we are receiving an update for a
@@ -143,51 +161,26 @@ func (re RetryableError) Error() string {
 	return fmt.Sprintf("Inner errors:\n\t%s", strings.Join(errorStrings, "\n\t"))
 }
 
-func (rcsw *RemoteClusterServiceWatcher) extractGatewayInfo(gateway *corev1.Service) ([]corev1.EndpointAddress, int32, string, string, error) {
-	if len(gateway.Status.LoadBalancer.Ingress) == 0 {
-		return nil, 0, "", "", errors.New("expected gateway to have at lest 1 external Ip address but it has none")
-	}
-
-	var foundPort = false
-	var port int32
-	for _, p := range gateway.Spec.Ports {
-		if p.Name == consts.GatewayPortName {
-			foundPort = true
-			port = p.Port
-			break
-		}
-	}
-
-	if !foundPort {
-		return nil, 0, "", "", fmt.Errorf("cannot find  port named %s on gateway", consts.GatewayPortName)
-	}
-
-	var gatewayEndpoints []corev1.EndpointAddress
-	for _, ingress := range gateway.Status.LoadBalancer.Ingress {
-		gatewayEndpoints = append(gatewayEndpoints, corev1.EndpointAddress{
-			IP:       ingress.IP,
-			Hostname: ingress.Hostname,
-		})
-	}
-
-	gatewayIdentity := gateway.Annotations[consts.GatewayIdentity]
-
-	return gatewayEndpoints, port, gateway.ResourceVersion, gatewayIdentity, nil
-}
-
 // When the gateway is resolved we need to produce a set of endpoint addresses that that
 // contain the external IPs that this gateway exposes. Therefore we return the IP addresses
 // as well as a single port on which the gateway is accessible.
-func (rcsw *RemoteClusterServiceWatcher) resolveGateway(metadata *gatewayMetadata) ([]corev1.EndpointAddress, int32, string, string, error) {
+func (rcsw *RemoteClusterServiceWatcher) resolveGateway(metadata *gatewayMetadata) (*GatewaySpec, error) {
 	gateway, err := rcsw.remoteAPIClient.Svc().Lister().Services(metadata.Namespace).Get(metadata.Name)
 	if err != nil {
-		return nil, 0, "", "", err
+		return nil, err
 	}
-	return rcsw.extractGatewayInfo(gateway)
+	return rcsw.extractGatewaySpec(gateway)
 }
 
 // NewRemoteClusterServiceWatcher constructs a new cluster watcher
-func NewRemoteClusterServiceWatcher(localAPI *k8s.API, cfg *rest.Config, clusterName string, requeueLimit int, clusterDomain string) (*RemoteClusterServiceWatcher, error) {
+func NewRemoteClusterServiceWatcher(
+	localAPI *k8s.API,
+	cfg *rest.Config,
+	clusterName string,
+	requeueLimit int,
+	clusterDomain string,
+	probeEventsSink ProbeEventSink,
+) (*RemoteClusterServiceWatcher, error) {
 	remoteAPI, err := k8s.InitializeAPIForConfig(cfg, false, k8s.Svc)
 	if err != nil {
 		return nil, fmt.Errorf("cannot initialize remote api for cluster %s: %s", clusterName, err)
@@ -203,8 +196,9 @@ func NewRemoteClusterServiceWatcher(localAPI *k8s.API, cfg *rest.Config, cluster
 			"cluster":    clusterName,
 			"apiAddress": cfg.Host,
 		}),
-		eventsQueue:  workqueue.NewRateLimitingQueue(workqueue.DefaultControllerRateLimiter()),
-		requeueLimit: requeueLimit,
+		eventsQueue:     workqueue.NewRateLimitingQueue(workqueue.DefaultControllerRateLimiter()),
+		requeueLimit:    requeueLimit,
+		probeEventsSink: probeEventsSink,
 	}, nil
 }
 
@@ -309,11 +303,25 @@ func (rcsw *RemoteClusterServiceWatcher) cleanupOrphanedServices() error {
 				// something went wrong getting the service, we can retry
 				errors = append(errors, err)
 			}
+		} else {
+			if gtwData := getGatewayMetadata(srv.Annotations); gtwData != nil {
+				gatewaySpec, err := rcsw.resolveGateway(gtwData)
+				if gatewaySpec != nil && err == nil {
+					rcsw.probeEventsSink.send(&MirroredServicePaired{
+						serviceName:      srv.Name,
+						serviceNamespace: srv.Namespace,
+						GatewaySpec:      *gatewaySpec,
+					})
+				} else {
+					rcsw.log.Debugf("Could not resolve gateway for %s/%s: %s", srv.Namespace, srv.Name, err)
+				}
+			}
 		}
 	}
 	if len(errors) > 0 {
 		return RetryableError{errors}
 	}
+
 	return nil
 }
 
@@ -396,6 +404,13 @@ func (rcsw *RemoteClusterServiceWatcher) handleRemoteServiceDeleted(ev *RemoteSe
 	}
 
 	rcsw.log.Debugf("Successfully deleted Service: %s/%s", ev.Namespace, localServiceName)
+	rcsw.probeEventsSink.send(&MirroredServiceUnpaired{
+		serviceName:      localServiceName,
+		serviceNamespace: ev.Namespace,
+		gatewayName:      ev.GatewayData.Name,
+		gatewayNs:        ev.GatewayData.Namespace,
+		clusterName:      rcsw.clusterName,
+	})
 	return nil
 }
 
@@ -405,24 +420,44 @@ func (rcsw *RemoteClusterServiceWatcher) handleRemoteServiceUpdated(ev *RemoteSe
 	serviceInfo := fmt.Sprintf("%s/%s", ev.remoteUpdate.Namespace, ev.remoteUpdate.Name)
 	rcsw.log.Debugf("Updating remote mirrored service %s/%s", ev.localService.Namespace, ev.localService.Name)
 
-	gatewayEndpoints, gatewayPort, resVersion, gatewayIdentity, err := rcsw.resolveGateway(ev.gatewayData)
+	gatewayChanged := false
+	if ev.localEndpoints.Labels[consts.RemoteGatewayNameLabel] != ev.gatewayData.Name || ev.localEndpoints.Labels[consts.RemoteGatewayNsLabel] != ev.gatewayData.Namespace {
+		gatewayChanged = true
+		rcsw.probeEventsSink.send(&MirroredServiceUnpaired{
+			serviceName:      ev.localService.Name,
+			serviceNamespace: ev.localService.Namespace,
+			gatewayName:      ev.localEndpoints.Labels[consts.RemoteGatewayNameLabel],
+			gatewayNs:        ev.localEndpoints.Labels[consts.RemoteGatewayNsLabel],
+			clusterName:      rcsw.clusterName,
+		})
+	}
+
+	gatewaySpec, err := rcsw.resolveGateway(&ev.gatewayData)
 	if err == nil {
 		ev.localEndpoints.Subsets = []corev1.EndpointSubset{
 			{
-				Addresses: gatewayEndpoints,
-				Ports:     rcsw.getEndpointsPorts(ev.remoteUpdate, gatewayPort),
+				Addresses: gatewaySpec.addresses,
+				Ports:     rcsw.getEndpointsPorts(ev.remoteUpdate, int32(gatewaySpec.incomingPort)),
 			},
 		}
 
 		ev.localEndpoints.Labels[consts.RemoteGatewayNameLabel] = ev.gatewayData.Name
 		ev.localEndpoints.Labels[consts.RemoteGatewayNsLabel] = ev.gatewayData.Namespace
 
-		if gatewayIdentity != "" {
-			ev.localEndpoints.Annotations[consts.RemoteGatewayIdentity] = gatewayIdentity
+		if gatewaySpec.identity != "" {
+			ev.localEndpoints.Annotations[consts.RemoteGatewayIdentity] = gatewaySpec.identity
 		} else {
 			delete(ev.localEndpoints.Annotations, consts.RemoteGatewayIdentity)
 		}
 
+		if gatewayChanged {
+			rcsw.probeEventsSink.send(&MirroredServicePaired{
+				serviceName:      ev.localService.Name,
+				serviceNamespace: ev.localService.Namespace,
+				GatewaySpec:      *gatewaySpec,
+			})
+		}
+
 	} else {
 		rcsw.log.Warnf("Could not resolve gateway for %s: %s, nulling endpoints", serviceInfo, err)
 		ev.localEndpoints.Subsets = nil
@@ -432,9 +467,9 @@ func (rcsw *RemoteClusterServiceWatcher) handleRemoteServiceUpdated(ev *RemoteSe
 		return RetryableError{[]error{err}}
 	}
 
-	ev.localService.Labels = rcsw.getMirroredServiceLabels(ev.gatewayData)
+	ev.localService.Labels = rcsw.getMirroredServiceLabels(&ev.gatewayData)
 	ev.localService.Annotations = rcsw.getMirroredServiceAnnotations(ev.remoteUpdate)
-	ev.localService.Annotations[consts.RemoteGatewayResourceVersionAnnotation] = resVersion
+	ev.localService.Annotations[consts.RemoteGatewayResourceVersionAnnotation] = gatewaySpec.resourceVersion
 	ev.localService.Spec.Ports = remapRemoteServicePorts(ev.remoteUpdate.Spec.Ports)
 
 	if _, err := rcsw.localAPIClient.Client.CoreV1().Services(ev.localService.Namespace).Update(ev.localService); err != nil {
@@ -496,20 +531,27 @@ func (rcsw *RemoteClusterServiceWatcher) handleRemoteServiceCreated(ev *RemoteSe
 	}
 
 	// Now we try to resolve the remote gateway
-	gatewayEndpoints, gatewayPort, resVersion, gatewayIdentity, err := rcsw.resolveGateway(ev.gatewayData)
+	gatewaySpec, err := rcsw.resolveGateway(ev.gatewayData)
 	if err == nil {
 		// only if we resolve it, we are updating the endpoints addresses and ports
-		rcsw.log.Debugf("Resolved remote gateway [%v:%d] for %s", gatewayEndpoints, gatewayPort, serviceInfo)
+		rcsw.log.Debugf("Resolved remote gateway [%v:%d] for %s", gatewaySpec.addresses, gatewaySpec.incomingPort, serviceInfo)
 		endpointsToCreate.Subsets = []corev1.EndpointSubset{
 			{
-				Addresses: gatewayEndpoints,
-				Ports:     rcsw.getEndpointsPorts(ev.service, gatewayPort),
+				Addresses: gatewaySpec.addresses,
+				Ports:     rcsw.getEndpointsPorts(ev.service, int32(gatewaySpec.incomingPort)),
 			},
 		}
-		serviceToCreate.Annotations[consts.RemoteGatewayResourceVersionAnnotation] = resVersion
-		if gatewayIdentity != "" {
-			endpointsToCreate.Annotations[consts.RemoteGatewayIdentity] = gatewayIdentity
+		serviceToCreate.Annotations[consts.RemoteGatewayResourceVersionAnnotation] = gatewaySpec.resourceVersion
+		if gatewaySpec.identity != "" {
+			endpointsToCreate.Annotations[consts.RemoteGatewayIdentity] = gatewaySpec.identity
 		}
+
+		rcsw.probeEventsSink.send(&MirroredServicePaired{
+			serviceName:      serviceToCreate.Name,
+			serviceNamespace: serviceToCreate.Namespace,
+			GatewaySpec:      *gatewaySpec,
+		})
+
 	} else {
 		rcsw.log.Warnf("Could not resolve gateway for %s: %s, skipping subsets", serviceInfo, err)
 		endpointsToCreate.Subsets = nil
@@ -534,7 +576,7 @@ func (rcsw *RemoteClusterServiceWatcher) handleRemoteServiceCreated(ev *RemoteSe
 }
 
 func (rcsw *RemoteClusterServiceWatcher) handleRemoteGatewayDeleted(ev *RemoteGatewayDeleted) error {
-	affectedEndpoints, err := rcsw.endpointsForGateway(ev.gatewayData)
+	affectedEndpoints, err := rcsw.endpointsForGateway(&ev.gatewayData)
 	if err != nil {
 		// if we cannot find the endpoints, we can give up
 		if kerrors.IsNotFound(err) {
@@ -563,13 +605,17 @@ func (rcsw *RemoteClusterServiceWatcher) handleRemoteGatewayDeleted(ev *RemoteGa
 }
 
 func (rcsw *RemoteClusterServiceWatcher) handleRemoteGatewayUpdated(ev *RemoteGatewayUpdated) error {
-	rcsw.log.Debugf("Updating %d services due to remote gateway [%s/%s] update", len(ev.affectedServices), ev.gatewayData.Namespace, ev.gatewayData.Name)
+	rcsw.log.Debugf("Updating %d services due to remote gateway [%s/%s] update", len(ev.affectedServices), ev.gatewaySpec.gatewayNamespace, ev.gatewaySpec.gatewayName)
+
+	rcsw.probeEventsSink.send(&GatewayUpdated{
+		GatewaySpec: ev.gatewaySpec,
+	})
 
 	var errors []error
 	for _, svc := range ev.affectedServices {
 		updatedService := svc.DeepCopy()
 		if updatedService.Labels != nil {
-			updatedService.Annotations[consts.RemoteGatewayResourceVersionAnnotation] = ev.newResourceVersion
+			updatedService.Annotations[consts.RemoteGatewayResourceVersionAnnotation] = ev.gatewaySpec.resourceVersion
 		}
 		endpoints, err := rcsw.localAPIClient.Endpoint().Lister().Endpoints(svc.Namespace).Get(svc.Name)
 		if err != nil {
@@ -580,13 +626,13 @@ func (rcsw *RemoteClusterServiceWatcher) handleRemoteGatewayUpdated(ev *RemoteGa
 		updatedEndpoints := endpoints.DeepCopy()
 		updatedEndpoints.Subsets = []corev1.EndpointSubset{
 			{
-				Addresses: ev.newEndpointAddresses,
-				Ports:     rcsw.getEndpointsPorts(updatedService, ev.newPort),
+				Addresses: ev.gatewaySpec.addresses,
+				Ports:     rcsw.getEndpointsPorts(updatedService, int32(ev.gatewaySpec.incomingPort)),
 			},
 		}
 
-		if ev.identity != "" {
-			updatedEndpoints.Annotations[consts.RemoteGatewayIdentity] = ev.identity
+		if ev.gatewaySpec.identity != "" {
+			updatedEndpoints.Annotations[consts.RemoteGatewayIdentity] = ev.gatewaySpec.identity
 		} else {
 			delete(updatedEndpoints.Annotations, consts.RemoteGatewayIdentity)
 		}
@@ -641,11 +687,11 @@ func (rcsw *RemoteClusterServiceWatcher) handleConsiderGatewayUpdateDispatch(eve
 	}
 
 	if len(services) > 0 {
-		gatewayMeta := &gatewayMetadata{
+		gatewayMeta := gatewayMetadata{
 			Name:      event.maybeGateway.Name,
 			Namespace: event.maybeGateway.Namespace,
 		}
-		if endpoints, port, resVersion, identity, err := rcsw.extractGatewayInfo(event.maybeGateway); err != nil {
+		if gatewaySpec, err := rcsw.extractGatewaySpec(event.maybeGateway); err != nil {
 			rcsw.log.Warnf("Gateway [%s/%s] is not a compliant gateway anymore, dispatching GatewayDeleted event: %s", event.maybeGateway.Namespace, event.maybeGateway.Name, err)
 			// in case something changed about this gateway and it is not really a gateway anymore,
 			// simply dispatch deletion event so all endpoints are nulled
@@ -662,12 +708,8 @@ func (rcsw *RemoteClusterServiceWatcher) handleConsiderGatewayUpdateDispatch(eve
 
 			if len(affectedServices) > 0 {
 				rcsw.eventsQueue.Add(&RemoteGatewayUpdated{
-					newPort:              port,
-					newEndpointAddresses: endpoints,
-					gatewayData:          gatewayMeta,
-					newResourceVersion:   resVersion,
-					affectedServices:     affectedServices,
-					identity:             identity,
+					gatewaySpec:      *gatewaySpec,
+					affectedServices: affectedServices,
 				})
 			}
 
@@ -718,7 +760,7 @@ func (rcsw *RemoteClusterServiceWatcher) createOrUpdateService(service *corev1.S
 						localService:   localService,
 						localEndpoints: endpoints,
 						remoteUpdate:   service,
-						gatewayData:    gtwData,
+						gatewayData:    *gtwData,
 					})
 				} else {
 					return RetryableError{[]error{err}}
@@ -785,12 +827,13 @@ func (rcsw *RemoteClusterServiceWatcher) endpointsForGateway(gatewayData *gatewa
 func (rcsw *RemoteClusterServiceWatcher) handleOnDelete(service *corev1.Service) {
 	if gtwData := getGatewayMetadata(service.Annotations); gtwData != nil {
 		rcsw.eventsQueue.Add(&RemoteServiceDeleted{
-			Name:      service.Name,
-			Namespace: service.Namespace,
+			Name:        service.Name,
+			Namespace:   service.Namespace,
+			GatewayData: *gtwData,
 		})
 	} else {
 		rcsw.eventsQueue.Add(&RemoteGatewayDeleted{
-			gatewayData: &gatewayMetadata{
+			gatewayData: gatewayMetadata{
 				Name:      service.Name,
 				Namespace: service.Namespace,
 			}})
@@ -904,9 +947,81 @@ func (rcsw *RemoteClusterServiceWatcher) Start() {
 
 // Stop stops watching the cluster and cleans up all mirrored resources
 func (rcsw *RemoteClusterServiceWatcher) Stop(cleanupState bool) {
+	rcsw.probeEventsSink.send(&ClusterNotRegistered{
+		clusterName: rcsw.clusterName,
+	})
 	close(rcsw.stopper)
 	if cleanupState {
 		rcsw.eventsQueue.Add(&ClusterUnregistered{})
 	}
 	rcsw.eventsQueue.ShutDown()
 }
+
+func parseProbeConfig(data map[string]string) (*ProbeConfig, error) {
+	probePath := data[consts.GatewayProbePath]
+	probePort, err := strconv.ParseUint(data[consts.GatewayProbePort], 10, 32)
+	if err != nil {
+		return nil, err
+	}
+
+	probePeriod, err := strconv.ParseUint(data[consts.GatewayProbePeriod], 10, 32)
+	if err != nil {
+		return nil, err
+	}
+
+	if probePath == "" {
+		return nil, errors.New("probe path is empty")
+	}
+
+	return &ProbeConfig{
+		path:            probePath,
+		port:            uint32(probePort),
+		periodInSeconds: uint32(probePeriod),
+	}, nil
+}
+
+func (rcsw *RemoteClusterServiceWatcher) extractGatewaySpec(gateway *corev1.Service) (*GatewaySpec, error) {
+	if len(gateway.Status.LoadBalancer.Ingress) == 0 {
+		return nil, fmt.Errorf("expected gateway %s/%s to have at lest 1 external Ip address but it has none", gateway.Namespace, gateway.Name)
+	}
+
+	var foundPort = false
+	var port uint32
+	for _, p := range gateway.Spec.Ports {
+		if p.Name == consts.GatewayPortName {
+			foundPort = true
+			port = uint32(p.Port)
+			break
+		}
+	}
+
+	if !foundPort {
+		return nil, fmt.Errorf("cannot find  port named %s on gateway", consts.GatewayPortName)
+	}
+
+	var gatewayEndpoints []corev1.EndpointAddress
+	for _, ingress := range gateway.Status.LoadBalancer.Ingress {
+		gatewayEndpoints = append(gatewayEndpoints, corev1.EndpointAddress{
+			IP:       ingress.IP,
+			Hostname: ingress.Hostname,
+		})
+	}
+
+	gatewayIdentity := gateway.Annotations[consts.GatewayIdentity]
+	probeConfig, err := parseProbeConfig(gateway.Annotations)
+
+	if err != nil {
+		rcsw.log.Debugf("could not parse probe config for gateway: %s/%s: %s", gateway.Namespace, gateway.Name, err)
+	}
+
+	return &GatewaySpec{
+		clusterName:      rcsw.clusterName,
+		gatewayName:      gateway.Name,
+		gatewayNamespace: gateway.Namespace,
+		addresses:        gatewayEndpoints,
+		incomingPort:     port,
+		resourceVersion:  gateway.ResourceVersion,
+		identity:         gatewayIdentity,
+		ProbeConfig:      probeConfig,
+	}, nil
+}
diff --git a/controller/cmd/service-mirror/config_watcher.go b/controller/cmd/service-mirror/config_watcher.go
index 32b9a2a27..a92df5f59 100644
--- a/controller/cmd/service-mirror/config_watcher.go
+++ b/controller/cmd/service-mirror/config_watcher.go
@@ -13,6 +13,11 @@ import (
 	consts "github.com/linkerd/linkerd2/pkg/k8s"
 )
 
+// ProbeEventSink is an interface for a type that can send events to the probe manager
+type ProbeEventSink interface {
+	send(event interface{})
+}
+
 // RemoteClusterConfigWatcher watches for secrets of type MirrorSecretType
 // and upon the detection of such secret created starts a RemoteClusterServiceWatcher
 type RemoteClusterConfigWatcher struct {
@@ -20,14 +25,16 @@ type RemoteClusterConfigWatcher struct {
 	clusterWatchers map[string]*RemoteClusterServiceWatcher
 	requeueLimit    int
 	sync.RWMutex
+	probeEventsSink ProbeEventSink
 }
 
 // NewRemoteClusterConfigWatcher Creates a new config watcher
-func NewRemoteClusterConfigWatcher(k8sAPI *k8s.API, requeueLimit int) *RemoteClusterConfigWatcher {
+func NewRemoteClusterConfigWatcher(k8sAPI *k8s.API, requeueLimit int, probeEventsSink ProbeEventSink) *RemoteClusterConfigWatcher {
 	rcw := &RemoteClusterConfigWatcher{
 		k8sAPI:          k8sAPI,
 		clusterWatchers: map[string]*RemoteClusterServiceWatcher{},
 		requeueLimit:    requeueLimit,
+		probeEventsSink: probeEventsSink,
 	}
 	k8sAPI.Secret().Informer().AddEventHandler(
 		cache.FilteringResourceEventHandler{
@@ -105,12 +112,12 @@ func (rcw *RemoteClusterConfigWatcher) Stop() {
 }
 
 func (rcw *RemoteClusterConfigWatcher) registerRemoteCluster(secret *corev1.Secret) error {
-	config, name, domain, err := parseRemoteClusterSecret(secret)
+	config, err := parseRemoteClusterSecret(secret)
 	if err != nil {
 		return err
 	}
 
-	clientConfig, err := clientcmd.RESTConfigFromKubeConfig(config)
+	clientConfig, err := clientcmd.RESTConfigFromKubeConfig(config.apiConfig)
 	if err != nil {
 		return fmt.Errorf("unable to parse kube config: %s", err)
 	}
@@ -118,50 +125,61 @@ func (rcw *RemoteClusterConfigWatcher) registerRemoteCluster(secret *corev1.Secr
 	rcw.Lock()
 	defer rcw.Unlock()
 
-	if _, ok := rcw.clusterWatchers[name]; ok {
-		return fmt.Errorf("there is already a cluster with name %s being watcher. Please delete its config before attempting to register a new one", name)
+	if _, ok := rcw.clusterWatchers[config.clusterName]; ok {
+		return fmt.Errorf("there is already a cluster with name %s being watcher. Please delete its config before attempting to register a new one", config.clusterName)
 	}
 
-	watcher, err := NewRemoteClusterServiceWatcher(rcw.k8sAPI, clientConfig, name, rcw.requeueLimit, domain)
+	watcher, err := NewRemoteClusterServiceWatcher(rcw.k8sAPI, clientConfig, config.clusterName, rcw.requeueLimit, config.clusterDomain, rcw.probeEventsSink)
 	if err != nil {
 		return err
 	}
 
-	rcw.clusterWatchers[name] = watcher
+	rcw.clusterWatchers[config.clusterName] = watcher
 	watcher.Start()
 	return nil
 
 }
 
 func (rcw *RemoteClusterConfigWatcher) unregisterRemoteCluster(secret *corev1.Secret, cleanState bool) error {
-	_, name, _, err := parseRemoteClusterSecret(secret)
+	config, err := parseRemoteClusterSecret(secret)
 	if err != nil {
 		return err
 	}
 	rcw.Lock()
 	defer rcw.Unlock()
-	if watcher, ok := rcw.clusterWatchers[name]; ok {
+	if watcher, ok := rcw.clusterWatchers[config.clusterName]; ok {
 		watcher.Stop(cleanState)
 	} else {
-		return fmt.Errorf("cannot find watcher for cluser: %s", name)
+		return fmt.Errorf("cannot find watcher for cluser: %s", config.clusterName)
 	}
-	delete(rcw.clusterWatchers, name)
+	delete(rcw.clusterWatchers, config.clusterName)
 
 	return nil
 }
 
-func parseRemoteClusterSecret(secret *corev1.Secret) ([]byte, string, string, error) {
+type watchedClusterConfig struct {
+	apiConfig     []byte
+	clusterName   string
+	clusterDomain string
+}
+
+func parseRemoteClusterSecret(secret *corev1.Secret) (*watchedClusterConfig, error) {
 	clusterName, hasClusterName := secret.Annotations[consts.RemoteClusterNameLabel]
 	config, hasConfig := secret.Data[consts.ConfigKeyName]
 	domain, hasDomain := secret.Annotations[consts.RemoteClusterDomainAnnotation]
 	if !hasClusterName {
-		return nil, "", "", fmt.Errorf("secret of type %s should contain key %s", consts.MirrorSecretType, consts.ConfigKeyName)
+		return nil, fmt.Errorf("secret of type %s should contain key %s", consts.MirrorSecretType, consts.ConfigKeyName)
 	}
 	if !hasConfig {
-		return nil, "", "", fmt.Errorf("secret should contain remote cluster name as annotation %s", consts.RemoteClusterNameLabel)
+		return nil, fmt.Errorf("secret should contain remote cluster name as annotation %s", consts.RemoteClusterNameLabel)
 	}
 	if !hasDomain {
-		return nil, "", "", fmt.Errorf("secret should contain remote cluster domain as annotation %s", consts.RemoteClusterDomainAnnotation)
+		return nil, fmt.Errorf("secret should contain remote cluster domain as annotation %s", consts.RemoteClusterDomainAnnotation)
 	}
-	return config, clusterName, domain, nil
+
+	return &watchedClusterConfig{
+		apiConfig:     config,
+		clusterName:   clusterName,
+		clusterDomain: domain,
+	}, nil
 }
diff --git a/controller/cmd/service-mirror/jittered_ticker.go b/controller/cmd/service-mirror/jittered_ticker.go
new file mode 100644
index 000000000..b57c0edd3
--- /dev/null
+++ b/controller/cmd/service-mirror/jittered_ticker.go
@@ -0,0 +1,77 @@
+// Copyright (c) 2016-2017 Tigera, Inc. All rights reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//NB: This file was copied from: https://github.com/projectcalico/felix/blob/master/jitter/jittered_ticker.go
+
+package servicemirror
+
+import (
+	"math/rand"
+	"time"
+
+	log "github.com/sirupsen/logrus"
+)
+
+// Ticker tries to emit events on channel C at minDuration intervals plus up to maxJitter.
+type Ticker struct {
+	C           <-chan time.Time
+	stop        chan bool
+	MinDuration time.Duration
+	MaxJitter   time.Duration
+}
+
+// NewTicker creates a new Ticker
+func NewTicker(minDuration time.Duration, maxJitter time.Duration) *Ticker {
+	if minDuration < 0 {
+		log.WithField("duration", minDuration).Panic("Negative duration")
+	}
+	if maxJitter < 0 {
+		log.WithField("jitter", minDuration).Panic("Negative jitter")
+	}
+	c := make(chan time.Time, 1)
+	ticker := &Ticker{
+		C:           c,
+		stop:        make(chan bool),
+		MinDuration: minDuration,
+		MaxJitter:   maxJitter,
+	}
+	go ticker.loop(c)
+	return ticker
+}
+
+func (t *Ticker) loop(c chan time.Time) {
+tickLoop:
+	for {
+		time.Sleep(t.calculateDelay())
+		// Send best-effort then go back to sleep.
+		select {
+		case <-t.stop:
+			log.Info("Stopping jittered ticker")
+			close(c)
+			break tickLoop
+		case c <- time.Now():
+		default:
+		}
+	}
+}
+
+func (t *Ticker) calculateDelay() time.Duration {
+	jitter := time.Duration(rand.Int63n(int64(t.MaxJitter)))
+	delay := t.MinDuration + jitter
+	return delay
+}
+
+// Stop the ticker
+func (t *Ticker) Stop() {
+	t.stop <- true
+}
diff --git a/controller/cmd/service-mirror/main.go b/controller/cmd/service-mirror/main.go
index 97265f48b..015c9a0d8 100644
--- a/controller/cmd/service-mirror/main.go
+++ b/controller/cmd/service-mirror/main.go
@@ -7,16 +7,24 @@ import (
 	"syscall"
 
 	"github.com/linkerd/linkerd2/controller/k8s"
+	"github.com/linkerd/linkerd2/pkg/admin"
 	"github.com/linkerd/linkerd2/pkg/flags"
 	log "github.com/sirupsen/logrus"
 )
 
+type chanProbeEventSink struct{ sender func(event interface{}) }
+
+func (s *chanProbeEventSink) send(event interface{}) {
+	s.sender(event)
+}
+
 // Main executes the tap service-mirror
 func Main(args []string) {
 	cmd := flag.NewFlagSet("service-mirror", flag.ExitOnError)
 
 	kubeConfigPath := cmd.String("kubeconfig", "", "path to the local kube config")
 	requeueLimit := cmd.Int("event-requeue-limit", 3, "requeue limit for events")
+	metricsAddr := cmd.String("metrics-addr", ":9999", "address to serve scrapable metrics on")
 
 	flags.ConfigureAndParse(cmd, args)
 
@@ -36,12 +44,17 @@ func Main(args []string) {
 		log.Fatalf("Failed to initialize K8s API: %s", err)
 	}
 
+	probeManager := NewProbeManager(k8sAPI)
+	probeManager.Start()
+
 	k8sAPI.Sync(nil)
-	watcher := NewRemoteClusterConfigWatcher(k8sAPI, *requeueLimit)
+	watcher := NewRemoteClusterConfigWatcher(k8sAPI, *requeueLimit, &chanProbeEventSink{probeManager.enqueueEvent})
 	log.Info("Started cluster config watcher")
 
-	<-stop
+	go admin.StartServer(*metricsAddr)
 
+	<-stop
 	log.Info("Stopping cluster config watcher")
 	watcher.Stop()
+	probeManager.Stop()
 }
diff --git a/controller/cmd/service-mirror/metrics.go b/controller/cmd/service-mirror/metrics.go
new file mode 100644
index 000000000..ec2fdf31f
--- /dev/null
+++ b/controller/cmd/service-mirror/metrics.go
@@ -0,0 +1,116 @@
+package servicemirror
+
+import (
+	"github.com/prometheus/client_golang/prometheus"
+	"github.com/prometheus/client_golang/prometheus/promauto"
+)
+
+const (
+	gatewayNameLabel      = "gateway_name"
+	gatewayNamespaceLabel = "gateway_namespace"
+	gatewayClusterName    = "remote_cluster_name"
+	eventTypeLabelName    = "event_type"
+	probeSuccessfulLabel  = "probe_successful"
+)
+
+type probeMetricVecs struct {
+	services  *prometheus.GaugeVec
+	alive     *prometheus.GaugeVec
+	latencies *prometheus.HistogramVec
+	enqueues  *prometheus.CounterVec
+	dequeues  *prometheus.CounterVec
+	probes    *prometheus.CounterVec
+}
+
+type probeMetrics struct {
+	services  prometheus.Gauge
+	alive     prometheus.Gauge
+	latencies prometheus.Observer
+	probes    *prometheus.CounterVec
+}
+
+func newProbeMetricVecs() probeMetricVecs {
+	labelNames := []string{gatewayNameLabel, gatewayNamespaceLabel, gatewayClusterName}
+
+	probes := promauto.NewCounterVec(
+		prometheus.CounterOpts{
+			Name: "gateway_probes",
+			Help: "A counter for the number of actual performed probes to a gateway",
+		},
+		[]string{gatewayNameLabel, gatewayNamespaceLabel, gatewayClusterName, probeSuccessfulLabel},
+	)
+
+	enqueues := promauto.NewCounterVec(
+		prometheus.CounterOpts{
+			Name: "probe_manager_event_enqueues",
+			Help: "A counter for the number of enqueued events to the probe manager",
+		},
+		[]string{eventTypeLabelName},
+	)
+
+	dequeues := promauto.NewCounterVec(
+		prometheus.CounterOpts{
+			Name: "probe_manager_event_dequeues",
+			Help: "A counter for the number of dequeued events to the probe manager",
+		},
+		[]string{eventTypeLabelName},
+	)
+
+	services := promauto.NewGaugeVec(
+		prometheus.GaugeOpts{
+			Name: "num_mirrored_services",
+			Help: "A gauge for the current number of mirrored services associated with a gateway",
+		},
+		labelNames,
+	)
+
+	alive := promauto.NewGaugeVec(
+		prometheus.GaugeOpts{
+			Name: "gateway_alive",
+			Help: "A gauge which is 1 if the gateway is alive and 0 if it is not.",
+		},
+		labelNames,
+	)
+
+	latencies := promauto.NewHistogramVec(
+		prometheus.HistogramOpts{
+			Name: "gateway_probe_latency_ms",
+			Help: "A histogram of latencies to a remote gateway.",
+			Buckets: []float64{
+				1, 2, 3, 4, 5,
+				10, 20, 30, 40, 50,
+				100, 200, 300, 400, 500,
+				1000, 2000, 3000, 4000, 5000,
+				10000, 20000, 30000, 40000, 50000,
+			},
+		},
+		labelNames)
+
+	return probeMetricVecs{
+		services:  services,
+		alive:     alive,
+		latencies: latencies,
+		enqueues:  enqueues,
+		dequeues:  dequeues,
+		probes:    probes,
+	}
+}
+func (mv probeMetricVecs) newWorkerMetrics(gatewayNamespace, gatewayName, remoteClusterName string) (*probeMetrics, error) {
+
+	labels := prometheus.Labels{
+		gatewayNameLabel:      gatewayName,
+		gatewayNamespaceLabel: gatewayNamespace,
+		gatewayClusterName:    remoteClusterName,
+	}
+
+	curriedProbes, err := mv.probes.CurryWith(labels)
+	if err != nil {
+		return nil, err
+	}
+	return &probeMetrics{
+		services:  mv.services.With(labels),
+		alive:     mv.alive.With(labels),
+		latencies: mv.latencies.With(labels),
+		probes:    curriedProbes,
+	}, nil
+}
diff --git a/controller/cmd/service-mirror/probe_manager.go b/controller/cmd/service-mirror/probe_manager.go
new file mode 100644
index 000000000..0febad05d
--- /dev/null
+++ b/controller/cmd/service-mirror/probe_manager.go
@@ -0,0 +1,239 @@
+package servicemirror
+
+import (
+	"fmt"
+
+	"github.com/prometheus/client_golang/prometheus"
+
+	"github.com/linkerd/linkerd2/controller/k8s"
+	consts "github.com/linkerd/linkerd2/pkg/k8s"
+	log "github.com/sirupsen/logrus"
+	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/labels"
+)
+
+const probeChanBufferSize = 500
+
+// ProbeManager takes care of managing the lifecycle of probe workers
+type ProbeManager struct {
+	k8sAPI       *k8s.API
+	probeWorkers map[string]*ProbeWorker
+
+	events     chan interface{}
+	metricVecs *probeMetricVecs
+	done       chan struct{}
+}
+
+// MirroredServiceUnpaired is emitted when a service is no longer mirrored
+type MirroredServiceUnpaired struct {
+	serviceName      string
+	serviceNamespace string
+	gatewayName      string
+	gatewayNs        string
+	clusterName      string
+}
+
+// MirroredServicePaired is emitted when a new mirrored service is created
+type MirroredServicePaired struct {
+	serviceName      string
+	serviceNamespace string
+	GatewaySpec
+}
+
+// ClusterNotRegistered is is emitted when the cluster is not monitored anymore
+type ClusterNotRegistered struct {
+	clusterName string
+}
+
+// GatewayUpdated is emitted when something about the gateway is updated (i.e. its external IP(s))
+type GatewayUpdated struct {
+	GatewaySpec
+}
+
+// NewProbeManager creates a new probe manager
+func NewProbeManager(k8sAPI *k8s.API) *ProbeManager {
+	metricVecs := newProbeMetricVecs()
+	return &ProbeManager{
+		k8sAPI:       k8sAPI,
+		probeWorkers: make(map[string]*ProbeWorker),
+		events:       make(chan interface{}, probeChanBufferSize),
+		metricVecs:   &metricVecs,
+		done:         make(chan struct{}),
+	}
+}
+
+func eventTypeString(ev interface{}) string {
+	switch ev.(type) {
+	case *MirroredServicePaired:
+		return "MirroredServicePaired"
+	case *MirroredServiceUnpaired:
+		return "MirroredServiceUnpaired"
+	case *GatewayUpdated:
+		return "GatewayUpdated"
+	case *ClusterNotRegistered:
+		return "ClusterNotRegistered"
+	default:
+		return "Unknown"
+	}
+}
+
+func (m *ProbeManager) enqueueEvent(event interface{}) {
+	m.metricVecs.enqueues.With(prometheus.Labels{eventTypeLabelName: eventTypeString(event)}).Inc()
+	m.events <- event
+}
+
+func probeKey(gatewayNamespace string, gatewayName string, clusterName string) string {
+	return fmt.Sprintf("%s-%s-%s", gatewayNamespace, gatewayName, clusterName)
+}
+
+func (m *ProbeManager) handleEvent(ev interface{}) {
+	switch ev := ev.(type) {
+	case *MirroredServicePaired:
+		m.handleMirroredServicePaired(ev)
+	case *MirroredServiceUnpaired:
+		m.handleMirroredServiceUnpaired(ev)
+	case *GatewayUpdated:
+		m.handleGatewayUpdated(ev)
+	case *ClusterNotRegistered:
+		m.handleClusterNotRegistered(ev)
+	default:
+		log.Errorf("Received unknown event: %v", ev)
+	}
+}
+
+func endpointAddressesToIps(addrs []corev1.EndpointAddress) []string {
+	result := []string{}
+
+	for _, a := range addrs {
+
+		result = append(result, a.IP)
+	}
+
+	return result
+}
+
+func gatewayToProbeSpec(gatewaySpec GatewaySpec) *probeSpec {
+
+	if gatewaySpec.ProbeConfig == nil {
+		return nil
+	}
+	return &probeSpec{
+		ips:             endpointAddressesToIps(gatewaySpec.addresses),
+		path:            gatewaySpec.path,
+		port:            gatewaySpec.port,
+		periodInSeconds: gatewaySpec.periodInSeconds,
+	}
+}
+
+func (m *ProbeManager) handleMirroredServicePaired(event *MirroredServicePaired) {
+	probeKey := probeKey(event.gatewayNamespace, event.gatewayName, event.clusterName)
+	worker, ok := m.probeWorkers[probeKey]
+	if ok {
+		log.Debugf("Probe worker %s already exists", probeKey)
+		worker.PairService(event.serviceName, event.serviceNamespace)
+	} else {
+		log.Debugf("Creating probe worker %s", probeKey)
+		probeMetrics, err := m.metricVecs.newWorkerMetrics(event.gatewayNamespace, event.gatewayName, event.clusterName)
+		if err != nil {
+			log.Errorf("Could not crete probe metrics: %s", err)
+		} else {
+			probeSpec := gatewayToProbeSpec(event.GatewaySpec)
+			if probeSpec != nil {
+				worker = NewProbeWorker(probeSpec, probeMetrics, probeKey)
+				worker.PairService(event.serviceName, event.serviceNamespace)
+				m.probeWorkers[probeKey] = worker
+				worker.Start()
+			} else {
+				log.Debugf("No probe spec for: %s", probeKey)
+			}
+		}
+	}
+}
+
+func (m *ProbeManager) handleMirroredServiceUnpaired(event *MirroredServiceUnpaired) {
+	probeKey := probeKey(event.gatewayNs, event.gatewayName, event.clusterName)
+	worker, ok := m.probeWorkers[probeKey]
+	if ok {
+		worker.UnPairService(event.serviceName, event.serviceNamespace)
+		if worker.NumPairedServices() < 1 {
+			log.Debugf("Probe worker's %s associated services dropped to 0, cleaning up", probeKey)
+			worker.Stop()
+			delete(m.probeWorkers, probeKey)
+		}
+	} else {
+		log.Debugf("Could not find a worker for %s while handling MirroredServiceUnpaired event", probeKey)
+	}
+}
+
+func (m *ProbeManager) handleGatewayUpdated(event *GatewayUpdated) {
+	probeKey := probeKey(event.gatewayNamespace, event.gatewayName, event.clusterName)
+	worker, ok := m.probeWorkers[probeKey]
+	if ok {
+		probeSpec := gatewayToProbeSpec(event.GatewaySpec)
+		if probeSpec != nil {
+			worker.UpdateProbeSpec(probeSpec)
+		} else {
+			log.Debugf("No probe spec for: %s", probeKey)
+		}
+	} else {
+		log.Debugf("Could not find a worker for %s while handling GatewayUpdated event", probeKey)
+	}
+}
+
+func (m *ProbeManager) stopProbe(key string) {
+	if worker, ok := m.probeWorkers[key]; ok {
+		worker.Stop()
+		delete(m.probeWorkers, key)
+	} else {
+		log.Infof("Could not find probe worker with key %s", key)
+	}
+}
+
+func (m *ProbeManager) handleClusterNotRegistered(event *ClusterNotRegistered) {
+	matchLabels := map[string]string{
+		consts.MirroredResourceLabel:  "true",
+		consts.RemoteClusterNameLabel: event.clusterName,
+	}
+
+	services, err := m.k8sAPI.Svc().Lister().List(labels.Set(matchLabels).AsSelector())
+	if err != nil {
+		log.Errorf("Was not able to unregister cluster %s, %s", event.clusterName, err)
+	}
+
+	stopped := make(map[string]bool)
+	for _, svc := range services {
+		probeKey := probeKey(svc.Labels[consts.RemoteGatewayNameLabel], svc.Labels[consts.RemoteGatewayNameLabel], event.clusterName)
+
+		if _, ok := stopped[probeKey]; !ok {
+			m.stopProbe(probeKey)
+			stopped[probeKey] = true
+		}
+	}
+}
+
+func (m *ProbeManager) run() {
+	for {
+		select {
+		case event := <-m.events:
+			log.Debugf("Received event: %v", event)
+			m.metricVecs.dequeues.With(prometheus.Labels{eventTypeLabelName: eventTypeString(event)}).Inc()
+			m.handleEvent(event)
+		case <-m.done:
+			log.Debug("Shutting down ProbeManager")
+			for key := range m.probeWorkers {
+				m.stopProbe(key)
+			}
+			return
+		}
+	}
+}
+
+// Start starts the probe manager
+func (m *ProbeManager) Start() {
+	go m.run()
+}
+
+// Stop stops the probe manager
+func (m *ProbeManager) Stop() {
+	close(m.done)
+}
diff --git a/controller/cmd/service-mirror/probe_worker.go b/controller/cmd/service-mirror/probe_worker.go
new file mode 100644
index 000000000..b5ffb0bd3
--- /dev/null
+++ b/controller/cmd/service-mirror/probe_worker.go
@@ -0,0 +1,154 @@
+package servicemirror
+
+import (
+	"fmt"
+	"math/rand"
+	"net/http"
+	"sync"
+	"time"
+
+	"github.com/prometheus/client_golang/prometheus"
+
+	logging "github.com/sirupsen/logrus"
+)
+
+const httpGatewayTimeoutMillis = 50000
+
+type probeSpec struct {
+	ips             []string
+	path            string
+	port            uint32
+	periodInSeconds uint32
+}
+
+// ProbeWorker is responsible for monitoring gateways using a probe specification
+type ProbeWorker struct {
+	*sync.RWMutex
+	probeSpec      *probeSpec
+	pairedServices map[string]struct{}
+	stopCh         chan struct{}
+	metrics        *probeMetrics
+	log            *logging.Entry
+}
+
+// NewProbeWorker creates a new probe worker associated with a particular gateway
+func NewProbeWorker(spec *probeSpec, metrics *probeMetrics, probekey string) *ProbeWorker {
+	return &ProbeWorker{
+		RWMutex:        &sync.RWMutex{},
+		probeSpec:      spec,
+		pairedServices: make(map[string]struct{}),
+		stopCh:         make(chan struct{}),
+		metrics:        metrics,
+		log: logging.WithFields(logging.Fields{
+			"probe-key": probekey,
+		}),
+	}
+}
+
+// NumPairedServices returns the number of paired services for this probe worker
+func (pw *ProbeWorker) NumPairedServices() int {
+	return len(pw.pairedServices)
+}
+
+// PairService increments the number of services that are routed by the gateway
+func (pw *ProbeWorker) PairService(serviceName, serviceNamespace string) {
+	svcKey := fmt.Sprintf("%s-%s", serviceNamespace, serviceName)
+	if _, ok := pw.pairedServices[svcKey]; !ok {
+		pw.pairedServices[svcKey] = struct{}{}
+		pw.metrics.services.Set(float64(len(pw.pairedServices)))
+	}
+}
+
+// UnPairService decrements the number of services that are routed by the gateway
+func (pw *ProbeWorker) UnPairService(serviceName, serviceNamespace string) {
+	svcKey := fmt.Sprintf("%s-%s", serviceNamespace, serviceName)
+	if _, ok := pw.pairedServices[svcKey]; ok {
+		delete(pw.pairedServices, svcKey)
+		pw.metrics.services.Set(float64(len(pw.pairedServices)))
+	}
+}
+
+// UpdateProbeSpec is used to update the probe specification when something about the gateway changes
+func (pw *ProbeWorker) UpdateProbeSpec(spec *probeSpec) {
+	pw.Lock()
+	pw.probeSpec = spec
+	pw.Unlock()
+}
+
+// Stop this probe worker
+func (pw *ProbeWorker) Stop() {
+	pw.log.Debug("Stopping probe worker")
+	close(pw.stopCh)
+}
+
+// Start this probe worker
+func (pw *ProbeWorker) Start() {
+	pw.log.Debug("Starting probe worker")
+	go pw.run()
+}
+
+func (pw *ProbeWorker) run() {
+	periodInMillis := pw.probeSpec.periodInSeconds * 1000
+	probeTickerPeriod := time.Duration(periodInMillis) * time.Millisecond
+	maxJitter := time.Duration(periodInMillis/10) * time.Millisecond // max jitter is 10% of period
+	probeTicker := NewTicker(probeTickerPeriod, maxJitter)
+
+probeLoop:
+	for {
+		select {
+		case <-pw.stopCh:
+			break probeLoop
+		case <-probeTicker.C:
+			pw.doProbe()
+		}
+	}
+}
+
+func (pw *ProbeWorker) pickAnIP() string {
+	numIps := len(pw.probeSpec.ips)
+	if numIps == 0 {
+		return ""
+	}
+	return pw.probeSpec.ips[rand.Int()%numIps]
+}
+
+func (pw *ProbeWorker) doProbe() {
+	pw.RLock()
+	defer pw.RUnlock()
+
+	successLabel := prometheus.Labels{probeSuccessfulLabel: "true"}
+	notSuccessLabel := prometheus.Labels{probeSuccessfulLabel: "false"}
+
+	ipToTry := pw.pickAnIP()
+	if ipToTry == "" {
+		pw.log.Debug("No ips. Marking as unhealthy")
+		pw.metrics.alive.Set(0)
+	} else {
+		client := http.Client{
+			Timeout: httpGatewayTimeoutMillis * time.Millisecond,
+		}
+		start := time.Now()
+		resp, err := client.Get(fmt.Sprintf("http://%s:%d/%s", ipToTry, pw.probeSpec.port, pw.probeSpec.path))
+		end := time.Since(start)
+		if err != nil {
+			pw.log.Errorf("Problem connecting with gateway. Marking as unhealthy %s", err)
+			pw.metrics.alive.Set(0)
+			pw.metrics.probes.With(notSuccessLabel).Inc()
+			return
+		} else if resp.StatusCode != 200 {
+			pw.log.Debugf("Gateway returned unexpected status %d. Marking as unhealthy", resp.StatusCode)
+			pw.metrics.alive.Set(0)
+			pw.metrics.probes.With(notSuccessLabel).Inc()
+		} else {
+			pw.log.Debug("Gateway is healthy")
+			pw.metrics.alive.Set(1)
+			pw.metrics.latencies.Observe(float64(end.Milliseconds()))
+			pw.metrics.probes.With(successLabel).Inc()
+		}
+
+		if err := resp.Body.Close(); err != nil {
+			pw.log.Debugf("Failed to close response body %s", err)
+		}
+
+	}
+}
diff --git a/pkg/charts/servicemirror/values.go b/pkg/charts/servicemirror/values.go
index c84d5ec7b..203c3ead8 100644
--- a/pkg/charts/servicemirror/values.go
+++ b/pkg/charts/servicemirror/values.go
@@ -14,12 +14,13 @@ const (
 
 // Values contains the top-level elements in the Helm charts
 type Values struct {
-	Namespace              string `json:"namespace"`
-	ControllerImage        string `json:"controllerImage"`
-	ControllerImageVersion string `json:"controllerImageVersion"`
-	ServiceMirrorUID       int64  `json:"serviceMirrorUID"`
-	LogLevel               string `json:"logLevel"`
-	EventRequeueLimit      int32  `json:"eventRequeueLimit"`
+	Namespace                string `json:"namespace"`
+	ControllerImage          string `json:"controllerImage"`
+	ControllerImageVersion   string `json:"controllerImageVersion"`
+	ControllerComponentLabel string `json:"controllerComponentLabel"`
+	ServiceMirrorUID         int64  `json:"serviceMirrorUID"`
+	LogLevel                 string `json:"logLevel"`
+	EventRequeueLimit        int32  `json:"eventRequeueLimit"`
 }
 
 // NewValues returns a new instance of the Values type.
diff --git a/pkg/k8s/labels.go b/pkg/k8s/labels.go
index 3dfbaffea..dd5380169 100644
--- a/pkg/k8s/labels.go
+++ b/pkg/k8s/labels.go
@@ -388,6 +388,15 @@ const (
 	// GatewayIdentity can be found on the remote gateway service
 	GatewayIdentity = SvcMirrorPrefix + "/gateway-identity"
 
+	// GatewayProbePort the port on which the gateway can be probed
+	GatewayProbePort = SvcMirrorPrefix + "/probe-port"
+
+	// GatewayProbePeriod the interval at which the health of the gateway should be probed
+	GatewayProbePeriod = SvcMirrorPrefix + "/probe-period"
+
+	// GatewayProbePath the path at which the health of the gateway should be probed
+	GatewayProbePath = SvcMirrorPrefix + "/probe-path"
+
 	// ConfigKeyName is the key in the secret that stores the kubeconfig needed to connect
 	// to a remote cluster
 	ConfigKeyName = "kubeconfig"
diff --git a/web/Dockerfile b/web/Dockerfile
index 1ac667c1c..651ac073f 100644
--- a/web/Dockerfile
+++ b/web/Dockerfile
@@ -1,5 +1,5 @@
 ## bundle web assets
-FROM node:10 as webpack-bundle
+FROM node:14-stretch as webpack-bundle
 RUN curl -o- -L https://yarnpkg.com/install.sh | bash -s -- --version 1.21.1 --network-concurrency 1
 
 ENV PATH /root/.yarn/bin:$PATH
diff --git a/web/app/webpack.config.js b/web/app/webpack.config.js
index 91fd2103f..5fd08182b 100644
--- a/web/app/webpack.config.js
+++ b/web/app/webpack.config.js
@@ -79,6 +79,10 @@ module.exports = {
       inject: false,
       filename: 'index_bundle.js',
       template: 'index_bundle.js.lodash.tmpl',
+      // this is important! Production builds minimize the template, causing
+      // the comment on the first line to be for the entire file and then
+      // nothing loads.
+      minify: false
     }),
   ],
 };


Test patch:
diff --git a/cli/cmd/testdata/install_addon_control-plane.golden b/cli/cmd/testdata/install_addon_control-plane.golden
index cc072326d..e454ca184 100644
--- a/cli/cmd/testdata/install_addon_control-plane.golden
+++ b/cli/cmd/testdata/install_addon_control-plane.golden
@@ -1033,6 +1033,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/cli/cmd/testdata/install_control-plane.golden b/cli/cmd/testdata/install_control-plane.golden
index addd5f19e..4a885918d 100644
--- a/cli/cmd/testdata/install_control-plane.golden
+++ b/cli/cmd/testdata/install_control-plane.golden
@@ -1033,6 +1033,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/cli/cmd/testdata/install_controlplane_tracing_output.golden b/cli/cmd/testdata/install_controlplane_tracing_output.golden
index 5a9f6cc53..da483a581 100644
--- a/cli/cmd/testdata/install_controlplane_tracing_output.golden
+++ b/cli/cmd/testdata/install_controlplane_tracing_output.golden
@@ -1933,6 +1933,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/cli/cmd/testdata/install_custom_registry.golden b/cli/cmd/testdata/install_custom_registry.golden
index 4e3a9ef07..a53bd3b26 100644
--- a/cli/cmd/testdata/install_custom_registry.golden
+++ b/cli/cmd/testdata/install_custom_registry.golden
@@ -1873,6 +1873,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/cli/cmd/testdata/install_default.golden b/cli/cmd/testdata/install_default.golden
index c02a1db05..030bc1e21 100644
--- a/cli/cmd/testdata/install_default.golden
+++ b/cli/cmd/testdata/install_default.golden
@@ -1873,6 +1873,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/cli/cmd/testdata/install_ha_output.golden b/cli/cmd/testdata/install_ha_output.golden
index 4f5a8926b..ce4b2ab6d 100644
--- a/cli/cmd/testdata/install_ha_output.golden
+++ b/cli/cmd/testdata/install_ha_output.golden
@@ -2001,6 +2001,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/cli/cmd/testdata/install_ha_with_overrides_output.golden b/cli/cmd/testdata/install_ha_with_overrides_output.golden
index a8b7efdea..81345d6dd 100644
--- a/cli/cmd/testdata/install_ha_with_overrides_output.golden
+++ b/cli/cmd/testdata/install_ha_with_overrides_output.golden
@@ -2001,6 +2001,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/cli/cmd/testdata/install_heartbeat_disabled_output.golden b/cli/cmd/testdata/install_heartbeat_disabled_output.golden
index ccbd78259..d625f3f5a 100644
--- a/cli/cmd/testdata/install_heartbeat_disabled_output.golden
+++ b/cli/cmd/testdata/install_heartbeat_disabled_output.golden
@@ -1784,6 +1784,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/cli/cmd/testdata/install_helm_output.golden b/cli/cmd/testdata/install_helm_output.golden
index 31c8c275e..5aa40e4a0 100644
--- a/cli/cmd/testdata/install_helm_output.golden
+++ b/cli/cmd/testdata/install_helm_output.golden
@@ -1939,6 +1939,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/cli/cmd/testdata/install_helm_output_addons.golden b/cli/cmd/testdata/install_helm_output_addons.golden
index 6db21f19a..cca17f2d3 100644
--- a/cli/cmd/testdata/install_helm_output_addons.golden
+++ b/cli/cmd/testdata/install_helm_output_addons.golden
@@ -1939,6 +1939,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/cli/cmd/testdata/install_helm_output_ha.golden b/cli/cmd/testdata/install_helm_output_ha.golden
index f1cd9c85e..8222a0261 100644
--- a/cli/cmd/testdata/install_helm_output_ha.golden
+++ b/cli/cmd/testdata/install_helm_output_ha.golden
@@ -2067,6 +2067,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/cli/cmd/testdata/install_no_init_container.golden b/cli/cmd/testdata/install_no_init_container.golden
index 63619164b..0c44685a1 100644
--- a/cli/cmd/testdata/install_no_init_container.golden
+++ b/cli/cmd/testdata/install_no_init_container.golden
@@ -1738,6 +1738,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/cli/cmd/testdata/install_output.golden b/cli/cmd/testdata/install_output.golden
index 714e882e9..f3e0cbdac 100644
--- a/cli/cmd/testdata/install_output.golden
+++ b/cli/cmd/testdata/install_output.golden
@@ -1869,6 +1869,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/cli/cmd/testdata/install_proxy_ignores.golden b/cli/cmd/testdata/install_proxy_ignores.golden
index a1c1898d0..b0f730399 100644
--- a/cli/cmd/testdata/install_proxy_ignores.golden
+++ b/cli/cmd/testdata/install_proxy_ignores.golden
@@ -1873,6 +1873,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/cli/cmd/testdata/install_restricted_dashboard.golden b/cli/cmd/testdata/install_restricted_dashboard.golden
index 0ff8a4030..8d4afdc65 100644
--- a/cli/cmd/testdata/install_restricted_dashboard.golden
+++ b/cli/cmd/testdata/install_restricted_dashboard.golden
@@ -1808,6 +1808,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/cli/cmd/testdata/install_tracing.golden b/cli/cmd/testdata/install_tracing.golden
index d2cf23fb6..36bf5563e 100644
--- a/cli/cmd/testdata/install_tracing.golden
+++ b/cli/cmd/testdata/install_tracing.golden
@@ -1873,6 +1873,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/cli/cmd/testdata/upgrade_add-on_controlplane.golden b/cli/cmd/testdata/upgrade_add-on_controlplane.golden
index 1e8226931..609979c4e 100644
--- a/cli/cmd/testdata/upgrade_add-on_controlplane.golden
+++ b/cli/cmd/testdata/upgrade_add-on_controlplane.golden
@@ -1041,6 +1041,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/cli/cmd/testdata/upgrade_add-on_overwrite.golden b/cli/cmd/testdata/upgrade_add-on_overwrite.golden
index bfa3066d9..3eaa1b550 100644
--- a/cli/cmd/testdata/upgrade_add-on_overwrite.golden
+++ b/cli/cmd/testdata/upgrade_add-on_overwrite.golden
@@ -1881,6 +1881,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/cli/cmd/testdata/upgrade_add_add-on.golden b/cli/cmd/testdata/upgrade_add_add-on.golden
index 8864449dc..1c12fbacf 100644
--- a/cli/cmd/testdata/upgrade_add_add-on.golden
+++ b/cli/cmd/testdata/upgrade_add_add-on.golden
@@ -1881,6 +1881,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/cli/cmd/testdata/upgrade_default.golden b/cli/cmd/testdata/upgrade_default.golden
index ea29509c5..fc2d4361b 100644
--- a/cli/cmd/testdata/upgrade_default.golden
+++ b/cli/cmd/testdata/upgrade_default.golden
@@ -1881,6 +1881,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/cli/cmd/testdata/upgrade_external_issuer.golden b/cli/cmd/testdata/upgrade_external_issuer.golden
index 24869ad0d..424312deb 100644
--- a/cli/cmd/testdata/upgrade_external_issuer.golden
+++ b/cli/cmd/testdata/upgrade_external_issuer.golden
@@ -1867,6 +1867,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/cli/cmd/testdata/upgrade_ha.golden b/cli/cmd/testdata/upgrade_ha.golden
index 657de0a1f..8ed0064c1 100644
--- a/cli/cmd/testdata/upgrade_ha.golden
+++ b/cli/cmd/testdata/upgrade_ha.golden
@@ -2009,6 +2009,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/cli/cmd/testdata/upgrade_overwrite_issuer.golden b/cli/cmd/testdata/upgrade_overwrite_issuer.golden
index 64f150ab5..2f840137b 100644
--- a/cli/cmd/testdata/upgrade_overwrite_issuer.golden
+++ b/cli/cmd/testdata/upgrade_overwrite_issuer.golden
@@ -1873,6 +1873,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/cli/cmd/testdata/upgrade_overwrite_trust_anchors-external-issuer.golden b/cli/cmd/testdata/upgrade_overwrite_trust_anchors-external-issuer.golden
index 81ef8639f..ea04f4aa3 100644
--- a/cli/cmd/testdata/upgrade_overwrite_trust_anchors-external-issuer.golden
+++ b/cli/cmd/testdata/upgrade_overwrite_trust_anchors-external-issuer.golden
@@ -1859,6 +1859,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/cli/cmd/testdata/upgrade_overwrite_trust_anchors.golden b/cli/cmd/testdata/upgrade_overwrite_trust_anchors.golden
index 64f150ab5..2f840137b 100644
--- a/cli/cmd/testdata/upgrade_overwrite_trust_anchors.golden
+++ b/cli/cmd/testdata/upgrade_overwrite_trust_anchors.golden
@@ -1873,6 +1873,19 @@ data:
         action: replace
         target_label: component
 
+    - job_name: 'linkerd-service-mirror'
+      kubernetes_sd_configs:
+      - role: pod
+      relabel_configs:
+      - source_labels:
+        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
+        - __meta_kubernetes_pod_container_port_name
+        action: keep
+        regex: linkerd-service-mirror;admin-http$
+      - source_labels: [__meta_kubernetes_pod_container_name]
+        action: replace
+        target_label: component
+
     - job_name: 'linkerd-proxy'
       kubernetes_sd_configs:
       - role: pod
diff --git a/controller/cmd/service-mirror/cluster_watcher_test.go b/controller/cmd/service-mirror/cluster_watcher_test.go
index e2189ed27..1637258e2 100644
--- a/controller/cmd/service-mirror/cluster_watcher_test.go
+++ b/controller/cmd/service-mirror/cluster_watcher_test.go
@@ -27,6 +27,10 @@ type testCase struct {
 	expectedEventsInQueue  []interface{}
 }
 
+type NoOpProbeEventSink struct{}
+
+func (s *NoOpProbeEventSink) send(event interface{}) {}
+
 func runTestCase(tc *testCase, t *testing.T) {
 	t.Run(tc.testDescription, func(t *testing.T) {
 		remoteAPI, err := k8s.NewFakeAPI(tc.remoteResources...)
@@ -53,6 +57,7 @@ func runTestCase(tc *testCase, t *testing.T) {
 			log:             logging.WithFields(logging.Fields{"cluster": clusterName}),
 			eventsQueue:     q,
 			requeueLimit:    0,
+			probeEventsSink: &NoOpProbeEventSink{},
 		}
 
 		for _, ev := range tc.events {
@@ -270,6 +275,10 @@ func TestRemoteServiceDeleted(t *testing.T) {
 				&RemoteServiceDeleted{
 					Name:      "test-service-remote-to-delete",
 					Namespace: "test-namespace-to-delete",
+					GatewayData: gatewayMetadata{
+						Name:      "gateway",
+						Namespace: "gateway-ns",
+					},
 				},
 			},
 
@@ -326,7 +335,7 @@ func TestRemoteServiceUpdated(t *testing.T) {
 							Protocol: "TCP",
 						},
 					}),
-					gatewayData: &gatewayMetadata{
+					gatewayData: gatewayMetadata{
 						Name:      "gateway-new",
 						Namespace: "gateway-ns",
 					},
@@ -437,7 +446,7 @@ func TestRemoteServiceUpdated(t *testing.T) {
 							Protocol: "TCP",
 						},
 					}),
-					gatewayData: &gatewayMetadata{
+					gatewayData: gatewayMetadata{
 						Name:      "gateway",
 						Namespace: "gateway-ns",
 					},
@@ -525,13 +534,16 @@ func TestRemoteGatewayUpdated(t *testing.T) {
 			testDescription: "endpoints ports are updated on gateway change",
 			events: []interface{}{
 				&RemoteGatewayUpdated{
-					newPort:              999,
-					newEndpointAddresses: []corev1.EndpointAddress{{IP: "0.0.0.0"}},
-					gatewayData: &gatewayMetadata{
-						Name:      "gateway",
-						Namespace: "gateway-ns",
+
+					gatewaySpec: GatewaySpec{
+						gatewayName:      "gateway",
+						gatewayNamespace: "gateway-ns",
+						clusterName:      "remote",
+						addresses:        []corev1.EndpointAddress{{IP: "0.0.0.0"}},
+						incomingPort:     999,
+						resourceVersion:  "currentGatewayResVersion",
+						ProbeConfig:      nil,
 					},
-					newResourceVersion: "currentGatewayResVersion",
 					affectedServices: []*corev1.Service{
 						mirroredService("test-service-1-remote", "test-namespace", "gateway", "gateway-ns", "", "pastGatewayResVersion",
 							[]corev1.ServicePort{
@@ -624,13 +636,14 @@ func TestRemoteGatewayUpdated(t *testing.T) {
 			testDescription: "endpoints addresses are updated on gateway change",
 			events: []interface{}{
 				&RemoteGatewayUpdated{
-					newPort:              888,
-					newEndpointAddresses: []corev1.EndpointAddress{{IP: "0.0.0.1"}},
-					gatewayData: &gatewayMetadata{
-						Name:      "gateway",
-						Namespace: "gateway-ns",
+					gatewaySpec: GatewaySpec{
+						gatewayName:      "gateway",
+						gatewayNamespace: "gateway-ns",
+						clusterName:      "remote",
+						addresses:        []corev1.EndpointAddress{{IP: "0.0.0.1"}},
+						incomingPort:     888,
+						resourceVersion:  "currentGatewayResVersion",
 					},
-					newResourceVersion: "currentGatewayResVersion",
 					affectedServices: []*corev1.Service{
 						mirroredService("test-service-1-remote", "test-namespace", "gateway", "gateway-ns", "", "pastGatewayResVersion",
 							[]corev1.ServicePort{
@@ -721,14 +734,16 @@ func TestRemoteGatewayUpdated(t *testing.T) {
 			testDescription: "identity is updated on gateway change",
 			events: []interface{}{
 				&RemoteGatewayUpdated{
-					identity:             "new-identity",
-					newPort:              888,
-					newEndpointAddresses: []corev1.EndpointAddress{{IP: "0.0.0.0"}},
-					gatewayData: &gatewayMetadata{
-						Name:      "gateway",
-						Namespace: "gateway-ns",
+					gatewaySpec: GatewaySpec{
+						gatewayName:      "gateway",
+						gatewayNamespace: "gateway-ns",
+						clusterName:      "",
+						addresses:        []corev1.EndpointAddress{{IP: "0.0.0.0"}},
+						incomingPort:     888,
+						resourceVersion:  "currentGatewayResVersion",
+						identity:         "new-identity",
+						ProbeConfig:      nil,
 					},
-					newResourceVersion: "currentGatewayResVersion",
 					affectedServices: []*corev1.Service{
 						mirroredService("test-service-1-remote", "test-namespace", "gateway", "gateway-ns", "", "pastGatewayResVersion",
 							[]corev1.ServicePort{
@@ -825,7 +840,7 @@ func TestRemoteGatewayDeleted(t *testing.T) {
 			testDescription: "removes endpoint subsets when gateway is deleted",
 			events: []interface{}{
 				&RemoteGatewayDeleted{
-					gatewayData: &gatewayMetadata{
+					gatewayData: gatewayMetadata{
 						Name:      "gateway",
 						Namespace: "gateway-ns",
 					},
@@ -995,7 +1010,7 @@ func onAddOrUpdateTestCases(t *testing.T, isAdd bool) []testCase {
 				localService:   mirroredService("test-service-remote", "test-namespace", "gateway", "gateway-ns", "pastResourceVersion", "gatewayResVersion", nil),
 				localEndpoints: endpoints("test-service-remote", "test-namespace", "gateway", "gateway-ns", "0.0.0.0", "", nil),
 				remoteUpdate:   remoteService("test-service", "test-namespace", "gateway", "gateway-ns", "currentResVersion", nil),
-				gatewayData: &gatewayMetadata{
+				gatewayData: gatewayMetadata{
 					Name:      "gateway",
 					Namespace: "gateway-ns",
 				},
@@ -1075,6 +1090,10 @@ func TestOnDelete(t *testing.T) {
 				&RemoteServiceDeleted{
 					Name:      "test-service",
 					Namespace: "test-namespace",
+					GatewayData: gatewayMetadata{
+						Name:      "gateway",
+						Namespace: "gateway-ns",
+					},
 				},
 			},
 		},
@@ -1087,7 +1106,7 @@ func TestOnDelete(t *testing.T) {
 			},
 			expectedEventsInQueue: []interface{}{
 				&RemoteGatewayDeleted{
-					gatewayData: &gatewayMetadata{
+					gatewayData: gatewayMetadata{
 						Name:      "gateway",
 						Namespace: "test-namespace",
 					},