Repository: prometheus/prometheus (language: go)

Issue:
Stale metrics for prometheus_notifications_sent_total
**What did you do?**
If prometheus was connected to one alertmanager and successfully sent alerts, but after that the addresses of alertmanagers have changed (new kubernetes have been rolled out), then in the metrics `prometheus_notifications_.*` there will be metrics on old alertmanager.  Is it possible to remove metrics from unused alertmanager when reloading prometheus?

**What did you expect to see?**
When we re-create pods with new addresses, we want old alertmanager metrics to disappear from prometheus. 

**What did you see instead? Under which circumstances?**
Prometheus continues to give metrics about old alertmanager's: http://joxi.ru/GrqqgplHk8dYnr?d=1
Because of the dependent metrics, we can't now understand if there are alertmanagers we use that we can't send the metrics through.

**Environment**

* System information:
```
Linux 5.3.0-28-generic x86_64
```

* Prometheus version:
```
prometheus, version 2.16.0 (branch: HEAD, revision: b90be6f32a33c03163d700e1452b54454ddce0ec)
  build user:       root@0fdd9501c22e
  build date:       20200220-10:44:36
  go version:       go1.13.8
```

* Alertmanager configuration file:
```
alerting:
  alertmanagers:
  - kubernetes_sd_configs:
    - role: endpoints
      namespaces:
        names:
        - monitoring
    scheme: http
    path_prefix: /
    timeout: 10s
    api_version: v1
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_name]
      separator: ;
      regex: madison-proxy
      replacement: $1
      action: keep
    - source_labels: [__meta_kubernetes_pod_container_port_number]
      separator: ;
      regex: "80"
      replacement: $1
      action: keep
```



Discussion / hints:
Counters don't become invalid due to the thing they refer to no longer existing, using the erors metrics should catch this.

Fix patch:
diff --git a/notifier/notifier.go b/notifier/notifier.go
index d1832402f78..5c4ad77c581 100644
--- a/notifier/notifier.go
+++ b/notifier/notifier.go
@@ -693,6 +693,7 @@ func (s *alertmanagerSet) sync(tgs []*targetgroup.Group) {
 
 	s.mtx.Lock()
 	defer s.mtx.Unlock()
+	previousAms := s.ams
 	// Set new Alertmanagers and deduplicate them along their unique URL.
 	s.ams = []alertmanager{}
 	s.droppedAms = []alertmanager{}
@@ -712,6 +713,17 @@ func (s *alertmanagerSet) sync(tgs []*targetgroup.Group) {
 		seen[us] = struct{}{}
 		s.ams = append(s.ams, am)
 	}
+	// Now remove counters for any removed Alertmanagers.
+	for _, am := range previousAms {
+		us := am.url().String()
+		if _, ok := seen[us]; ok {
+			continue
+		}
+		s.metrics.latency.DeleteLabelValues(us)
+		s.metrics.sent.DeleteLabelValues(us)
+		s.metrics.errors.DeleteLabelValues(us)
+		seen[us] = struct{}{}
+	}
 }
 
 func postPath(pre string, v config.AlertmanagerAPIVersion) string {


Test patch: