Repository: traefik/traefik (language: go)

Issue:
Prometheus metrics: Duplicated services in server up gauge
### Welcome!

- [X] Yes, I've searched similar issues on [GitHub](https://github.com/traefik/traefik/issues) and didn't find any.
- [X] Yes, I've searched similar issues on the [Traefik community forum](https://community.traefik.io) and didn't find any.

### What did you do?

We run several environments as independent clusters.
After updating one of the environments to 2.8 we experienced the traefik_service_server_up gauge in the Prometheus metrics to duplicate the metric for every service.

2.7.2
```
# TYPE traefik_service_server_up gauge
traefik_service_server_up{service="serv1@docker",url="http://172.18.0.10:8888"} 1
traefik_service_server_up{service="serv2@docker",url="http://172.18.0.7:8080"} 1
```

### What did you see instead?

Output with duplicated entries, the URLs match over all of our services (shortened to two here for legibility):

2.8.1
```
# TYPE traefik_service_server_up gauge
traefik_service_server_up{service="serv1@docker",url="http://172.18.0.10:8080"} 1
traefik_service_server_up{service="serv1@docker",url="http://172.18.0.11:8080"} 1
traefik_service_server_up{service="serv2@docker",url="http://172.18.0.10:8080"} 1
traefik_service_server_up{service="serv2@docker",url="http://172.18.0.11:8080"} 1
```

### What version of Traefik are you using?

Correct output:
```
Version:      2.7.2
Codename:     epoisses
Go version:   go1.17.11
Built:        2022-06-27T13:58:55Z
OS/Arch:      linux/amd64
```

Incorrect output:
```
Version:      2.8.1
Codename:     vacherin
Go version:   go1.17.11
Built:        2022-07-11T14:06:57Z
OS/Arch:      linux/amd64
```

### What is your environment & configuration?

```yaml
api:
  dashboard: true
  insecure: false

ping: {}

metrics:
  prometheus:
    manualRouting: true
    addRoutersLabels: true

providers:
  docker:
    exposedByDefault: false
    watch: true
    network: "proxy"
  file:
    directory: "/dynamic"
    watch: true

entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: "https"
  https:
    address: ":443"
    http:
      tls: { }
```

Example labels for one of the services:
```yaml
    labels:
      - "traefik.enable=true"
      # Serv1 application
      - "traefik.http.routers.serv1.rule=Host(`qa.serv1.domain.de`)"
      - "traefik.http.routers.serv1.priority=20"
      - "traefik.http.routers.serv1.entrypoints=https"
      - "traefik.http.routers.serv1.service=serv1"
      - "traefik.http.services.serv1.loadbalancer.server.port=8080"
      # Serv1 actuator endpoint
      - "traefik.http.routers.serv1-actuator.rule=(Host(`qa.serv1.domain.de`) && PathPrefix(`/actuator`))"
      - "traefik.http.routers.serv1-actuator.priority=30"
      - "traefik.http.routers.serv1-actuator.entrypoints=https"
      - "traefik.http.routers.serv1-actuator.service=serv1-actuator"
      - "traefik.http.services.serv1-actuator.loadbalancer.server.port=8081"
      # Traefik Health Check for Serv1 application
      - "traefik.http.services.serv1.loadbalancer.healthcheck.scheme=http"
      - "traefik.http.services.serv1.loadbalancer.healthcheck.hostname=serv1"
      - "traefik.http.services.serv1.loadbalancer.healthcheck.port=8081"
      - "traefik.http.services.serv1.loadbalancer.healthcheck.path=/actuator"
      - "traefik.http.services.serv1.loadbalancer.healthcheck.interval=5s"
      - "traefik.http.services.serv1.loadbalancer.healthcheck.timeout=100ms"
```


### If applicable, please paste the log output in DEBUG level

_No response_


Discussion / hints:
Hello @<user>,

Thanks for your interest in Traefik!

Your `v2.8.1` example shows that:
> ```
> # TYPE traefik_service_server_up gauge
> traefik_service_server_up{service="serv1@docker",url="http://172.18.0.10:8080"} 1
> traefik_service_server_up{service="serv1@docker",url="http://172.18.0.11:8080"} 1
> traefik_service_server_up{service="serv2@docker",url="http://172.18.0.10:8080"} 1
> traefik_service_server_up{service="serv2@docker",url="http://172.18.0.11:8080"} 1
> ```

It seems appropriate to have four entries in the Prometheus output as there are 2 Traefik services, with both 2 different docker services.

So either, you configured both your Traefik services to have 2 docker services (e.g. scaling your docker services), or each of your docker services is attached to the two same Traefik services(e.g. each of your docker services is associated with the same Traefik service).

As It seems your issue might be a configuration problem, this issue will be closed.
To confirm this, please join our [Community Forum](https://community.traefik.io/) and reach out to us on the [Traefik section](https://community.traefik.io/c/traefik/5).

In the meantime, you can double-check [Traefik's documentation](https://doc.traefik.io/traefik/).
This is working completely fine in 2.7 and broken in 2.8, so it would be strange if this was a configuration problem.

We have 6 applications running on this server, each has one service configuration (plus one for actuator endpoints, but due to that not having a healthcheck, it does not show in the server up gauge as far as I understood).
On 2.7 we have 6 entries in Prometheus in total (as expected), on 2.8 it is multiplied to 36.
It seems that for any given service it creates an entry for each service known to Traefik.

So basically:
```
service1_up_gauge(service1_container_ip)
service1_up_gauge(service2_container_ip)
service1_up_gauge(service3_container_ip)
service1_up_gauge(service4_container_ip)
service1_up_gauge(service5_container_ip)
service1_up_gauge(service6_container_ip)
```

Fix patch:
diff --git a/pkg/metrics/prometheus.go b/pkg/metrics/prometheus.go
index 019279b0b3..1f23dcf2d9 100644
--- a/pkg/metrics/prometheus.go
+++ b/pkg/metrics/prometheus.go
@@ -310,7 +310,7 @@ func OnConfigurationUpdate(conf dynamic.Configuration, entryPoints []string) {
 func newPrometheusState() *prometheusState {
 	return &prometheusState{
 		dynamicConfig: newDynamicConfig(),
-		deletedURLs:   make(map[string]string),
+		deletedURLs:   make(map[string][]string),
 	}
 }
 
@@ -327,7 +327,7 @@ type prometheusState struct {
 	deletedEP       []string
 	deletedRouters  []string
 	deletedServices []string
-	deletedURLs     map[string]string
+	deletedURLs     map[string][]string
 }
 
 func (ps *prometheusState) SetDynamicConfig(dynamicConfig *dynamicConfig) {
@@ -350,11 +350,10 @@ func (ps *prometheusState) SetDynamicConfig(dynamicConfig *dynamicConfig) {
 		actualService, ok := dynamicConfig.services[service]
 		if !ok {
 			ps.deletedServices = append(ps.deletedServices, service)
-			continue
 		}
 		for url := range serV {
 			if _, ok := actualService[url]; !ok {
-				ps.deletedURLs[service] = url
+				ps.deletedURLs[service] = append(ps.deletedURLs[service], url)
 			}
 		}
 	}
@@ -401,16 +400,18 @@ func (ps *prometheusState) Collect(ch chan<- stdprometheus.Metric) {
 		}
 	}
 
-	for service, url := range ps.deletedURLs {
-		if !ps.dynamicConfig.hasServerURL(service, url) {
-			ps.DeletePartialMatch(map[string]string{"service": service, "url": url})
+	for service, urls := range ps.deletedURLs {
+		for _, url := range urls {
+			if !ps.dynamicConfig.hasServerURL(service, url) {
+				ps.DeletePartialMatch(map[string]string{"service": service, "url": url})
+			}
 		}
 	}
 
 	ps.deletedEP = nil
 	ps.deletedRouters = nil
 	ps.deletedServices = nil
-	ps.deletedURLs = make(map[string]string)
+	ps.deletedURLs = make(map[string][]string)
 }
 
 // DeletePartialMatch deletes all metrics where the variable labels contain all of those passed in as labels.


Test patch:
diff --git a/pkg/metrics/prometheus_test.go b/pkg/metrics/prometheus_test.go
index 02688dfc3a..30627c3bd9 100644
--- a/pkg/metrics/prometheus_test.go
+++ b/pkg/metrics/prometheus_test.go
@@ -364,6 +364,7 @@ func TestPrometheusMetricRemoval(t *testing.T) {
 				th.WithService("bar@providerName", th.WithServers(
 					th.WithServer("http://localhost:9000"),
 					th.WithServer("http://localhost:9999"),
+					th.WithServer("http://localhost:9998"),
 				)),
 				th.WithService("service1", th.WithServers(th.WithServer("http://localhost:9000"))),
 			),
@@ -403,6 +404,10 @@ func TestPrometheusMetricRemoval(t *testing.T) {
 		ServiceServerUpGauge().
 		With("service", "bar@providerName", "url", "http://localhost:9999").
 		Set(1)
+	prometheusRegistry.
+		ServiceServerUpGauge().
+		With("service", "bar@providerName", "url", "http://localhost:9998").
+		Set(1)
 
 	assertMetricsExist(t, mustScrape(), entryPointReqsTotalName, routerReqsTotalName, serviceReqsTotalName, serviceServerUpName)
 	assertMetricsAbsent(t, mustScrape(), entryPointReqsTotalName, routerReqsTotalName, serviceReqsTotalName, serviceServerUpName)
@@ -432,6 +437,47 @@ func TestPrometheusMetricRemoval(t *testing.T) {
 	assertMetricsExist(t, mustScrape(), entryPointReqsTotalName, serviceReqsTotalName, serviceServerUpName, routerReqsTotalName)
 }
 
+func TestPrometheusMetricRemoveEndpointForRecoveredService(t *testing.T) {
+	promState = newPrometheusState()
+	promRegistry = prometheus.NewRegistry()
+	t.Cleanup(promState.reset)
+
+	prometheusRegistry := RegisterPrometheus(context.Background(), &types.Prometheus{AddServicesLabels: true})
+	defer promRegistry.Unregister(promState)
+
+	conf1 := dynamic.Configuration{
+		HTTP: th.BuildConfiguration(
+			th.WithLoadBalancerServices(
+				th.WithService("service1", th.WithServers(th.WithServer("http://localhost:9000"))),
+			),
+		),
+	}
+
+	conf2 := dynamic.Configuration{
+		HTTP: th.BuildConfiguration(),
+	}
+
+	conf3 := dynamic.Configuration{
+		HTTP: th.BuildConfiguration(
+			th.WithLoadBalancerServices(
+				th.WithService("service1", th.WithServers(th.WithServer("http://localhost:9001"))),
+			),
+		),
+	}
+
+	OnConfigurationUpdate(conf1, []string{})
+	OnConfigurationUpdate(conf2, []string{})
+	OnConfigurationUpdate(conf3, []string{})
+
+	prometheusRegistry.
+		ServiceServerUpGauge().
+		With("service", "service1", "url", "http://localhost:9000").
+		Add(1)
+
+	assertMetricsExist(t, mustScrape(), serviceServerUpName)
+	assertMetricsAbsent(t, mustScrape(), serviceServerUpName)
+}
+
 func TestPrometheusRemovedMetricsReset(t *testing.T) {
 	t.Cleanup(promState.reset)
 
@@ -488,7 +534,7 @@ func (ps *prometheusState) reset() {
 	ps.deletedEP = nil
 	ps.deletedRouters = nil
 	ps.deletedServices = nil
-	ps.deletedURLs = make(map[string]string)
+	ps.deletedURLs = make(map[string][]string)
 }
 
 // Tracking and gathering the metrics happens concurrently.