Repository: traefik/traefik (language: go)

Issue:
Turning on full metrics will lose 40% performance and needs to be optimized
<!-- PLEASE FOLLOW THE ISSUE TEMPLATE TO HELP TRIAGE AND SUPPORT! -->

### Do you want to request a *feature* or report a *bug*?

<!--
DO NOT FILE ISSUES FOR GENERAL SUPPORT QUESTIONS.

The issue tracker is for reporting bugs and feature requests only.
For end-user related support questions, please refer to one of the following:

- the Traefik community forum: https://community.containo.us/

-->

Feature

### What did you expect to see?
Version: v2.4, contain routers metrics.
Metrics provider: prometheus
Metrics: entryPoints, services, routers
Trun off accessLog.
Case 1:
Turn off all metrics,  traefik can get 14000 QPS.
Case 2:
Turn on all metrics, can get only 9400 QPS.

Performance of metrics needs to be optimized.


Discussion / hints:
Hello @letian0805,

Thanks for your interest in Traefik!

Which metric provider are you using?
Can you share something reproducible or the methodology you've used to reach your conclusions?

Without knowing exactly the context of the performance drop you are facing, it will be difficult to address your issue.
Nonetheless, it comes without saying that enabling metrics or access log, or any middleware, which remain as additional computations, the performances will drop accordingly to those additional computations.
Hi @rtribotte 

I'm using prometheus. No other middlewares.

Metrics configures of case 1:
```toml
#[accesslog]

[metrics]
    [metrics.prometheus]
        entryPoint = "metrics"
        addEntryPointsLabels = false
        addServicesLabels = false
        addRoutersLabels = false
```
case 2:
```toml
#[accesslog]

[metrics]
    [metrics.prometheus]
        entryPoint = "metrics"
        addEntryPointsLabels = true
        addServicesLabels = true
        addRoutersLabels = true
```
Test service: a simple echo service.
```go
    m := http.NewServeMux()
    m.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        clientIP := r.RemoteAddr
        realIP := r.Header.Get("X-Real-IP")
        xForwardedFor := r.Header.Get("X-Forwarded-For")
        msg := fmt.Sprintf("handle %s  from client_ip: %s, x-real-ip: %s, x-forwarded-for: %s, tag: %s\n", r.RequestURI, clientIP, realIP, xForwardedFor, config.Consul.Tag)
        ioutil.ReadAll(r.Body)
        w.Write([]byte(msg))
        logrus.Info(msg)
        r.Body.Close()
    }) 
```

Request body: 300 Bytes

accessLog false
addEntryPointsLabels = false
addServicesLabels = false
addRoutersLabels = false
total: 100000,  cost: 7.007286 s, requests/s: 14270.859992, success rate: 100%

accessLog false
addEntryPointsLabels = true
addServicesLabels = false
addRoutersLabels = false
total: 100000,  cost: 8.156821 s, requests/s: 12259.677966, success rate: 100%

accessLog false
addEntryPointsLabels = true
addServicesLabels = true
addRoutersLabels = false
total: 100000,  cost: 9.188745 s, requests/s: 10882.879146, success rate: 100%

accessLog false
addEntryPointsLabels = true
addServicesLabels = true
addRoutersLabels = true
total: 100000,  cost: 10.635392 s, requests/s: 9402.567991, success rate: 100%
```
accessLog false
addEntryPointsLabels = true
addServicesLabels = true
addRoutersLabels = true
```

This configuration, in theory, isn't optimal, as Services are included in Router metrics so you're duplicating the information for no benefit as the same information can be collected from router metrics.  Can you run it with the following config?

```
accessLog false
addEntryPointsLabels = true
addServicesLabels = false
addRoutersLabels = true
```
@jorge07 Ok. I'll try it. But i have tried this
```
accessLog false
addEntryPointsLabels = true
addServicesLabels = true
addRoutersLabels = false
total: 100000, cost: 9.188745 s, requests/s: 10882.879146, success rate: 100%
```
Yeah, with this config you've full visibility and Service labels aren't required anymore. The performance penalty it's important that's right. 
From the test results, each item will increase the performance loss by at least 10%.
There's a big performance drop.
Any update?

Fix patch:
diff --git a/go.mod b/go.mod
index 99258d188c..5f2930f8c6 100644
--- a/go.mod
+++ b/go.mod
@@ -49,7 +49,7 @@ require (
 	github.com/patrickmn/go-cache v2.1.0+incompatible
 	github.com/pires/go-proxyproto v0.6.1
 	github.com/pmezard/go-difflib v1.0.0
-	github.com/prometheus/client_golang v1.11.0
+	github.com/prometheus/client_golang v1.12.2-0.20220704083116-e8f91604d835
 	github.com/prometheus/client_model v0.2.0
 	github.com/rancher/go-rancher-metadata v0.0.0-20200311180630-7f4c936a06ac
 	github.com/sirupsen/logrus v1.8.1
@@ -155,7 +155,7 @@ require (
 	github.com/fsnotify/fsnotify v1.5.1 // indirect
 	github.com/fvbommel/sortorder v1.0.1 // indirect
 	github.com/go-errors/errors v1.0.1 // indirect
-	github.com/go-logfmt/logfmt v0.5.0 // indirect
+	github.com/go-logfmt/logfmt v0.5.1 // indirect
 	github.com/go-logr/logr v0.4.0 // indirect
 	github.com/go-resty/resty/v2 v2.1.1-0.20191201195748-d7b97669fe48 // indirect
 	github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0 // indirect
@@ -266,8 +266,8 @@ require (
 	github.com/philhofer/fwd v1.1.1 // indirect
 	github.com/pkg/errors v0.9.1 // indirect
 	github.com/pquerna/otp v1.3.0 // indirect
-	github.com/prometheus/common v0.26.0 // indirect
-	github.com/prometheus/procfs v0.6.0 // indirect
+	github.com/prometheus/common v0.35.0 // indirect
+	github.com/prometheus/procfs v0.7.3 // indirect
 	github.com/sacloud/libsacloud v1.36.2 // indirect
 	github.com/sanathkr/go-yaml v0.0.0-20170819195128-ed9d249f429b // indirect
 	github.com/santhosh-tekuri/jsonschema v1.2.4 // indirect
@@ -307,16 +307,16 @@ require (
 	go.uber.org/zap v1.18.1 // indirect
 	golang.org/x/crypto v0.0.0-20220427172511-eb4f295cb31f // indirect
 	golang.org/x/lint v0.0.0-20210508222113-6edffad5e616 // indirect
-	golang.org/x/oauth2 v0.0.0-20210402161424-2e8d93401602 // indirect
+	golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b // indirect
 	golang.org/x/sync v0.0.0-20210220032951-036812b2e83c // indirect
-	golang.org/x/sys v0.0.0-20220307203707-22a9840ba4d7 // indirect
+	golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a // indirect
 	golang.org/x/term v0.0.0-20210927222741-03fcf44c2211 // indirect
 	golang.org/x/text v0.3.7 // indirect
 	golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
 	google.golang.org/api v0.44.0 // indirect
 	google.golang.org/appengine v1.6.7 // indirect
 	google.golang.org/genproto v0.0.0-20210602131652-f16073e35f0c // indirect
-	google.golang.org/protobuf v1.27.1 // indirect
+	google.golang.org/protobuf v1.28.0 // indirect
 	gopkg.in/inf.v0 v0.9.1 // indirect
 	gopkg.in/ini.v1 v1.62.0 // indirect
 	gopkg.in/ns1/ns1-go.v2 v2.6.2 // indirect
diff --git a/go.sum b/go.sum
index bd4656a307..2eecd9497b 100644
--- a/go.sum
+++ b/go.sum
@@ -686,11 +686,13 @@ github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2
 github.com/go-kit/kit v0.10.1-0.20200915143503-439c4d2ed3ea h1:CnEQOUv4ilElSwFB9g/lVmz206oLE4aNZDYngIY1Gvg=
 github.com/go-kit/kit v0.10.1-0.20200915143503-439c4d2ed3ea/go.mod h1:xUsJbQ/Fp4kEt7AFgCuvyX4a71u8h9jB8tj/ORgOZ7o=
 github.com/go-kit/log v0.1.0/go.mod h1:zbhenjAZHb184qTLMA9ZjW7ThYL0H2mk7Q6pNt4vbaY=
+github.com/go-kit/log v0.2.0/go.mod h1:NwTd00d/i8cPZ3xOwwiv2PO5MOcx78fFErGNcVmBjv0=
 github.com/go-ldap/ldap/v3 v3.1.3/go.mod h1:3rbOH3jRS2u6jg2rJnKAMLE/xQyCKIveG2Sa/Cohzb8=
 github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
 github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
-github.com/go-logfmt/logfmt v0.5.0 h1:TrB8swr/68K7m9CcGut2g3UOihhbcbiMAYiuTXdEih4=
 github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG1KdI/P7A=
+github.com/go-logfmt/logfmt v0.5.1 h1:otpy5pqBCBZ1ng9RQ0dPu4PN7ba75Y/aA+UpowDyNVA=
+github.com/go-logfmt/logfmt v0.5.1/go.mod h1:WYhtIu8zTZfxdn5+rREduYbwxfcBr/Vr6KEVveWlfTs=
 github.com/go-logr/logr v0.1.0/go.mod h1:ixOQHD9gLJUVQQ2ZOR7zLEifBX6tGkNJF4QyIY7sIas=
 github.com/go-logr/logr v0.2.0/go.mod h1:z6/tIYblkpsD+a4lm/fGIIU9mZ+XfAiaFtq7xTgseGU=
 github.com/go-logr/logr v0.4.0 h1:K7/B1jt6fIBQVd4Owv2MqGQClcgf0R266+7C/QjRcLc=
@@ -1653,8 +1655,10 @@ github.com/prometheus/client_golang v1.1.0/go.mod h1:I1FGZT9+L76gKKOs5djB6ezCbFQ
 github.com/prometheus/client_golang v1.3.0/go.mod h1:hJaj2vgQTGQmVCsAACORcieXFeDPbaTKGT+JTgUa3og=
 github.com/prometheus/client_golang v1.4.0/go.mod h1:e9GMxYsXl05ICDXkRhurwBS4Q3OK1iX/F2sw+iXX5zU=
 github.com/prometheus/client_golang v1.7.1/go.mod h1:PY5Wy2awLA44sXw4AOSfFBetzPP4j5+D6mVACh+pe2M=
-github.com/prometheus/client_golang v1.11.0 h1:HNkLOAEQMIDv/K+04rukrLx6ch7msSRwf3/SASFAGtQ=
 github.com/prometheus/client_golang v1.11.0/go.mod h1:Z6t4BnS23TR94PD6BsDNk8yVqroYurpAkEiz0P2BEV0=
+github.com/prometheus/client_golang v1.12.1/go.mod h1:3Z9XVyYiZYEO+YQWt3RD2R3jrbd179Rt297l4aS6nDY=
+github.com/prometheus/client_golang v1.12.2-0.20220704083116-e8f91604d835 h1:sYuFGkrz0PtewSFk0Bg7p7jjiiklc6FUIWz+mFGQfD0=
+github.com/prometheus/client_golang v1.12.2-0.20220704083116-e8f91604d835/go.mod h1:RjnYTcBFM8s+WRft6oBqj4p5OgXJASPw5UFiI7w+GSs=
 github.com/prometheus/client_model v0.0.0-20171117100541-99fa1f4be8e5/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
 github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
 github.com/prometheus/client_model v0.0.0-20190115171406-56726106282f/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
@@ -1674,8 +1678,10 @@ github.com/prometheus/common v0.6.0/go.mod h1:eBmuwkDJBwy6iBfxCBob6t6dR6ENT/y+J+
 github.com/prometheus/common v0.7.0/go.mod h1:DjGbpBbp5NYNiECxcL/VnbXCCaQpKd3tt26CguLLsqA=
 github.com/prometheus/common v0.9.1/go.mod h1:yhUN8i9wzaXS3w1O07YhxHEBxD+W35wd8bs7vj7HSQ4=
 github.com/prometheus/common v0.10.0/go.mod h1:Tlit/dnDKsSWFlCLTWaA1cyBgKHSMdTB80sz/V91rCo=
-github.com/prometheus/common v0.26.0 h1:iMAkS2TDoNWnKM+Kopnx/8tnEStIfpYA0ur0xQzzhMQ=
 github.com/prometheus/common v0.26.0/go.mod h1:M7rCNAaPfAosfx8veZJCuw84e35h3Cfd9VFqTh1DIvc=
+github.com/prometheus/common v0.32.1/go.mod h1:vu+V0TpY+O6vW9J44gczi3Ap/oXXR10b+M/gUGO4Hls=
+github.com/prometheus/common v0.35.0 h1:Eyr+Pw2VymWejHqCugNaQXkAi6KayVNxaHeu6khmFBE=
+github.com/prometheus/common v0.35.0/go.mod h1:phzohg0JFMnBEFGxTDbfu3QyL5GI8gTQJFhYO5B3mfA=
 github.com/prometheus/procfs v0.0.0-20180125133057-cb4147076ac7/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
 github.com/prometheus/procfs v0.0.0-20180725123919-05ee40e3a273/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
 github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
@@ -1690,8 +1696,9 @@ github.com/prometheus/procfs v0.0.5/go.mod h1:4A/X28fw3Fc593LaREMrKMqOKvUAntwMDa
 github.com/prometheus/procfs v0.0.8/go.mod h1:7Qr8sr6344vo1JqZ6HhLceV9o3AJ1Ff+GxbHq6oeK9A=
 github.com/prometheus/procfs v0.1.3/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU=
 github.com/prometheus/procfs v0.2.0/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU=
-github.com/prometheus/procfs v0.6.0 h1:mxy4L2jP6qMonqmq+aTtOx1ifVWUgG/TAmntgbh3xv4=
 github.com/prometheus/procfs v0.6.0/go.mod h1:cz+aTbrPOrUb4q7XlbU9ygM+/jj0fzG6c1xBZuNvfVA=
+github.com/prometheus/procfs v0.7.3 h1:4jVXhlkAyzOScmCkXBTOLRLTz8EeU+eyjrwB/EPq0VU=
+github.com/prometheus/procfs v0.7.3/go.mod h1:cz+aTbrPOrUb4q7XlbU9ygM+/jj0fzG6c1xBZuNvfVA=
 github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU=
 github.com/qri-io/jsonpointer v0.1.0/go.mod h1:DnJPaYgiKu56EuDp8TU5wFLdZIcAnb/uH9v37ZaMV64=
 github.com/qri-io/jsonschema v0.1.1/go.mod h1:QpzJ6gBQ0GYgGmh7mDQ1YsvvhSgE4rYj0k8t5MBOmUY=
@@ -2232,9 +2239,12 @@ golang.org/x/net v0.0.0-20210410081132-afb366fc7cd1/go.mod h1:9tjilg8BloeKEkVJvy
 golang.org/x/net v0.0.0-20210428140749-89ef3d95e781/go.mod h1:OJAsFXCWl8Ukc7SiCT/9KSuxbyM7479/AVlXFRxuMCk=
 golang.org/x/net v0.0.0-20210510120150-4163338589ed/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
 golang.org/x/net v0.0.0-20210520170846-37e1c6afe023/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
+golang.org/x/net v0.0.0-20210525063256-abc453219eb5/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
 golang.org/x/net v0.0.0-20210726213435-c6fcb2dbf985/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
 golang.org/x/net v0.0.0-20211020060615-d418f374d309/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
 golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
+golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
+golang.org/x/net v0.0.0-20220225172249-27dd8689420f/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
 golang.org/x/net v0.0.0-20220425223048-2871e0cb64e4 h1:HVyaeDAYux4pnY+D/SiwmLOR36ewZ4iGQIIrtnuCjFA=
 golang.org/x/net v0.0.0-20220425223048-2871e0cb64e4/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
 golang.org/x/oauth2 v0.0.0-20180724155351-3d292e4d0cdc/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
@@ -2251,8 +2261,10 @@ golang.org/x/oauth2 v0.0.0-20201208152858-08078c50e5b5/go.mod h1:KelEdhl1UZF7XfJ
 golang.org/x/oauth2 v0.0.0-20210218202405-ba52d332ba99/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
 golang.org/x/oauth2 v0.0.0-20210220000619-9bb904979d93/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
 golang.org/x/oauth2 v0.0.0-20210313182246-cd4f82c27b84/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
-golang.org/x/oauth2 v0.0.0-20210402161424-2e8d93401602 h1:0Ja1LBD+yisY6RWM/BH7TJVXWsSjs2VwBSmvSX4HdBc=
 golang.org/x/oauth2 v0.0.0-20210402161424-2e8d93401602/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
+golang.org/x/oauth2 v0.0.0-20210514164344-f6687ab2804c/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
+golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b h1:clP8eMhB30EHdc0bd2Twtq6kgU7yl5ub2cQLSdrv1Dg=
+golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b/go.mod h1:DAh4E804XQdzx2j+YRIaUnCqCV2RuMz24cGBJ5QYIrc=
 golang.org/x/perf v0.0.0-20180704124530-6e6d33e29852/go.mod h1:JLpeXjPJfIyPr5TlbXLkXWLhP8nz10XfvxElABhCtcw=
 golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -2391,10 +2403,12 @@ golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBc
 golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20211103235746-7861aae1554b/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220114195835-da31bd327af9/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220209214540-3681064d5158/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220227234510-4e6760a101f9/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.0.0-20220307203707-22a9840ba4d7 h1:8IVLkfbr2cLhv0a/vKq4UFUcJym8RmDoDboxCFWEjYE=
 golang.org/x/sys v0.0.0-20220307203707-22a9840ba4d7/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a h1:dGzPydgVsqGcTRVwiLJ1jVbufYwmzD3LfVPLKsKg+0k=
+golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
@@ -2658,8 +2672,9 @@ google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGj
 google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c=
 google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
 google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
-google.golang.org/protobuf v1.27.1 h1:SnqbnDw1V7RiZcXPx5MEeqPv2s79L9i7BJUlG/+RurQ=
 google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
+google.golang.org/protobuf v1.28.0 h1:w43yiav+6bVFTBQFZX0r7ipe9JQ1QsbMgHwbBziscLw=
+google.golang.org/protobuf v1.28.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
 gopkg.in/DataDog/dd-trace-go.v1 v1.38.1 h1:nAKgcpJLXRHF56cKCP3bN8gTTQmmNAZFEblbyGKhKTo=
 gopkg.in/DataDog/dd-trace-go.v1 v1.38.1/go.mod h1:GBhK4yaMJ1h329ivtKAqRNe1EZ944UnZwtz5lh7CnJc=
 gopkg.in/airbrake/gobrake.v2 v2.0.9/go.mod h1:/h5ZAUhDkGaJfjzjKLSjv6zCL6O0LLBxU4K+aSYdM/U=
diff --git a/pkg/metrics/prometheus.go b/pkg/metrics/prometheus.go
index cad4461799..019279b0b3 100644
--- a/pkg/metrics/prometheus.go
+++ b/pkg/metrics/prometheus.go
@@ -4,8 +4,6 @@ import (
 	"context"
 	"errors"
 	"net/http"
-	"sort"
-	"strings"
 	"sync"
 	"time"
 
@@ -15,7 +13,6 @@ import (
 	"github.com/prometheus/client_golang/prometheus/promhttp"
 	"github.com/traefik/traefik/v2/pkg/config/dynamic"
 	"github.com/traefik/traefik/v2/pkg/log"
-	"github.com/traefik/traefik/v2/pkg/safe"
 	"github.com/traefik/traefik/v2/pkg/types"
 )
 
@@ -111,37 +108,33 @@ func initStandardRegistry(config *types.Prometheus) Registry {
 		buckets = config.Buckets
 	}
 
-	safe.Go(func() {
-		promState.ListenValueUpdates()
-	})
-
-	configReloads := newCounterFrom(promState.collectors, stdprometheus.CounterOpts{
+	configReloads := newCounterFrom(stdprometheus.CounterOpts{
 		Name: configReloadsTotalName,
 		Help: "Config reloads",
 	}, []string{})
-	configReloadsFailures := newCounterFrom(promState.collectors, stdprometheus.CounterOpts{
+	configReloadsFailures := newCounterFrom(stdprometheus.CounterOpts{
 		Name: configReloadsFailuresTotalName,
 		Help: "Config failure reloads",
 	}, []string{})
-	lastConfigReloadSuccess := newGaugeFrom(promState.collectors, stdprometheus.GaugeOpts{
+	lastConfigReloadSuccess := newGaugeFrom(stdprometheus.GaugeOpts{
 		Name: configLastReloadSuccessName,
 		Help: "Last config reload success",
 	}, []string{})
-	lastConfigReloadFailure := newGaugeFrom(promState.collectors, stdprometheus.GaugeOpts{
+	lastConfigReloadFailure := newGaugeFrom(stdprometheus.GaugeOpts{
 		Name: configLastReloadFailureName,
 		Help: "Last config reload failure",
 	}, []string{})
-	tlsCertsNotAfterTimestamp := newGaugeFrom(promState.collectors, stdprometheus.GaugeOpts{
+	tlsCertsNotAfterTimestamp := newGaugeFrom(stdprometheus.GaugeOpts{
 		Name: tlsCertsNotAfterTimestamp,
 		Help: "Certificate expiration timestamp",
 	}, []string{"cn", "serial", "sans"})
 
-	promState.describers = []func(chan<- *stdprometheus.Desc){
-		configReloads.cv.Describe,
-		configReloadsFailures.cv.Describe,
-		lastConfigReloadSuccess.gv.Describe,
-		lastConfigReloadFailure.gv.Describe,
-		tlsCertsNotAfterTimestamp.gv.Describe,
+	promState.vectors = []vector{
+		configReloads.cv,
+		configReloadsFailures.cv,
+		lastConfigReloadSuccess.gv,
+		lastConfigReloadFailure.gv,
+		tlsCertsNotAfterTimestamp.gv,
 	}
 
 	reg := &standardRegistry{
@@ -156,30 +149,30 @@ func initStandardRegistry(config *types.Prometheus) Registry {
 	}
 
 	if config.AddEntryPointsLabels {
-		entryPointReqs := newCounterFrom(promState.collectors, stdprometheus.CounterOpts{
+		entryPointReqs := newCounterFrom(stdprometheus.CounterOpts{
 			Name: entryPointReqsTotalName,
 			Help: "How many HTTP requests processed on an entrypoint, partitioned by status code, protocol, and method.",
 		}, []string{"code", "method", "protocol", "entrypoint"})
-		entryPointReqsTLS := newCounterFrom(promState.collectors, stdprometheus.CounterOpts{
+		entryPointReqsTLS := newCounterFrom(stdprometheus.CounterOpts{
 			Name: entryPointReqsTLSTotalName,
 			Help: "How many HTTP requests with TLS processed on an entrypoint, partitioned by TLS Version and TLS cipher Used.",
 		}, []string{"tls_version", "tls_cipher", "entrypoint"})
-		entryPointReqDurations := newHistogramFrom(promState.collectors, stdprometheus.HistogramOpts{
+		entryPointReqDurations := newHistogramFrom(stdprometheus.HistogramOpts{
 			Name:    entryPointReqDurationName,
 			Help:    "How long it took to process the request on an entrypoint, partitioned by status code, protocol, and method.",
 			Buckets: buckets,
 		}, []string{"code", "method", "protocol", "entrypoint"})
-		entryPointOpenConns := newGaugeFrom(promState.collectors, stdprometheus.GaugeOpts{
+		entryPointOpenConns := newGaugeFrom(stdprometheus.GaugeOpts{
 			Name: entryPointOpenConnsName,
 			Help: "How many open connections exist on an entrypoint, partitioned by method and protocol.",
 		}, []string{"method", "protocol", "entrypoint"})
 
-		promState.describers = append(promState.describers, []func(chan<- *stdprometheus.Desc){
-			entryPointReqs.cv.Describe,
-			entryPointReqsTLS.cv.Describe,
-			entryPointReqDurations.hv.Describe,
-			entryPointOpenConns.gv.Describe,
-		}...)
+		promState.vectors = append(promState.vectors,
+			entryPointReqs.cv,
+			entryPointReqsTLS.cv,
+			entryPointReqDurations.hv,
+			entryPointOpenConns.gv,
+		)
 
 		reg.entryPointReqsCounter = entryPointReqs
 		reg.entryPointReqsTLSCounter = entryPointReqsTLS
@@ -188,30 +181,30 @@ func initStandardRegistry(config *types.Prometheus) Registry {
 	}
 
 	if config.AddRoutersLabels {
-		routerReqs := newCounterFrom(promState.collectors, stdprometheus.CounterOpts{
+		routerReqs := newCounterFrom(stdprometheus.CounterOpts{
 			Name: routerReqsTotalName,
 			Help: "How many HTTP requests are processed on a router, partitioned by service, status code, protocol, and method.",
 		}, []string{"code", "method", "protocol", "router", "service"})
-		routerReqsTLS := newCounterFrom(promState.collectors, stdprometheus.CounterOpts{
+		routerReqsTLS := newCounterFrom(stdprometheus.CounterOpts{
 			Name: routerReqsTLSTotalName,
 			Help: "How many HTTP requests with TLS are processed on a router, partitioned by service, TLS Version, and TLS cipher Used.",
 		}, []string{"tls_version", "tls_cipher", "router", "service"})
-		routerReqDurations := newHistogramFrom(promState.collectors, stdprometheus.HistogramOpts{
+		routerReqDurations := newHistogramFrom(stdprometheus.HistogramOpts{
 			Name:    routerReqDurationName,
 			Help:    "How long it took to process the request on a router, partitioned by service, status code, protocol, and method.",
 			Buckets: buckets,
 		}, []string{"code", "method", "protocol", "router", "service"})
-		routerOpenConns := newGaugeFrom(promState.collectors, stdprometheus.GaugeOpts{
+		routerOpenConns := newGaugeFrom(stdprometheus.GaugeOpts{
 			Name: routerOpenConnsName,
 			Help: "How many open connections exist on a router, partitioned by service, method, and protocol.",
 		}, []string{"method", "protocol", "router", "service"})
 
-		promState.describers = append(promState.describers, []func(chan<- *stdprometheus.Desc){
-			routerReqs.cv.Describe,
-			routerReqsTLS.cv.Describe,
-			routerReqDurations.hv.Describe,
-			routerOpenConns.gv.Describe,
-		}...)
+		promState.vectors = append(promState.vectors,
+			routerReqs.cv,
+			routerReqsTLS.cv,
+			routerReqDurations.hv,
+			routerOpenConns.gv,
+		)
 		reg.routerReqsCounter = routerReqs
 		reg.routerReqsTLSCounter = routerReqsTLS
 		reg.routerReqDurationHistogram, _ = NewHistogramWithScale(routerReqDurations, time.Second)
@@ -219,40 +212,40 @@ func initStandardRegistry(config *types.Prometheus) Registry {
 	}
 
 	if config.AddServicesLabels {
-		serviceReqs := newCounterFrom(promState.collectors, stdprometheus.CounterOpts{
+		serviceReqs := newCounterFrom(stdprometheus.CounterOpts{
 			Name: serviceReqsTotalName,
 			Help: "How many HTTP requests processed on a service, partitioned by status code, protocol, and method.",
 		}, []string{"code", "method", "protocol", "service"})
-		serviceReqsTLS := newCounterFrom(promState.collectors, stdprometheus.CounterOpts{
+		serviceReqsTLS := newCounterFrom(stdprometheus.CounterOpts{
 			Name: serviceReqsTLSTotalName,
 			Help: "How many HTTP requests with TLS processed on a service, partitioned by TLS version and TLS cipher.",
 		}, []string{"tls_version", "tls_cipher", "service"})
-		serviceReqDurations := newHistogramFrom(promState.collectors, stdprometheus.HistogramOpts{
+		serviceReqDurations := newHistogramFrom(stdprometheus.HistogramOpts{
 			Name:    serviceReqDurationName,
 			Help:    "How long it took to process the request on a service, partitioned by status code, protocol, and method.",
 			Buckets: buckets,
 		}, []string{"code", "method", "protocol", "service"})
-		serviceOpenConns := newGaugeFrom(promState.collectors, stdprometheus.GaugeOpts{
+		serviceOpenConns := newGaugeFrom(stdprometheus.GaugeOpts{
 			Name: serviceOpenConnsName,
 			Help: "How many open connections exist on a service, partitioned by method and protocol.",
 		}, []string{"method", "protocol", "service"})
-		serviceRetries := newCounterFrom(promState.collectors, stdprometheus.CounterOpts{
+		serviceRetries := newCounterFrom(stdprometheus.CounterOpts{
 			Name: serviceRetriesTotalName,
 			Help: "How many request retries happened on a service.",
 		}, []string{"service"})
-		serviceServerUp := newGaugeFrom(promState.collectors, stdprometheus.GaugeOpts{
+		serviceServerUp := newGaugeFrom(stdprometheus.GaugeOpts{
 			Name: serviceServerUpName,
 			Help: "service server is up, described by gauge value of 0 or 1.",
 		}, []string{"service", "url"})
 
-		promState.describers = append(promState.describers, []func(chan<- *stdprometheus.Desc){
-			serviceReqs.cv.Describe,
-			serviceReqsTLS.cv.Describe,
-			serviceReqDurations.hv.Describe,
-			serviceOpenConns.gv.Describe,
-			serviceRetries.cv.Describe,
-			serviceServerUp.gv.Describe,
-		}...)
+		promState.vectors = append(promState.vectors,
+			serviceReqs.cv,
+			serviceReqsTLS.cv,
+			serviceReqDurations.hv,
+			serviceOpenConns.gv,
+			serviceRetries.cv,
+			serviceServerUp.gv,
+		)
 
 		reg.serviceReqsCounter = serviceReqs
 		reg.serviceReqsTLSCounter = serviceReqsTLS
@@ -287,64 +280,93 @@ func registerPromState(ctx context.Context) bool {
 // It then converts the configuration to the optimized package internal format
 // and sets it to the promState.
 func OnConfigurationUpdate(conf dynamic.Configuration, entryPoints []string) {
-	dynamicConfig := newDynamicConfig()
+	dynCfg := newDynamicConfig()
 
 	for _, value := range entryPoints {
-		dynamicConfig.entryPoints[value] = true
+		dynCfg.entryPoints[value] = true
+	}
+
+	if conf.HTTP == nil {
+		promState.SetDynamicConfig(dynCfg)
+		return
 	}
 
 	for name := range conf.HTTP.Routers {
-		dynamicConfig.routers[name] = true
+		dynCfg.routers[name] = true
 	}
 
 	for serviceName, service := range conf.HTTP.Services {
-		dynamicConfig.services[serviceName] = make(map[string]bool)
+		dynCfg.services[serviceName] = make(map[string]bool)
 		if service.LoadBalancer != nil {
 			for _, server := range service.LoadBalancer.Servers {
-				dynamicConfig.services[serviceName][server.URL] = true
+				dynCfg.services[serviceName][server.URL] = true
 			}
 		}
 	}
 
-	promState.SetDynamicConfig(dynamicConfig)
+	promState.SetDynamicConfig(dynCfg)
 }
 
 func newPrometheusState() *prometheusState {
 	return &prometheusState{
-		collectors:    make(chan *collector),
 		dynamicConfig: newDynamicConfig(),
-		state:         make(map[string]*collector),
+		deletedURLs:   make(map[string]string),
 	}
 }
 
+type vector interface {
+	stdprometheus.Collector
+	DeletePartialMatch(labels stdprometheus.Labels) int
+}
+
 type prometheusState struct {
-	collectors chan *collector
-	describers []func(ch chan<- *stdprometheus.Desc)
+	vectors []vector
 
-	mtx           sync.Mutex
-	dynamicConfig *dynamicConfig
-	state         map[string]*collector
+	mtx             sync.Mutex
+	dynamicConfig   *dynamicConfig
+	deletedEP       []string
+	deletedRouters  []string
+	deletedServices []string
+	deletedURLs     map[string]string
 }
 
 func (ps *prometheusState) SetDynamicConfig(dynamicConfig *dynamicConfig) {
 	ps.mtx.Lock()
 	defer ps.mtx.Unlock()
-	ps.dynamicConfig = dynamicConfig
-}
 
-func (ps *prometheusState) ListenValueUpdates() {
-	for collector := range ps.collectors {
-		ps.mtx.Lock()
-		ps.state[collector.id] = collector
-		ps.mtx.Unlock()
+	for ep := range ps.dynamicConfig.entryPoints {
+		if _, ok := dynamicConfig.entryPoints[ep]; !ok {
+			ps.deletedEP = append(ps.deletedEP, ep)
+		}
+	}
+
+	for router := range ps.dynamicConfig.routers {
+		if _, ok := dynamicConfig.routers[router]; !ok {
+			ps.deletedRouters = append(ps.deletedRouters, router)
+		}
 	}
+
+	for service, serV := range ps.dynamicConfig.services {
+		actualService, ok := dynamicConfig.services[service]
+		if !ok {
+			ps.deletedServices = append(ps.deletedServices, service)
+			continue
+		}
+		for url := range serV {
+			if _, ok := actualService[url]; !ok {
+				ps.deletedURLs[service] = url
+			}
+		}
+	}
+
+	ps.dynamicConfig = dynamicConfig
 }
 
 // Describe implements prometheus.Collector and simply calls
 // the registered describer functions.
 func (ps *prometheusState) Describe(ch chan<- *stdprometheus.Desc) {
-	for _, desc := range ps.describers {
-		desc(ch)
+	for _, v := range ps.vectors {
+		v.Describe(ch)
 	}
 }
 
@@ -354,49 +376,52 @@ func (ps *prometheusState) Describe(ch chan<- *stdprometheus.Desc) {
 // The removal happens only after their Collect method was called to ensure that
 // also those metrics will be exported on the current scrape.
 func (ps *prometheusState) Collect(ch chan<- stdprometheus.Metric) {
+	for _, v := range ps.vectors {
+		v.Collect(ch)
+	}
+
 	ps.mtx.Lock()
 	defer ps.mtx.Unlock()
 
-	var outdatedKeys []string
-	for key, cs := range ps.state {
-		cs.collector.Collect(ch)
-
-		if ps.isOutdated(cs) {
-			outdatedKeys = append(outdatedKeys, key)
+	for _, ep := range ps.deletedEP {
+		if !ps.dynamicConfig.hasEntryPoint(ep) {
+			ps.DeletePartialMatch(map[string]string{"entrypoint": ep})
 		}
 	}
 
-	for _, key := range outdatedKeys {
-		ps.state[key].delete()
-		delete(ps.state, key)
-	}
-}
-
-// isOutdated checks whether the passed collector has labels that mark
-// it as belonging to an outdated configuration of Traefik.
-func (ps *prometheusState) isOutdated(collector *collector) bool {
-	labels := collector.labels
-
-	if entrypointName, ok := labels["entrypoint"]; ok && !ps.dynamicConfig.hasEntryPoint(entrypointName) {
-		return true
+	for _, router := range ps.deletedRouters {
+		if !ps.dynamicConfig.hasRouter(router) {
+			ps.DeletePartialMatch(map[string]string{"router": router})
+		}
 	}
 
-	if routerName, ok := labels["router"]; ok {
-		if !ps.dynamicConfig.hasRouter(routerName) {
-			return true
+	for _, service := range ps.deletedServices {
+		if !ps.dynamicConfig.hasService(service) {
+			ps.DeletePartialMatch(map[string]string{"service": service})
 		}
 	}
 
-	if serviceName, ok := labels["service"]; ok {
-		if !ps.dynamicConfig.hasService(serviceName) {
-			return true
-		}
-		if url, ok := labels["url"]; ok && !ps.dynamicConfig.hasServerURL(serviceName, url) {
-			return true
+	for service, url := range ps.deletedURLs {
+		if !ps.dynamicConfig.hasServerURL(service, url) {
+			ps.DeletePartialMatch(map[string]string{"service": service, "url": url})
 		}
 	}
 
-	return false
+	ps.deletedEP = nil
+	ps.deletedRouters = nil
+	ps.deletedServices = nil
+	ps.deletedURLs = make(map[string]string)
+}
+
+// DeletePartialMatch deletes all metrics where the variable labels contain all of those passed in as labels.
+// The order of the labels does not matter.
+// It returns the number of metrics deleted.
+func (ps *prometheusState) DeletePartialMatch(labels stdprometheus.Labels) int {
+	var count int
+	for _, elem := range ps.vectors {
+		count += elem.DeletePartialMatch(labels)
+	}
+	return count
 }
 
 func newDynamicConfig() *dynamicConfig {
@@ -440,42 +465,15 @@ func (d *dynamicConfig) hasServerURL(serviceName, serverURL string) bool {
 	return false
 }
 
-func newCollector(metricName string, labels stdprometheus.Labels, c stdprometheus.Collector, deleteFn func()) *collector {
-	return &collector{
-		id:        buildMetricID(metricName, labels),
-		labels:    labels,
-		collector: c,
-		delete:    deleteFn,
-	}
-}
-
-// collector wraps a Collector object from the Prometheus client library.
-// It adds information on how many generations this metric should be present
-// in the /metrics output, relative to the time it was last tracked.
-type collector struct {
-	id        string
-	labels    stdprometheus.Labels
-	collector stdprometheus.Collector
-	delete    func()
-}
-
-func buildMetricID(metricName string, labels stdprometheus.Labels) string {
-	var labelNamesValues []string
-	for name, value := range labels {
-		labelNamesValues = append(labelNamesValues, name, value)
-	}
-	sort.Strings(labelNamesValues)
-	return metricName + ":" + strings.Join(labelNamesValues, "|")
-}
-
-func newCounterFrom(collectors chan<- *collector, opts stdprometheus.CounterOpts, labelNames []string) *counter {
+func newCounterFrom(opts stdprometheus.CounterOpts, labelNames []string) *counter {
 	cv := stdprometheus.NewCounterVec(opts, labelNames)
 	c := &counter{
-		name:       opts.Name,
-		cv:         cv,
-		collectors: collectors,
+		name:             opts.Name,
+		cv:               cv,
+		labelNamesValues: make([]string, 0, 16),
 	}
 	if len(labelNames) == 0 {
+		c.collector = cv.WithLabelValues()
 		c.Add(0)
 	}
 	return c
@@ -485,39 +483,37 @@ type counter struct {
 	name             string
 	cv               *stdprometheus.CounterVec
 	labelNamesValues labelNamesValues
-	collectors       chan<- *collector
+	collector        stdprometheus.Counter
 }
 
 func (c *counter) With(labelValues ...string) metrics.Counter {
+	lnv := c.labelNamesValues.With(labelValues...)
 	return &counter{
 		name:             c.name,
 		cv:               c.cv,
-		labelNamesValues: c.labelNamesValues.With(labelValues...),
-		collectors:       c.collectors,
+		labelNamesValues: lnv,
+		collector:        c.cv.With(lnv.ToLabels()),
 	}
 }
 
 func (c *counter) Add(delta float64) {
-	labels := c.labelNamesValues.ToLabels()
-	collector := c.cv.With(labels)
-	collector.Add(delta)
-	c.collectors <- newCollector(c.name, labels, collector, func() {
-		c.cv.Delete(labels)
-	})
+	c.collector.Add(delta)
 }
 
 func (c *counter) Describe(ch chan<- *stdprometheus.Desc) {
 	c.cv.Describe(ch)
 }
 
-func newGaugeFrom(collectors chan<- *collector, opts stdprometheus.GaugeOpts, labelNames []string) *gauge {
+func newGaugeFrom(opts stdprometheus.GaugeOpts, labelNames []string) *gauge {
 	gv := stdprometheus.NewGaugeVec(opts, labelNames)
 	g := &gauge{
-		name:       opts.Name,
-		gv:         gv,
-		collectors: collectors,
+		name:             opts.Name,
+		gv:               gv,
+		labelNamesValues: make([]string, 0, 16),
 	}
+
 	if len(labelNames) == 0 {
+		g.collector = gv.WithLabelValues()
 		g.Set(0)
 	}
 	return g
@@ -527,46 +523,37 @@ type gauge struct {
 	name             string
 	gv               *stdprometheus.GaugeVec
 	labelNamesValues labelNamesValues
-	collectors       chan<- *collector
+	collector        stdprometheus.Gauge
 }
 
 func (g *gauge) With(labelValues ...string) metrics.Gauge {
+	lnv := g.labelNamesValues.With(labelValues...)
 	return &gauge{
 		name:             g.name,
 		gv:               g.gv,
-		labelNamesValues: g.labelNamesValues.With(labelValues...),
-		collectors:       g.collectors,
+		labelNamesValues: lnv,
+		collector:        g.gv.With(lnv.ToLabels()),
 	}
 }
 
 func (g *gauge) Add(delta float64) {
-	labels := g.labelNamesValues.ToLabels()
-	collector := g.gv.With(labels)
-	collector.Add(delta)
-	g.collectors <- newCollector(g.name, labels, collector, func() {
-		g.gv.Delete(labels)
-	})
+	g.collector.Add(delta)
 }
 
 func (g *gauge) Set(value float64) {
-	labels := g.labelNamesValues.ToLabels()
-	collector := g.gv.With(labels)
-	collector.Set(value)
-	g.collectors <- newCollector(g.name, labels, collector, func() {
-		g.gv.Delete(labels)
-	})
+	g.collector.Set(value)
 }
 
 func (g *gauge) Describe(ch chan<- *stdprometheus.Desc) {
 	g.gv.Describe(ch)
 }
 
-func newHistogramFrom(collectors chan<- *collector, opts stdprometheus.HistogramOpts, labelNames []string) *histogram {
+func newHistogramFrom(opts stdprometheus.HistogramOpts, labelNames []string) *histogram {
 	hv := stdprometheus.NewHistogramVec(opts, labelNames)
 	return &histogram{
-		name:       opts.Name,
-		hv:         hv,
-		collectors: collectors,
+		name:             opts.Name,
+		hv:               hv,
+		labelNamesValues: make([]string, 0, 16),
 	}
 }
 
@@ -574,28 +561,21 @@ type histogram struct {
 	name             string
 	hv               *stdprometheus.HistogramVec
 	labelNamesValues labelNamesValues
-	collectors       chan<- *collector
+	collector        stdprometheus.Observer
 }
 
 func (h *histogram) With(labelValues ...string) metrics.Histogram {
+	lnv := h.labelNamesValues.With(labelValues...)
 	return &histogram{
 		name:             h.name,
 		hv:               h.hv,
-		labelNamesValues: h.labelNamesValues.With(labelValues...),
-		collectors:       h.collectors,
+		labelNamesValues: lnv,
+		collector:        h.hv.With(lnv.ToLabels()),
 	}
 }
 
 func (h *histogram) Observe(value float64) {
-	labels := h.labelNamesValues.ToLabels()
-	observer := h.hv.With(labels)
-	observer.Observe(value)
-	// Do a type assertion to be sure that prometheus will be able to call the Collect method.
-	if collector, ok := observer.(stdprometheus.Histogram); ok {
-		h.collectors <- newCollector(h.name, labels, collector, func() {
-			h.hv.Delete(labels)
-		})
-	}
+	h.collector.Observe(value)
 }
 
 func (h *histogram) Describe(ch chan<- *stdprometheus.Desc) {
@@ -618,7 +598,7 @@ func (lvs labelNamesValues) With(labelValues ...string) labelNamesValues {
 // ToLabels is a convenience method to convert a labelNamesValues
 // to the native prometheus.Labels.
 func (lvs labelNamesValues) ToLabels() stdprometheus.Labels {
-	labels := stdprometheus.Labels{}
+	labels := make(map[string]string, len(lvs)/2)
 	for i := 0; i < len(lvs); i += 2 {
 		labels[lvs[i]] = lvs[i+1]
 	}
diff --git a/pkg/middlewares/metrics/metrics.go b/pkg/middlewares/metrics/metrics.go
index a491401cb7..327e61e18d 100644
--- a/pkg/middlewares/metrics/metrics.go
+++ b/pkg/middlewares/metrics/metrics.go
@@ -103,8 +103,9 @@ func (m *metricsMiddleware) ServeHTTP(rw http.ResponseWriter, req *http.Request)
 	labels = append(labels, m.baseLabels...)
 	labels = append(labels, "method", getMethod(req), "protocol", getRequestProtocol(req))
 
-	m.openConnsGauge.With(labels...).Add(1)
-	defer m.openConnsGauge.With(labels...).Add(-1)
+	openConnsGauge := m.openConnsGauge.With(labels...)
+	openConnsGauge.Add(1)
+	defer openConnsGauge.Add(-1)
 
 	// TLS metrics
 	if req.TLS != nil {
@@ -122,8 +123,7 @@ func (m *metricsMiddleware) ServeHTTP(rw http.ResponseWriter, req *http.Request)
 
 	labels = append(labels, "code", strconv.Itoa(recorder.getCode()))
 
-	histograms := m.reqDurationHistogram.With(labels...)
-	histograms.ObserveFromStart(start)
+	m.reqDurationHistogram.With(labels...).ObserveFromStart(start)
 
 	m.reqsCounter.With(labels...).Add(1)
 }


Test patch:
diff --git a/pkg/metrics/prometheus_test.go b/pkg/metrics/prometheus_test.go
index 7e1154a038..02688dfc3a 100644
--- a/pkg/metrics/prometheus_test.go
+++ b/pkg/metrics/prometheus_test.go
@@ -17,8 +17,7 @@ import (
 )
 
 func TestRegisterPromState(t *testing.T) {
-	// Reset state of global promState.
-	defer promState.reset()
+	t.Cleanup(promState.reset)
 
 	testCases := []struct {
 		desc                 string
@@ -88,21 +87,10 @@ func TestRegisterPromState(t *testing.T) {
 	}
 }
 
-// reset is a utility method for unit testing. It should be called after each
-// test run that changes promState internally in order to avoid dependencies
-// between unit tests.
-func (ps *prometheusState) reset() {
-	ps.collectors = make(chan *collector)
-	ps.describers = []func(ch chan<- *prometheus.Desc){}
-	ps.dynamicConfig = newDynamicConfig()
-	ps.state = make(map[string]*collector)
-}
-
 func TestPrometheus(t *testing.T) {
 	promState = newPrometheusState()
 	promRegistry = prometheus.NewRegistry()
-	// Reset state of global promState.
-	defer promState.reset()
+	t.Cleanup(promState.reset)
 
 	prometheusRegistry := RegisterPrometheus(context.Background(), &types.Prometheus{AddEntryPointsLabels: true, AddRoutersLabels: true, AddServicesLabels: true})
 	defer promRegistry.Unregister(promState)
@@ -361,30 +349,40 @@ func TestPrometheus(t *testing.T) {
 func TestPrometheusMetricRemoval(t *testing.T) {
 	promState = newPrometheusState()
 	promRegistry = prometheus.NewRegistry()
-	// Reset state of global promState.
-	defer promState.reset()
+	t.Cleanup(promState.reset)
 
 	prometheusRegistry := RegisterPrometheus(context.Background(), &types.Prometheus{AddEntryPointsLabels: true, AddServicesLabels: true, AddRoutersLabels: true})
 	defer promRegistry.Unregister(promState)
 
-	conf := dynamic.Configuration{
+	conf1 := dynamic.Configuration{
 		HTTP: th.BuildConfiguration(
 			th.WithRouters(
-				th.WithRouter("foo@providerName",
-					th.WithServiceName("bar")),
+				th.WithRouter("foo@providerName", th.WithServiceName("bar")),
+				th.WithRouter("router2", th.WithServiceName("bar@providerName")),
 			),
-			th.WithLoadBalancerServices(th.WithService("bar@providerName",
-				th.WithServers(th.WithServer("http://localhost:9000"))),
+			th.WithLoadBalancerServices(
+				th.WithService("bar@providerName", th.WithServers(
+					th.WithServer("http://localhost:9000"),
+					th.WithServer("http://localhost:9999"),
+				)),
+				th.WithService("service1", th.WithServers(th.WithServer("http://localhost:9000"))),
 			),
-			func(cfg *dynamic.HTTPConfiguration) {
-				cfg.Services["fii"] = &dynamic.Service{
-					Weighted: &dynamic.WeightedRoundRobin{},
-				}
-			},
 		),
 	}
 
-	OnConfigurationUpdate(conf, []string{"entrypoint1"})
+	conf2 := dynamic.Configuration{
+		HTTP: th.BuildConfiguration(
+			th.WithRouters(
+				th.WithRouter("foo@providerName", th.WithServiceName("bar")),
+			),
+			th.WithLoadBalancerServices(
+				th.WithService("bar@providerName", th.WithServers(th.WithServer("http://localhost:9000"))),
+			),
+		),
+	}
+
+	OnConfigurationUpdate(conf1, []string{"entrypoint1", "entrypoint2"})
+	OnConfigurationUpdate(conf2, []string{"entrypoint1"})
 
 	// Register some metrics manually that are not part of the active configuration.
 	// Those metrics should be part of the /metrics output on the first scrape but
@@ -393,22 +391,21 @@ func TestPrometheusMetricRemoval(t *testing.T) {
 		EntryPointReqsCounter().
 		With("entrypoint", "entrypoint2", "code", strconv.Itoa(http.StatusOK), "method", http.MethodGet, "protocol", "http").
 		Add(1)
+	prometheusRegistry.
+		RouterReqsCounter().
+		With("router", "router2", "service", "bar@providerName", "code", strconv.Itoa(http.StatusOK), "method", http.MethodGet, "protocol", "http").
+		Add(1)
 	prometheusRegistry.
 		ServiceReqsCounter().
-		With("service", "service2", "code", strconv.Itoa(http.StatusOK), "method", http.MethodGet, "protocol", "http").
+		With("service", "service1", "code", strconv.Itoa(http.StatusOK), "method", http.MethodGet, "protocol", "http").
 		Add(1)
 	prometheusRegistry.
 		ServiceServerUpGauge().
-		With("service", "service1", "url", "http://localhost:9999").
+		With("service", "bar@providerName", "url", "http://localhost:9999").
 		Set(1)
-	prometheusRegistry.
-		RouterReqsCounter().
-		With("router", "router2", "service", "service2", "code", strconv.Itoa(http.StatusOK), "method", http.MethodGet, "protocol", "http").
-		Add(1)
 
-	assertMetricsExist(t, mustScrape(), entryPointReqsTotalName, serviceReqsTotalName, serviceServerUpName)
-	assertMetricsAbsent(t, mustScrape(), entryPointReqsTotalName, serviceReqsTotalName, serviceServerUpName)
-	assertMetricsAbsent(t, mustScrape(), routerReqsTotalName, routerReqDurationName, routerOpenConnsName)
+	assertMetricsExist(t, mustScrape(), entryPointReqsTotalName, routerReqsTotalName, serviceReqsTotalName, serviceServerUpName)
+	assertMetricsAbsent(t, mustScrape(), entryPointReqsTotalName, routerReqsTotalName, serviceReqsTotalName, serviceServerUpName)
 
 	// To verify that metrics belonging to active configurations are not removed
 	// here the counter examples.
@@ -418,24 +415,39 @@ func TestPrometheusMetricRemoval(t *testing.T) {
 		Add(1)
 	prometheusRegistry.
 		RouterReqsCounter().
-		With("router", "foo@providerName", "service", "bar@providerName", "code", strconv.Itoa(http.StatusOK), "method", http.MethodGet, "protocol", "http").
+		With("router", "foo@providerName", "service", "bar", "code", strconv.Itoa(http.StatusOK), "method", http.MethodGet, "protocol", "http").
+		Add(1)
+	prometheusRegistry.
+		ServiceReqsCounter().
+		With("service", "bar@providerName", "code", strconv.Itoa(http.StatusOK), "method", http.MethodGet, "protocol", "http").
 		Add(1)
+	prometheusRegistry.
+		ServiceServerUpGauge().
+		With("service", "bar@providerName", "url", "http://localhost:9000").
+		Set(1)
 
 	delayForTrackingCompletion()
 
-	assertMetricsExist(t, mustScrape(), entryPointReqsTotalName)
-	assertMetricsExist(t, mustScrape(), entryPointReqsTotalName)
-	assertMetricsExist(t, mustScrape(), routerReqsTotalName)
-	assertMetricsExist(t, mustScrape(), routerReqsTotalName)
+	assertMetricsExist(t, mustScrape(), entryPointReqsTotalName, serviceReqsTotalName, serviceServerUpName, routerReqsTotalName)
+	assertMetricsExist(t, mustScrape(), entryPointReqsTotalName, serviceReqsTotalName, serviceServerUpName, routerReqsTotalName)
 }
 
 func TestPrometheusRemovedMetricsReset(t *testing.T) {
-	// Reset state of global promState.
-	defer promState.reset()
+	t.Cleanup(promState.reset)
 
 	prometheusRegistry := RegisterPrometheus(context.Background(), &types.Prometheus{AddEntryPointsLabels: true, AddServicesLabels: true})
 	defer promRegistry.Unregister(promState)
 
+	conf1 := dynamic.Configuration{
+		HTTP: th.BuildConfiguration(
+			th.WithLoadBalancerServices(th.WithService("service",
+				th.WithServers(th.WithServer("http://localhost:9000"))),
+			),
+		),
+	}
+	OnConfigurationUpdate(conf1, []string{"entrypoint1", "entrypoint2"})
+	OnConfigurationUpdate(dynamic.Configuration{}, nil)
+
 	labelNamesValues := []string{
 		"service", "service",
 		"code", strconv.Itoa(http.StatusOK),
@@ -467,12 +479,24 @@ func TestPrometheusRemovedMetricsReset(t *testing.T) {
 	assertCounterValue(t, 1, findMetricFamily(serviceReqsTotalName, metricsFamilies), labelNamesValues...)
 }
 
+// reset is a utility method for unit testing.
+// It should be called after each test run that changes promState internally
+// in order to avoid dependencies between unit tests.
+func (ps *prometheusState) reset() {
+	ps.dynamicConfig = newDynamicConfig()
+	ps.vectors = nil
+	ps.deletedEP = nil
+	ps.deletedRouters = nil
+	ps.deletedServices = nil
+	ps.deletedURLs = make(map[string]string)
+}
+
 // Tracking and gathering the metrics happens concurrently.
-// In practice this is no problem, because in case a tracked metric would miss
-// the current scrape, it would just be there in the next one.
-// That we can test reliably the tracking of all metrics here, we sleep
-// for a short amount of time, to make sure the metric will be present
-// in the next scrape.
+// In practice this is no problem, because in case a tracked metric would miss the current scrape,
+// it would just be there in the next one.
+// That we can test reliably the tracking of all metrics here,
+// we sleep for a short amount of time,
+// to make sure the metric will be present in the next scrape.
 func delayForTrackingCompletion() {
 	time.Sleep(250 * time.Millisecond)
 }