Repository: traefik/traefik (language: go)

Issue:
Possible memory leak in v2.10.5, maybe cause by consul/consulcatalog provider
### Welcome!

- [X] Yes, I've searched similar issues on [GitHub](https://github.com/traefik/traefik/issues) and didn't find any.
- [X] Yes, I've searched similar issues on the [Traefik community forum](https://community.traefik.io) and didn't find any.

### What did you do?

I deployed traefik in k8s and enabled the `consul/consulcatalog provider`.

### What did you see instead?

Everything is ok, BUT memory leak!
<img width="2105" alt="截屏2024-02-08 14 59 11" src="https://github.com/traefik/traefik/assets/128664758/4c24da0a-33a1-4ef7-84e5-f2cf807ed425">

> If I disable consul provider, memory leaks have been significantly improved.
> If I disable consul/consulcatalog provider, memory leak disappears.

#####  [after 72h] [just enable consulcatalog provider] pprof
- [goroutines20240208103528.pb.gz](https://github.com/traefik/traefik/files/14203882/goroutines20240208103528.pb.gz)
- [heap20240208103528.pb.gz](https://github.com/traefik/traefik/files/14203883/heap20240208103528.pb.gz)

##### [after 4h] [just enable file provider] pprof
- [goroutines20240208150500.pb.gz](https://github.com/traefik/traefik/files/14205416/goroutines20240208150500.pb.gz)
- [heap20240208150500.pb.gz](https://github.com/traefik/traefik/files/14205417/heap20240208150500.pb.gz)


### What version of Traefik are you using?

v2.10.5

### What is your environment & configuration?
- traefik: v2.10.5  (replicas: 12)
- k8s: v1.26
- consul-k8s: v1.3.1

### k8s deployment.yaml
```yaml
kind: Deployment
apiVersion: apps/v1
metadata:
  name: rdsec-one-apigateway-deployment
  labels:
    app: rdsec-one-apigateway

spec:
  replicas: 12
  selector:
    matchLabels:
      app: rdsec-one-apigateway
  template:
    metadata:
      labels:
        app: rdsec-one-apigateway
    spec:
      terminationGracePeriodSeconds: 30
      nodeSelector:
        usage: rdsec-01
      tolerations:
      - key: "HIGH_Critical"
        operator: "Equal"
        value: "r5.4xlarge"
        effect: "NoSchedule"
      containers:
        - name: rdsec-one-apigateway
          image: traefik:v2.10.5
          ports:
            - name: web
              containerPort: 80
            - name: websecure
              containerPort: 443
            - name: dashboard
              containerPort: 8080
          envFrom:
          - configMapRef:
              name: rdsec-one-apigateway-env-config
          - secretRef:  
              name: rdsec-one-apigateway-secret
          volumeMounts:
            - name: rdsec-one-apigateway-traefik-config
              mountPath: /etc/traefik
            - name: rdsec-one-apigateway-traefik-file-provider-config
              mountPath: "/etc/traefik/config"
          resources:
            requests:
              memory: "256Mi"
              cpu: "256m"
            limits:
              memory: "2048Mi"
              cpu: "1000m"
      volumes:
        - name: rdsec-one-apigateway-traefik-config
          configMap:
            name: rdsec-one-apigateway-traefik-config
        - name: rdsec-one-apigateway-traefik-file-provider-config
          configMap:
            name: rdsec-one-apigateway-traefik-file-provider-config
```

### traefik.yaml
```yaml
global:
  checkNewVersion: false

serversTransport:
  maxIdleConnsPerHost: 1000
  forwardingTimeouts:
    dialTimeout: 60s
    responseHeaderTimeout: 60s
    idleConnTimeout: 90s

entryPoints:
  web:
    address: :80
    transport:
      respondingTimeouts:
        idleTimeout: 120s
      lifeCycle:
        graceTimeOut: 30s
    http2:
      maxConcurrentStreams: 1000
    forwardedHeaders:
      insecure: true
    proxyProtocol:
      insecure: true
  # websecure:
  #   address: ":443"

api:
  dashboard: true
  insecure: true
  debug: true

ping:
  entryPoint: traefik

log:
  level: DEBUG
  # filePath: /var/log/traefik.log
  format: json

accessLog:
  # filePath: /var/log/access.log
  bufferingSize: 100
  format: json
  filters:
    minDuration: "3ms"
  fields:
    defaultMode: keep
    headers:
      defaultMode: keep
      names:
        Lares-Authorization: redact
        Authorization: redact
        token: redact

metrics:
  prometheus:
    entryPoint: web

tracing:
  serviceName: traefik
  spanNameLimit: 150

  jaeger:
    localAgentHostPort: ""
    collector:
      endpoint: ***

providers:
  file:
    directory: "/etc/traefik/config"
    watch: true
  consul:
    endpoints:
      - "***"
    token: "***"
  consulCatalog:
    refreshInterval: 30s
    watch: false
    endpoint:
      address: ***
      scheme: "https"
      datacenter: aws1
      token: "***"
    exposedByDefault: false
    defaultRule: PathPrefix(`/stag/{{ normalize .Name }}`)
    connectAware: true
    connectByDefault: true

experimental:
  localPlugins:
    jwt-auth:
      moduleName: github.com/traefik/jwt-auth
    validate-request-schema:
      moduleName: github.com/traefik/validate-request-schema
```

### If applicable, please paste the log output in DEBUG level

_No response_


Discussion / hints:
Hello @caiuskong ,

Thanks for your interest in Traefik.
It appears that your profiles are related to 2 different setups. To conduct a more accurate comparison, I recommend generating pprof profiles on the same setup (e.g., with consulcatalog enabled) at different intervals (e.g., 4 hours and 72 hours).
After analyzing your profiles it seems that the issue may be related to plugins.
To be able to dig further into this issue, can you send us a minimal reproducible setup?
Thanks for your(@lbenguigui) reply. 

I tried deploy the `local environment` and testing, unfortunately I couldn't reproduce the memory leak.
In the local environment, I either file provider or consul provider for API stress testing (router using middlewares), and the memory changes were similar.

In the `stag environment`, I enabled both consul/consulcatalog provider and did the following tests, potential memory leaks have occurred.
**By analyzing the pprof, I suspect a goroutine leak and it is caused by the plugin (yaegi)?**

### Testing Data
| time      | action |  pprof |
| ----------- | ----------- |  ----------- |
| 5m      | do nothing, only start traefik       | [goroutine-profile.pb.gz](https://github.com/traefik/traefik/files/14356867/goroutine-profile.pb.gz) , [heap-profile.pb.gz](https://github.com/traefik/traefik/files/14356868/heap-profile.pb.gz)         |
| 25m   | API stress testing (106540 requests in 5.00m). Router from consul provider.        |  [goroutine-profile.pb.gz](https://github.com/traefik/traefik/files/14356883/goroutine-profile.pb.gz), [heap-profile.pb.gz](https://github.com/traefik/traefik/files/14356884/heap-profile.pb.gz)       |
| 45m   | API stress testing (288594 requests in 5.00m). Router from consulcatalog. provider        |  [goroutine-profile.pb.gz](https://github.com/traefik/traefik/files/14356897/goroutine-profile.pb.gz), [heap-profile.pb.gz](https://github.com/traefik/traefik/files/14356898/heap-profile.pb.gz)      |
| 22h   | Regular requests, about 20,000+ per hour        |  [goroutine-profile.pb.gz](https://github.com/traefik/traefik/files/14357402/goroutine-profile.pb.gz), [heap-profile.pb.gz](https://github.com/traefik/traefik/files/14357403/heap-profile.pb.gz)      |

### Traefik Static Configration
```yml
global:
  checkNewVersion: false

serversTransport:
  maxIdleConnsPerHost: 1000
  forwardingTimeouts:
    dialTimeout: 60s
    responseHeaderTimeout: 60s
    idleConnTimeout: 90s

entryPoints:
  web:
    address: :80
    transport:
      respondingTimeouts:
        idleTimeout: 120s
      lifeCycle:
        graceTimeOut: 30s
    http2:
      maxConcurrentStreams: 1000
    forwardedHeaders:
      insecure: true
    proxyProtocol:
      insecure: true
  # websecure:
  #   address: ":443"

api:
  dashboard: true
  insecure: true
  debug: true

ping:
  entryPoint: traefik

log:
  level: DEBUG
  # filePath: /var/log/traefik.log
  format: json

accessLog:
  # filePath: /var/log/access.log
  bufferingSize: 100
  format: json
  filters:
    minDuration: "3ms"
  fields:
    defaultMode: keep
    headers:
      defaultMode: keep
      names:
        Lares-Authorization: redact
        Authorization: redact
        token: redact

metrics:
  prometheus:
    entryPoint: web

tracing:
  serviceName: traefik
  spanNameLimit: 150

  jaeger:
    localAgentHostPort: ""
    collector:
      endpoint: https://jaeger-collector-stg.***.com/api/traces

providers:
  file:
    directory: "/etc/traefik/config"
    watch: true
  consul:
    endpoints:
      - "consul-stag.***.com"
    token: "${CONSUL_HTTP_TOKEN}"
  consulCatalog:
    refreshInterval: 30s
    watch: false
    endpoint:
      address: consul-stag.***.com
      scheme: "https"
      datacenter: aws1
      token: "${CONSUL_HTTP_TOKEN}"
    exposedByDefault: false
    defaultRule: PathPrefix(`/stag/{{ normalize .Name }}`)
    connectAware: true
    connectByDefault: true

experimental:
  localPlugins:
    jwt-auth:
      moduleName: github.com/traefik/jwt-auth
    validate-request-schema:
      moduleName: github.com/traefik/validate-request-schema
```

### Traefik Dynamic Configuration Raw Data (Part)
```json
{
    "routers": {
        "caius-ex-service@consul": {
            "entryPoints": [
                "web"
            ],
            "middlewares": [
                "secured-chain@consul"
            ],
            "service": "caius-ex-service",
            "rule": "PathPrefix(`/stag/caius-ex-service`)",
            "status": "enabled",
            "using": [
                "web"
            ]
        },
        "caius-in-service@consulcatalog": {
            "entryPoints": [
                "web"
            ],
            "middlewares": [
                "secured-chain@consul"
            ],
            "service": "caius-in-service",
            "rule": "PathPrefix(`/stag/caius-in-service`)",
            "status": "enabled",
            "using": [
                "web"
            ]
        },
    },
    "middlewares": {
        "auth@consul": {
            "plugin": {
                "jwt-auth": {
                    "redisUrl": "***",
                    "secret": "***"
                }
            },
            "status": "enabled"
        },
        "validateRequestSchema@consul": {
            "plugin": {
                "validate-request-schema": {
                    "extension": {
                        "name": "x-amazon-apigateway-integration",
                        "replacePathServices": "***"
                    },
                    "redisUrl": "***"
                }
            },
            "status": "enabled"
        },
        "replacepathregex-gw-prefix@consul": {
            "replacePathRegex": {
                "regex": "^/(stag|prod)/[^/]+/?(.*?)/?$",
                "replacement": "/${2}"
            },
            "status": "enabled"
        },
        "secured-chain@consul": {
            "chain": {
                "middlewares": [
                    "auth@consul",
                    "validateRequestSchema@consul",
                    "replacepathregex-gw-prefix@consul"
                ]
            },
            "status": "enabled",
            "usedBy": [
                "caius-ex-service@consul",
            ]
        },
    },
    "services": {
        "caius-ex-service@consul": {
            "loadBalancer": {
                "servers": [
                    {
                        "url": "***"
                    }
                ],
                "passHostHeader": false
            },
            "status": "enabled",
            "usedBy": [
                "caius-ex-service@consul"
            ],
            "serverStatus": {
                "***": "UP"
            }
        },
        "caius-in-service@consulcatalog": {
            "loadBalancer": {
                "servers": [
                    {
                        "url": "***"
                    }
                ],
                "passHostHeader": true,
                "serversTransport": "tls-default-aws1-caius-in-service@consulcatalog"
            },
            "status": "enabled",
            "usedBy": [
                "caius-in-service@consulcatalog"
            ],
            "serverStatus": {
                "***": "UP"
            }
        },
    }
}
```
### [4d18h] - goroutine pprof
- [profile.pb.gz](https://github.com/traefik/traefik/files/14399076/profile.pb.gz)
```
      File: traefik
Type: goroutine
Time: Feb 26, 2024 at 10:48am (CST)
Showing nodes accounting for 21546, 100% of 21546 total
----------------------------------------------------------+-------------
      flat  flat%   sum%        cum   cum%   calls calls% + context 	 	 
----------------------------------------------------------+-------------
                                             21531 99.94% |   runtime.selectgo runtime/select.go:327
                                                 8 0.037% |   runtime.netpollblock runtime/netpoll.go:564
                                                 4 0.019% |   runtime.chanrecv runtime/chan.go:583
     21543   100%   100%      21543   100%                | runtime.gopark runtime/proc.go:398
----------------------------------------------------------+-------------
...
...
----------------------------------------------------------+-------------
                                             21492 99.82% |   reflect.rselect runtime/select.go:589
                                                 6 0.028% |   io.(*pipe).read io/pipe.go:57
                                                 5 0.023% |   github.com/patrickmn/go-cache.(*janitor).Run github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1079
                                                 4 0.019% |   github.com/traefik/traefik/v2/pkg/server.(*httpForwarder).Accept github.com/traefik/traefik/v2/pkg/server/server_entrypoint_tcp.go:58
                                                 3 0.014% |   github.com/traefik/traefik/v2/pkg/provider/aggregator.(*RingChannel).ringBuffer github.com/traefik/traefik/v2/pkg/provider/aggregator/ring_channel.go:48
                                                 3 0.014% |   github.com/traefik/traefik/v2/pkg/provider/aggregator.maybeThrottledProvide.func1.1 github.com/traefik/traefik/v2/pkg/provider/aggregator/aggregator.go:47
                                                 3 0.014% |   net/http.(*http2ClientConn).RoundTrip net/http/h2_bundle.go:8381
                                                 3 0.014% |   net/http.(*http2clientStream).writeRequest net/http/h2_bundle.go:8549
                                                 1 0.0046% |   github.com/traefik/traefik/v2/pkg/provider/consulcatalog.(*Provider).Provide.func1.1 github.com/traefik/traefik/v2/pkg/provider/consulcatalog/consul_catalog.go:227
                                                 1 0.0046% |   github.com/traefik/traefik/v2/pkg/provider/consulcatalog.(*Provider).watchConnectTLS github.com/traefik/traefik/v2/pkg/provider/consulcatalog/consul_catalog.go:560
                                                 1 0.0046% |   github.com/traefik/traefik/v2/pkg/provider/consulcatalog.repeatSend github.com/traefik/traefik/v2/pkg/provider/consulcatalog/consul_catalog.go:623
                                                 1 0.0046% |   github.com/traefik/traefik/v2/pkg/provider/file.(*Provider).addWatcher.func1 github.com/traefik/traefik/v2/pkg/provider/file/file.go:110
                                                 1 0.0046% |   github.com/traefik/traefik/v2/pkg/provider/kv.(*Provider).watchKv.func1 github.com/traefik/traefik/v2/pkg/provider/kv/kv.go:101
                                                 1 0.0046% |   github.com/traefik/traefik/v2/pkg/server.(*ConfigurationWatcher).applyConfigurations github.com/traefik/traefik/v2/pkg/server/configurationwatcher.go:151
                                                 1 0.0046% |   github.com/traefik/traefik/v2/pkg/server.(*ConfigurationWatcher).receiveConfigurations github.com/traefik/traefik/v2/pkg/server/configurationwatcher.go:103
                                                 1 0.0046% |   github.com/traefik/traefik/v2/pkg/server.(*Server).listenSignals github.com/traefik/traefik/v2/pkg/server/server_signals.go:20
                                                 1 0.0046% |   github.com/uber/jaeger-client-go.(*remoteReporter).processQueue github.com/uber/jaeger-client-go@v2.30.0+incompatible/reporter.go:296
                                                 1 0.0046% |   go.opencensus.io/stats/view.(*worker).start go.opencensus.io@v0.24.0/stats/view/worker.go:292
                                                 1 0.0046% |   net/http.(*persistConn).writeLoop net/http/transport.go:2421
                                                 1 0.0046% |   os/signal.NotifyContext.func1 os/signal/signal.go:288
         0     0%   100%      21531 99.93%                | runtime.selectgo runtime/select.go:327
                                             21531   100% |   runtime.gopark runtime/proc.go:398
----------------------------------------------------------+-------------
```
**In context, I still suspect a consul catalog provider issue** :question:

Hello, 
Is this not all service discoveries ?

we may be faced to same issue where traefik is oomkiiled every day on our 400 nodes clusters.
We noticed this after disabling our HPA which hided the issue. We disabled mostly every feature we use but we kept kubernetes SD and isolated one ingress only in a single ingress class, to observe traefik over time.

Here you see a fresh 6Mb hop without any traffic added.

![image](https://github.com/traefik/traefik/assets/119752/22f92ebb-35fc-415e-8baf-7095bee5f46e)

Here is what we see onto our production clusters with 5k ingresses, 1 oomkiil per day (and a resync of the oomkill moment since yesterday where we added 1Gb to see)

![image](https://github.com/traefik/traefik/assets/119752/373a07a1-14cc-48bb-b02c-358c68fc856c)


There is a memory leak in Yaegi's handling of goroutines.

The following program reproduces it:
```go
package main

import (
	"fmt"
	"runtime"
	"sync"
)

func humanizeBytes(bytes uint64) string {
	const (
		_         = iota
		kB uint64 = 1 << (10 * iota)
		mB
		gB
		tB
		pB
	)

	switch {
	case bytes < kB:
		return fmt.Sprintf("%dB", bytes)
	case bytes < mB:
		return fmt.Sprintf("%.2fKB", float64(bytes)/float64(kB))
	case bytes < gB:
		return fmt.Sprintf("%.2fMB", float64(bytes)/float64(mB))
	case bytes < tB:
		return fmt.Sprintf("%.2fGB", float64(bytes)/float64(gB))
	case bytes < pB:
		return fmt.Sprintf("%.2fTB", float64(bytes)/float64(tB))
	default:
		return fmt.Sprintf("%dB", bytes)
	}
}

func main() {
	i := 0
	wg := sync.WaitGroup{}
	for {
		var m runtime.MemStats
		runtime.ReadMemStats(&m)
		fmt.Printf("#%d: alloc = %s, routines = %d, gc = %d\n", i, humanizeBytes(m.Alloc), runtime.NumGoroutine(), m.NumGC)

		wg.Add(1)
		go func() {
			wg.Done()
		}()
		wg.Wait()
		i = i + 1
	}
}
```

After 1.000.000 iterations, Golang's interpreter will consume KB<n<2MB of memory wheras Yaegi's interpreter will have leaked about 10GB of memory on my machine (and going).


![yaegi](https://github.com/traefik/traefik/assets/895503/1a358343-aec3-4eee-aba1-51bc79516164)

This isn't a goroutine leak, as the goroutines count remains stable and it isn't a garbage collector contention as the garbage collector is in the same range of iterations.

This hints at a leak in the Yaegi wrapper code around the execution of goStmt (yaegi/interp/run.go) and I'm wondering if a reference is not kept to a frame preventing its garbage collection and leaking a frame per goroutine call (just a theory at this point, I'm 0% familiar with Yaegi).

EDIT:
It is actually worse than that, it is any binary function call that will leak, not just goroutines, so any lambda will also cause the issue. I have opened an issue on the Yaegi tracker there:

https://github.com/traefik/yaegi/issues/1618
Following research from @poolpOrg, it appears the issue is there since Traefik [2.9.4](https://github.com/traefik/traefik/releases/tag/v2.9.4), which introduced yaegi 0.14.3
FWIW, the following graph shows a config reload without and with my fix:

![image](https://github.com/traefik/traefik/assets/895503/5afd8d2e-f76a-4e1a-ada0-9b0ba1ff6bf7)

I removed the details of my analysis for reasons I won't disclose here but which I'll hand over to Traefik Labs developers should they contact me.

Cheers,

>I'll hand over to Traefik Labs developers should they contact me

Hi @poolpOrg, I sent you an email yesterday, happy to chat about that
the problem is fixed by https://github.com/traefik/yaegi/pull/1621
Hi @emilevauge, you must have missed my reply at 4:54PM :-)

Thanks for the fix @ldez and @mvertes !


Fix patch:
diff --git a/go.mod b/go.mod
index d8441118e4..c9cd7a9ab9 100644
--- a/go.mod
+++ b/go.mod
@@ -58,7 +58,7 @@ require (
 	github.com/stvp/go-udp-testing v0.0.0-20191102171040-06b61409b154
 	github.com/testcontainers/testcontainers-go v0.27.0
 	github.com/traefik/paerser v0.2.0
-	github.com/traefik/yaegi v0.15.1
+	github.com/traefik/yaegi v0.16.1
 	github.com/uber/jaeger-client-go v2.30.0+incompatible
 	github.com/uber/jaeger-lib v2.2.0+incompatible
 	github.com/unrolled/render v1.0.2
diff --git a/go.sum b/go.sum
index b3a1be0b66..cb7a4cfd6b 100644
--- a/go.sum
+++ b/go.sum
@@ -1244,8 +1244,8 @@ github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8/go.mod h1
 github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
 github.com/traefik/paerser v0.2.0 h1:zqCLGSXoNlcBd+mzqSCLjon/I6phqIjeJL2xFB2ysgQ=
 github.com/traefik/paerser v0.2.0/go.mod h1:afzaVcgF8A+MpTnPG4wBr4whjanCSYA6vK5RwaYVtRc=
-github.com/traefik/yaegi v0.15.1 h1:YA5SbaL6HZA0Exh9T/oArRHqGN2HQ+zgmCY7dkoTXu4=
-github.com/traefik/yaegi v0.15.1/go.mod h1:AVRxhaI2G+nUsaM1zyktzwXn69G3t/AuTDrCiTds9p0=
+github.com/traefik/yaegi v0.16.1 h1:f1De3DVJqIDKmnasUF6MwmWv1dSEEat0wcpXhD2On3E=
+github.com/traefik/yaegi v0.16.1/go.mod h1:4eVhbPb3LnD2VigQjhYbEJ69vDRFdT2HQNrXx8eEwUY=
 github.com/transip/gotransip/v6 v6.23.0 h1:PsTdjortrEZ8IFFifEryzjVjOy9SgK4ahlnhKBBIQgA=
 github.com/transip/gotransip/v6 v6.23.0/go.mod h1:nzv9eN2tdsUrm5nG5ZX6AugYIU4qgsMwIn2c0EZLk8c=
 github.com/tv42/httpunix v0.0.0-20150427012821-b75d8614f926/go.mod h1:9ESjWnEqriFuLhtthL60Sar/7RFoluCcXsuvEwTV5KM=


Test patch: