Repository: traefik/traefik (language: go)

Issue:
Increased memory usage in version 2.11.18
### Welcome!

- [x] Yes, I've searched similar issues on [GitHub](https://github.com/traefik/traefik/issues) and didn't find any.
- [x] Yes, I've searched similar issues on the [Traefik community forum](https://community.traefik.io) and didn't find any.

### What did you do?

Upgrade from version 2.11.16 to 2.11.18. We did not expect any change in memory usage.

### What did you see instead?

It seems Træfik now uses at least 2x as much memory as before, and it may even be leaking memory.

![Image](https://github.com/user-attachments/assets/0c2a328a-d219-4a2f-9e03-7191f90c2aa8)
![Image](https://github.com/user-attachments/assets/9118c042-7778-4d8e-806d-611d23ef7bd2)


### What version of Traefik are you using?

2.11.18

### What is your environment & configuration?

N/A

### If applicable, please paste the log output in DEBUG level

_No response_


Discussion / hints:
Hey @aucampia,

Thanks for reaching out.

Could you provide a minimal reproducible case (for instance, full Kubernetes manifest to reproduce the issue)?

Hi,

We are also experiencing this issue with the latest Traefik version 2.11.18.
We are noticing memory usage spikes, which ultimately lead to our Docker Swarm Managers being killed.

On January 8th, we updated to version 2.11.18.
![Image](https://github.com/user-attachments/assets/a6bb9917-805f-473f-b877-1a5084f006d2)

Traefik is running in our Docker Swarm environment. We have 3 managers, 2 workers, and over 20 services.

``` yaml
version: '3.3'

services:
  proxy:
    image: traefik:v2.11.16
    command: 
      - --api=true
      - --providers.docker
      - --providers.docker.swarmMode=true
      - --entryPoints.web-secure.address=:443
      - --entryPoints.web-secure.transport.respondingTimeouts.readTimeout=150
      - --entryPoints.metrics.address=:8082
      - --metrics=true
      - --metrics.prometheus=true
      - --metrics.prometheus.entryPoint=metrics
      - --ping=true
      - --entryPoints.ping.address=:8083
      - --ping.entryPoint=ping
    ports:
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      sample:
      management:
    deploy:
      mode: global
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.traefik.rule=(Host(`${url}`) && PathPrefix(`/api`)) || (Host(`${url}`) && PathPrefix(`/dashboard`))"
        - "traefik.http.services.traefik.loadbalancer.server.port=8080"
        - "traefik.http.routers.traefik.service=api@internal"
        - "traefik.http.routers.traefik.tls=true"
        - "traefik.http.routers.traefik.middlewares=auth"
        - "traefik.http.middlewares.auth.basicauth.users=sample:sample"
        - "traefik.http.routers.traefikping.rule=(Host(`${url}`) && PathPrefix(`/ping`))"
        - "traefik.http.routers.traefikping.service=ping@internal"
        - "traefik.http.routers.traefikping.tls=true"
        - "traefik.http.services.traefikping.loadbalancer.server.port=8083"
      placement:
        constraints:
          - node.role == manager
networks:
  sample:
    external: true
  management:
    external: true

volumes:
  logs:
```
We also faced this problem

![Image](https://github.com/user-attachments/assets/edd9810a-9c25-4c50-9ccd-dd01928936f5)
We have same problem after upgrading from 2.11.16 to 2.11.18, pods starting to be OOM killed. 
Sorry I accidentally closed this, please re-open it.
Here are the PRs merged between 2.11.16 and 2.11.18:
- [websocket,server] Disable http2 connect setting for websocket by default (https://github.com/traefik/traefik/pull/11412 by [rtribotte](https://github.com/rtribotte))
- [acme] Update go-acme/lego to v4.21.0 (https://github.com/traefik/traefik/pull/11368 by [ldez](https://github.com/ldez))
- [middleware] Fix typo in basicauth note (https://github.com/traefik/traefik/pull/11397 by [tieje](https://github.com/tieje))
- [service] Configure ErrorLog in httputil.ReverseProxy (https://github.com/traefik/traefik/pull/11344 by [peacewalker122](https://github.com/peacewalker122))
- Bump golang.org/x/net to v0.33.0 (https://github.com/traefik/traefik/pull/11365 by [kevinpollet](https://github.com/kevinpollet))

The most likely root cause is [the bump of the lib golang.org/x/net to v0.33.0](https://github.com/traefik/traefik/pull/11365). There is no [release note](https://pkg.go.dev/golang.org/x/net?tab=versions) associated with v0.33.0. The go team released v0.34.0 on Jan 6, it might be worth a try.

In the meantime, can anyone provide a [pprof](https://doc.traefik.io/traefik/operations/api/#debug) of Traefik while facing the issue? 
Have the same problem. Memory consumption is too large and increasing.
Hi,

We’ve built a test version of `v2.11.18` with the `x/net` dependency downgraded to version `v0.30.0`. You can find the changes [here](https://github.com/traefik/traefik/compare/master...mmatur:traefik:fix/v2.11.18-issue-11423).

A corresponding Docker image is available on [Docker Hub](https://hub.docker.com/repository/docker/traefik/traefik/tags/v2.11.18-issue-11423/sha256-a8a5ceac6265a7ad1e7c469229eca8e010bf51c09cb87b3f7762000ba8a617f4).

⚠ **Warning**:  
This is a test image intended solely for debugging purposes, to determine whether the upgrade of `x/net` to version `v0.33.0` could be impacting memory consumption.

Let us know your findings!

I saw this also with upgrade from 2.11.16 to 2.11.17
a pprof show huge usage of logrus maybe it's https://github.com/traefik/traefik/pull/11344

![Image](https://github.com/user-attachments/assets/31157340-566e-4d28-be75-4f030db548b0)
I think I found something. In #11344, a new logger object is initialized per request.
Indeed, this is the issue:
https://github.com/traefik/traefik/pull/11344/files#diff-f7d7f0e8fef165ce3ca78be8f4d887b323d564a29b25d416a6a7d2b0e9ff7df7R86
`ErrorLog: stdlog.New(log.WithoutContext().WriterLevel(logrus.DebugLevel), "", 0),`
Thanks a lot @RemiBou for helping on this, we will be able to fix it ASAP.
I think it's missing a close on the logger and happens mostly hen you have a lot of config changes

Fix patch:
diff --git a/pkg/server/service/proxy.go b/pkg/server/service/proxy.go
index 13d107a32c..8f3c0a418a 100644
--- a/pkg/server/service/proxy.go
+++ b/pkg/server/service/proxy.go
@@ -26,6 +26,12 @@ const StatusClientClosedRequest = 499
 // StatusClientClosedRequestText non-standard HTTP status for client disconnection.
 const StatusClientClosedRequestText = "Client Closed Request"
 
+// errorLogger is a logger instance used to log proxy errors.
+// This logger is a shared instance as having one instance by proxy introduces a memory and go routine leak.
+// The writer go routine is never stopped as the finalizer is never called.
+// See https://github.com/sirupsen/logrus/blob/d1e6332644483cfee14de11099f03645561d55f8/writer.go#L57).
+var errorLogger = stdlog.New(log.WithoutContext().WriterLevel(logrus.DebugLevel), "", 0)
+
 func buildProxy(passHostHeader *bool, responseForwarding *dynamic.ResponseForwarding, roundTripper http.RoundTripper, bufferPool httputil.BufferPool) (http.Handler, error) {
 	var flushInterval ptypes.Duration
 	if responseForwarding != nil {
@@ -83,7 +89,7 @@ func buildProxy(passHostHeader *bool, responseForwarding *dynamic.ResponseForwar
 		Transport:     roundTripper,
 		FlushInterval: time.Duration(flushInterval),
 		BufferPool:    bufferPool,
-		ErrorLog:      stdlog.New(log.WithoutContext().WriterLevel(logrus.DebugLevel), "", 0),
+		ErrorLog:      errorLogger,
 		ErrorHandler: func(w http.ResponseWriter, request *http.Request, err error) {
 			statusCode := http.StatusInternalServerError
 


Test patch: