Repository: traefik/traefik (language: go)

Issue:
Memory leak using the Docker provider (Swarm mode)
### Welcome!

- [X] Yes, I've searched similar issues on [GitHub](https://github.com/traefik/traefik/issues) and didn't find any.
- [X] Yes, I've searched similar issues on the [Traefik community forum](https://community.containo.us) and didn't find any.

### What did you do?

Configured Traefik as a service in Docker Swarm, with command-line options and labels, relying mostly on the defaults.

I must note here that I have an unorthodox setup: the swarm consists of two manager nodes, one of which regularly goes down. As long as it is down, Traefik constantly logs errors like:
```
frontend_traefik.1.qsk3rquh4y98@srvu    | time="2022-03-22T13:12:11Z" level=error msg="Failed to list services for docker swarm mode, error Error response from daemon: rpc error: code = Unknown desc = The swarm does not have a leader. It's possible that too few managers are online. Make sure more than half of the managers are online." providerName=docker
```
That in itself wouldn't be a big deal, however...


### What did you see instead?

...I noticed memory usage steadily increasing, too.
I'm not sure whether the excessive logging and memory usage are related, but just in a day, it grew from ~25MB to ~250MB.
As soon as it reached its resource limits, it went into periodic CPU bursts, which I suspect is Go's GC kicking in (?). Once restarted, memory usage went back to ~25MB.

I enabled debug mode and took these snapshots after about a day:
![allocs](https://user-images.githubusercontent.com/123012/159799045-690a8c6a-ac05-4fd8-b79b-b7eabb0f2f4d.png)
![heap](https://user-images.githubusercontent.com/123012/159799056-a70d233a-9fe7-4bcf-aa15-5c846d1b74e2.png)

I noticed most memory is consumed in data structures related to the Docker provider and HTTP request processing, so I first tried setting:
`--providers.docker.httpClientTimeout=60`
but that didn't help.

Setting this, however, fixed the issue!
`--providers.docker.watch=false`

The docs aren't exactly wordy on this one, but I suppose this means if my services bounce/get relocated, then Traefik will be none the wiser. Like I said, my setup is weird, so I can live with this... but wanted to let you know in case it uncovers some deeper issue.

### What version of Traefik are you using?

Version:      2.6.1
Codename:     rocamadour
Go version:   go1.17.7
Built:        2022-02-14T16:50:25Z
OS/Arch:      linux/amd64

### What is your environment & configuration?

```
--entrypoints.web.address=:80
--providers.docker.swarmMode=true
--providers.docker.exposedbydefault=false
```

### If applicable, please paste the log output in DEBUG level

_No response_


Discussion / hints:
I've been bit by the same problem on Traefik 2.8.3.
Leak happens when swarm status can't be retrieved.
It can easily be reproduced by demoting the Swarm node configured as Docker provider in Traefik.

My relevant configuration is:
```
--providers.docker=true
--providers.docker.endpoint=unix:///var/run/docker.sock
--providers.docker.swarmMode=true
--providers.docker.watch=true
```

Open file descriptors count is steady until Docker node is demoted, causing Docker's control socket to return error and file descriptors leaking begin:
![traefik-file-descriptors-leak](https://user-images.githubusercontent.com/6118420/186432490-a4e2a8f8-5cf4-4483-837f-deb8337318fe.png)
Leaking stops when promoting the node back:
![traefik-file-descriptors-leak-end](https://user-images.githubusercontent.com/6118420/186434541-2109f350-48d9-4814-bcf5-bdd69e1f1a9f.png)

Latest Traefik log entries, when leak happens:
```
time="2022-08-24T13:21:13Z" level=error msg="Failed to list services for docker swarm mode, error Error response from daemon: This node is not a swarm manager. Worker nodes can't be used to view or modify cluster state. Please run this command on a manager node or promote the current node to a manager." providerName=docker
time="2022-08-24T13:21:13Z" level=error msg="Provider connection error Error response from daemon: This node is not a swarm manager. Worker nodes can't be used to view or modify cluster state. Please run this command on a manager node or promote the current node to a manager., retrying in 5.45179408s" providerName=docker
time="2022-08-24T13:21:18Z" level=error msg="Failed to list services for docker swarm mode, error Error response from daemon: This node is not a swarm manager. Worker nodes can't be used to view or modify cluster state. Please run this command on a manager node or promote the current node to a manager." providerName=docker

```
Traefik's process opened file descriptor list:
[traefik-file-descriptors-leak.txt](https://github.com/traefik/traefik/files/9416529/traefik-file-descriptors-leak.txt)

The following set of file descriptors stay open for each unsuccessful provider endpoint request:
```
lrwx------ 1 root root 64 Aug 24 13:20 9 -> 'socket:[362439]'
lrwx------ 1 root root 64 Aug 24 13:20 90 -> 'socket:[511925]'
lrwx------ 1 root root 64 Aug 24 13:20 91 -> 'socket:[512472]'
lrwx------ 1 root root 64 Aug 24 13:20 92 -> 'socket:[511948]'
lrwx------ 1 root root 64 Aug 24 13:20 93 -> 'socket:[512483]'
lrwx------ 1 root root 64 Aug 24 13:20 94 -> 'socket:[511974]'
lrwx------ 1 root root 64 Aug 24 13:20 95 -> 'socket:[511991]'
lrwx------ 1 root root 64 Aug 24 13:20 96 -> 'socket:[512520]'
lrwx------ 1 root root 64 Aug 24 13:20 97 -> 'socket:[512584]'
lrwx------ 1 root root 64 Aug 24 13:20 98 -> 'socket:[512608]'
lrwx------ 1 root root 64 Aug 24 13:20 99 -> 'socket:[512617]'
```

@aeburriel thanks for the tip, I can now reproduce.
it's barely noticeable in my case, so I'm not a 100% sure, but I'm fairly convinced.

Fix patch:
diff --git a/pkg/provider/docker/docker.go b/pkg/provider/docker/docker.go
index eda5318fc3..6de0012750 100644
--- a/pkg/provider/docker/docker.go
+++ b/pkg/provider/docker/docker.go
@@ -205,6 +205,7 @@ func (p *Provider) Provide(configurationChan chan<- dynamic.Message, pool *safe.
 				logger.Errorf("Failed to create a client for docker, error: %s", err)
 				return err
 			}
+			defer dockerClient.Close()
 
 			serverVersion, err := dockerClient.ServerVersion(ctx)
 			if err != nil {
@@ -249,7 +250,7 @@ func (p *Provider) Provide(configurationChan chan<- dynamic.Message, pool *safe.
 							case <-ticker.C:
 								services, err := p.listServices(ctx, dockerClient)
 								if err != nil {
-									logger.Errorf("Failed to list services for docker, error %s", err)
+									logger.Errorf("Failed to list services for docker swarm mode, error %s", err)
 									errChan <- err
 									return
 								}


Test patch: