Repository: amir20/dozzle (language: go)

Issue:
Possible Memory Leak With Dozzle In Swarm Mode Deployed Globally
**Describe the bug**
The memory usage has continued to climb with Dozzle in Swarm Mode, screen shots and my config are below. I am not watching any logs, dozzle is simply running in the background

**To Reproduce**
Deploy Dozzle in Swarm mode with the config and monitor with a tool like Grafana

**Expected behavior**
Memory to level off at some point that is resonable, according to the graph usage before I went to swarm mode was around and under 20MB

**Screenshots**
![Capture](https://github.com/amir20/dozzle/assets/51446227/b807e9a3-f35a-4c3b-8ff1-48d38cb175f6)


**Desktop (please complete the following information):**

- OS: Ubuntu 22.04
- Docker version 26.1.4
- Browser & version N/A
- Version 8.0.1

Below is my config
```
version: '3.5'

services:
  dozzle:
    image: dockreg.biswb.com:5000/amir20-dozzle:070524
    environment:
      DOZZLE_MODE: swarm
      DOZZLE_AUTH_PROVIDER: simple
    networks:
      - DockInternalComms      
    volumes:
      - data:/data
      - /var/run/docker.sock:/var/run/docker.sock:ro
    deploy:
      mode: global
      update_config:
        delay: 20s
        failure_action: rollback     

networks:
  DockInternalComms:
    external: true
``` 


Discussion / hints:
Can you send the logs if you have them? 

There isn't much difference between agent mode vs swarm mode.  But perhaps, agents being destroyed are not being cleaned up correctly. 

Questions: 

1. Looks like this happened over a period of 8 hours? 
2. Are all instances like these? 
3. How many are you running? 


I have been running multiple agents remotely and they have been hovering around ~20MB. So something else might be happening. 
> Can you send the logs if you have them?

Of course, with this handy log tool called dozzle its easy ;)
```
doz_dozzle

    time=2024-07-07T22:28:53Zlevel=infomsg=Dozzle version v8.0.1

    addtime=2024-07-07T22:28:53Z
    addlevel=info
    addmsg=Dozzle version v8.0.1

doz_dozzle

    time=2024-07-07T22:28:53Zlevel=infomsg=Starting in Swarm mode

doz_dozzle

    time=2024-07-07T22:28:53Zlevel=infomsg=gRPC server listening on [::]:7007

doz_dozzle

    time=2024-07-07T22:28:53Zlevel=infomsg=Accepting connections on :8080

doz_dozzle

    time=2024-07-07T22:28:53Zlevel=infomsg=found 4 new dozzle replicas

doz_dozzle

    time=2024-07-07T22:36:42Zlevel=infomsg=Dozzle version v8.0.1

doz_dozzle

    time=2024-07-07T22:36:42Zlevel=infomsg=Starting in Swarm mode

doz_dozzle

    time=2024-07-07T22:36:42Zlevel=infomsg=gRPC server listening on [::]:7007

doz_dozzle

    time=2024-07-07T22:36:42Zlevel=infomsg=Accepting connections on :8080

doz_dozzle

    time=2024-07-07T22:36:42Zlevel=infomsg=found 4 new dozzle replicas

doz_dozzle

    time=2024-07-07T22:37:54Zlevel=infomsg=replaced 1 dozzle replicas

doz_dozzle

    time=2024-07-07T22:43:43Zlevel=infomsg=Dozzle version v8.0.1

doz_dozzle

    time=2024-07-07T22:43:43Zlevel=infomsg=Starting in Swarm mode

doz_dozzle

    time=2024-07-07T22:43:43Zlevel=infomsg=gRPC server listening on [::]:7007

doz_dozzle

    time=2024-07-07T22:43:43Zlevel=infomsg=Accepting connections on :8080

doz_dozzle

    time=2024-07-07T22:43:43Zlevel=infomsg=found 4 new dozzle replicas

doz_dozzle

    time=2024-07-07T22:44:27Zlevel=infomsg=replaced 1 dozzle replicas

doz_dozzle

    time=2024-07-07T22:44:37Zlevel=infomsg=replaced 1 dozzle replicas

doz_dozzle

    time=2024-07-07T22:52:37Zlevel=infomsg=Dozzle version v8.0.1

doz_dozzle

    time=2024-07-07T22:52:37Zlevel=infomsg=Starting in Swarm mode

doz_dozzle

    time=2024-07-07T22:52:37Zlevel=infomsg=gRPC server listening on [::]:7007

doz_dozzle

    time=2024-07-07T22:52:37Zlevel=infomsg=Accepting connections on :8080

doz_dozzle

    time=2024-07-07T22:52:37Zlevel=infomsg=found 4 new dozzle replicas

doz_dozzle

    time=2024-07-07T22:53:07Zlevel=infomsg=replaced 1 dozzle replicas

doz_dozzle

    time=2024-07-07T22:53:16Zlevel=infomsg=replaced 1 dozzle replicas

doz_dozzle

    time=2024-07-07T22:53:24Zlevel=infomsg=replaced 1 dozzle replicas

doz_dozzle

    time=2024-07-07T22:58:16Zlevel=infomsg=Dozzle version v8.0.1

doz_dozzle

    time=2024-07-07T22:58:16Zlevel=infomsg=Starting in Swarm mode

doz_dozzle

    time=2024-07-07T22:58:16Zlevel=infomsg=gRPC server listening on [::]:7007

doz_dozzle

    time=2024-07-07T22:58:16Zlevel=infomsg=Accepting connections on :8080

doz_dozzle

    time=2024-07-07T22:58:16Zlevel=infomsg=found 4 new dozzle replicas

doz_dozzle

    time=2024-07-07T22:58:31Zlevel=infomsg=replaced 1 dozzle replicas

doz_dozzle

    time=2024-07-07T22:58:39Zlevel=infomsg=replaced 1 dozzle replicas

doz_dozzle

    time=2024-07-07T22:58:43Zlevel=infomsg=replaced 1 dozzle replicas

doz_dozzle

    time=2024-07-07T22:59:15Zlevel=infomsg=replaced 1 dozzle replicas

doz_dozzle

    time=2024-07-08T04:22:25Zlevel=infomsg=Token created for user admin

doz_dozzle

    time=2024-07-08T10:23:05Zlevel=infomsg=stopped collecting container stats

doz_dozzle

    time=2024-07-08T10:23:05Zlevel=infomsg=stopped collecting container stats

doz_dozzle

    time=2024-07-08T10:23:05Zlevel=infomsg=stopped collecting container stats

doz_dozzle

    time=2024-07-08T10:23:05Zlevel=infomsg=stopped collecting container stats

doz_dozzle

    time=2024-07-08T10:23:05Zlevel=infomsg=stopped collecting container stats

doz_dozzle

    time=2024-07-08T12:11:55Zlevel=infomsg=Token created for user admin

doz_dozzle

    time=2024-07-08T15:28:23Zlevel=infomsg=Token created for user admin
```
 
> There isn't much difference between agent mode vs swarm mode. But perhaps, agents being destroyed are not being cleaned up correctly.
> 
> Questions:
> 
>     1. Looks like this happened over a period of 8 hours?

That is correct
> 
>     2. Are all instances like these?

I have just the one running now with 5 nodes, I plan to implement tomorrow with a cluster of 12

> 
>     3. How many are you running?
>

Just one dozzle container per node

> 
> I have been running multiple agents remotely and they have been hovering around ~20MB. So something else might be happening.

I could also try memory limits and see what happens when the container reaches those limits, does it stop functioning or just level off at that spot, perhaps get real slow, unknown. But wanted to drop the issue here and screenshots before I tried mitigating it.  

It's going to take me some time to reproduce this. And since it's Monday I am back at work.  

Your logs look ok. After a deploy I see `replaced 1 dozzle replicas` which is to be expected. Then I don't see any more agents being connected. So perhaps the issue isn't in disposing the clients. 


> I could also try memory limits and see what happens when the container reaches those limits, does it stop functioning or just level off at that spot, perhaps get real slow, unknown. But wanted to drop the issue here and screenshots before I tried mitigating it.

This would probably just kill it with OOM error. 

I am trying to think if any other information would help me. I may enable profiling and have you test that if I can't get it to work. 

I can open up debug logging if you think it would help, I think I saw that dozzle has that feature, just let me know. 

Plus with the 12 nodes i am curious if the path of how much memory is used is the same or if it grows exponentially 
Ok definitely seeing it creep up slowly. Let me see what I find in the day. 
I found a leaking goroutine. Can you try `amir20/dozzle:pr-3076`? 
I still see a little memory increase so will keep investigating. 
> I still see a little memory increase so will keep investigating.

I am seeing it still leaking as well, and will let you know what I find tomorrow with the other cluster
Any update? I noticed something interesting that this only happens in swarm mode. Seems like an agent with UI don't have this issue. 

Further debugging suggests it is something outside of Dozzle and related to gRPC. I wasted too much time researching and couldn't find something in my code. 
I see it mention gRPC in the logs and wasn't sure. As far as agent mode, I did not have the issue when I used the docker socket proxy, and also don't think I had the issue when dozzle was the agent in agent mode. We had the other issues of course we talked about in the other ticket, but i don't recall seeing memory leaks. 

So it looks like we should go agent mode/global and a sperate dozzle container with connection strings to those instead of using swarm mode
> So it looks like we should go agent mode/global and a sperate dozzle container with connection strings to those instead of using swarm mode

As a temporary memory fix yes. 

But I do not know why there is a leak.  
I have changed my setup back to the dozzle agent mode and will monitor it closely to make sure I didn't just miss it the first time, although I don't think I did and then will be able to confirm its only an issue with swarm mode
Thanks! I may ask you to go back to swarm mode to help me debug. 
> Thanks! I may ask you to go back to swarm mode to help me debug.

Happy to do it just let me know
I think I may have a better fix. :) But will try tomorrow. 

Fix patch:
diff --git a/internal/agent/client.go b/internal/agent/client.go
index 7e1e78e4ccb5..38384df6f0a0 100644
--- a/internal/agent/client.go
+++ b/internal/agent/client.go
@@ -25,6 +25,7 @@ import (
 type Client struct {
 	client pb.AgentServiceClient
 	host   docker.Host
+	conn   *grpc.ClientConn
 }
 
 func NewClient(endpoint string, certificates tls.Certificate, opts ...grpc.DialOption) (*Client, error) {
@@ -46,17 +47,19 @@ func NewClient(endpoint string, certificates tls.Certificate, opts ...grpc.DialO
 	opts = append(opts, grpc.WithTransportCredentials(creds))
 	conn, err := grpc.NewClient(endpoint, opts...)
 	if err != nil {
-		log.Fatalf("failed to connect to server: %v", err)
+		return nil, err
 	}
 
 	client := pb.NewAgentServiceClient(conn)
 	info, err := client.HostInfo(context.Background(), &pb.HostInfoRequest{})
 	if err != nil {
+		conn.Close()
 		return nil, err
 	}
 
 	return &Client{
 		client: client,
+		conn:   conn,
 
 		host: docker.Host{
 			ID:       info.Host.Id,
@@ -362,6 +365,10 @@ func (c *Client) Host() docker.Host {
 	return c.host
 }
 
+func (c *Client) Close() error {
+	return c.conn.Close()
+}
+
 func jsonBytesToOrderedMap(b []byte) *orderedmap.OrderedMap[string, any] {
 	var data *orderedmap.OrderedMap[string, any]
 	reader := bytes.NewReader(b)
diff --git a/internal/support/docker/multi_host_service.go b/internal/support/docker/multi_host_service.go
index 1dbca8643117..2cc703ef6a75 100644
--- a/internal/support/docker/multi_host_service.go
+++ b/internal/support/docker/multi_host_service.go
@@ -45,14 +45,14 @@ func NewMultiHostService(clients []ClientService) *MultiHostService {
 	return m
 }
 
-func NewSwarmService(client docker.Client, certificates tls.Certificate) *MultiHostService {
+func NewSwarmService(localClient docker.Client, certificates tls.Certificate) *MultiHostService {
 	m := &MultiHostService{
 		clients:   make(map[string]ClientService),
 		SwarmMode: true,
 	}
 
-	localClient := NewDockerClientService(client)
-	m.clients[localClient.Host().ID] = localClient
+	localService := NewDockerClientService(localClient)
+	m.clients[localClient.Host().ID] = localService
 
 	discover := func() {
 		ips, err := net.LookupIP("tasks.dozzle")
@@ -63,17 +63,18 @@ func NewSwarmService(client docker.Client, certificates tls.Certificate) *MultiH
 		found := 0
 		replaced := 0
 		for _, ip := range ips {
-			client, err := agent.NewClient(ip.String()+":7007", certificates)
+			clientAgent, err := agent.NewClient(ip.String()+":7007", certificates)
 			if err != nil {
 				log.Warnf("error creating client for %s: %v", ip, err)
 				continue
 			}
 
-			if client.Host().ID == localClient.Host().ID {
+			if clientAgent.Host().ID == localClient.Host().ID {
+				closeAgent(clientAgent)
 				continue
 			}
 
-			service := NewAgentService(client)
+			service := NewAgentService(clientAgent)
 			if existing, ok := m.clients[service.Host().ID]; !ok {
 				log.Debugf("adding swarm service %s", service.Host().ID)
 				m.clients[service.Host().ID] = service
@@ -83,6 +84,11 @@ func NewSwarmService(client docker.Client, certificates tls.Certificate) *MultiH
 				delete(m.clients, existing.Host().ID)
 				m.clients[service.Host().ID] = service
 				replaced++
+				if existingAgent, ok := existing.(*agentService); ok {
+					closeAgent(existingAgent.client)
+				}
+			} else {
+				closeAgent(clientAgent)
 			}
 		}
 
@@ -107,6 +113,13 @@ func NewSwarmService(client docker.Client, certificates tls.Certificate) *MultiH
 	return m
 }
 
+func closeAgent(agent *agent.Client) {
+	log.Debugf("closing agent %s", agent.Host().ID)
+	if err := agent.Close(); err != nil {
+		log.Warnf("error closing agent: %v", err)
+	}
+}
+
 func (m *MultiHostService) FindContainer(host string, id string) (*containerService, error) {
 	client, ok := m.clients[host]
 	if !ok {


Test patch: