Repository: caddyserver/caddy (language: go)

Issue:
Memory leak in v2.6.2
Hello,

We are observing a memory leak after upgrading from v2.6.1 to v2.6.2.

![leak](https://user-images.githubusercontent.com/19559599/196632965-da509b8d-ea80-4e26-a319-62cf143d30c8.png)

The moment the memory issue starts occurring coincides with the release of v2.6.2. The drops in memory are caused by the container getting terminated after reaching the maximum allowed memory.

**Dockerfile**
```
FROM caddy:builder AS builder

RUN xcaddy build v2.6.2 \
    --with github.com/greenpau/caddy-security@v1.1.15

FROM caddy:latest

COPY --from=builder /usr/bin/caddy /usr/bin/caddy
```

**Patch**
```
@@ -1,6 +1,6 @@
 FROM caddy:builder AS builder

-RUN xcaddy build v2.6.1 \
+RUN xcaddy build v2.6.2 \
     --with github.com/greenpau/caddy-security@v1.1.15

 FROM caddy:latest
```

**Initial config**
```json
{
  "admin": {
    "config": {
      "load_delay": "1s",
      "load": {
        "module": "http",
        "url": "http://caddy-config.caddy:4000/config"
      }
    }
  }
}
```

**External config**
```json
{
  "admin": {
    "config": {
      "load_delay": "60s",
      "load": {
        "module": "http",
        "url": "http://caddy-config.caddy:4000/config"
      }
    }
  },
  "apps": {
    ...
  }
}
```

**Logs**

Logs don't show anything weird. There are only info logs about the configuration not changing (correct behavior).

v2.6.1

![log-before](https://user-images.githubusercontent.com/19559599/196641812-98def875-38e4-4adb-809a-5d671c7f640d.png)

v2.6.2

![log-after](https://user-images.githubusercontent.com/19559599/196641830-3f6512a1-b95d-494f-bbf6-af5c27d389cb.png)

Please let me know if you need any additional information,

Thank you,

Chris



Discussion / hints:
Thanks, that's very interesting.

Can you please provide the full configs (without modifications, except you can redact private keys or passwords only please) as well as a CPU and memory profile:

Go to localhost:2019/debug/pprof (on the server) and start by collecting a goroutine dump. Then collect a CPU profile and memory/heap profile. They can take about 30s to capture. It should download a file after that delay. Then please upload them here along with the goroutine dump. Try to do this near the peak of memory use (or at least not right after a restart).

Could you also enable access/request and debug level logs if you haven't already?

And if you have time you could do a git bisect to determine which commit causes the bug? That would be extremely helpful and certainly speed up a patch.
My hunch is that it has to do with https://github.com/caddyserver/caddy/pull/5077 but I didn't dig 
@francislavoie Possibly, although now that I look at it, I think it would only leak if the connection failed or the response status was an error status:

https://github.com/caddyserver/caddy/pull/5077/files#diff-30c30eb9f50b002df80f2b88d4ecf25bf2f2b80721f414fc940bea0d644c0216R123-R128

It seems unlikely for that to be occurring all the time. I'd think the logs would mention that. (But in any case, we should probably close the response body in those error returns.)

Fix patch:
diff --git a/caddyconfig/httploader.go b/caddyconfig/httploader.go
index f1992199809..04dec34f1d7 100644
--- a/caddyconfig/httploader.go
+++ b/caddyconfig/httploader.go
@@ -134,16 +134,16 @@ func doHttpCallWithRetries(ctx caddy.Context, client *http.Client, request *http
 	var err error
 	const maxAttempts = 10
 
-	// attempt up to 10 times
 	for i := 0; i < maxAttempts; i++ {
 		resp, err = attemptHttpCall(client, request)
 		if err != nil && i < maxAttempts-1 {
-			// wait 500ms before reattempting, or until context is done
 			select {
 			case <-time.After(time.Millisecond * 500):
 			case <-ctx.Done():
 				return resp, ctx.Err()
 			}
+		} else {
+			break
 		}
 	}
 


Test patch: