Repository: caddyserver/caddy (language: go)

Issue:
Panic after "update quic-go to v0.48.0" commit
Caddy regularly crashes for me after upgrading to commit a211c65, update quic-go to v0.48.0. The log shows:

```
panic: close of closed channel
github.com/quic-go/quic-go@v0.48.0/http3/server.go:289
github.com/quic-go/quic-go/http3.(*Server).serveListener.func1()
github.com/quic-go/quic-go@v0.48.0/http3/server.go:340 +0x128
created by github.com/quic-go/quic-go/http3.(*Server).serveListener in goroutine 115
github.com/quic-go/quic-go@v0.48.0/http3/server.go:333 +0x39
```

Reverting to the previous commit stops the crashes.


Discussion / hints:
@marten-seemann FYI

(Graceful shutdown is hard. I assume this is related.)
@mattxtaz did you try c6f2979986d87d7236b132c687c8887c92248dd8 ? It's a followup which changes how we do graceful shutdown.
Yes. Sorry I should have mentioned that. It crashes with that one too so that wasn't a fix in this case I'm afraid.
@mattxtaz Can you try `xcaddy build master --with github.com/quic-go/quic-go=github.com/WeidiDeng/quic-go@shutdown-fix`?
Weidi, I've gotta say, thanks for taking on so many of the challenging problems :+1: :100:
Just be sure, you are not reloading caddy at that moment right?
> [@mattxtaz](https://github.com/mattxtaz) Can you try `xcaddy build master --with github.com/quic-go/quic-go=github.com/WeidiDeng/quic-go@shutdown-fix`?

I've just applied this fix and am now running master. This seems to work. I could reliably crash it before just refreshing a couple of web pages but it's been running for several minutes. I'll leave it running and keep an eye on it. Thank you for coming up with a patch so quickly!

I am not reloading caddy. All I was doing was restarting it, then refreshing a couple of web pages and it crashed within seconds.
I've just experimented a little with restarting caddy. Sometimes it shuts down cleanly with exit code 0, other times it seems to time out waiting and shuts down with exit code 137 which causes podman to log that the container failed. Is that expected? But other than this, it appears that the crash I was reporting looks fixed at the moment.

```
Oct 19 09:01:49 example.com caddy[474553]: {"level":"info","ts":1729324909.3985527,"msg":"shutting down apps, then terminating","signal":"SIGTERM"}
Oct 19 09:01:49 example.com caddy[474553]: {"level":"warn","ts":1729324909.398607,"msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
Oct 19 09:01:49 example.com caddy[474553]: {"level":"info","ts":1729324909.3986526,"logger":"http","msg":"servers shutting down with eternal grace period"}
Oct 19 09:01:59 example.com systemd[1103]: caddy.service: Main process exited, code=exited, status=137/n/a                                                               
Oct 19 09:01:59 example.com systemd[1103]: caddy.service: Failed with result 'exit-code'.
```

Verses one that worked:

```
Oct 19 09:01:27 example.com caddy[474378]: {"level":"info","ts":1729324887.230141,"msg":"shutting down apps, then terminating","signal":"SIGTERM"}                             
Oct 19 09:01:27 example.com caddy[474378]: {"level":"warn","ts":1729324887.2301776,"msg":"exiting; byeee!! 👋","signal":"SIGTERM"}                                       
Oct 19 09:01:27 example.com caddy[474378]: {"level":"info","ts":1729324887.2302244,"logger":"http","msg":"servers shutting down with eternal grace period"}                                
Oct 19 09:01:27 example.com caddy[474378]: {"level":"info","ts":1729324887.230502,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}                             
Oct 19 09:01:27 example.com caddy[474378]: {"level":"info","ts":1729324887.2305396,"msg":"shutdown complete","signal":"SIGTERM","exit_code":0}         
```
@marten-seemann Can you take a look at this?
@mattxtaz What if you just caddy running using the previous commit and not restart it? Does caddy still crash?
To confirm,

Commit 48ce47f caddy always runs normally and never crashes.
Commit a211c65 caddy always crashes after refreshing a page a few times.
Commit c6f2979 caddy still crashes after refreshing a page a few times.
Commit 388c7e8 and --with github.com/quic-go/quic-go=github.com/WeidiDeng/quic-go@shutdown-fix caddy never crashes, problem is fixed/worked around.

All of this is after restarting Caddy (after a crash) and then refreshing some pages several times. I am not reloading or restarting Caddy myself at all whilst testing this first problem. Only refreshing some web pages.

The other second problem I mentioned about error code 137 is a totally different problem, sorry if I confused things there. With that one I am manually restarting my Caddy container to test shutdown/restart,

Commit 48ce47f caddy always shuts down and restarts with error code 0.
Commit 388c7e8 and --with github.com/quic-go/quic-go=github.com/WeidiDeng/quic-go@shutdown-fix caddy  caddy sometimes shuts down with error code 0, and sometimes exits with error code 137 after 10 seconds.

I have not tried this with any other commits.
I see, so the only reason you restarted caddy is due to the crash with the reported error message, right?

The 137 is graceful shutdown timeout, and caddy is forcibly killed.
@mattxtaz would you be able to try it again with [f9d0171](https://github.com/quic-go/quic-go/pull/4707/commits/f9d01716d9f091963ffc334f874d392734db43ca)?

I think you can use this:

```
xcaddy build master --with github.com/quic-go/quic-go=github.com/WeidiDeng/quic-go@f9d0171
```

That would help @marten-seemann have confirmation that the patch is good to go, see https://github.com/quic-go/quic-go/pull/4707#issuecomment-2424982883
@francislavoie , @marten-seemann I can confirm that running Caddy with this commit works. It's not crashing at all during page refreshes.
> Can you try `xcaddy build master --with github.com/quic-go/quic-go=github.com/WeidiDeng/quic-go@shutdown-fix`?

I was running into the same issue, which is fixed by PR https://github.com/quic-go/quic-go/pull/4707.

Fix patch:
diff --git a/go.mod b/go.mod
index 67fe8e59c89..d847c7bfbef 100644
--- a/go.mod
+++ b/go.mod
@@ -19,7 +19,7 @@ require (
 	github.com/klauspost/cpuid/v2 v2.2.8
 	github.com/mholt/acmez/v2 v2.0.3
 	github.com/prometheus/client_golang v1.19.1
-	github.com/quic-go/quic-go v0.48.0
+	github.com/quic-go/quic-go v0.48.1
 	github.com/smallstep/certificates v0.26.1
 	github.com/smallstep/nosql v0.6.1
 	github.com/smallstep/truststore v0.13.0
diff --git a/go.sum b/go.sum
index 0956dffd2f0..794ad6fc80b 100644
--- a/go.sum
+++ b/go.sum
@@ -392,8 +392,8 @@ github.com/prometheus/procfs v0.12.0 h1:jluTpSng7V9hY0O2R9DzzJHYb2xULk9VTR1V1R/k
 github.com/prometheus/procfs v0.12.0/go.mod h1:pcuDEFsWDnvcgNzo4EEweacyhjeA9Zk3cnaOZAZEfOo=
 github.com/quic-go/qpack v0.5.1 h1:giqksBPnT/HDtZ6VhtFKgoLOWmlyo9Ei6u9PqzIMbhI=
 github.com/quic-go/qpack v0.5.1/go.mod h1:+PC4XFrEskIVkcLzpEkbLqq1uCoxPhQuvK5rH1ZgaEg=
-github.com/quic-go/quic-go v0.48.0 h1:2TCyvBrMu1Z25rvIAlnp2dPT4lgh/uTqLqiXVpp5AeU=
-github.com/quic-go/quic-go v0.48.0/go.mod h1:yBgs3rWBOADpga7F+jJsb6Ybg1LSYiQvwWlLX+/6HMs=
+github.com/quic-go/quic-go v0.48.1 h1:y/8xmfWI9qmGTc+lBr4jKRUWLGSlSigv847ULJ4hYXA=
+github.com/quic-go/quic-go v0.48.1/go.mod h1:yBgs3rWBOADpga7F+jJsb6Ybg1LSYiQvwWlLX+/6HMs=
 github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
 github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ=
 github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog=


Test patch: